Leonx AI

LiteLLM Integration with Langfuse

Table of Contents

Why Integrate LiteLLM with Langfuse? #

In the previous article, we introduced LiteLLM (How to Use AI Gateway (LiteLLM): Unified Management of AI Large Language Model Access). However, in enterprise AI application deployment, having just a unified API gateway is not enough. As AI applications scale, we need comprehensive observability and monitoring capabilities to:

  1. Track Model Performance - Understand response times, success rates, and quality across different models in various scenarios
  2. Analyze Usage Patterns - Identify high-frequency queries, user behavior patterns, and potential optimization opportunities
  3. Cost Management - Accurately track AI usage costs per team, project, or feature
  4. Quality Assurance - Monitor model output quality and promptly detect anomalies or degradation

This is why integrating LiteLLM with Langfuse becomes so important.

Introduction to Langfuse #

Langfuse is an open-source observability platform specifically designed for LLM applications, providing comprehensive tracking, evaluation, and monitoring capabilities. It can:

  • Record detailed information about all LLM interactions
  • Provide intuitive dashboards displaying performance metrics and usage statistics
  • Support custom tags and grouping for multi-dimensional analysis
  • Implement cost tracking and allocation
  • Offer scoring and feedback mechanisms to help improve model output quality

Integration Benefits #

Combining LiteLLM with Langfuse offers the following advantages:

  1. Unified Management and Comprehensive Monitoring - LiteLLM provides a unified API access layer, while Langfuse offers in-depth observability
  2. Seamless Data Flow - All requests through LiteLLM are automatically recorded in Langfuse without additional code
  3. Granular Analysis - Performance and costs can be analyzed by model, team, project, or any custom dimension
  4. Closed-Loop Optimization - Based on Langfuse data insights, LiteLLM routing strategies and model selection can be optimized

Implementation Steps #

1. Install Langfuse #

  1. Clone Langfuse Repository
git clone https://github.com/langfuse/langfuse.git
cd langfuse
  1. Configure Environment Variables Configure necessary environment variables in the docker-compose.yml file:
  • POSTGRES_PASSWORD: PostgreSQL database password
  • NEXTAUTH_SECRET: NextAuth secret key
  • NEXTAUTH_URL: Application access URL (default: http://localhost:3000)
  1. Start Services
docker compose up -d

  1. Wait for Services to be Ready Wait approximately 2-3 minutes until the langfuse-web-1 container logs show “Ready”.

  2. Access Interface Open your browser and visit http://localhost:3000 to access the Langfuse management interface.

2. Configure LiteLLM Integration #

If you haven’t installed LiteLLM yet, please refer to: How to Use AI Gateway (LiteLLM): Unified Management of AI Large Language Model Access

2.1 Configure LiteLLM Configuration File #

Edit config.yaml:

# https://docs.litellm.ai/docs/proxy/logging#langfuse
litellm_settings:
  success_callback: ["langfuse"]

2.2 Configure Environment Variables #

Edit .env file:

# Langfuse Configuration
LANGFUSE_PUBLIC_KEY="pk-lf-xxxxxxxx-0000-xxxx-xxxx-xxxxxxxxxxxx"
LANGFUSE_SECRET_KEY="sk-lf-xxxxxxxx-6767-xxxx-xxxx-xxxxxxxxxxxx"
# Optional, defaults to https://cloud.langfuse.com
LANGFUSE_HOST="http://localhost:3000"

2.3 Restart LiteLLM Service #

# Restart LiteLLM service to apply configuration
cd litellm
docker-compose restart

3. Usage Examples #

LiteLLM supports passing trace information through both HTTP Headers and request body.

3.1 Passing Through HTTP Headers #

curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H 'langfuse_trace_user_id: user-grayson' \
  -H 'langfuse_trace_session_id: oa-test-id-hello-word' \
  -H 'langfuse_trace_environment: local' \
  -d '{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant. Your name is Li Si 1"
    },
    {
      "role": "user",
      "content": "how are you? what your name?"
    }
  ]
}'

3.2 Passing Through Request Body #

curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
  "model": "gpt-4o-mini",
  "metadata": {
    "session_id": "oa-test-id-hello-word",
    "user_id": "user-grayson",
    "trace_environment": "staging",
    "trace_metadata": {"subdomain": "demo-full", "custom": "hello"}
  },
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant. Your name is Li Si San"
    },
    {
      "role": "user",
      "content": "how are you? what your name?"
    }
  ]
}'

4. Results Showcase #

Langfuse Tracking Interface Langfuse Session Details Langfuse Performance Analysis

Reference Documentation #