What's New​

Four new endpoints for video character operations:

• Create character - Upload a video to create a reusable asset
• Get character - Retrieve character metadata
• Edit video - Modify generated videos
• Extend video - Continue clips with character consistency

Available from: LiteLLM v1.83.0+

Quick Example​

import litellm

# Create character from video
character = litellm.avideo_create_character(
    name="Luna",
    video=open("luna.mp4", "rb"),
    custom_llm_provider="openai",
    model="sora-2"
)
print(f"Character: {character.id}")

# Use in generation
video = litellm.avideo(
    model="sora-2",
    prompt="Luna dances through a magical forest.",
    characters=[{"id": character.id}],
    seconds="8"
)

# Get character info
fetched = litellm.avideo_get_character(
    character_id=character.id,
    custom_llm_provider="openai"
)

# Edit with character preserved
edited = litellm.avideo_edit(
    video_id=video.id,
    prompt="Add warm golden lighting"
)

# Extend sequence
extended = litellm.avideo_extension(
    video_id=video.id,
    prompt="Luna waves goodbye",
    seconds="5"
)

Via Proxy​

# Create character
curl -X POST "http://localhost:4000/v1/videos/characters" \
  -H "Authorization: Bearer sk-litellm-key" \
  -F "video=@luna.mp4" \
  -F "name=Luna"

# Get character
curl -X GET "http://localhost:4000/v1/videos/characters/char_abc123def456" \
  -H "Authorization: Bearer sk-litellm-key"

# Edit video
curl -X POST "http://localhost:4000/v1/videos/edits" \
  -H "Authorization: Bearer sk-litellm-key" \
  -H "Content-Type: application/json" \
  -d '{
    "video": {"id": "video_xyz789"},
    "prompt": "Add warm golden lighting and enhance colors"
  }'

# Extend video
curl -X POST "http://localhost:4000/v1/videos/extensions" \
  -H "Authorization: Bearer sk-litellm-key" \
  -H "Content-Type: application/json" \
  -d '{
    "video": {"id": "video_xyz789"},
    "prompt": "Luna waves goodbye and walks into the sunset",
    "seconds": "5"
  }'

Managed Character IDs​

LiteLLM automatically encodes provider and model metadata into character IDs:

What happens:

Upload character "Luna" with model "sora-2" on OpenAI
  ↓
LiteLLM creates: char_abc123def456 (contains provider + model_id)
  ↓
When you reference it later, LiteLLM decodes automatically
  ↓
Router knows exactly which deployment to use

Behind the scenes:

• Character ID format: character_<base64_encoded_metadata>
• Metadata includes: provider, model_id, original_character_id
• Transparent to you - just use the ID, LiteLLM handles routing

How it works​

Flow of generating ephemeral token

Proxy Setup​

model_list:
  - model_name: gpt-4o-realtime
    litellm_params:
      model: openai/gpt-4o-realtime-preview-2024-12-17
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      mode: realtime

Azure: use model: azure/gpt-4o-realtime-preview, api_key, api_base.

litellm --config /path/to/config.yaml

Try it live​

Client Usage​

1. Get token - POST /v1/realtime/client_secrets with LiteLLM API key and { model }.

2. WebRTC handshake - Create RTCPeerConnection, add mic track, create data channel oai-events, send SDP offer to POST /v1/realtime/calls with Authorization: Bearer <encrypted_token> and Content-Type: application/sdp.

3. Events - Use the data channel for session.update and other events.

// 1. Token
const r = await fetch("http://proxy:4000/v1/realtime/client_secrets", {
  method: "POST",
  headers: { "Authorization": "Bearer sk-litellm-key", "Content-Type": "application/json" },
  body: JSON.stringify({ model: "gpt-4o-realtime" }),
});
const { client_secret } = await r.json();
const token = client_secret.value;

// 2. WebRTC
const pc = new RTCPeerConnection();
const audio = document.createElement("audio");
audio.autoplay = true;
pc.ontrack = (e) => (audio.srcObject = e.streams[0]);
const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(ms.getTracks()[0]);
const dc = pc.createDataChannel("oai-events");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const sdpRes = await fetch("http://proxy:4000/v1/realtime/calls", {
  method: "POST",
  headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/sdp" },
  body: offer.sdp,
});
await pc.setRemoteDescription({ type: "answer", sdp: await sdpRes.text() });

// 3. Events
dc.send(JSON.stringify({ type: "session.update", session: { instructions: "..." } }));

FAQ​

Q: What do I do if I get a 401 Token expired error?
A: Tokens are short-lived. Get a fresh token right before creating the WebRTC offer.

Q: Which key should I use for /v1/realtime/calls?
A: Use the encrypted token from client_secrets, not your raw API key.

Q: Should I pass the model parameter when making the call?
A: No, the encrypted token already encodes all routing information including model.

Q: How do I resolve Azure api-version errors?
A: Set the correct api_version in litellm_params (or via the AZURE_API_VERSION environment variable), along with the right api_base and deployment values.

Q: What if I get no audio?
A: Make sure you grant microphone permission, ensure pc.ontrack assigns the audio element with autoplay enabled, check your network/firewall for WebRTC traffic, and inspect the browser console for ICE or SDP errors.

Docker Image​

docker pull ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch

Usage​

model_list:
  - model_name: gpt-5.4
    litellm_params:
      model: openai/gpt-5.4
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.4",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ]
  }'

from litellm import completion

response = completion(
    model="openai/gpt-5.4",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

Notes​

• Restart your container to get the cost tracking for this model.
• Use /responses for better model performance.
• GPT-5.4 supports reasoning, function calling, vision, and tool-use — see the OpenAI provider docs for advanced usage.

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.80.8-stable.1

pip install litellm==v1.80.8-stable.1

What's New​

Supports all four thinking levels:

• MINIMAL: Ultra-fast responses with minimal reasoning
• LOW: Simple instruction following
• MEDIUM: Balanced reasoning for complex tasks
• HIGH: Maximum reasoning depth (dynamic)

Quick Start​

from litellm import completion

response = completion(
    model="gemini/gemini-3.1-flash-lite-preview",
    messages=[{"role": "user", "content": "Extract key entities from this text: ..."}],
)

print(response.choices[0].message.content)

from litellm import completion

# Use MEDIUM thinking for complex reasoning tasks
response = completion(
    model="gemini/gemini-3.1-flash-lite-preview",
    messages=[{"role": "user", "content": "Analyze this dataset and identify patterns"}],
    reasoning_effort="medium",  # low, medium , high
)

print(response.choices[0].message.content)

model_list:
  - model_name: gemini-3.1-flash-lite
    litellm_params:
      model: gemini/gemini-3.1-flash-lite-preview
      api_key: os.environ/GEMINI_API_KEY
  
  # Or use Vertex AI
  - model_name: vertex-gemini-3.1-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.1-flash-lite-preview
      vertex_project: your-project-id
      vertex_location: us-central1

litellm --config /path/to/config.yaml

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
  -d '{
    "model": "gemini-3.1-flash-lite",
    "messages": [{"role": "user", "content": "Extract structured data from this text"}],
    "reasoning_effort": "low"
  }'

Supported Endpoints​

LiteLLM provides full end-to-end support for Gemini 3.1 Flash Lite Preview on:

• ✅ /v1/chat/completions - OpenAI-compatible chat completions endpoint
• ✅ /v1/responses - OpenAI Responses API endpoint (streaming and non-streaming)
• ✅ /v1/messages - Anthropic-compatible messages endpoint
• ✅ /v1/generateContent – Google Gemini API compatible endpoint

All endpoints support:

• Streaming and non-streaming responses
• Function calling with thought signatures
• Multi-turn conversations
• All Gemini 3-specific features (thinking levels, thought signatures)
• Full multimodal support (text, image, audio, video)

reasoning_effort Mapping for Gemini 3.1​

LiteLLM automatically maps OpenAI's reasoning_effort parameter to Gemini's thinkingLevel:

Note: This fix is available starting from LiteLLM v1.81.14.rc.2 or higher.

Summary​

A change to improve Redis connection pool cleanup introduced a regression that closed httpx clients that were still actively being used by the proxy. The LLMClientCache (an in-memory TTL cache) stores both Redis clients and httpx clients under the same eviction policy. When a cache entry expired or was evicted, the new cleanup code called aclose()/close() on the evicted value which worked correctly for Redis clients, but destroyed httpx clients that other parts of the system still held references to and were actively using for LLM API calls.

Impact: Any proxy instance that hit the cache TTL (default 10 minutes) or capacity limit (200 entries) would have its httpx clients closed out from under it, causing requests to LLM providers to fail with connection errors.

Background​

LLMClientCache extends InMemoryCache and is used to cache SDK clients (OpenAI, Anthropic, etc.) to avoid re-creating them on every request. These clients are keyed by configuration + event loop ID. The cache has:

• Max size: 200 entries
• Default TTL: 10 minutes

When the cache is full or entries expire, InMemoryCache.evict_cache() calls _remove_key() to drop entries.

The cached values are a mix of:

• Redis/async Redis clients — owned exclusively by the cache, safe to close on eviction
• httpx-backed SDK clients (OpenAI, Anthropic, etc.) — shared references, still in use by router/model instances

Root Cause​

PR #21717 overrode _remove_key() in LLMClientCache to close async clients on eviction:

class LLMClientCache(InMemoryCache):
    def _remove_key(self, key: str) -> None:
        value = self.cache_dict.get(key)
        super()._remove_key(key)
        if value is not None:
            close_fn = getattr(value, "aclose", None) or getattr(value, "close", None)
            if close_fn and asyncio.iscoroutinefunction(close_fn):
                try:
                    asyncio.get_running_loop().create_task(close_fn())
                except RuntimeError:
                    pass
            elif close_fn and callable(close_fn):
                try:
                    close_fn()
                except Exception:
                    pass

The intent was correct for Redis clients — prevent connection pool leaks when cached Redis clients expire. But LLMClientCache also stores httpx-backed SDK clients (e.g., AsyncOpenAI, AsyncAnthropic). These clients:

1. Have an aclose() method (inherited from httpx)
2. Are still held by references elsewhere in the codebase (router, model instances)
3. Were being closed without any check on whether they were still in use

So when the cache evicted an entry, it would call aclose() on an httpx client that was still being used for active LLM requests → closed transport → connection errors.

The Fix​

PR #22247 removed the _remove_key override entirely:

 class LLMClientCache(InMemoryCache):
-    def _remove_key(self, key: str) -> None:
-        """Close async clients before evicting them to prevent connection pool leaks."""
-        value = self.cache_dict.get(key)
-        super()._remove_key(key)
-        if value is not None:
-            close_fn = getattr(value, "aclose", None) or getattr(
-                value, "close", None
-            )
-            ...
-
     def update_cache_key_with_event_loop(self, key):

The eviction now simply drops the reference and lets Python's GC handle cleanup, which is safe because:

• httpx clients that are still referenced elsewhere stay alive
• Unreferenced clients get cleaned up by GC naturally

The other improvements from PR #21717 were kept:

• max_connections respected for URL-based Redis configs, previously silently dropped
• disconnect() now closes both sync and async Redis clients, sync client was previously leaked
• Connection pool passthrough, when a pool is provided with a URL config, it's used directly instead of creating a duplicate

Remediation​

The e2e tests go through get_async_httpx_client() the same code path the proxy uses in production and assert the client is still functional after eviction. These run in CI on every PR against main. If anyone modifies LLMClientCache eviction behavior, overrides _remove_key, or adds any form of client cleanup on eviction, these tests will fail regardless of the implementation approach.

Why phase matters for GPT-5.3-Codex​

phase appears on assistant output items and helps distinguish preamble/commentary turns from final closeout responses.

Reference: Phase parameter docs

Supported values:

• null
• "commentary"
• "final_answer"

Important:

• Persist assistant output items with phase exactly as returned.
• Send those assistant items back on the next turn.
• Do not add phase to user messages.

Docker Image​

docker pull ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3

Usage​

model_list:
  - model_name: gpt-5.3-codex
    litellm_params:
      model: openai/gpt-5.3-codex

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3 \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.3-codex",
    "input": "Write a Python script that checks if a number is prime."
  }'

Python Example: Persist phase with OpenAI Client + LiteLLM Base URL​

from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000/v1",  # LiteLLM Proxy
    api_key="your-litellm-api-key",
)

items = []  # Persist this per conversation/thread


def _item_get(item, key, default=None):
    if isinstance(item, dict):
        return item.get(key, default)
    return getattr(item, key, default)


def run_turn(user_text: str):
    global items

    # User message: no phase field
    items.append(
        {
            "type": "message",
            "role": "user",
            "content": [{"type": "input_text", "text": user_text}],
        }
    )

    resp = client.responses.create(
        model="gpt-5.3-codex",
        input=items,
    )

    # Persist assistant output items verbatim, including phase
    for out_item in (resp.output or []):
        items.append(out_item)

    # Optional: inspect latest phase for UI/telemetry routing
    latest_phase = None
    for out_item in reversed(resp.output or []):
        if _item_get(out_item, "type") == "output_item.done" and _item_get(out_item, "phase") is not None:
            latest_phase = _item_get(out_item, "phase")
            break

    return resp, latest_phase

Notes​

• Use /v1/responses for GPT Codex models.
• Preserve full assistant output history for best multi-turn behavior.
• If phase metadata is dropped during history reconstruction, output quality can degrade on long-running tasks.

Summary​

When load balancing OpenAI's Responses API across deployments with different API keys (e.g., different Azure regions or OpenAI organizations), follow-up requests containing encrypted content items (like rs_... reasoning items) would fail with:

{
  "error": {
    "message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
    "type": "invalid_request_error",
    "code": "invalid_encrypted_content"
  }
}

Encrypted content items are cryptographically tied to the API key's organization that created them. When the router load balanced a follow-up request to a deployment with a different API key, decryption failed.

• Responses API calls with encrypted content: Complete failure when routed to wrong deployment
• Initial requests: Unaffected — only follow-up requests containing encrypted items failed
• Other API endpoints: No impact — chat completions, embeddings, etc. functioned normally

Background​

OpenAI's Responses API can return encrypted "reasoning items" (with IDs like rs_...) that contain intermediate reasoning steps. These items are encrypted with the organization's key and can only be decrypted by the same organization's API key.

When load balancing across deployments with different API keys, the existing affinity mechanisms were insufficient:

• responses_api_deployment_check: Requires previous_response_id which some clients (like Codex) don't provide
• deployment_affinity: Too broad — pins all requests from a user to one deployment, reducing effective quota by the number of users
• session_affinity: Requires explicit session IDs and still reduces quota

Root Cause​

LiteLLM's router had no mechanism to track which deployment created specific encrypted content items and route follow-up requests accordingly. The router treated all deployments as interchangeable, leading to decryption failures when encrypted content crossed organizational boundaries.

The Problem Flow:

1. User calls router.aresponses() with model gpt-5.1-codex
2. Router load balances to Deployment A (Azure East US, API Key 1)
3. Response contains encrypted reasoning item rs_abc123 (encrypted with Org 1's key)
4. User makes follow-up request with rs_abc123 in the input
5. Router load balances to Deployment B (Azure West Europe, API Key 2)
6. Deployment B tries to decrypt rs_abc123 with Org 2's key → fails

Why Existing Solutions Didn't Work:

• previous_response_id: Not provided by all clients (e.g., Codex)
• deployment_affinity: Pins all user requests to one deployment → reduces quota to 1/N where N = number of deployments
• session_affinity: Requires explicit session management and still reduces quota

Timeline:

1. Users configured multi-region Responses API load balancing with different API keys
2. Initial requests succeeded, but follow-up requests with encrypted content failed intermittently
3. Error rate correlated with number of deployments (more deployments = higher chance of routing to wrong one)
4. Investigation revealed encrypted content was organization-bound
5. Existing affinity mechanisms deemed unsuitable (quota reduction, missing previous_response_id)
6. New solution designed and implemented: encrypted_content_affinity

The Fix​

Implemented a new encrypted_content_affinity pre-call check that intelligently tracks encrypted content and routes follow-up requests only when necessary.

Implementation​

1. Encoding model_id into output items (responses/utils.py)

The same approach used for previous_response_id affinity — no cache needed. When a response contains output items with encrypted_content, LiteLLM encodes the originating deployment's model_id in two places for redundancy:

1. Into the item ID (if present): rs_abc123 → encitem_{base64("litellm:model_id:{model_id};item_id:rs_abc123")}
2. Into the encrypted_content itself: Wraps the content with litellm_enc:{base64("model_id:{model_id}")};{original_encrypted_content}

# Encoding item IDs (when present)
def _build_encrypted_item_id(model_id: str, item_id: str) -> str:
    assembled = f"litellm:model_id:{model_id};item_id:{item_id}"
    encoded = base64.b64encode(assembled.encode("utf-8")).decode("utf-8")
    return f"encitem_{encoded}"

# Wrapping encrypted_content (always, for redundancy)
def _wrap_encrypted_content_with_model_id(encrypted_content: str, model_id: str) -> str:
    metadata = f"model_id:{model_id}"
    encoded_metadata = base64.b64encode(metadata.encode("utf-8")).decode("utf-8")
    return f"litellm_enc:{encoded_metadata};{encrypted_content}"

Why wrap encrypted_content directly? Some clients (like Codex) don't consistently send item IDs in follow-up requests, but they always send the encrypted_content itself. By embedding model_id into the content, affinity works even when IDs are missing.

Streaming responses: The wrapping logic is applied to both:

• Final response objects (non-streaming)
• Individual streaming events (response.output_item.added, response.output_item.done)

This ensures clients receiving streaming responses get wrapped content they can send back.

Before forwarding to the upstream provider, LiteLLM restores the original item IDs and unwraps encrypted_content so the provider never sees the encoded form:

# In responses/main.py — before calling the handler
input = ResponsesAPIRequestUtils._restore_encrypted_content_item_ids_in_input(input)

2. EncryptedContentAffinityCheck — routing only (encrypted_content_affinity_check.py)

No async_log_success_event or cache lookups — the model_id is decoded directly from the item ID or encrypted_content:

class EncryptedContentAffinityCheck(CustomLogger):
    async def async_filter_deployments(self, model, healthy_deployments, ...):
        """Extract model_id from input items (ID or encrypted_content) and pin to that deployment."""
        for item in request_kwargs.get("input", []):
            # Try to extract model_id from two sources:
            model_id = self._extract_model_id_from_input(item)
            
            if model_id:
                deployment = self._find_deployment_by_model_id(
                    healthy_deployments, model_id
                )
                if deployment:
                    request_kwargs["_encrypted_content_affinity_pinned"] = True
                    return [deployment]
        return healthy_deployments
    
    def _extract_model_id_from_input(self, item: dict) -> Optional[str]:
        """Extract model_id from either encoded ID or wrapped encrypted_content."""
        # 1. Try decoding from item ID (if present)
        item_id = item.get("id", "")
        if item_id:
            decoded = ResponsesAPIRequestUtils._decode_encrypted_item_id(item_id)
            if decoded:
                return decoded["model_id"]
        
        # 2. Try unwrapping from encrypted_content (fallback for clients that omit IDs)
        encrypted_content = item.get("encrypted_content", "")
        if encrypted_content and encrypted_content.startswith("litellm_enc:"):
            model_id, _ = ResponsesAPIRequestUtils._unwrap_encrypted_content_with_model_id(
                encrypted_content
            )
            return model_id
        
        return None

3. Rate Limit Bypass (router.py)

When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway):

# In async_get_available_deployment, after filtering healthy deployments:
if (
    request_kwargs.get("_encrypted_content_affinity_pinned")
    and len(healthy_deployments) == 1
):
    return healthy_deployments[0]  # Bypass routing strategy (RPM/TPM checks)

3. Configuration

router_settings:
  routing_strategy: usage-based-routing-v2
  enable_pre_call_checks: true
  optional_pre_call_checks:
    - encrypted_content_affinity
  deployment_affinity_ttl_seconds: 86400  # 24 hours

Key Benefits​

✅ No quota reduction: Only pins requests containing encrypted items
✅ Bypasses rate limits: When encrypted content requires a specific deployment, RPM/TPM limits don't block it
✅ No previous_response_id required: Works by encoding model_id directly into the item ID
✅ No cache required: model_id is decoded on-the-fly from the item ID — no Redis, no TTL
✅ Globally safe: Can be enabled for all models; non-Responses-API calls are unaffected
✅ Surgical precision: Normal requests continue to load balance freely

Remediation​

Follow-up Fix: Streaming Responses (Mar 3, 2026)​

The Issue​

After the initial fix was deployed, users reported that the invalid_encrypted_content error still occurred when using streaming responses with clients like Codex. Investigation revealed:

• ✅ Non-streaming responses: encrypted_content was correctly wrapped with litellm_enc: prefix
• ❌ Streaming responses: Individual response.output_item.added and response.output_item.done events contained raw, unwrapped encrypted_content

Since Codex and other clients consume responses as streams, they received unwrapped content in these events and sent it back in follow-up requests, causing the affinity check to fail.

The Root Cause​

The _update_encrypted_content_item_ids_in_response function only modified the final response object, which is used for non-streaming responses. For streaming responses, individual chunks are processed by ResponsesAPIStreamingIterator._process_chunk, which was not applying the wrapping logic to streaming events.

The Fix​

Modified litellm/litellm/responses/streaming_iterator.py to wrap encrypted_content in streaming events:

# In ResponsesAPIStreamingIterator._process_chunk
if (
    self.litellm_metadata
    and self.litellm_metadata.get("encrypted_content_affinity_enabled")
):
    event_type = getattr(openai_responses_api_chunk, "type", None)
    if event_type in (
        ResponsesAPIStreamEvents.OUTPUT_ITEM_ADDED,
        ResponsesAPIStreamEvents.OUTPUT_ITEM_DONE,
    ):
        item = getattr(openai_responses_api_chunk, "item", None)
        if item:
            encrypted_content = getattr(item, "encrypted_content", None)
            if encrypted_content and isinstance(encrypted_content, str):
                model_id = (
                    self.litellm_metadata.get("model_info", {}).get("id")
                    if self.litellm_metadata
                    else None
                )
                if model_id:
                    wrapped_content = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
                        encrypted_content, model_id
                    )
                    setattr(item, "encrypted_content", wrapped_content)

This ensures that all encrypted_content sent to clients (streaming or non-streaming) is wrapped with model_id metadata, enabling consistent affinity routing.

Migration Guide​

Before (Using deployment_affinity)​

router_settings:
  optional_pre_call_checks:
    - deployment_affinity  # ❌ Reduces quota by number of users

Problem: All requests from a user pin to one deployment, reducing effective quota to 1/N.

After (Using encrypted_content_affinity)​

router_settings:
  optional_pre_call_checks:
    - encrypted_content_affinity  # ✅ Only pins requests with encrypted content

Benefit: Normal requests load balance freely, only encrypted content requests pin when necessary.

Summary​

When a new Anthropic model (e.g. claude-sonnet-4-6) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:

key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.

The reload updated litellm.model_cost correctly but never re-ran add_known_models(), so litellm.anthropic_models (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the anthropic/* wildcard even though the cost map knew about it.

• LLM calls: All requests to newly-added Anthropic models were blocked with a 401.
• Existing models: Unaffected — only models missing from the stale provider set were impacted.
• Other providers: Same bug class existed for any provider wildcard (e.g. openai/*, gemini/*).

Background​

LiteLLM supports provider-level wildcard access rules. When an admin configures a key or team with models=['anthropic/*'], any model whose provider resolves to anthropic should be allowed. The resolution happens in _model_custom_llm_provider_matches_wildcard_pattern:

litellm.anthropic_models is a Python set populated at import time by add_known_models(). It is the source get_llm_provider() consults to map a bare model name like claude-sonnet-4-6 to the provider string "anthropic".

Root Cause​

add_known_models() is called once at module import time. Both reload paths in proxy_server.py updated litellm.model_cost with the fresh map but never called add_known_models() again:

# Before the fix — both reload paths looked like this:
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map          # ✅ cost map updated
_invalidate_model_cost_lowercase_map()           # ✅ cache cleared
# ❌ add_known_models() never called
#    → litellm.anthropic_models still has the old set
#    → new model not in the set
#    → get_llm_provider() raises for the new model
#    → wildcard match returns False
#    → 401 for every request to the new model

The gap existed in two places:

1. _check_and_reload_model_cost_map — the periodic automatic reload (every 10 s)
2. The /reload/model_cost_map admin endpoint — the manual reload

Timeline:

1. New model (claude-sonnet-4-6) added to model_prices_and_context_window.json
2. Admin triggers cost map reload via UI → litellm.model_cost updated
3. Users with anthropic/* wildcard keys attempt requests to claude-sonnet-4-6
4. get_llm_provider('claude-sonnet-4-6') raises → wildcard returns False → 401
5. Admin reloads cost map again — same result (root cause not addressed)
6. ~3 hours of investigation → root cause identified → fix deployed

The Fix​

After each reload, add_known_models() is called with the freshly fetched map passed explicitly. Passing the map directly (rather than relying on the module-level reference) removes any ambiguity about which dict is iterated:

# After the fix — both reload paths now do:
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map
_invalidate_model_cost_lowercase_map()
litellm.add_known_models(model_cost_map=new_model_cost_map)  # ✅ sets repopulated

add_known_models() was also updated to accept an optional explicit map so callers cannot accidentally iterate a stale module-level reference:

# Before
def add_known_models():
    for key, value in model_cost.items():   # reads module global — ambiguous after reload
        ...

# After
def add_known_models(model_cost_map: Optional[Dict] = None):
    _map = model_cost_map if model_cost_map is not None else model_cost
    for key, value in _map.items():         # always iterates the map you just fetched
        ...

After the fix, the provider sets (anthropic_models, open_ai_chat_completion_models, etc.) are always consistent with litellm.model_cost immediately after every reload. New models become accessible via wildcard rules without any proxy restart.

Remediation​

Note: This fix is available starting from LiteLLM v1.81.3.rc.6 or higher.

Summary​

A PR (#19467) accidentally removed the root_path=server_root_path parameter from the FastAPI app initialization in proxy_server.py. This caused the proxy to ignore the SERVER_ROOT_PATH environment variable when serving the UI. Users who deploy LiteLLM behind a reverse proxy with a path prefix (e.g., /api/v1 or /llmproxy) found that all UI pages returned 404 Not Found.

• LLM API calls: No impact. API routing was unaffected.
• UI pages: All UI pages returned 404 for deployments using SERVER_ROOT_PATH.
• Swagger/OpenAPI docs: Broken when accessed through the configured root path.

Background​

Many LiteLLM deployments run behind a reverse proxy (e.g., Nginx, Traefik, AWS ALB) that routes traffic to LiteLLM under a path prefix. FastAPI's root_path parameter tells the application about this prefix so it can correctly serve static files, generate URLs, and handle routing.

The root_path parameter was present in proxy_server.py since early versions of LiteLLM. It was removed as a side effect of PR #19467, which was intended to fix a different UI 404 issue.

Root cause​

PR #19467 (73d49f8) removed the root_path=server_root_path line from the FastAPI() constructor in proxy_server.py:

 app = FastAPI(
     docs_url=_get_docs_url(),
     redoc_url=_get_redoc_url(),
     title=_title,
     description=_description,
     version=version,
-    root_path=server_root_path,
     lifespan=proxy_startup_event,
 )

Without root_path, FastAPI treated all requests as if the application was mounted at /, causing path mismatches for any deployment using SERVER_ROOT_PATH.

The regression went undetected because:

1. No automated test verified that root_path was set on the FastAPI app.
2. No manual test procedure existed for SERVER_ROOT_PATH functionality.
3. Default deployments (without SERVER_ROOT_PATH) were unaffected, so most CI tests passed.

Remediation​

CI workflow details​

The new test_server_root_path.yml workflow runs on every PR against main. It:

1. Builds the LiteLLM Docker image
2. Starts a container with SERVER_ROOT_PATH set (tests both /api/v1 and /llmproxy)
3. Verifies the UI returns valid HTML at {ROOT_PATH}/ui/
4. Fails the workflow if the UI is unreachable

This prevents future regressions where changes to proxy_server.py accidentally break SERVER_ROOT_PATH support.

Timeline​

Resolution steps for users​

For users still experiencing issues, update to the latest LiteLLM version:

pip install --upgrade litellm

Verify your SERVER_ROOT_PATH is correctly set:

# In your environment or docker-compose.yml
SERVER_ROOT_PATH="/your-prefix"

Then confirm the UI is accessible at http://your-host:4000/your-prefix/ui/.

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.81.9-stable.gemini.3.1-pro

pip install litellm==v1.81.9-stable.gemini.3.1-pro

What's New​

1. New Thinking Levels: thinkingLevel with MINIMAL & MEDIUM​

Gemini 3.1 Pro introduces support for medium thinking level

LiteLLM automatically maps the OpenAI reasoning_effort parameter to Gemini's thinkingLevel, so you can use familiar reasoning_effort values (minimal, low, medium, high) without changing your code!

Supported Endpoints​

LiteLLM provides full end-to-end support for Gemini 3.1 Pro on:

• ✅ /v1/chat/completions - OpenAI-compatible chat completions endpoint
• ✅ /v1/responses - OpenAI Responses API endpoint (streaming and non-streaming)
• ✅ /v1/messages - Anthropic-compatible messages endpoint
• ✅ /v1/generateContent – Google Gemini API compatible endpoint

All endpoints support:

• Streaming and non-streaming responses
• Function calling with thought signatures
• Multi-turn conversations
• All Gemini 3-specific features
• Conversion of provider specific thinking related param to thinkingLevel

Quick Start​

from litellm import completion

# No need to make any changes to your code as we map openai reasoning param to thinkingLevel
response = completion(
    model="gemini/gemini-3.1-pro-preview",
    messages=[{"role": "user", "content": "Solve this complex math problem: 25 * 4 + 10"}],
    reasoning_effort="medium",  # NEW: MEDIUM thinking level
)

print(response.choices[0].message.content)

model_list:
  - model_name: gemini-3.1-pro-preview
    litellm_params:
      model: gemini/gemini-3.1-pro-preview
      api_key: os.environ/GEMINI_API_KEY
  - model_name: vertex-gemini-3.1-pro-preview
    litellm_params:
      model: vertex_ai/gemini-3.1-pro-preview

litellm --config /path/to/config.yaml

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
  -d '{
    "model": "gemini-3.1-pro-preview",
    "messages": [{"role": "user", "content": "Complex reasoning task"}],
    "reasoning_effort": "medium"
  }'

reasoning_effort Mapping for Gemini 3+​

Summary​

A commit (dbcae4a) intended to fix OpenAI SDK behavior broke vLLM embeddings by explicitly passing encoding_format=None in API requests. vLLM rejects this with error: "unknown variant \`, expected float or base64"`.

• vLLM embedding calls: Complete failure - all requests rejected
• Other providers: No impact - OpenAI and other providers functioned normally
• Other vLLM functionality: No impact - only embeddings were affected

Background​

The encoding_format parameter for embeddings specifies whether vectors should be returned as float arrays or base64 encoded strings. Different providers have different expectations:

• OpenAI SDK: If encoding_format is omitted, the SDK adds a default value of "float"
• vLLM: Strictly validates encoding_format - only accepts "float", "base64", or complete omission. Rejects None or empty string values.

Root cause​

A well-intentioned fix for OpenAI SDK behavior inadvertently broke vLLM embeddings:

The Breaking Change (dbcae4a):

In litellm/main.py, the code was changed to explicitly set encoding_format=None instead of omitting it:

# Added in dbcae4a
if encoding_format is not None:
    optional_params["encoding_format"] = encoding_format
else:
    # Omitting causes openai sdk to add default value of "float"
    optional_params["encoding_format"] = None

This fix worked correctly for OpenAI - explicitly passing None prevented the SDK from adding its default value. However, vLLM's strict parameter validation rejected None values, causing all embedding requests to fail.

The Fix​

Fix deployed (55348dd). The solution filters out None and empty string values from optional_params before sending requests to OpenAI-like providers (including vLLM).

In litellm/llms/openai_like/embedding/handler.py:

# Before (broken)
data = {"model": model, "input": input, **optional_params}

# After (fixed)
filtered_optional_params = {k: v for k, v in optional_params.items() if v not in (None, '')}
data = {"model": model, "input": input, **filtered_optional_params}

This ensures:

• Valid values ("float", "base64") are preserved and sent
• None and empty string values are filtered out (parameter omitted entirely)
• OpenAI SDK no longer adds defaults because liteLLM handles the parameter upstream

Remediation​

Docker Image​

docker pull ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6

Usage - Anthropic​

model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

from litellm import completion

response = completion(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": "what llm are you"}]
)
print(response.choices[0].message.content)

Usage - Azure​

model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: azure_ai/claude-sonnet-4-6
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

from litellm import completion

response = completion(
    model="azure_ai/claude-sonnet-4-6",
    api_key="your-azure-api-key",
    api_base="https://<resource>.services.ai.azure.com",
    messages=[{"role": "user", "content": "what llm are you"}]
)
print(response.choices[0].message.content)

Usage - Vertex AI​

model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: vertex_ai/claude-sonnet-4-6
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: us-east5

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

from litellm import completion

response = completion(
    model="vertex_ai/claude-sonnet-4-6",
    vertex_project="your-project-id",
    vertex_location="us-east5",
    messages=[{"role": "user", "content": "what llm are you"}]
)
print(response.choices[0].message.content)

Usage - Bedrock​

model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-4-6-v1
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

from litellm import completion

response = completion(
    model="bedrock/anthropic.claude-sonnet-4-6-v1",
    aws_access_key_id="your-access-key",
    aws_secret_access_key="your-secret-key",
    aws_region_name="us-east-1",
    messages=[{"role": "user", "content": "what llm are you"}]
)
print(response.choices[0].message.content)

Note: This fix will be available starting from v1.81.13-nightly or higher of LiteLLM.

Summary​

Claude Code began sending unsupported Anthropic beta headers to non-Anthropic providers (Bedrock, Azure AI, Vertex AI), causing invalid beta flag errors. LiteLLM was forwarding all beta headers without provider-specific validation. Users experienced request failures when routing Claude Code requests through LiteLLM to these providers.

• LLM calls to Anthropic: No impact.
• LLM calls to Bedrock/Azure/Vertex: Failed with invalid beta flag errors when unsupported headers were present.
• Cost tracking and routing: No impact.

Background​

Anthropic uses beta headers to enable experimental features in Claude. When Claude Code makes API requests, it includes headers like anthropic-beta: prompt-caching-scope-2026-01-05,advanced-tool-use-2025-11-20. However, not all providers support all Anthropic beta features.

Before this incident, LiteLLM forwarded all beta headers to all providers without validation:

Requests succeeded for Anthropic (native support) but failed for other providers when Claude Code sent headers those providers didn't support.

Root cause​

LiteLLM lacked provider-specific beta header validation. When Claude Code introduced new beta features or sent headers that specific providers didn't support, those headers were blindly forwarded, causing provider API errors.

Remediation​

Now LiteLLM validates and transforms headers per-provider:

Dynamic configuration updates​

A key improvement is zero-downtime configuration updates. When Anthropic releases new beta features, users can update their configuration without restarting:

# Manually trigger reload (no restart needed)
curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"

# Or schedule automatic reloads every 24 hours
curl -X POST "https://your-proxy-url/schedule/anthropic_beta_headers_reload?hours=24" \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"

This prevents future incidents where Claude Code introduces new headers before LiteLLM configuration is updated.

Configuration format​

The anthropic_beta_headers_config.json file maps input headers to provider-specific output headers:

{
  "description": "Mapping of Anthropic beta headers for each provider.",
  "anthropic": {
    "advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
    "computer-use-2025-01-24": "computer-use-2025-01-24"
  },
  "bedrock_converse": {
    "advanced-tool-use-2025-11-20": null,
    "computer-use-2025-01-24": "computer-use-2025-01-24"
  },
  "azure_ai": {
    "advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
    "computer-use-2025-01-24": "computer-use-2025-01-24"
  }
}

Validation rules:

1. Headers must exist in the mapping for the target provider
2. Headers with null values are filtered out (unsupported)
3. Header names can be transformed per-provider (e.g., Bedrock uses different names for some features)

Resolution steps for users​

For users still experiencing issues, update to the latest LiteLLM version if < v1.81.11-nightly:

pip install --upgrade litellm

Or manually reload the configuration without restarting:

curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"

Related documentation​

• Managing Anthropic Beta Headers - Complete configuration guide
• anthropic_beta_headers_config.json - Current configuration file

Supported Models​

LiteLLM supports the following MiniMax models:

Features Supported​

• Prompt Caching: Reduce costs with cached prompts ($0.03/M tokens for cache read, $0.375/M tokens for cache write)
• Function Calling: Built-in tool calling support
• Reasoning: Advanced reasoning capabilities with thinking support
• System Messages: Full system message support
• Cost Tracking: Automatic cost calculation for all requests

Docker Image​

docker pull litellm/litellm:v1.81.3-stable

Usage - OpenAI Compatible API (/v1/chat/completions)​

model_list:
  - model_name: minimax-m2-5
    litellm_params:
      model: minimax/MiniMax-M2.5
      api_key: os.environ/MINIMAX_API_KEY
      api_base: https://api.minimax.io/v1

docker run -d \
  -p 4000:4000 \
  -e MINIMAX_API_KEY=$MINIMAX_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.3-stable \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "minimax-m2-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

With Reasoning Split​

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "minimax-m2-5",
  "messages": [
    {
      "role": "user",
      "content": "Solve: 2+2=?"
    }
  ],
  "extra_body": {
    "reasoning_split": true
  }
}'

Usage - Anthropic Compatible API (/v1/messages)​

model_list:
  - model_name: minimax-m2-5
    litellm_params:
      model: minimax/MiniMax-M2.5
      api_key: os.environ/MINIMAX_API_KEY
      api_base: https://api.minimax.io/anthropic/v1/messages

docker run -d \
  -p 4000:4000 \
  -e MINIMAX_API_KEY=$MINIMAX_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.3-stable \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "minimax-m2-5",
  "max_tokens": 1000,
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

With Thinking​

curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "minimax-m2-5",
  "max_tokens": 1000,
  "thinking": {
    "type": "enabled",
    "budget_tokens": 1000
  },
  "messages": [
    {
      "role": "user",
      "content": "Solve: 2+2=?"
    }
  ]
}'

Usage - LiteLLM SDK​

OpenAI-compatible API​

import litellm

response = litellm.completion(
    model="minimax/MiniMax-M2.5",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
    api_key="your-minimax-api-key",
    api_base="https://api.minimax.io/v1"
)

print(response.choices[0].message.content)

Anthropic-compatible API​

import litellm

response = litellm.anthropic.messages.acreate(
    model="minimax/MiniMax-M2.5",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    api_key="your-minimax-api-key",
    api_base="https://api.minimax.io/anthropic/v1/messages",
    max_tokens=1000
)

print(response.choices[0].message.content)

With Thinking​

response = litellm.anthropic.messages.acreate(
    model="minimax/MiniMax-M2.5",
    messages=[{"role": "user", "content": "Solve: 2+2=?"}],
    thinking={"type": "enabled", "budget_tokens": 1000},
    api_key="your-minimax-api-key"
)

# Access thinking content
for block in response.choices[0].message.content:
    if hasattr(block, 'type') and block.type == 'thinking':
        print(f"Thinking: {block.thinking}")

With Reasoning Split (OpenAI API)​

response = litellm.completion(
    model="minimax/MiniMax-M2.5",
    messages=[
        {"role": "user", "content": "Solve: 2+2=?"}
    ],
    extra_body={"reasoning_split": True},
    api_key="your-minimax-api-key",
    api_base="https://api.minimax.io/v1"
)

# Access thinking and response
if hasattr(response.choices[0].message, 'reasoning_details'):
    print(f"Thinking: {response.choices[0].message.reasoning_details}")
print(f"Response: {response.choices[0].message.content}")

Cost Tracking​

LiteLLM automatically tracks costs for MiniMax-M2.5 requests. The pricing is:

• Input: $0.3 per 1M tokens
• Output: $1.2 per 1M tokens
• Cache Read: $0.03 per 1M tokens
• Cache Write: $0.375 per 1M tokens

Accessing Cost Information​

response = litellm.completion(
    model="minimax/MiniMax-M2.5",
    messages=[{"role": "user", "content": "Hello!"}],
    api_key="your-minimax-api-key"
)

# Access cost information
print(f"Cost: ${response._hidden_params.get('response_cost', 0)}")

Streaming Support​

OpenAI API​

response = litellm.completion(
    model="minimax/MiniMax-M2.5",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
    api_key="your-minimax-api-key",
    api_base="https://api.minimax.io/v1"
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Streaming with Reasoning Split​

stream = litellm.completion(
    model="minimax/MiniMax-M2.5",
    messages=[
        {"role": "user", "content": "Tell me a story"},
    ],
    extra_body={"reasoning_split": True},
    stream=True,
    api_key="your-minimax-api-key",
    api_base="https://api.minimax.io/v1"
)

reasoning_buffer = ""
text_buffer = ""

for chunk in stream:
    if hasattr(chunk.choices[0].delta, "reasoning_details") and chunk.choices[0].delta.reasoning_details:
        for detail in chunk.choices[0].delta.reasoning_details:
            if "text" in detail:
                reasoning_text = detail["text"]
                new_reasoning = reasoning_text[len(reasoning_buffer):]
                if new_reasoning:
                    print(new_reasoning, end="", flush=True)
                    reasoning_buffer = reasoning_text

    if chunk.choices[0].delta.content:
        content_text = chunk.choices[0].delta.content
        new_text = content_text[len(text_buffer):] if text_buffer else content_text
        if new_text:
            print(new_text, end="", flush=True)
            text_buffer = content_text

Using with Native SDKs​

Anthropic SDK via LiteLLM Proxy​

import os
os.environ["ANTHROPIC_BASE_URL"] = "http://localhost:4000"
os.environ["ANTHROPIC_API_KEY"] = "sk-1234"  # Your LiteLLM proxy key

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="minimax-m2-5",
    max_tokens=1000,
    system="You are a helpful assistant.",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Hi, how are you?"
                }
            ]
        }
    ]
)

for block in message.content:
    if block.type == "thinking":
        print(f"Thinking:\n{block.thinking}\n")
    elif block.type == "text":
        print(f"Text:\n{block.text}\n")

OpenAI SDK via LiteLLM Proxy​

import os
os.environ["OPENAI_BASE_URL"] = "http://localhost:4000"
os.environ["OPENAI_API_KEY"] = "sk-1234"  # Your LiteLLM proxy key

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="minimax-m2-5",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hi, how are you?"},
    ],
    extra_body={"reasoning_split": True},
)

# Access thinking and response
if hasattr(response.choices[0].message, 'reasoning_details'):
    print(f"Thinking:\n{response.choices[0].message.reasoning_details[0]['text']}\n")
print(f"Text:\n{response.choices[0].message.content}\n")

Summary​

A malformed JSON entry in model_prices_and_context_window.json was merged to main (562f0a0). This caused LiteLLM to silently fall back to a stale local copy of the model cost map. Users on older package versions lost cost tracking for newer models only (e.g. azure/gpt-5.2). No LLM calls were blocked.

• LLM calls and proxy routing: No impact.
• Cost tracking: Impacted for newer models not present in the local backup. Older models were unaffected. The incident lasted ~20 minutes until the commit was reverted.

Background​

The model cost map is not in the request path. It is used after the LLM response comes back, inside a try/catch, to calculate spend. A missing entry never blocks a call.

Both paths return a response to the caller. When the cost map lookup fails, the only difference is cost=0 on that request.

Root cause​

LiteLLM fetches the model cost map from GitHub main at import time. If the fetch fails, it falls back to a local backup bundled with the package. Before this incident, the fallback was completely silent -- no warning was logged.

A contributor PR introduced an extra { bracket, producing invalid JSON. The remote fetch failed with JSONDecodeError, triggering the silent fallback. Users on older package versions had backup files missing newer models.

Timeline:

1. Malformed JSON merged to main
2. LiteLLM installations fall back to local backup on next import
3. Users report "This model isn't mapped yet" for newer models
4. Bad commit identified and reverted (~20 minutes)

Remediation​