LiteLLM User Guide

LiteLLM is an open-source AI Gateway that gives you one unified, OpenAI-format interface to 100+ large language model providers — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, and many more. You use it in one of two ways:

• Python SDK — pip install litellm, then call completion(...) from your code. One function, every provider.
• AI Gateway (Proxy Server) — run litellm as a standalone OpenAI-compatible HTTP service for your whole team, with virtual API keys, spend tracking, load balancing, guardrails, and an admin dashboard.

Both speak the OpenAI request/response format, so you can swap providers — or move from the SDK to the gateway — without rewriting your application code.

Note

This guide covers everyday usage of both the SDK and the gateway. For architecture, deployment topology, the full configuration reference, authentication internals, and operations, see the LiteLLM Admin Guide.

1. Quick Start

1.1 Python SDK in 30 seconds

pip install litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "sk-your-openai-key"

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

print(response.choices[0].message.content)

The return value is an OpenAI-style ModelResponse object: response.choices[0].message.content holds the text, and response.usage holds token counts. Switching to a different provider is a one-line change to the model string:

os.environ["ANTHROPIC_API_KEY"] = "sk-ant-your-key"

response = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

Tip

The model string follows the pattern provider/model-name. The prefix before the slash (openai/, anthropic/, gemini/, bedrock/, …) tells LiteLLM which provider to route to and which API key to read. See §5 Supported Providers.

1.2 AI Gateway in 60 seconds

pip install 'litellm[proxy]'
litellm --model gpt-4o

This starts an OpenAI-compatible server on http://0.0.0.0:4000. Point any OpenAI client at it:

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

For real deployments you start the gateway from a config.yaml instead of --model:

litellm --config config.yaml

A minimal config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-1234

Important

model_name is the public alias clients request. litellm_params.model is the real upstream model LiteLLM calls. This indirection is what lets you load-balance, rename, and swap models without touching client code. The full config reference lives in the Admin Guide.

2. Installation

LiteLLM is published on PyPI as litellm. The current version documented here is v1.88.0.

pip (SDK only)

pip install litellm

pip (with proxy server)

pip install 'litellm[proxy]'

uv

uv add litellm            # SDK in a project
uv tool install 'litellm[proxy]'   # proxy as a CLI tool

Docker (gateway)

docker run -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Note

Requirements: Python 3.8+ for the SDK. The proxy server adds FastAPI/uvicorn and (optionally) a PostgreSQL database when you enable virtual keys, teams, and spend tracking. Database setup is covered in the Admin Guide.

3. Core Concepts

LiteLLM (litellm) : The Python library. Import functions like completion, embedding, and acompletion directly. No server required.

AI Gateway / Proxy Server : A long-running HTTP service (litellm --config config.yaml) that exposes OpenAI-compatible endpoints, manages virtual keys and budgets, and routes to many upstream models. Also called the LiteLLM Proxy.

The model string (provider/model) : Every call identifies a model with a provider/model-name string. The provider prefix selects the adapter and the environment variable that holds its credentials. Examples: openai/gpt-4o, anthropic/claude-sonnet-4-20250514, gemini/gemini-2.5-pro, bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0, azure/<your-deployment>.

OpenAI format : Requests use OpenAI's messages array and parameters (temperature, max_tokens, tools, stream, …); responses use OpenAI's choices/usage shape — regardless of the underlying provider. LiteLLM translates to and from each provider's native API for you.

Router : The SDK component (and the engine inside the gateway) that load-balances across multiple deployments of the same model, applies retries, fallbacks, and rate limits. See §6.

4. Using the Python SDK

All SDK functions accept the OpenAI parameter set and return OpenAI-shaped objects. Every function has an a-prefixed async twin (acompletion, aembedding, …) usable with await.

4.1 Chat completions

from litellm import completion

response = completion(
    model="openai/gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain async I/O in one sentence."},
    ],
    temperature=0.7,
    max_tokens=200,
)
print(response.choices[0].message.content)
print(response.usage)   # prompt_tokens, completion_tokens, total_tokens

ModelResponse(... choices=[Choices(message=Message(content="Async I/O lets a single thread ..."))],
usage=Usage(prompt_tokens=24, completion_tokens=18, total_tokens=42))

4.2 Streaming

Set stream=True to receive incremental OpenAI-style chunks:

from litellm import completion

stream = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

To reassemble a streamed response into a single object, use stream_chunk_builder:

from litellm import completion, stream_chunk_builder

chunks = list(completion(model="openai/gpt-4o", messages=messages, stream=True))
final = stream_chunk_builder(chunks, messages=messages)
print(final.choices[0].message.content)

4.3 Async usage

import asyncio
from litellm import acompletion

async def main():
    response = await acompletion(
        model="gemini/gemini-2.5-pro",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())

4.4 Embeddings

from litellm import embedding

response = embedding(
    model="openai/text-embedding-3-small",
    input=["The quick brown fox", "jumps over the lazy dog"],
)
print(len(response.data))            # 2 vectors
print(len(response.data[0]["embedding"]))  # vector dimensions

Async: from litellm import aembedding.

4.5 Image generation

from litellm import image_generation

response = image_generation(
    model="openai/dall-e-3",
    prompt="A red panda coding at a laptop, watercolor",
    n=1,
    size="1024x1024",
)
print(response.data[0]["url"])

4.6 Audio — transcription and speech

from litellm import transcription, speech

# Speech-to-text (Whisper-compatible)
with open("audio.mp3", "rb") as f:
    result = transcription(model="openai/whisper-1", file=f)
print(result.text)

# Text-to-speech
tts = speech(model="openai/tts-1", input="Hello from LiteLLM", voice="alloy")
tts.stream_to_file("speech.mp3")

Async twins: atranscription, aspeech.

4.7 Moderation

from litellm import moderation

result = moderation(model="openai/omni-moderation-latest", input="some text to check")
print(result.results[0].flagged)

4.8 Reranking

from litellm import rerank

result = rerank(
    model="cohere/rerank-english-v3.0",
    query="What is the capital of the US?",
    documents=["Paris is in France.", "Washington, D.C. is the US capital."],
    top_n=2,
)
print(result.results)

Async: arerank.

4.9 The Responses API

LiteLLM mirrors OpenAI's newer Responses API (/responses) across providers:

from litellm import responses

resp = responses(
    model="openai/gpt-4o",
    input="Summarize the theory of relativity in two sentences.",
)
print(resp.output_text)

4.10 Token counting

from litellm import token_counter

n = token_counter(model="openai/gpt-4o", messages=[{"role": "user", "content": "hi"}])
print(n)

4.11 Setting provider credentials

LiteLLM reads provider keys from environment variables by default. The variable name is provider-specific (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, COHERE_API_KEY; AWS uses the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME). You can also pass credentials inline per call:

response = completion(
    model="openai/gpt-4o",
    messages=messages,
    api_key="sk-...",          # overrides the env var
    api_base="https://my-proxy.example.com/v1",  # custom endpoint
)

Tip

Provider-specific parameters that aren't part of the OpenAI schema can be passed straight through (e.g. top_k, reasoning_effort). If a provider doesn't support a parameter you sent, set drop_params=True to have LiteLLM silently drop unsupported params instead of erroring.

4.12 Handling exceptions

LiteLLM maps every provider's errors onto the OpenAI exception types, so you write one set of handlers:

from litellm import completion
from litellm.exceptions import (
    AuthenticationError, RateLimitError, ContextWindowExceededError,
    BadRequestError, Timeout, APIConnectionError, ServiceUnavailableError,
)

try:
    response = completion(model="openai/gpt-4o", messages=messages)
except RateLimitError:
    ...   # back off and retry
except ContextWindowExceededError:
    ...   # trim the prompt
except AuthenticationError:
    ...   # bad/missing API key

5. Supported Providers

LiteLLM ships adapters for 100+ providers (123 provider integrations in this version). You select one with the model-string prefix; the full, always-current matrix of which providers support which endpoints lives at models.litellm.ai and in the provider docs.

A selection of commonly used providers and their model-string prefixes:

Note

Not every provider supports every capability. Bedrock and Cohere support /rerank; Azure and Vertex support image and audio endpoints; many OpenAI-compatible providers support only /chat/completions. Check the capability matrix on models.litellm.ai before relying on a less common endpoint for a given provider.

Calling any OpenAI-compatible endpoint

If a provider speaks the OpenAI API but isn't a built-in integration, use the generic openai/ prefix with a custom api_base:

response = completion(
    model="openai/my-model",
    messages=messages,
    api_base="https://my-openai-compatible-host/v1",
    api_key="sk-...",
)

6. Reliability: Routing, Retries, and Fallbacks

For production workloads you usually want more than a single model call: load balancing across several API keys or regions, automatic retries, and fallbacks to a backup model. The Router provides this in the SDK (the gateway uses the same engine — configure it in config.yaml, covered in the Admin Guide).

6.1 Load balancing across deployments

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o",                       # public alias
            "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-key-1"},
            "tpm": 100000, "rpm": 1000,
        },
        {
            "model_name": "gpt-4o",                       # same alias, 2nd deployment
            "litellm_params": {"model": "azure/gpt-4o-deployment",
                                "api_key": "azure-key", "api_base": "https://...azure.com"},
        },
    ],
    routing_strategy="usage-based-routing-v2",
)

response = router.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

The router spreads requests across both deployments of gpt-4o according to the chosen strategy and the per-deployment tpm/rpm limits.

6.2 Retries and fallbacks

router = Router(
    model_list=model_list,
    num_retries=2,                       # retry transient failures
    fallbacks=[{"gpt-4o": ["claude-sonnet"]}],   # if gpt-4o fails, try claude-sonnet
    context_window_fallbacks=[{"gpt-4o": ["gpt-4o-large-context"]}],
)

Tip

Without the Router, completion_with_retries(...) gives you simple retry behavior for a single model. Use the Router when you have multiple deployments or want fallbacks.

6.3 One-off retries without a router

from litellm import completion_with_retries

response = completion_with_retries(
    model="openai/gpt-4o",
    messages=messages,
    num_retries=3,
)

7. Caching (SDK)

LiteLLM can cache responses so identical requests don't re-hit the provider:

import litellm
from litellm import completion
from litellm.caching.caching import Cache

litellm.cache = Cache()   # in-memory by default; supports Redis, S3, disk, etc.

# First call hits the provider; the second identical call is served from cache.
completion(model="openai/gpt-4o", messages=messages, caching=True)
completion(model="openai/gpt-4o", messages=messages, caching=True)

Note

For Redis-backed caching, pass Cache(type="redis", host=..., port=..., password=...). The gateway exposes cache configuration in config.yaml and cache-management endpoints — see the Admin Guide.

8. Using the AI Gateway (as a client)

Once an admin has deployed the LiteLLM gateway (see the Admin Guide), you interact with it as an OpenAI-compatible service. You authenticate with a virtual key (a sk-... token your admin issues), not the raw provider keys.

8.1 Making requests

OpenAI Python SDK

import openai

client = openai.OpenAI(
    api_key="sk-your-virtual-key",
    base_url="http://your-gateway:4000",
)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

curl

curl http://your-gateway:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

LiteLLM SDK → gateway

from litellm import completion

response = completion(
    model="litellm_proxy/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    api_key="sk-your-virtual-key",
    api_base="http://your-gateway:4000",
)

The gateway exposes the standard OpenAI surface — /chat/completions, /completions, /embeddings, /images/generations, /audio/transcriptions, /audio/speech, /moderations, /rerank, /responses, /messages (Anthropic format), and /models — plus /v1/... equivalents.

Important

You request the public model name your admin configured (e.g. gpt-4o), not the upstream model string. Run GET /models (or litellm-proxy models list) to see what's available to your key.

8.2 The litellm-proxy CLI

Installing litellm[proxy] also gives you the litellm-proxy command-line client for talking to a running gateway. Point it at your gateway and key:

export LITELLM_PROXY_URL=http://your-gateway:4000
export LITELLM_PROXY_API_KEY=sk-your-virtual-key

# Quick interactive chat
litellm-proxy chat gpt-4o --system "You are concise." --temperature 0.3

# See which models you can call
litellm-proxy models list

Note

Key, user, team, and credential management commands require an admin (master-key or admin-role) token. As a regular user you'll mostly use chat, models list, and http request.

9. Advanced Features

9.1 Tool / function calling

Tool calling works in the OpenAI format across every provider that supports it:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
    },
}]

response = completion(model="openai/gpt-4o", messages=messages, tools=tools)
tool_calls = response.choices[0].message.tool_calls

9.2 MCP tools (Model Context Protocol)

LiteLLM can load tools from an MCP server and expose them to any model. Via the SDK bridge:

from litellm import experimental_mcp_client
import litellm

tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
response = await litellm.acompletion(model="gpt-4o", messages=messages, tools=tools)

Via the gateway, reference a registered MCP server directly in the request:

curl -X POST 'http://your-gateway:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-your-virtual-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

The gateway can also be used as an MCP endpoint directly from tools like Cursor by pointing them at http://your-gateway:4000/mcp/ with an x-litellm-api-key header.

9.3 A2A agents

LiteLLM exposes an Agent-to-Agent (A2A) gateway for invoking agents (LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI) behind a virtual key:

base_url = "http://your-gateway:4000/a2a/my-agent"   # gateway + agent name
headers = {"Authorization": "Bearer sk-your-virtual-key"}
# ...use the a2a-sdk A2AClient to send messages (see README for the full example)

9.4 Batch completions

Send the same prompt to multiple models, or run many prompts, in one call:

from litellm import batch_completion

responses = batch_completion(
    model="openai/gpt-4o",
    messages=[[{"role": "user", "content": q}] for q in questions],
)

10. Common Tasks (Recipes)

Swap a provider without code changes : Change only the model string: openai/gpt-4o → anthropic/claude-sonnet-4-20250514. Set the matching API key env var. Everything else stays the same.

Drop unsupported parameters automatically : Pass drop_params=True (or set litellm.drop_params = True globally) so provider-incompatible params are dropped instead of raising BadRequestError.

Get consistent token usage across providers : Read response.usage.total_tokens — LiteLLM normalizes usage reporting even when providers report it differently.

Use a local model : Run Ollama locally and call completion(model="ollama/llama3", messages=...). No API key needed.

Point the SDK at your team's gateway : Prefix the model with litellm_proxy/ and pass api_base/api_key for the gateway, so the SDK routes through your gateway's keys, budgets, and logging.

Verbose debugging : Set litellm.set_verbose = True (SDK) or start the gateway with --detailed_debug to see the exact upstream request/response.

11. Troubleshooting

Error: AuthenticationError - Missing API Key

Cause: The environment variable for the provider isn't set, or the wrong variable name is used. Solution: Set the provider's key (e.g. export OPENAI_API_KEY=sk-...). Confirm the variable name matches the provider (ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.). Prevention: Load keys from a .env file at startup, or pass api_key=... explicitly per call.

Error: BadRequestError - <param> is not supported

Cause: You passed a parameter the target provider doesn't accept. Solution: Set drop_params=True so LiteLLM drops unsupported params, or remove the parameter. Prevention: Set litellm.drop_params = True globally when calling many heterogeneous providers.

Error: ContextWindowExceededError

Cause: Prompt tokens plus max_tokens exceed the model's context window. Solution: Trim the prompt, lower max_tokens, or configure context_window_fallbacks to a larger-context model. Prevention: Use token_counter(...) to check size before sending.

Error: RateLimitError (HTTP 429)

Cause: The upstream provider throttled you. Solution: Use a Router with num_retries and multiple deployments to spread load; add fallbacks to a secondary provider. Prevention: Set per-deployment tpm/rpm so the router paces requests.

Error: Timeout

Cause: The provider didn't respond within the timeout. Solution: Increase timeout= on the call (or --request_timeout on the gateway). Add retries. Prevention: Set a sensible default timeout and retry policy for production traffic.

Gateway: Invalid proxy server token passed

Cause: The Authorization: Bearer token isn't a valid virtual key (or you used the master key where a virtual key is expected). Solution: Generate a virtual key (litellm-proxy keys generate or the dashboard) and use it. Confirm base_url points at the gateway, not a provider. Prevention: Never ship the master key to clients — issue scoped virtual keys.

Streaming returns nothing / buffers

Cause: A client or intermediary is buffering the response, or the provider's streaming isn't enabled. Solution: Confirm stream=True and that your HTTP client reads incrementally. When going through the gateway, ensure no proxy in front buffers SSE responses.

Warning

Some providers silently degrade unsupported features rather than erroring — e.g. strict JSON-schema response formats may fall back to looser JSON on certain providers, and tool_choice can be dropped on cached requests for some providers. If strict behavior matters, verify the provider supports it on models.litellm.ai and test end-to-end.

12. FAQ

Q: Do I have to use the OpenAI format? Yes — that's the point. You always send OpenAI-format requests; LiteLLM translates to each provider. It also accepts and exposes the Anthropic /messages format through a compatibility layer if you prefer it.

Q: What's the difference between the SDK and the gateway? The SDK is a library you import into one application. The gateway is a shared HTTP service for many applications/users, adding virtual keys, budgets, spend tracking, load balancing, guardrails, and a dashboard. They share the same routing engine.

Q: Can I use LiteLLM with the official OpenAI client? Yes. Point openai.OpenAI(base_url="http://your-gateway:4000", api_key="sk-virtual-key") at the gateway. No code changes beyond the base URL and key.

Q: How do I add a provider that isn't built in? If it's OpenAI-compatible, use the openai/ prefix with a custom api_base. Otherwise LiteLLM supports custom provider handlers (see the Admin Guide / provider docs).

Q: Does LiteLLM support local models? Yes — Ollama (ollama/...), self-hosted vLLM (hosted_vllm/...), LM Studio, llamafile, and others. No API key needed for local servers.

Q: How do I control which models a key can use? On the gateway, virtual keys can be scoped to specific models, teams, and budgets. Admins manage this via the dashboard, the management API, or litellm-proxy keys generate --models .... See the Admin Guide.

Q: How do I see how many tokens / how much I'm spending? Through the gateway: spend is tracked per key/user/team and viewable in the dashboard and via /spend/* endpoints. In the SDK, read response.usage and use LiteLLM's cost helpers.

Q: Is streaming supported everywhere? Streaming is supported for chat completions across providers that offer it. Use stream=True and iterate the response; use stream_chunk_builder to reassemble.

Q: Which Python version do I need? Python 3.8 or newer for the SDK.

Q: Where's the full, always-current provider/model list? models.litellm.ai and docs.litellm.ai/docs/providers.

13. Quick Reference