vLLM — User Guide

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models. Originally developed at UC Berkeley's Sky Computing Lab, vLLM enables production-grade LLM serving with PagedAttention, continuous batching, and a fully OpenAI-compatible API — all on your own hardware.

This guide covers everything you need to deploy, configure, and use vLLM: from running your first model in minutes to advanced multi-GPU parallelism, quantization, LoRA adapters, multimodal inference, and beyond. For architecture internals, deployment operations, and configuration deep-dives, see the vLLM Admin Guide.

Table of Contents

1. Quick Start
2. Installation
3. Configuration
4. Core Usage
5. Common Tasks & Workflow Recipes
6. Troubleshooting
7. FAQ

1. Quick Start

Get vLLM serving a model in under five minutes.

1.1 Prerequisites

• Python 3.9 or later (3.12 recommended)
• NVIDIA GPU with CUDA 12.1+ or AMD GPU with ROCm 6.x or a CPU-only environment
• At least 16 GiB of VRAM for a 7B parameter model (more for larger models)
• Internet access to download models from HuggingFace Hub

Tip

For your very first test, the facebook/opt-125m model runs on any GPU with 2+ GiB VRAM and downloads in seconds. Use it to verify your installation before pulling a multi-gigabyte model.

1.2 Install vLLM

pip install vllm

1.3 Start a Server

vllm serve meta-llama/Llama-3.1-8B-Instruct

This starts an OpenAI-compatible HTTP server on http://localhost:8000. The first run downloads model weights from HuggingFace Hub.

1.4 Send Your First Request

In a second terminal, send a chat completion:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "What is PagedAttention?"}
    ]
  }'

Or use the built-in interactive chat CLI:

vllm chat

Tip

The vllm chat and vllm complete commands connect to a running server at localhost:8000 by default. Start the server first, then open these in a second terminal for a quick interactive test without writing any code.

1.5 Quick Offline Inference (Python)

No server needed for batch offline workloads:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(
    ["Explain quantum computing in one paragraph."],
    SamplingParams(temperature=0.8, max_tokens=256)
)
for output in outputs:
    print(output.outputs[0].text)

1.6 CLI at a Glance

2. Installation

2.1 Platform Support Matrix

Important

The default pip install vllm installs the CUDA build. For ROCm, XPU, or CPU targets, you must use the appropriate index URL or build from source. Installing the CUDA wheel on a non-CUDA system will fail at runtime.

2.2 NVIDIA CUDA — pip (Recommended for Development)

# Create a virtual environment (strongly recommended)
python -m venv vllm-env
source vllm-env/bin/activate   # Linux/macOS
# vllm-env\Scripts\activate    # Windows

# Install
pip install vllm

Verify the installation:

python -c "import vllm; print(vllm.__version__)"
vllm --version

Warning

vLLM requires PyTorch 2.4+ and is sensitive to CUDA/PyTorch version mismatches. If you already have PyTorch installed, ensure it matches vLLM's requirements. Installing into a fresh virtual environment avoids version conflicts.

2.3 NVIDIA CUDA — Docker (Recommended for Production)

The official Docker image bundles CUDA, FlashInfer, and all dependencies:

# Pull the latest image
docker pull vllm/vllm-openai:latest

# Run with GPU access
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct

Default image base: nvidia/cuda:13.0.0-devel-ubuntu22.04, Python 3.12, FlashInfer 0.6.7.

Pass a HuggingFace token for gated models:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=hf_your_token_here \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4

Important

Mount the HuggingFace cache directory (~/.cache/huggingface) into the container so model weights persist between container restarts. Without this mount, the container re-downloads weights on every run.

2.4 AMD ROCm

# Install the ROCm-specific build
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.2

# Set the target device
export VLLM_TARGET_DEVICE=rocm

# Verify GPU visibility
export HIP_VISIBLE_DEVICES=0,1  # adjust as needed

2.5 Intel XPU

export VLLM_TARGET_DEVICE=xpu
pip install vllm  # XPU build from appropriate index

2.6 CPU-Only Installation

CPU mode works on Intel/AMD x86, ARM AArch64, Apple Silicon, and IBM Z. It is slower than GPU but requires no specialized hardware:

export VLLM_TARGET_DEVICE=cpu
pip install vllm

For CPU inference, also configure KV cache space:

export VLLM_CPU_KVCACHE_SPACE=8   # GiB of RAM for KV cache

2.7 Build Parallelism (Speeding Up Source Builds)

If building from source, control compilation parallelism:

export MAX_JOBS=8       # parallel compile jobs
export NVCC_THREADS=4  # NVCC threads per job
pip install -e .

2.8 Verifying Installation

# Check version
vllm --version

# Collect full environment info (share when filing bug reports)
vllm collect-env

# Quick smoke test (tiny model, minimal VRAM)
vllm serve facebook/opt-125m --port 8001 &
curl http://localhost:8001/health
curl http://localhost:8001/v1/models
kill %1

3. Configuration

vLLM can be configured via CLI flags, a YAML config file, and environment variables. All three can be combined; CLI flags take highest precedence, then YAML, then environment variables.

3.1 YAML Config File

Create a vllm_config.yaml to avoid long command lines:

model: meta-llama/Llama-3.1-8B-Instruct
host: 0.0.0.0
port: 8000
tensor-parallel-size: 2
gpu-memory-utilization: 0.90
max-model-len: 8192
enable-prefix-caching: true
dtype: bfloat16

Load it with:

vllm serve --config vllm_config.yaml

CLI flags override YAML values when both are specified.

3.2 Complete vllm serve Flag Reference

Model & Tokenizer

Networking

Memory & Performance

Parallelism

Quantization

LoRA

Advanced Features

Observability

3.3 Environment Variable Reference

3.4 API Key Authentication

To require an API key for all server requests:

export VLLM_API_KEY=my-secret-key
vllm serve meta-llama/Llama-3.1-8B-Instruct

Clients must then pass the key:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'

3.5 Disabling Usage Statistics

vLLM may collect anonymous usage statistics. To opt out:

export VLLM_NO_USAGE_STATS=1
# or equivalently:
export VLLM_DO_NOT_TRACK=1

4. Core Usage

4.1 OpenAI-Compatible API

vLLM's server is a drop-in replacement for the OpenAI API. Any client that works with OpenAI works with vLLM — point it at your vLLM host and port.

Chat Completions

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Summarize the French Revolution."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'

Text Completions

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The capital of France is",
    "max_tokens": 20,
    "temperature": 0
  }'

Streaming Responses

Add "stream": true to any request for Server-Sent Events:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Write a haiku about the ocean."}],
    "stream": true
  }'

Tip

Streaming dramatically improves perceived latency for long responses. Enable it in any production user-facing application — the first token arrives as soon as it is generated rather than waiting for the complete response.

List Available Models

curl http://localhost:8000/v1/models

Health Check

curl http://localhost:8000/health

4.2 Complete API Endpoint Reference

vLLM also supports:

• Anthropic Messages API — use with Anthropic-format clients
• Responses API — stateful multi-turn via the Responses protocol

4.3 Python Client (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-used"  # or your VLLM_API_KEY value
)

# Chat completion
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is PagedAttention?"}
    ],
    temperature=0.7,
    max_tokens=256
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "List 5 programming languages."}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

4.4 Offline Python API

The LLM class provides a high-level interface for batch inference without a server:

from vllm import LLM, SamplingParams

# Initialize (downloads model on first run)
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

# Define sampling parameters
params = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    max_tokens=512
)

# Batch generation
prompts = [
    "Explain the Pythagorean theorem.",
    "What is the capital of Japan?",
    "Write a limerick about Python programming."
]
outputs = llm.generate(prompts, params)

for output in outputs:
    print(f"Prompt: {output.prompt!r}")
    print(f"Output: {output.outputs[0].text!r}")
    print("---")

Chat Interface (Offline)

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

conversation = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
]

outputs = llm.chat(conversation, SamplingParams(temperature=0, max_tokens=50))
print(outputs[0].outputs[0].text)

Embeddings (Offline)

from vllm import LLM

llm = LLM(model="intfloat/e5-mistral-7b-instruct", task="embed")
outputs = llm.embed(["Hello, world!", "vLLM is fast."])
for output in outputs:
    print(output.outputs.embedding[:5])  # first 5 dimensions

Classification (Offline)

from vllm import LLM

llm = LLM(model="your-classification-model", task="classify")
outputs = llm.classify(["This movie is great!", "Terrible experience."])
for output in outputs:
    print(output.outputs.probs)

Scoring / Reranking (Offline)

from vllm import LLM

llm = LLM(model="your-reranker-model", task="score")
outputs = llm.score(
    query="What is PagedAttention?",
    texts=["PagedAttention is...", "Paris is the capital..."]
)
for output in outputs:
    print(output.outputs.score)

4.5 Quantization Methods

Quantization reduces VRAM usage and often increases throughput, at a small quality cost:

Apply quantization:

vllm serve TheBloke/Llama-2-7B-GPTQ --quantization gptq
vllm serve your-fp8-model --quantization fp8

Note

Online quantization (quantizing at load time) is supported for some methods. Pre-quantized models from HuggingFace Hub load faster. Check the model card to see if a pre-quantized variant exists before quantizing at load time.

4.6 Model Types Supported

vLLM supports 200+ model architectures. If your model is not listed, check the official vLLM docs for the full architecture support matrix.

5. Common Tasks & Workflow Recipes

5.1 Single-GPU Serving (Default)

# Serve Llama 3.1 8B on a single GPU
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

5.2 Multi-GPU with Tensor Parallelism

Tensor parallelism splits individual layers across GPUs. Use it when a model does not fit on one GPU, or to increase throughput:

# 4-GPU tensor parallel for Llama 3.1 70B
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --dtype bfloat16

Important

Tensor parallel size must divide evenly into the number of attention heads in the model. For example, a model with 32 attention heads supports -tp 1, -tp 2, -tp 4, -tp 8, and -tp 16, but not -tp 3 or -tp 5. Check the model config if you get a validation error.

5.3 Multi-Node with Pipeline Parallelism

Pipeline parallelism splits layers across nodes. Combine with tensor parallelism for very large models:

# 8 GPUs across 2 nodes: 4-way TP, 2-way PP
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --dtype bfloat16

See the Admin Guide for multi-node Ray cluster setup.

5.4 Data Parallelism

Run multiple full model replicas for higher throughput:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 4 \
  --tensor-parallel-size 1

5.5 Expert Parallelism for MoE Models

For Mixture-of-Experts models like Mixtral or DeepSeek-V3:

vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --dtype bfloat16

5.6 Serving a Quantized Model

# AWQ-quantized Mistral 7B
vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ --quantization awq

# GPTQ-quantized model
vllm serve TheBloke/Llama-2-13B-GPTQ --quantization gptq

# FP8
vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --quantization fp8

5.7 LoRA Adapter Serving

Enable multi-LoRA serving to dynamically load adapters per request:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-loras 4

Then specify a LoRA in your request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Translate to Spanish: Hello world"}],
    "lora_request": {
      "lora_name": "my-spanish-lora",
      "lora_int_id": 1,
      "lora_path": "/path/to/lora/adapter"
    }
  }'

5.8 Prefix Caching

Enable prefix caching to speed up requests that share a common prompt prefix (system prompts, few-shot examples, RAG context):

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-prefix-caching

Tip

Prefix caching is particularly effective for RAG applications where many requests share the same large retrieved context, and for chat applications where the system prompt is identical across sessions. Time-to-first-token drops dramatically for repeated prefixes.

5.9 Chunked Prefill

Chunked prefill allows long prompts to be processed incrementally alongside generation, preventing latency spikes for other requests:

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-chunked-prefill

5.10 Speculative Decoding

Speculative decoding accelerates generation by using a smaller draft model to propose tokens, then verifying with the main model in parallel. Several strategies are supported:

Draft Model

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"model": "meta-llama/Llama-3.1-8B-Instruct", "num_speculative_tokens": 5}'

EAGLE (Efficient Autoregressive Language Modeling)

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"method": "eagle", "model": "path/to/eagle-head"}'

N-gram Speculative Decoding

Uses n-gram matching in the prompt for speculation — no draft model required:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5}'

Note

Speculative decoding reduces latency when generation is memory-bandwidth bound and draft acceptance rates are high. It has no effect (or may slightly reduce throughput) at high request rates. Best suited for low-concurrency, latency-sensitive workloads. Known issue: thinking token budgets may not be enforced when MTP speculative decoding is active (see GitHub issue #39573).

5.11 Structured Output Generation

Force the model to produce valid JSON, matching a schema, or following a grammar:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --structured-outputs-config '{"backend": "xgrammar"}'

Then in your request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Extract: John Smith, age 30, engineer"}],
    "guided_json": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "occupation": {"type": "string"}
      },
      "required": ["name", "age", "occupation"]
    }
  }'

5.12 Multimodal Inference (Vision)

For vision-language models like LLaVA or Qwen-VL:

vllm serve llava-hf/llava-1.5-7b-hf --tokenizer-mode auto

Send images in your request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llava-hf/llava-1.5-7b-hf",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
          {"type": "text", "text": "What is in this image?"}
        ]
      }
    ],
    "max_tokens": 256
  }'

For video and audio, use the same multi-content format with appropriate media types.

Control media fetch timeouts:

export VLLM_IMAGE_FETCH_TIMEOUT=10   # seconds
export VLLM_VIDEO_FETCH_TIMEOUT=60   # seconds
export VLLM_AUDIO_FETCH_TIMEOUT=20   # seconds
export VLLM_MEDIA_CACHE_MAX_SIZE_MB=10240  # 10 GiB cache

5.13 Embeddings Server

vllm serve intfloat/e5-mistral-7b-instruct

Generate embeddings:

curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "intfloat/e5-mistral-7b-instruct",
    "input": ["Hello, world!", "vLLM embeddings are fast."]
  }'

For Cohere-compatible clients:

curl http://localhost:8000/v2/embed \
  -H "Content-Type: application/json" \
  -d '{
    "model": "intfloat/e5-mistral-7b-instruct",
    "texts": ["Hello", "World"]
  }'

5.14 Tokenization Endpoints

Inspect tokens without generating:

# Tokenize
curl http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello, world!"}'

# Detokenize
curl http://localhost:8000/detokenize \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "tokens": [15496, 11, 995, 0]}'

5.15 Tool Calling

vLLM supports OpenAI-format tool calling:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string"}
            },
            "required": ["city"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

5.16 Reasoning Models

Enable reasoning output parsing for thinking models:

vllm serve your-reasoning-model \
  --reasoning-config '{"backend": "deepseek_r1"}'

Warning

Known limitation: when using MTP speculative decoding alongside reasoning models, thinking token budgets may not be enforced correctly (GitHub issue #39573). If you need strict thinking budget control, disable speculative decoding with these models.

5.17 Serving with SSL/TLS

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --ssl-keyfile /path/to/server.key \
  --ssl-certfile /path/to/server.crt \
  --host 0.0.0.0 \
  --port 443

5.18 Limiting GPU Selection

Use environment variables to restrict which GPUs vLLM sees:

# NVIDIA — use only GPUs 0 and 1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 2

# AMD ROCm — use only GPU 0
HIP_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-3.1-8B-Instruct

5.19 Named API Model Alias

Expose a model under a different name for API compatibility:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name gpt-3.5-turbo

Clients can then use "model": "gpt-3.5-turbo" in requests without knowing the actual model path.

5.20 Benchmarking

vLLM includes a comprehensive benchmarking suite:

# Benchmark serving throughput (requires running server)
vllm bench serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --num-prompts 100

# Benchmark per-request latency (standalone, no server needed)
vllm bench latency \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-iters 20

# Benchmark offline throughput
vllm bench throughput \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-prompts 200

# Benchmark multimodal preprocessor
vllm bench mm-processor --model llava-hf/llava-1.5-7b-hf

# Benchmark server startup time
vllm bench startup --model meta-llama/Llama-3.1-8B-Instruct

# Parameter sweep
vllm bench sweep --model meta-llama/Llama-3.1-8B-Instruct

5.21 Disabling CUDA Graphs (Eager Mode)

CUDA graphs improve throughput but increase startup time and VRAM. Disable them for debugging or on older GPUs:

vllm serve meta-llama/Llama-3.1-8B-Instruct --enforce-eager

5.22 Custom Tokenizer Mode

Select the tokenizer backend explicitly:

# Default — auto-detect
vllm serve your-model --tokenizer-mode auto

# For Mistral models (use official Mistral tokenizer)
vllm serve mistralai/Mistral-7B-Instruct-v0.3 --tokenizer-mode mistral

# For DeepSeek V3.2
vllm serve deepseek-ai/DeepSeek-V3 --tokenizer-mode deepseek_v32

# For Qwen VL models
vllm serve Qwen/Qwen-VL-Chat --tokenizer-mode qwen_vl

5.23 torch.compile Integration

Enable torch.compile for potential throughput improvements on supported hardware:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --compilation-config '{"level": 3}'

5.24 Disaggregated Prefill/Decode

Separate prefill and decode phases across different instances for latency optimization:

# Configure KV transfer for disaggregated serving
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-transfer-config '{"kv_connector": "PyNcclConnector", "kv_role": "kv_producer"}'

Warning

Known issue: the OffloadingConnector for GPU-to-CPU KV offload may crash in certain configurations (GitHub issue #39491). If you need KV offloading, test thoroughly before deploying to production and monitor for crashes.

See the Admin Guide for full disaggregated prefill/decode architecture.

5.25 CPU Inference

For environments without GPUs:

export VLLM_TARGET_DEVICE=cpu
export VLLM_CPU_KVCACHE_SPACE=16  # GiB of RAM for KV cache

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 4096

Note

CPU inference is significantly slower than GPU inference for most workloads. It is best suited for development, testing, or low-concurrency use cases where GPU hardware is unavailable. For Apple Silicon Macs, performance is improved by using the MPS backend.

5.26 Collect Environment Info for Bug Reports

Before filing a GitHub issue, collect environment information:

vllm collect-env

This outputs CUDA version, PyTorch version, GPU info, installed packages, and other diagnostic data. Include this output in all bug reports.

5.27 Rendering Server (GPU-less)

Start a rendering server that does not require a GPU:

vllm launch render

5.28 Logging Configuration

Control log verbosity:

# Set log level
export VLLM_LOGGING_LEVEL=DEBUG

# Use a custom Python logging config
export VLLM_LOGGING_CONFIG_PATH=/path/to/logging.json

# Disable internal stats logging
vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-log-stats

# Disable all vLLM logging
export VLLM_CONFIGURE_LOGGING=0

5.29 Serving a Local Model

Use a local directory path instead of a HuggingFace model name:

vllm serve /models/my-fine-tuned-llama \
  --trust-remote-code \
  --dtype bfloat16

Tip

Point --model at any directory containing a config.json and model weights in HuggingFace format. This works with fine-tuned models, merged LoRA adapters, and models downloaded with huggingface-cli download.

5.30 Serving a Gated HuggingFace Model

# Authenticate with HuggingFace CLI first
huggingface-cli login

# Then serve normally
vllm serve meta-llama/Meta-Llama-3-70B-Instruct --tensor-parallel-size 4

Or pass the token via environment variable:

HUGGING_FACE_HUB_TOKEN=hf_your_token vllm serve meta-llama/Meta-Llama-3-70B-Instruct

6. Troubleshooting

Error: CUDA out of memory

Cause: The model plus KV cache exceeds available GPU VRAM.

Solution:

1. Reduce --gpu-memory-utilization (try 0.80 or 0.75).
2. Reduce --max-model-len to limit KV cache size.
3. Enable quantization: --quantization awq or --quantization gptq.
4. Reduce --max-num-seqs to limit concurrent sequences.
5. Use tensor parallelism across more GPUs: --tensor-parallel-size 2.
6. Enable chunked prefill: --enable-chunked-prefill.

Prevention: Leave at least 10% GPU memory headroom. Always set --gpu-memory-utilization 0.90 or lower for production deployments.

Error: The model's max seq len (X) is larger than the maximum number of tokens that can be stored in KV cache

Cause: --max-model-len is set higher than what fits in GPU memory given current --gpu-memory-utilization.

Solution:

1. Reduce --max-model-len to match your actual use case.
2. Increase --gpu-memory-utilization (up to ~0.95 on a dedicated GPU).
3. Add more GPUs and increase --tensor-parallel-size.
4. Use quantization to reduce memory per parameter.

Prevention: Set --max-model-len explicitly to the maximum context length your application needs, rather than using the model's full supported length.

Error: ValueError: tensor parallel size (X) does not divide attention head count (Y)

Cause: The --tensor-parallel-size value does not evenly divide the number of attention heads.

Solution:

1. Check the model's config.json for num_attention_heads.
2. Choose a -tp value that divides that number evenly (power-of-2 values typically work: 1, 2, 4, 8).

Prevention: Always verify that -tp N divides the model's attention head count before starting the server.

Error: ImportError: No module named 'vllm'

Cause: vLLM is not installed in the current Python environment.

Solution:

1. Activate the correct virtual environment: source vllm-env/bin/activate.
2. Verify installation: pip show vllm.
3. Reinstall if needed: pip install vllm.

Prevention: Always activate your virtual environment before running vLLM commands.

Error: requests.exceptions.ConnectionError: Failed to establish a new connection

Cause: The vLLM server is not running, or the client is pointed at the wrong host/port.

Solution:

1. Check that vllm serve is running and did not exit with an error.
2. Verify the server address and port match (--host and --port).
3. Check health: curl http://localhost:8000/health.
4. Check for firewall rules blocking the port.

Prevention: Always run a health check after starting the server before sending inference requests.

Error: Model is not found (404 from /v1/chat/completions)

Cause: The "model" field in your request does not match the model name vLLM loaded.

Solution:

1. Call GET /v1/models to see the exact model name the server reports.
2. Use --served-model-name your-alias if you want to use a custom name.
3. Update the "model" field in your request to match.

Prevention: Use GET /v1/models programmatically in your client to discover the active model name rather than hardcoding it.

Error: NCCL error during multi-GPU startup

Cause: NCCL communication initialization failed, usually due to network interface configuration or firewall rules between GPUs/nodes.

Solution:

1. Set NCCL_DEBUG=INFO to get detailed NCCL logs.
2. Verify all GPUs are visible: nvidia-smi.
3. For multi-node: ensure all nodes can reach each other on NCCL ports.
4. Try --enforce-eager to bypass CUDA graph issues.

Prevention: Test single-GPU serving before scaling to multi-GPU. For multi-node, test network connectivity between nodes before starting vLLM.

Error: RuntimeError: Expected all tensors to be on the same device

Cause: Version mismatch between vLLM, PyTorch, and CUDA, or a bug in a specific model's code path.

Solution:

1. Run vllm collect-env and check for version mismatches.
2. Reinstall vLLM in a fresh virtual environment.
3. Try --enforce-eager to disable CUDA graphs.
4. File a bug report with the output of vllm collect-env.

Prevention: Always install vLLM in an isolated virtual environment. Check the release notes for supported PyTorch versions.

Error: ValueError: trust_remote_code must be set to True

Cause: The model requires executing custom modeling code from HuggingFace Hub, but --trust-remote-code was not passed.

Solution:

1. Add --trust-remote-code to your vllm serve command.

Prevention: Review the model card before trusting remote code. Only use this flag for models from sources you trust.

Error: httpx.ReadTimeout or client timeout

Cause: The model is taking longer to respond than the client's configured timeout.

Solution:

1. Increase the client timeout (e.g., httpx.Client(timeout=120.0)).
2. Reduce max_tokens in your request.
3. Enable streaming so partial output arrives before the timeout.
4. Check GPU utilization — if the GPU is not being used, there may be a larger problem.

Prevention: Set client timeouts appropriate to your expected max_tokens. For long outputs, always use streaming.

Error: TokenizerMismatch or garbled output

Cause: The tokenizer mode does not match the model's expected tokenizer.

Solution:

1. Specify the correct --tokenizer-mode for your model family.
2. For Mistral models: --tokenizer-mode mistral.
3. For DeepSeek V3.2: --tokenizer-mode deepseek_v32.
4. For Qwen-VL: --tokenizer-mode qwen_vl.

Prevention: Always check the model card for tokenizer requirements before serving.

Error: MoE kernel config fails on GH200

Cause: Benchmark tuning for MoE kernels fails on NVIDIA GH200 hardware (GitHub issue #37191).

Solution:

1. Skip benchmark tuning when running on GH200.
2. Monitor the GitHub issue for a fix.

Prevention: Check hardware compatibility in the vLLM issue tracker when using bleeding-edge NVIDIA hardware.

Error: Ngram speculative decoding produces corrupted output on hybrid models

Cause: N-gram speculative decoding produces incorrect output on hybrid GDN (gated-dense-network) models (GitHub issue #39273).

Solution:

1. Disable ngram speculative decoding for hybrid GDN models.
2. Use a draft model or EAGLE instead.

Prevention: Do not use ngram speculative decoding on Mamba or hybrid SSM/attention architectures.

Error: Mamba prefix caching + MTP crashes on NemotronH

Cause: Combining Mamba prefix caching with MTP (multi-token prediction) speculative decoding on NemotronH models causes a crash (GitHub issue #39809).

Solution:

1. Disable prefix caching when using MTP with Mamba-based models: remove --enable-prefix-caching.
2. Or disable MTP speculative decoding.

Prevention: Do not combine prefix caching and MTP speculative decoding on Mamba/NemotronH architectures until this issue is resolved.

Error: OffloadingConnector GPU-to-CPU KV crash

Cause: The OffloadingConnector for disaggregated prefill/decode GPU-to-CPU KV transfer crashes in certain configurations (GitHub issue #39491).

Solution:

1. Do not use OffloadingConnector in production until the issue is resolved.
2. Use an alternative KV connector (PyNcclConnector if applicable).

Prevention: Monitor GitHub issue #39491 for status and patches.

Error: Thinking token budget not enforced with MTP speculative decoding

Cause: When using MTP (multi-token prediction) speculative decoding with reasoning models, the configured thinking token budget is not respected (GitHub issue #39573).

Solution:

1. Disable speculative decoding when strict thinking budget enforcement is required.
2. Set longer budgets as a workaround.

Prevention: Do not combine MTP speculative decoding with thinking-budget-sensitive reasoning workflows.

Server starts but requests hang indefinitely

Cause: The model is still loading (can take several minutes for large models), or there is a deadlock in the scheduler.

Solution:

1. Check server logs for progress messages — model loading is normal and can take 1-10 minutes.
2. Poll GET /health — it returns 200 only when the server is ready.
3. If the server is stuck after model loading completes, restart it.

Prevention: Always wait for the server to report healthy before sending requests. Implement a health check loop in your startup scripts.

High latency under load

Cause: GPU memory is full causing KV cache eviction, or --max-num-batched-tokens is too low.

Solution:

1. Enable chunked prefill: --enable-chunked-prefill.
2. Enable prefix caching: --enable-prefix-caching.
3. Increase --max-num-batched-tokens.
4. Reduce --max-model-len to free KV cache space.
5. Add more GPUs or replicas.

Prevention: Benchmark with vllm bench serve at your expected concurrency level before going to production.

Model loads but generates empty or nonsensical output

Cause: Wrong dtype, quantization mismatch, or the prompt format does not match the model's training.

Solution:

1. Ensure you are using the model's expected chat template format.
2. Try --dtype bfloat16 explicitly.
3. Verify the quantization method matches the model's format.
4. Use --enforce-eager to rule out CUDA graph issues.

Prevention: Check the model card for the recommended dtype and prompt format.

Out of CPU memory during model loading

Cause: Model is too large for system RAM during weight loading and conversion.

Solution:

1. Reduce the number of worker processes.
2. Use a pre-quantized model to reduce weight size.
3. Close other applications consuming RAM.
4. Use a machine with more RAM.

Prevention: Ensure system RAM is at least 2x the model's disk size to accommodate loading and conversion buffers.

Image/video/audio fetch fails or times out in multimodal requests

Cause: Default fetch timeouts are too short for large media files or slow network connections.

Solution:

1. Increase timeouts:

   export VLLM_IMAGE_FETCH_TIMEOUT=30
   export VLLM_VIDEO_FETCH_TIMEOUT=120
   export VLLM_AUDIO_FETCH_TIMEOUT=60
   

2. Pre-download media and serve from localhost.
3. Increase VLLM_MEDIA_CACHE_MAX_SIZE_MB to cache more media.

Prevention: For production multimodal workloads, serve media from a local or fast CDN rather than fetching from arbitrary URLs at inference time.

Port already in use

Cause: Another process (or a previous vLLM instance) is already using port 8000.

Solution:

1. Find the process: lsof -i :8000 (Linux/macOS) or netstat -ano | findstr :8000 (Windows).
2. Kill the old process or use a different port: vllm serve <model> --port 8001.

Prevention: Use a process manager (systemd, supervisor) to ensure only one vLLM instance runs per port.

ROCm: Model not running on GPU

Cause: VLLM_TARGET_DEVICE not set, or HIP_VISIBLE_DEVICES not configured correctly.

Solution:

1. Set export VLLM_TARGET_DEVICE=rocm.
2. Verify GPU visibility: rocm-smi.
3. Set HIP_VISIBLE_DEVICES=0,1,... explicitly.

Prevention: Always set VLLM_TARGET_DEVICE=rocm in your environment when running on AMD GPUs.

Structured output backend errors

Cause: The xgrammar or guidance backend may not be installed or may conflict with your Python environment.

Solution:

1. Install the appropriate backend: pip install xgrammar or pip install guidance.
2. Ensure no version conflicts with existing packages.

Prevention: Install structured output dependencies before enabling structured output configs.

WARNING: Your system has insufficient swap space

Cause: vLLM tries to use swap for KV cache overflows, but swap is unavailable.

Solution:

1. Add swap space on Linux: sudo fallocate -l 32G /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile.
2. Reduce --gpu-memory-utilization to leave more headroom.

Prevention: Provision servers with adequate RAM and swap for your workload.

Server exits immediately after starting

Cause: Missing required flags, unsupported GPU, or dependency error.

Solution:

1. Check the terminal output for the specific error message.
2. Run vllm collect-env to verify environment compatibility.
3. Ensure --model points to a valid model name or path.
4. Try with --enforce-eager and minimal flags to isolate the issue.

Prevention: Always test with a small known-good model (e.g., facebook/opt-125m) before serving your target model.

AttributeError or ModuleNotFoundError after upgrading vLLM

Cause: Breaking API changes between vLLM versions, or partially upgraded dependencies.

Solution:

1. Reinstall vLLM in a clean virtual environment: pip install --force-reinstall vllm.
2. Check the vLLM changelog for breaking changes between versions.
3. Update any vLLM-specific code to match the new API.

Prevention: Pin vLLM to a specific version in production and upgrade intentionally with testing.

Caution

Never expose the vLLM server on a public IP without authentication (VLLM_API_KEY) and rate limiting. An unauthenticated vLLM server allows anyone to consume your GPU resources and generate arbitrary text. Always place the server behind a reverse proxy or firewall in production.

Warning

Using --trust-remote-code executes Python code downloaded from HuggingFace Hub. Only use this flag with models from organizations you fully trust. Malicious model code could exfiltrate data, install backdoors, or compromise your server.

7. FAQ

Q: What is PagedAttention?

PagedAttention is vLLM's core memory management innovation. It manages the KV cache in fixed-size "pages" (similar to OS virtual memory) rather than allocating contiguous blocks. This eliminates internal fragmentation and allows much higher GPU memory utilization, enabling more concurrent requests than systems that pre-allocate per-sequence contiguous memory.

Q: How does vLLM compare to other serving frameworks?

vLLM focuses on high throughput and memory efficiency via PagedAttention and continuous batching. It provides a fully OpenAI-compatible API, making it a drop-in replacement. For most multi-user serving scenarios, vLLM's continuous batching provides best-in-class GPU utilization.

Q: What is continuous batching?

Continuous batching (also called in-flight batching) processes new requests as soon as GPU capacity is available, rather than waiting for an entire batch to finish. This means each GPU step can contain tokens from different requests at different stages of generation, maximizing utilization and reducing average latency.

Q: Can I run vLLM without a GPU?

Yes. Set VLLM_TARGET_DEVICE=cpu and use VLLM_CPU_KVCACHE_SPACE to allocate RAM for the KV cache. CPU inference is significantly slower than GPU inference but works on any modern Intel/AMD x86 machine, ARM AArch64, Apple Silicon, or IBM Z.

Q: What models does vLLM support?

vLLM supports 200+ model architectures, including decoder-only LLMs (Llama, Qwen, Gemma), MoE models (Mixtral, DeepSeek-V3), multimodal models (LLaVA, Qwen-VL, Pixtral), embedding models (E5-Mistral, GTE, ColBERT), and classification/reward models. Check the official vLLM documentation for the full, up-to-date list.

Q: What is the difference between tensor parallelism and pipeline parallelism?

Tensor parallelism (-tp) splits individual model layers across GPUs — each GPU computes part of every layer in every step. It reduces per-layer latency but requires fast GPU interconnect (NVLink preferred). Pipeline parallelism (-pp) assigns different layers to different GPUs. It works better over slower interconnects but introduces pipeline bubble overhead. They can be combined.

Q: Can I serve multiple models at once?

A single vllm serve instance serves one model. To serve multiple models, run multiple vllm serve instances on different ports and route requests with a load balancer or API gateway.

Q: What is the difference between LoRA adapters and full model serving?

A base model plus LoRA adapters allows dynamic switching of fine-tuned behaviors without loading separate full models. With --enable-lora --max-loras 4, vLLM keeps up to 4 adapters hot in GPU memory and applies the requested one per request. This is far more memory-efficient than loading multiple full fine-tunes when the adapters share the same base model.

Q: Does vLLM support streaming responses?

Yes. Add "stream": true to any /v1/chat/completions or /v1/completions request to receive Server-Sent Events as tokens are generated.

Q: How do I serve a fine-tuned model?

Point --model at the local directory containing your fine-tuned weights in HuggingFace format. If your fine-tune merges the LoRA into the base model, serve it like any other model. If it is a separate LoRA adapter, use --enable-lora and specify the adapter path per request.

Q: What quantization method should I use?

For NVIDIA H100/H200 or AMD MI300X, FP8 is the best tradeoff of quality and speed with native hardware support. For older NVIDIA GPUs, AWQ or GPTQ offer good quality at 4-bit. BitsAndBytes is the easiest to set up and works broadly. See §4.5 for the full list.

Q: How do I check if my model loaded successfully?

Poll GET /health — it returns HTTP 200 when the server is ready. Also call GET /v1/models to confirm the model name is registered.

Q: What is prefix caching and when should I enable it?

Prefix caching stores the KV activations for the shared prefix of many requests (e.g., a system prompt or RAG context) so they are not recomputed for each request. Enable it with --enable-prefix-caching. It is most beneficial when many requests share a long common prefix. It has negligible overhead when prefixes are not repeated.

Q: What is chunked prefill?

Chunked prefill processes long prompt prefills incrementally instead of all at once. This prevents a single long request from monopolizing the GPU and stalling shorter requests waiting in the queue. Enable with --enable-chunked-prefill. It generally improves latency fairness at the cost of slightly lower peak throughput.

Q: How do I use speculative decoding?

Pass a --speculative-config JSON string with the method and parameters. Options include draft_model (small draft model), eagle, ngram, suffix, pard, and mlp. Speculative decoding reduces latency when the draft model's token predictions are accepted at high rates by the main model. See §5.10 for examples.

Q: Is vLLM's API compatible with the OpenAI Python SDK?

Yes. Set base_url="http://your-vllm-host:8000/v1" in the OpenAI client constructor. All standard OpenAI API methods work. vLLM also supports the Anthropic Messages API format.

Q: How do I set up authentication?

Set the VLLM_API_KEY environment variable before starting the server. Clients must then pass Authorization: Bearer <key> in all requests. See §3.4 for details.

Q: How do I disable usage statistics collection?

Set VLLM_NO_USAGE_STATS=1 or VLLM_DO_NOT_TRACK=1 in your environment before starting vLLM.

Q: What Python versions are supported?

Python 3.9 or later is required. Python 3.12 is the recommended version (used in the official Docker image).

Q: What is the --served-model-name flag for?

It sets the model name returned by GET /v1/models and accepted in request "model" fields. Use it to present a friendly alias (e.g., gpt-4-turbo) without exposing the actual HuggingFace model path, or to ensure compatibility with clients that expect a specific model name.

Q: How do I report a bug?

1. Run vllm collect-env and copy the output.
2. Reproduce the issue with the minimal configuration.
3. File an issue at https://github.com/vllm-project/vllm/issues with the env output, minimal reproduction steps, and error messages.

Q: What is --dtype auto and when should I change it?

auto selects the dtype based on the model's config (usually bfloat16 for modern models). Override with --dtype float16 for GPUs that don't support bfloat16, or --dtype float32 for full precision (very high VRAM cost). Only change from auto if you know your GPU or model requires a specific dtype.

Q: How does vLLM handle requests when the GPU is at capacity?

vLLM queues incoming requests. The continuous batching scheduler adds new requests to in-flight batches as GPU capacity becomes available. No request is dropped unless you implement rate limiting at a proxy layer.

Q: Can I use vLLM on Windows?

vLLM is primarily developed for Linux. Windows users should use WSL2 (Windows Subsystem for Linux) with CUDA support, or Docker Desktop with GPU access. Native Windows installation is not officially supported.

Q: What is VLLM_TARGET_DEVICE=empty?

empty is a special device target for testing and development — it loads the vLLM framework without initializing any hardware backend. It is used for CI tests and development workflows that don't require actual GPU computation.

Q: How do I maximize throughput?

1. Enable continuous batching (enabled by default).
2. Enable prefix caching (--enable-prefix-caching) if requests share prefixes.
3. Enable chunked prefill (--enable-chunked-prefill).
4. Tune --max-num-seqs and --max-num-batched-tokens for your workload.
5. Use quantization to fit more KV cache in VRAM.
6. Use tensor parallelism to spread across multiple GPUs.
7. Benchmark with vllm bench serve to find the optimal configuration.

Q: Does vLLM support the Anthropic API format?

Yes. vLLM includes Anthropic Messages API support, allowing Anthropic SDK clients to connect directly to a vLLM server without modification.

Q: What is the Admin Guide for?

The vLLM Admin Guide covers architecture internals, deployment operations, Docker/container setup, Ray cluster configuration, monitoring, security hardening, and detailed configuration reference for operators and architects running vLLM in production.

Q: Where can I find the full model support list?

The complete list of 200+ supported architectures is maintained in the official vLLM documentation. The briefing data covers the major categories; check the official docs for your specific model.

Q: How are memory blocks allocated for the KV cache?

vLLM uses PagedAttention with fixed-size blocks. The --block-size flag controls block size. Smaller blocks reduce memory fragmentation for short sequences; larger blocks improve memory bandwidth for long sequences. The default is tuned automatically for most use cases.

Q: Does vLLM support tool calling with streaming?

Yes. Tool calling works with streaming responses. When the model outputs a tool call, the tool call arguments stream incrementally via SSE before the final response.

Q: Can I run multiple vLLM instances on the same machine?

Yes. Run each instance on a different port and use different CUDA_VISIBLE_DEVICES values to assign specific GPUs to each instance. This is useful for serving different models simultaneously or for A/B testing.

Q: What is the difference between /v1/embeddings and /v2/embed?

/v1/embeddings follows the OpenAI embeddings API format. /v2/embed follows the Cohere embeddings API format. Use whichever matches your client library. Both are served by the same underlying embedding model.

Q: How does vLLM handle very long contexts?

vLLM supports models with context lengths up to their maximum trained length (e.g., 128K or 1M tokens for some models). Use --max-model-len to cap the effective length. PagedAttention manages KV cache memory efficiently for long contexts. For very long inputs, also enable chunked prefill to avoid latency spikes.