llama.cpp — User Guide

Table of Contents

1. Quick Start
2. Installation
3. Configuration
4. Core Usage
5. Common Tasks
6. Troubleshooting
7. FAQ

1. Quick Start

llama.cpp is a high-performance C/C++ inference engine for running Large Language Models (LLMs) locally on consumer hardware. It supports the GGUF model format with aggressive quantization, enabling models that would normally require datacenter hardware to run on a laptop, desktop, or even a mobile device. The library provides a complete toolkit: an interactive CLI, an OpenAI-compatible HTTP server, model conversion scripts, quantization tools, and SDKs for Android and iOS.

Key Features

• Local LLM inference — no cloud, no API fees, full privacy; runs entirely on your hardware
• GGUF model format — unified, self-describing format carrying weights, tokenizer, and metadata in one file
• 30+ quantization types — from 1-bit ternary (TQ1_0) to full float16, tuned for every RAM/quality tradeoff
• Multi-backend GPU acceleration — CUDA (NVIDIA), Metal (Apple Silicon), Vulkan (cross-platform), OpenCL, SYCL (Intel), Hexagon NPU (Snapdragon), CANN (Ascend), OpenVINO
• OpenAI-compatible server — drop-in replacement for OpenAI's /v1/chat/completions and /v1/completions APIs with built-in web UI
• Interactive CLI — colored REPL with session commands, file input, image/audio multimodal support
• Multimodal support — vision models (image + text) and audio models via llava, minicpmv, and related architectures
• Model conversion — scripts to convert HuggingFace checkpoints directly to GGUF, with LoRA adapter support

Prerequisites

Quick Build and Run

# Clone the repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Configure and build (CPU only)
cmake -B build
cmake --build build --config Release

# Download a model (example — substitute your own GGUF)
# wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

# Run interactive chat
./build/bin/llama-cli -m llama-2-7b.Q4_K_M.gguf -c 4096

# Or run a single prompt and exit
./build/bin/llama-cli -m llama-2-7b.Q4_K_M.gguf -p "Explain quantum entanglement in plain English." -n 256

Tip

On Windows the binary is at build\bin\Release\llama-cli.exe. On macOS with Apple Silicon, Metal acceleration is enabled automatically — no extra flags needed.

What to Try Next

• GPU acceleration → §2.3 Build Options (GPU Backends)
• HTTP server → §4.3 Server Mode (llama-server)
• Quantize a model → §4.5 Quantization
• Convert a HuggingFace model → §4.4 Model Conversion
• Common recipes → §5 Common Tasks

2. Installation

2.1 System Requirements

RAM requirements by model size:

Note

These are approximate figures for CPU-only inference. GPU VRAM requirements match the RAM figures above since the model weights must fit in VRAM when fully offloaded.

2.2 Build from Source (Primary Method)

Building from source is the standard and recommended installation method. Pre-built binaries are available on the GitHub Releases page as a convenience, but source builds give you full control over backends and optimizations.

Basic CPU Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release -j$(nproc)

Binaries are placed in build/bin/ on Linux/macOS and build/bin/Release/ on Windows.

Parallel Build Threads

# Use all available cores
cmake --build build --config Release -j$(nproc)

# Or a fixed count
cmake --build build --config Release -j8

2.3 Build Options (GPU Backends)

CUDA (NVIDIA GPUs)

Requires CUDA Toolkit 11.8 or later installed and nvcc on your PATH.

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

To target a specific GPU compute capability:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89;90"
cmake --build build --config Release -j$(nproc)

Tip

Common architectures: 75 (Turing / RTX 20xx), 86 (Ampere / RTX 30xx), 89 (Ada / RTX 40xx), 90 (Hopper / H100).

Metal (Apple Silicon and Intel Mac)

Metal is detected and enabled automatically on macOS. No extra CMake flag is required. If you want to force-disable it:

cmake -B build -DGGML_METAL=OFF
cmake --build build --config Release

For Apple Silicon, the build also links against Apple's Accelerate framework (BLAS) automatically.

Vulkan (Cross-Platform GPU)

Works on Linux, Windows, and macOS. Requires the Vulkan SDK (available from vulkan.lunarg.com).

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

OpenCL

For AMD GPUs and other OpenCL-capable devices on platforms where Vulkan is not available.

cmake -B build -DGGML_OPENCL=ON
cmake --build build --config Release -j$(nproc)

SYCL (Intel GPUs / oneAPI)

Requires Intel oneAPI Base Toolkit with DPC++ compiler.

# After sourcing Intel oneAPI environment:
source /opt/intel/oneapi/setvars.sh

cmake -B build -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
cmake --build build --config Release -j$(nproc)

BLAS (CPU Matrix Acceleration)

Improves prompt processing speed on CPU-only builds. Requires OpenBLAS or another BLAS implementation.

# OpenBLAS
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build --config Release -j$(nproc)

Available vendors: OpenBLAS, ATLAS, Intel10_64lp (MKL), Apple (Accelerate).

Additional CMake Options

2.4 CMake Presets

The repository ships a CMakePresets.json with ready-to-use configurations. List available presets with:

cmake --list-presets

Common presets:

# Linux, GCC, release
cmake --preset x64-linux-gcc-release
cmake --build --preset x64-linux-gcc-release

# Windows, MSVC, release
cmake --preset x64-windows-msvc-release
cmake --build --preset x64-windows-msvc-release

# Apple Silicon, Clang, release
cmake --preset arm64-apple-clang-release
cmake --build --preset arm64-apple-clang-release

2.5 Docker

A Dockerfile is provided in the repository. The quickest path:

# CPU-only image
docker build -t llama-cpp .

# Run interactive chat
docker run --rm -it \
  -v /path/to/models:/models \
  llama-cpp \
  llama-cli -m /models/model.gguf -c 4096

For CUDA support, use the CUDA variant:

docker build --build-arg GGML_CUDA=1 -t llama-cpp-cuda .

docker run --rm -it --gpus all \
  -v /path/to/models:/models \
  llama-cpp-cuda \
  llama-cli -m /models/model.gguf -ngl 99 -c 4096

2.6 Python Package (gguf-py)

The GGUF Python library enables programmatic reading and writing of GGUF files.

pip install gguf

This installs the gguf package (from gguf-py/) with GGUFReader and GGUFWriter classes, plus command-line utilities for inspecting GGUF files.

2.7 Nix

A flake.nix is present at the repository root:

# Enter dev shell
nix develop

# Or build directly
nix build

2.8 iOS / macOS XCFramework

For embedding llama.cpp in an Apple platform app, use the XCFramework build script:

./build-xcframework.sh

This produces a universal XCFramework with Metal and Accelerate backends, targeting:

• iOS 16.4+
• macOS 13.3+
• visionOS 1.0+
• tvOS 16.4+

2.9 Android

A native Android application is provided in examples/llama.android/. Open the directory in Android Studio, build, and deploy. The app uses JNI to call into llama.cpp with CPU inference (NEON SIMD on ARM64).

For Snapdragon Hexagon NPU acceleration, see the Hexagon environment variables in §3.3.

2.10 Post-Install Verification

# Check the binary runs
./build/bin/llama-cli --version

# Verify GPU backend loaded (look for "CUDA" or "Metal" or "Vulkan" in output)
./build/bin/llama-cli --list-devices

# Quick smoke test with a model
./build/bin/llama-cli -m model.gguf -p "Say hello" -n 16 --no-display-prompt

2.11 Build Troubleshooting

Error: cmake: command not found

Install CMake from your package manager or cmake.org. Minimum version is 3.14.

# Ubuntu/Debian
sudo apt install cmake

# macOS
brew install cmake

# Windows (via winget)
winget install Kitware.CMake

Error: CUDA not found during cmake configure

Ensure nvcc is on your PATH and CUDA_HOME / CUDA_PATH is set:

export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH

Error: undefined reference to __atomic_fetch_add_8

On older GCC versions, link against -latomic:

cmake -B build -DCMAKE_EXE_LINKER_FLAGS="-latomic"

Error: Metal shader compilation fails

Ensure Xcode Command Line Tools are fully installed:

xcode-select --install
sudo xcodebuild -license accept

Error: Vulkan SDK not found

Download and install the Vulkan SDK from vulkan.lunarg.com, then set:

export VULKAN_SDK=/path/to/vulkan-sdk
export PATH=$VULKAN_SDK/bin:$PATH
export LD_LIBRARY_PATH=$VULKAN_SDK/lib:$LD_LIBRARY_PATH

Windows MSVC: Build fails with /bigobj

cmake -B build -DCMAKE_CXX_FLAGS="/bigobj"

macOS: ld: warning: ignoring file ... built for archive

Clean build directory and rebuild:

rm -rf build
cmake -B build
cmake --build build --config Release

OpenBLAS not detected

# Ubuntu
sudo apt install libopenblas-dev

cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS \
  -DBLAS_LIBRARIES=/usr/lib/x86_64-linux-gnu/libopenblas.so

SYCL: icpx not found

Source the oneAPI environment before configuring:

source /opt/intel/oneapi/setvars.sh

Build very slow (single-threaded)

Always pass -j$(nproc) to use all CPU cores:

cmake --build build --config Release -j$(nproc)

Tests fail to compile

Tests are disabled by default. If you enabled them and encounter failures:

cmake -B build -DLLAMA_BUILD_TESTS=OFF

3. Configuration

3.1 Model Files

llama.cpp uses the GGUF (GGML Universal Format) for all model files. GGUF is a self-describing binary format that bundles model weights, tokenizer vocabulary, metadata (architecture, hyperparameters, prompt templates), and quantization configuration in a single .gguf file.

Obtaining models:

• HuggingFace Hub — search for GGUF on huggingface.co. Popular uploaders include TheBloke, bartowski, and the model authors themselves.
• Convert yourself — use convert_hf_to_gguf.py on any HuggingFace-format checkpoint (see §4.4).
• Direct download via CLI — use --hf-repo and --hf-file flags (see Task 1).

3.2 Quantization Types

Quantization reduces model size and memory bandwidth by storing weights at lower precision. More aggressive quantization = faster inference and smaller files, but some quality loss. Use perplexity (ppl) as the quality metric — lower is better, and the delta from the baseline (F16) matters more than the absolute value.

Tip

Recommended starting point: Q4_K_M offers the best quality-per-gigabyte ratio for most use cases. If quality matters more, try Q5_K_M or Q6_K. If RAM is severely constrained, IQ4_XS or IQ3_S with an importance matrix will outperform legacy Q3/Q4 at the same size.

K-quants vs I-quants:

• K-quants (Q4_K_M, Q5_K_M, etc.) — block-wise quantization with separate scales; faster to decode, well-supported across all backends
• I-quants (IQ4_XS, IQ3_S, etc.) — importance-matrix-aware quantization; require calibration data but achieve better quality at extreme compression levels

3.3 Environment Variables

llama.cpp reads LLAMA_ARG_* environment variables that map to every CLI flag, enabling configuration without modifying command lines (useful for Docker/systemd deployments).

# Equivalent to: llama-server -m model.gguf --port 9090 -ngl 35
export LLAMA_ARG_MODEL=model.gguf
export LLAMA_ARG_PORT=9090
export LLAMA_ARG_N_GPU_LAYERS=35
llama-server

HuggingFace authentication:

export HF_TOKEN=hf_your_token_here
# Now --hf-repo can access private/gated models

Hexagon NPU (Snapdragon) environment variables:

3.4 Key CLI Parameters Overview

The tools share a common parameter set. This is a quick reference; see the Admin Guide for complete documentation of every parameter.

Model loading:

Generation:

Context and batching:

3.5 Choosing Context Size

# Minimum context (saves memory, limits conversation length)
llama-cli -m model.gguf -c 512

# Typical chat context
llama-cli -m model.gguf -c 4096

# Long document processing (if model supports it)
llama-cli -m model.gguf -c 32768

# Maximum context (check model's training context limit)
llama-cli -m model.gguf -c 131072

Warning

KV cache memory scales linearly with context size. A 7B model with 32K context requires roughly 4 GB for KV cache alone at F16. Use --cache-type-k q8_0 --cache-type-v q8_0 to halve this, or --cache-type-k q4_0 --cache-type-v q4_0 to quarter it, with modest quality impact.

4. Core Usage

4.1 Interactive Chat (llama-cli)

llama-cli is the primary interactive interface. It provides a colored REPL, session management commands, and supports both chat and completion modes.

Basic Interactive Session

# Start interactive chat (uses model's built-in chat template)
./build/bin/llama-cli -m model.gguf

# With specific context size and GPU offload
./build/bin/llama-cli -m model.gguf -c 8192 -ngl 35

# With a system prompt
./build/bin/llama-cli -m model.gguf \
  --system-prompt "You are a helpful coding assistant specializing in Python."

# With custom generation parameters
./build/bin/llama-cli -m model.gguf \
  --temp 0.7 --top-k 40 --top-p 0.9 --repeat-penalty 1.1

When you launch llama-cli without -p/--prompt, it enters interactive mode. The ASCII llama logo is displayed, then you type at the > prompt.

Session Commands

While in interactive mode, type these commands at the prompt:

Single-Turn Completion Mode

# Generate text and exit (non-interactive)
./build/bin/llama-cli -m model.gguf \
  -p "Write a haiku about programming." \
  -n 128

# No-display-prompt: output only the generated text
./build/bin/llama-cli -m model.gguf \
  -p "The capital of France is" \
  -n 32 \
  --no-display-prompt

Reasoning / Thinking Models

For models that support chain-of-thought reasoning (e.g., DeepSeek-R1):

# Enable reasoning display (shows <think>...</think> blocks)
./build/bin/llama-cli -m deepseek-r1-q4.gguf --thinking

# Set reasoning token budget
./build/bin/llama-cli -m deepseek-r1-q4.gguf --reasoning-budget 2048

Multiline Input

While in interactive mode, use \ at end of line to continue on the next line. Alternatively, set --multiline-input to switch the newline behavior.

Color Output

Color is enabled by default on terminals that support it. Disable with --no-color if you are piping output to a file.

4.2 Text Completion (llama-completion)

llama-completion is a non-interactive completion tool optimized for scripting and batch use.

# Basic completion
./build/bin/llama-completion -m model.gguf \
  -p "Once upon a time in a land far away" \
  -n 256

# Read prompt from file
./build/bin/llama-completion -m model.gguf \
  --file prompt.txt \
  -n 512

# Output only generated text (no prompt echo)
./build/bin/llama-completion -m model.gguf \
  -p "Translate to French: Hello world" \
  -n 64 \
  --no-display-prompt

4.3 Server Mode (llama-server)

llama-server provides an OpenAI-compatible HTTP API and a built-in web UI. It is the primary integration point for applications that want to use llama.cpp as a backend.

Starting the Server

# Basic start
./build/bin/llama-server -m model.gguf

# Custom port and host
./build/bin/llama-server -m model.gguf --host 0.0.0.0 --port 8080

# With GPU offload and larger context
./build/bin/llama-server -m model.gguf \
  -ngl 99 \
  -c 8192 \
  --port 8080

# Multiple concurrent users (parallel slots)
./build/bin/llama-server -m model.gguf \
  --parallel 4 \
  -c 16384 \
  --port 8080

# With API key authentication
./build/bin/llama-server -m model.gguf \
  --api-key "your-secret-key" \
  --port 8080

Web UI

Once the server is running, open your browser to:

http://localhost:8080

The web UI provides a full chat interface with model parameter controls, conversation history, and streaming output — no additional software needed.

OpenAI-Compatible API Endpoints

Example API Calls

Chat completion:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the speed of light?"}
    ],
    "temperature": 0.7,
    "max_tokens": 256,
    "stream": false
  }'

Streaming chat completion:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Tell me a story"}],
    "stream": true
  }'

Text embeddings:

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "The quick brown fox jumps over the lazy dog",
    "model": "local-model"
  }'

With API key:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{"model": "local-model", "messages": [{"role": "user", "content": "Hi"}]}'

Using with OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # or your --api-key value
)

response = client.chat.completions.create(
    model="local-model",
    messages=[
        {"role": "user", "content": "Explain recursion in one paragraph."}
    ],
    temperature=0.7,
    max_tokens=256
)
print(response.choices[0].message.content)

Note

The model field in API requests is accepted but ignored — the server always uses the model it was started with. Pass any string; local-model is a common convention.

4.4 Model Conversion

HuggingFace to GGUF

The primary conversion script handles most HuggingFace-format models (PyTorch .bin or SafeTensors .safetensors).

# Install Python dependencies first
pip install -r requirements.txt

# Convert to F16 (lossless, large file)
python convert_hf_to_gguf.py /path/to/hf-model --outtype f16

# Convert to BF16
python convert_hf_to_gguf.py /path/to/hf-model --outtype bf16

# Convert to Q8_0 directly (saves a quantize step)
python convert_hf_to_gguf.py /path/to/hf-model --outtype q8_0

# Specify output file
python convert_hf_to_gguf.py /path/to/hf-model \
  --outtype f16 \
  --outfile /path/to/output/model-f16.gguf

# Auto-detect best output type
python convert_hf_to_gguf.py /path/to/hf-model --outtype auto

# Use a HuggingFace token for gated models
HF_TOKEN=hf_your_token python convert_hf_to_gguf.py /path/to/hf-model

Output types: f32, f16, bf16, q8_0, auto

Tip

When in doubt, use --outtype f16 as the intermediate step, then run llama-quantize separately to get any target quantization. This preserves the highest quality starting point.

LoRA Adapters to GGUF

# Convert a LoRA adapter (requires the base model)
python convert_lora_to_gguf.py \
  /path/to/lora-adapter \
  --base /path/to/base-model \
  --outtype f16 \
  --outfile lora-adapter.gguf

Legacy GGML to GGUF

For older models in the original GGML binary format:

python convert_llama_ggml_to_gguf.py \
  --input model.bin \
  --output model.gguf

4.5 Quantization

The llama-quantize tool converts a high-precision GGUF (typically F16 or F32) to a quantized GGUF.

Basic Quantization

# Quantize to Q4_K_M (recommended balance)
./build/bin/llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

# Quantize to Q5_K_M (higher quality)
./build/bin/llama-quantize model-f16.gguf model-q5_k_m.gguf Q5_K_M

# List all available quantization types
./build/bin/llama-quantize --help

With Importance Matrix (Better Quality)

An importance matrix (imatrix) records which weights matter most during inference on calibration data. Using one with I-quants or when quantizing aggressively significantly improves output quality.

# Step 1: Generate importance matrix (uses calibration text)
./build/bin/llama-imatrix \
  -m model-f16.gguf \
  -f calibration-data.txt \
  -o model.imatrix \
  --chunks 128

# Step 2: Quantize using the importance matrix
./build/bin/llama-quantize \
  --imatrix model.imatrix \
  model-f16.gguf \
  model-iq4_xs.gguf \
  IQ4_XS

Advanced Quantization Options

# Keep the output tensor at higher precision (reduces quality loss on last layer)
./build/bin/llama-quantize \
  --leave-output-tensor \
  model-f16.gguf model-q4_k_m.gguf Q4_K_M

# Force all tensors to pure quantization (no mixed precision)
./build/bin/llama-quantize \
  --pure \
  model-f16.gguf model-q4_0.gguf Q4_0

# Override quantization type for specific tensor patterns
./build/bin/llama-quantize \
  --tensor-type "attn_q=q8_0" \
  --tensor-type "attn_k=q8_0" \
  model-f16.gguf model-mixed.gguf Q4_K_M

# Dry run — show what would be done without writing output
./build/bin/llama-quantize \
  --dry-run \
  model-f16.gguf model-q4_k_m.gguf Q4_K_M

4.6 Embeddings

llama-embedding generates vector embeddings from text, suitable for semantic search, RAG pipelines, and clustering.

# Generate embedding for a single text
./build/bin/llama-embedding \
  -m embedding-model.gguf \
  -p "The quick brown fox jumps over the lazy dog"

# Batch input from file (one text per line)
./build/bin/llama-embedding \
  -m embedding-model.gguf \
  --file texts.txt

# Specify pooling type
./build/bin/llama-embedding \
  -m embedding-model.gguf \
  -p "text" \
  --pooling mean

# Output normalized embeddings
./build/bin/llama-embedding \
  -m embedding-model.gguf \
  -p "text" \
  --normalize

Pooling types:

Note

Use embedding models (e.g., nomic-embed-text, mxbai-embed-large, bge-m3) rather than generative models for embedding tasks. Check the model card for recommended pooling type.

4.7 Benchmarking

llama-bench measures prompt processing (prefill) and token generation (decode) performance.

# Basic benchmark
./build/bin/llama-bench -m model.gguf

# Specific prompt and generation lengths
./build/bin/llama-bench -m model.gguf -p 128 -n 64

# Multiple configurations in one run
./build/bin/llama-bench -m model.gguf \
  -p "128,512,1024" \
  -n "32,64,128"

# With GPU layers
./build/bin/llama-bench -m model.gguf -ngl 99 -p 512 -n 128

# Output as CSV
./build/bin/llama-bench -m model.gguf -o csv

# Output as JSON
./build/bin/llama-bench -m model.gguf -o json

Output includes:

• pp — prompt processing speed (tokens/second)
• tg — token generation speed (tokens/second)

5. Common Tasks

Task 1: Download and Run a Model from HuggingFace

Goal: Get a model from HuggingFace and run it in one command.

# Download and run directly (no separate download step)
./build/bin/llama-cli \
  --hf-repo bartowski/Llama-3.2-3B-Instruct-GGUF \
  --hf-file Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -c 4096

# For gated models (e.g., Llama 3), set HF_TOKEN first
export HF_TOKEN=hf_your_token_here
./build/bin/llama-cli \
  --hf-repo meta-llama/Meta-Llama-3-8B-Instruct-GGUF \
  --hf-file Meta-Llama-3-8B-Instruct-Q4_K_M.gguf \
  -c 8192 -ngl 99

Verify: The model loads and you see the interactive prompt.

Task 2: Set Up a Local ChatGPT-Like Server

Goal: Run a local HTTP server compatible with OpenAI client libraries.

# Start the server
./build/bin/llama-server \
  -m model-q4_k_m.gguf \
  -c 8192 \
  -ngl 99 \
  --parallel 2 \
  --host 0.0.0.0 \
  --port 8080

# Test with curl
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Verify: curl http://localhost:8080/health returns {"status":"ok"}. Open http://localhost:8080 in browser to see the web UI.

Task 3: Quantize a Model for Your Hardware

Goal: Convert a F16 GGUF to a quantized version that fits your available RAM.

# Check how much RAM you have, then pick quantization:
# 4GB free  → Q2_K or IQ3_XS for 7B
# 6GB free  → Q4_K_M for 7B (recommended)
# 8GB free  → Q5_K_M or Q6_K for 7B
# 16GB free → Q8_0 or F16 for 7B

./build/bin/llama-quantize \
  model-f16.gguf \
  model-q4_k_m.gguf \
  Q4_K_M

# Verify the output file is valid
./build/bin/llama-cli -m model-q4_k_m.gguf -p "Test" -n 8

Verify: New GGUF file is present and runs successfully.

Task 4: Generate an Importance Matrix for Better Quantization

Goal: Create calibration data that improves quality for aggressive quantization.

# Generate importance matrix using a representative text corpus
./build/bin/llama-imatrix \
  -m model-f16.gguf \
  -f calibration.txt \
  -o model.imatrix \
  --chunks 256 \
  -ngl 99

# Now quantize with imatrix for significantly better quality at low bpw
./build/bin/llama-quantize \
  --imatrix model.imatrix \
  model-f16.gguf \
  model-iq4_xs.gguf \
  IQ4_XS

Verify: Perplexity (llama-perplexity) on the quantized model is closer to the F16 baseline than without imatrix.

Task 5: Use a LoRA Adapter

Goal: Apply a LoRA fine-tune on top of a base model at runtime.

# Method 1: Load LoRA at runtime (merged on-the-fly)
./build/bin/llama-cli \
  -m base-model.gguf \
  --lora lora-adapter.gguf \
  -c 4096

# Method 2: Merge LoRA permanently into the base model
./build/bin/llama-export-lora \
  -m base-model.gguf \
  --lora lora-adapter.gguf \
  -o merged-model.gguf

# Use the merged model
./build/bin/llama-cli -m merged-model.gguf -c 4096

Verify: Model responds with the fine-tuned persona/style.

Task 6: Run on GPU with Layer Offloading

Goal: Maximize GPU utilization to speed up inference.

# Offload all layers to GPU (fastest, requires enough VRAM)
./build/bin/llama-cli -m model.gguf -ngl 999

# List available GPU devices
./build/bin/llama-cli --list-devices

# Partial offload (when VRAM is insufficient for full offload)
./build/bin/llama-cli -m model.gguf -ngl 20

# Use a specific GPU (index 0, 1, 2...)
./build/bin/llama-cli -m model.gguf -ngl 999 --main-gpu 1

Tip

Start with -ngl 999 and watch VRAM usage. If you get out-of-memory errors, reduce -ngl until it fits. Even partial offload (e.g., 20–30 layers out of 32) provides significant speedup.

Verify: Run llama-bench before and after to compare token generation speed.

Task 7: Serve Multiple Parallel Users

Goal: Handle concurrent requests efficiently.

# 4 parallel slots, expanded context to accommodate all users
./build/bin/llama-server \
  -m model.gguf \
  --parallel 4 \
  -c 32768 \
  -ngl 99 \
  --port 8080

# Check slot status
curl http://localhost:8080/slots

Verify: Send multiple concurrent requests and observe them all being served. Each slot maintains its own KV cache.

Task 8: Use Grammar-Constrained Output

Goal: Force the model to output valid JSON, specific formats, or custom grammars.

# Force JSON output using JSON schema
./build/bin/llama-cli \
  -m model.gguf \
  -p "Extract the name and age from: John Smith is 42 years old." \
  --json-schema '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}}}'

# Use a GBNF grammar file
./build/bin/llama-cli \
  -m model.gguf \
  -p "Generate a list" \
  --grammar-file grammars/json.gbnf

# Inline grammar
./build/bin/llama-cli \
  -m model.gguf \
  -p "What is 2+2?" \
  --grammar 'root ::= [0-9]+'

Via the server API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "List 3 fruits as JSON array"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "schema": {
          "type": "array",
          "items": {"type": "string"}
        }
      }
    }
  }'

Task 9: Generate Embeddings for RAG

Goal: Embed documents and queries for retrieval-augmented generation.

# Use a dedicated embedding model
./build/bin/llama-embedding \
  -m nomic-embed-text-v1.5-Q4_K_M.gguf \
  -p "llama.cpp is a high-performance LLM inference library" \
  --pooling mean \
  --normalize

# Via server API (start server with embedding model)
./build/bin/llama-server \
  -m nomic-embed-text-v1.5-Q4_K_M.gguf \
  --embedding \
  --port 8080

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": ["document one", "document two", "query text"],
    "model": "local"
  }'

Verify: Response includes a data array with embedding vectors of consistent dimensionality.

Task 10: Run Multimodal Inference (Image + Text)

Goal: Describe an image or answer questions about it.

# Interactive mode with vision model (LLaVA, MiniCPM-V, etc.)
./build/bin/llama-cli \
  -m llava-v1.6-mistral-7b-q4_k_m.gguf \
  --mmproj llava-v1.6-mistral-7b-mmproj-f16.gguf \
  -c 4096

# Then at the prompt type: /image /path/to/photo.jpg
# Followed by your question about the image.

# Or pass image directly as CLI argument
./build/bin/llama-cli \
  -m llava-v1.6-mistral-7b-q4_k_m.gguf \
  --mmproj llava-v1.6-mistral-7b-mmproj-f16.gguf \
  --image /path/to/photo.jpg \
  -p "What objects are in this image?" \
  -n 256

Important

Multimodal models require two GGUF files: the language model (-m) and the vision projector (--mmproj). Both must match — download them together from the same HuggingFace repository.

Task 11: Use Speculative Decoding for Faster Generation

Goal: Speed up generation using a small draft model to propose tokens verified by the large model.

# Speculative decoding: large model verifies draft model proposals
./build/bin/llama-cli \
  -m large-model-q4_k_m.gguf \
  --draft-model small-draft-model-q4_k_m.gguf \
  -c 4096 \
  -ngl 99 \
  --draft-max 8 \
  -p "Write a detailed explanation of"

Tip

Best speedup comes when the draft model is 5–10x smaller than the main model and trained on similar data (ideally a distilled version of the same model family). Expect 2–3x generation speedup on average.

Task 12: Text-to-Speech with llama-tts

Goal: Generate spoken audio from text using a TTS-capable model.

# Generate audio from text
./build/bin/llama-tts \
  -m tts-model.gguf \
  --vocoder vocoder-model.gguf \
  -p "Hello, this is a test of the text to speech system." \
  -o output.wav

# Use the server's TTS endpoint
./build/bin/llama-server \
  -m tts-model.gguf \
  --port 8080

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello world", "voice": "default"}' \
  --output speech.wav

Task 13: Split and Merge Large Models (llama-gguf-split)

Goal: Split a large GGUF across multiple files for storage or transfer, or merge shards back.

# Split a model into 4 GB shards
./build/bin/llama-gguf-split \
  --split-max-size 4G \
  large-model.gguf \
  large-model-split

# Creates: large-model-split-00001-of-00004.gguf, etc.

# Merge shards back into one file
./build/bin/llama-gguf-split \
  --merge \
  large-model-split-00001-of-00004.gguf \
  large-model-merged.gguf

Task 14: Run on Android Device

Goal: Run a model on an Android phone or tablet.

1. Open examples/llama.android/ in Android Studio.
2. Build the project (it compiles llama.cpp via CMake + NDK).
3. Deploy to device and launch the app.
4. Use the in-app file picker to select a GGUF from device storage.
5. Alternatively, push a model to the device:

adb push model-q4_k_m.gguf /sdcard/Download/model.gguf

Note

Android devices use ARM64 CPUs with NEON SIMD, which llama.cpp supports natively. Expect 3–8 tokens/second for a 1–3B parameter model on a modern phone.

Task 15: Run on Snapdragon / Hexagon NPU

Goal: Use the Hexagon NPU on Qualcomm Snapdragon for accelerated inference.

# Build with Hexagon backend
cmake -B build -DGGML_HEXAGON=ON
cmake --build build --config Release

# Configure via environment variables
export GGML_HEXAGON_NDEV=1
export GGML_HEXAGON_NHVX=4
export GGML_HEXAGON_VERBOSE=1

./build/bin/llama-cli -m model.gguf -c 2048

Task 16: Use the Web UI for Chat

Goal: Use the browser-based chat interface without any additional software.

# Start server
./build/bin/llama-server -m model.gguf --port 8080

# Open browser at:
# http://localhost:8080

The web UI supports:

• Streaming chat with markdown rendering
• System prompt configuration
• Sampling parameter sliders (temperature, top-p, top-k)
• Conversation export
• Model info panel

Task 17: Compute Model Perplexity

Goal: Measure how well a model predicts text — compare quantization quality objectively.

# Compute perplexity on a test corpus
./build/bin/llama-perplexity \
  -m model.gguf \
  -f wikitext-2-raw/wiki.test.raw \
  -c 512

# Compare two quantizations
./build/bin/llama-perplexity -m model-f16.gguf -f wiki.test.raw -c 512
./build/bin/llama-perplexity -m model-q4_k_m.gguf -f wiki.test.raw -c 512

Lower perplexity = better quality. The delta between F16 and a quantized version tells you the quality cost of that quantization.

Task 18: Use Docker Image

Goal: Run llama.cpp without building from source.

# CPU image
docker run --rm -it \
  -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:full \
  llama-cli -m /models/model.gguf -c 4096

# CUDA image (NVIDIA)
docker run --rm -it --gpus all \
  -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  llama-cli -m /models/model.gguf -ngl 99 -c 4096

# Server mode
docker run --rm -it \
  -p 8080:8080 \
  -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  llama-server -m /models/model.gguf --host 0.0.0.0 --port 8080

Task 19: Use RPC for Distributed Inference

Goal: Offload model layers to remote machines via the RPC backend.

# On the remote machine (the "server"):
./build/bin/llama-rpc-server --host 0.0.0.0 --port 50052

# On the local machine (the "client"), specify RPC endpoints:
./build/bin/llama-cli \
  -m model.gguf \
  --rpc 192.168.1.100:50052 \
  -ngl 99 \
  -c 4096

Multiple RPC servers can be specified comma-separated: --rpc host1:50052,host2:50052. This distributes layer offloading across multiple machines.

6. Troubleshooting

Error: CUDA out of memory

Cause: Model layers exceed available GPU VRAM.

Solution: Reduce the number of layers offloaded to GPU:

# Reduce -ngl until the error goes away
./build/bin/llama-cli -m model.gguf -ngl 20  # instead of -ngl 99

Or use a more aggressively quantized model (Q4_K_M instead of Q8_0), or reduce context size (-c 2048 instead of -c 8192). KV cache also occupies VRAM, so reducing -c frees significant VRAM.

Prevention: Use llama-bench to find the optimal -ngl for your VRAM. Rule of thumb: each layer of a 7B model uses roughly 100 MB VRAM.

Error: failed to load model from file

Cause: The file does not exist, is corrupted, or is not a valid GGUF file.

Solution:

# Check the file exists
ls -la model.gguf

# Verify it is GGUF format (should start with "GGUF" magic bytes)
xxd model.gguf | head -1

# Re-download if corrupted (resume partial download)
wget -c https://huggingface.co/.../model.gguf

Error: unknown model architecture

Cause: The GGUF was built for a newer version of llama.cpp than you have, or uses an architecture not yet supported.

Solution: Update llama.cpp to the latest version:

git pull
cmake -B build && cmake --build build --config Release -j$(nproc)

Error: Slow performance / low tokens per second

Cause (CPU): Not using SIMD optimizations, wrong thread count, or model too large for CPU cache.

Solution:

# Ensure GGML_NATIVE is ON (default) for AVX2/AVX512
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

# Set thread count to physical cores (not hyperthreads)
./build/bin/llama-cli -m model.gguf -t 8  # for an 8-core CPU

# On NUMA systems, pin to one node
numactl --cpunodebind=0 --membind=0 ./build/bin/llama-cli -m model.gguf

Cause (GPU): Layers not offloaded. Add -ngl 99:

./build/bin/llama-cli -m model.gguf -ngl 99

Error: Build failed — missing <cuda_runtime.h>

Cause: CUDA Toolkit not properly installed or not on PATH.

Solution:

# Ubuntu
sudo apt install cuda-toolkit-12-0
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH

# Verify
nvcc --version

Error: Server connection refused when connecting to llama-server

Cause: Server not started, wrong port, or binding to loopback only.

Solution:

# Check server is running
ps aux | grep llama-server

# Start with explicit port
./build/bin/llama-server -m model.gguf --port 8080

# Bind to all interfaces for remote access
./build/bin/llama-server -m model.gguf --host 0.0.0.0 --port 8080

# Check firewall (Linux)
sudo ufw allow 8080

Error: Context size too small / conversation gets cut off

Cause: The conversation history exceeds the configured context window.

Solution:

# Increase context size
./build/bin/llama-cli -m model.gguf -c 32768

# Use context extension if model supports RoPE scaling
./build/bin/llama-cli -m model.gguf -c 32768 --rope-scale 2.0

Error: GGUF version mismatch

Cause: The GGUF file was created with a different GGUF spec version than your build supports.

Solution: Update llama.cpp. If the model was created with a very old tool, re-convert from HuggingFace format:

python convert_hf_to_gguf.py /path/to/hf-model --outtype f16

Error: Metal shader compilation failure on macOS

Cause: Xcode Command Line Tools not installed, or Metal SDK version mismatch.

Solution:

# Install / update Xcode CLT
xcode-select --install
sudo xcodebuild -runFirstLaunch
sudo xcodebuild -license accept

# Fallback: disable Metal (CPU only)
./build/bin/llama-cli -m model.gguf -ngl 0

Error: Vulkan: no suitable device found

Cause: Vulkan driver not installed, or no Vulkan-capable GPU detected.

Solution:

# Linux — install Vulkan ICD for your GPU:

# NVIDIA
sudo apt install nvidia-vulkan-icd

# AMD (Mesa)
sudo apt install mesa-vulkan-drivers

# Intel
sudo apt install intel-media-va-driver

# Verify Vulkan works
vulkaninfo | grep deviceName

Error: OpenCL initialization failure

Cause: OpenCL runtime not installed for your GPU.

Solution:

# Install OpenCL for AMD on Ubuntu
sudo apt install clinfo ocl-icd-opencl-dev

# Intel OpenCL
sudo apt install intel-opencl-icd

# Verify
clinfo | grep "Platform Name"

Error: Python conversion script fails with ImportError or ModuleNotFoundError

Cause: Python dependencies not installed.

Solution:

# Install all requirements
pip install -r requirements.txt

# Or install specific packages
pip install torch transformers sentencepiece gguf protobuf

Error: tokenizer mismatch or garbled / repetitive output

Cause: Prompt template does not match the model, or wrong chat template applied.

Solution:

# Use the model's built-in template (if GGUF contains it — default behavior)
./build/bin/llama-cli -m model.gguf

# Override with a specific template name
./build/bin/llama-cli -m model.gguf --chat-template chatml

# List available templates
./build/bin/llama-cli --list-chat-templates

Error: NaN or Inf values in output / model produces garbage text

Cause: Numerical instability from very high temperature, misconfigured rope settings, or corrupted model weights.

Solution:

# Reset to safe sampling parameters
./build/bin/llama-cli -m model.gguf --temp 0.7 --top-k 40 --top-p 0.9

# Disable flash attention if enabled
./build/bin/llama-cli -m model.gguf --no-flash-attn

# Verify model tensor integrity
./build/bin/llama-cli -m model.gguf --check-tensors

Error: Multimodal model not loading images

Cause: Missing --mmproj argument, or vision projector file missing / mismatched.

Solution:

# Both files are required
./build/bin/llama-cli \
  -m llava-model.gguf \
  --mmproj llava-mmproj.gguf \
  --image photo.jpg \
  -p "Describe this image."

Ensure both GGUF files are from the same model version. Download them together from the same HuggingFace release.

Error: High memory usage / system swap thrashing

Cause: Model too large for available RAM, or KV cache is oversized.

Solution:

# Use a smaller quantization (switch from Q8_0 to Q4_K_M — saves ~50% RAM)

# Reduce context size (KV cache scales linearly with context)
./build/bin/llama-cli -m model.gguf -c 2048

# Use quantized KV cache (halves KV cache memory)
./build/bin/llama-cli -m model.gguf \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

# Lock model in RAM to prevent swap (requires enough RAM)
./build/bin/llama-cli -m model.gguf --mlock

Error: Chat template application failures / Jinja2 errors

Cause: Model's built-in Jinja2 template is incompatible with llama.cpp's template engine.

Solution:

# Override with a compatible template
./build/bin/llama-cli -m model.gguf --chat-template llama3

# Available templates include: llama2, llama3, chatml, gemma, phi3, mistral, etc.
./build/bin/llama-cli --list-chat-templates

Error: llama-quantize produces unexpectedly large output file

Cause: Some tensors are kept at higher precision due to --leave-output-tensor or per-tensor minimum size thresholds.

Solution: Use --dry-run to inspect per-tensor decisions before writing:

./build/bin/llama-quantize --dry-run model-f16.gguf model-q4.gguf Q4_K_M 2>&1 | grep "tensor"

Error: no space left on device during model download

Cause: Insufficient disk space for the GGUF file.

Solution:

df -h .
# Free space or specify a different path with more space
./build/bin/llama-cli --hf-repo owner/repo --hf-file model.gguf \
  --model /path/with/space/model.gguf

Error: permission denied on model file

Cause: File ownership or permissions prevent reading.

Solution:

chmod 644 model.gguf
# If in a Docker volume:
chown $(id -u):$(id -g) model.gguf

Error: llama-server returns HTTP 400 on /v1/chat/completions

Cause: Malformed JSON, missing required fields, or unsupported parameters.

Solution: Ensure the request body contains a messages array with at least one object with role and content:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"x","messages":[{"role":"user","content":"hi"}]}'

Error: Slow cold start / long model loading time

Cause: Large model loading from spinning disk, or memory mapping disabled.

Solution:

# Ensure mmap is enabled (default — do NOT pass --no-mmap unless needed)
# Store model files on SSD

# Pre-warm the file cache (Linux):
vmtouch -t model.gguf
# Or: cat model.gguf > /dev/null

Error: llama-imatrix runs out of memory

Cause: Context size too large during importance matrix generation.

Solution:

# Reduce context size and chunk count
./build/bin/llama-imatrix \
  -m model-f16.gguf \
  -f calibration.txt \
  -o model.imatrix \
  -c 512 \
  --chunks 64

Error: Flash attention errors or degraded output quality with --flash-attn

Cause: Flash attention requires specific GPU support and may not be available for all model architectures.

Solution:

# Let llama.cpp auto-detect (default AUTO mode)
./build/bin/llama-cli -m model.gguf

# Explicitly disable if causing issues
./build/bin/llama-cli -m model.gguf --no-flash-attn

Error: RoPE scaling not working / context extension degraded quality

Cause: RoPE scaling parameters not matched to model requirements, or model does not support dynamic NTK scaling.

Solution: Check the model card for recommended RoPE scaling parameters. Common settings for 2x context extension:

./build/bin/llama-cli -m model.gguf \
  -c 8192 \
  --rope-scale 2.0 \
  --rope-freq-base 1000000

7. FAQ

Models

Q: What models does llama.cpp support?

llama.cpp supports a very wide range of transformer-based LLM architectures via GGUF format. This includes Llama 1/2/3/3.1/3.2/3.3, Mistral, Mixtral MoE, Gemma 1/2/3, Phi-2/3/4, Qwen 1/1.5/2/2.5/3, DeepSeek 1/2/3/R1, Command-R, Falcon, MPT, StarCoder, CodeLlama, WizardLM, and many more. Architecture support is regularly added; check the GitHub repository for the current list.

Q: Where do I get GGUF models?

The primary source is HuggingFace Hub. Search for "GGUF" plus the model name. Popular community quantizers include bartowski, TheBloke, unsloth, and official model authors who publish their own GGUF releases. You can also convert any HuggingFace-format checkpoint yourself with convert_hf_to_gguf.py.

Q: How do I choose the right quantization level?

Start with Q4_K_M — it balances quality and size well for most use cases. If you have more VRAM/RAM and want higher quality, try Q5_K_M or Q6_K. If you are RAM-constrained, IQ4_XS (with imatrix) rivals Q4_K_M quality at slightly smaller size. Avoid Q2/Q3 for long-form coherent output unless memory is severely limited.

Q: Can I use models from other formats (GGML, PyTorch, SafeTensors)?

Yes. Old GGML binary format: convert_llama_ggml_to_gguf.py. HuggingFace PyTorch/SafeTensors: convert_hf_to_gguf.py. Native GGUF: use directly. ONNX format is not natively supported — convert to HuggingFace format first.

Q: What is the largest model I can run?

Limited by your total RAM (for CPU inference) or VRAM (for full GPU offload). With 64 GB RAM, you can run a 70B model at Q4_K_M (~38 GB). With 128 GB RAM, a 70B at Q8_0 (~70 GB) or a 405B model at Q2_K (~100 GB). Use partial GPU offload to mix VRAM + RAM for large models that exceed VRAM.

Q: Are vision and multimodal models supported?

Yes. LLaVA, MiniCPM-V, Qwen-VL, and similar architectures are supported. These require a separate vision projector GGUF (--mmproj). Audio input is also supported for compatible models via the /audio command in the CLI.

Q: What model families work best for coding tasks?

DeepSeek-Coder, CodeLlama, StarCoder2, Qwen2.5-Coder, and Phi-4 are among the strongest coding models available in GGUF. For fill-in-the-middle (infill) code completion, use the /infill server endpoint.

Performance

Q: How much RAM do I need?

Roughly the GGUF file size plus 10–20% overhead. A Q4_K_M 7B model is ~4.5 GB, so 6 GB total RAM works. Add KV cache memory: at 4096 context, a 7B model needs ~512 MB additional; at 32K context, ~4 GB additional. Use --cache-type-k q8_0 --cache-type-v q8_0 to halve KV cache memory.

Q: How much faster is GPU vs CPU?

Typically 5–20x faster for token generation, depending on the GPU and model size. A modern NVIDIA RTX 4090 can generate 80–150 tokens/second on a 7B Q4 model. A 16-core CPU typically manages 8–20 tokens/second. Prompt processing (prefill) shows even larger GPU speedups.

Q: How do I maximize throughput for a production server?

• Offload all layers to GPU (-ngl 999)
• Enable Flash Attention (--flash-attn)
• Use quantized KV cache (--cache-type-k q8_0 --cache-type-v q8_0)
• Set --parallel to match expected concurrent users
• Use Q4_K_M for speed vs a larger model
• Increase batch size (--batch-size 2048)

Q: How many CPU threads should I use?

Set --threads to your number of physical CPU cores (not logical/hyperthreaded). Check with lscpu | grep "Core(s) per socket". Hyperthreading generally does not help and can slow inference due to cache contention.

Q: Does llama.cpp use all my GPU memory?

No — it only uses what is needed for the model layers you offload (-ngl) and the KV cache. You can control VRAM usage precisely by adjusting -ngl, -c, and --cache-type-k/--cache-type-v.

Q: Is Apple Silicon (M1/M2/M3/M4) well supported?

Yes, and it is one of the primary development targets. Metal GPU acceleration is automatic on macOS, Accelerate BLAS is linked automatically, and Apple Silicon's unified memory means there is no CPU/GPU transfer overhead. Apple Silicon Macs routinely achieve 20–60 tokens/second on 7B models.

Usage

Q: Can I use llama-server as a drop-in replacement for the OpenAI API?

For most use cases, yes. The /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models endpoints follow the OpenAI API specification. Point your client's base_url at http://localhost:8080/v1. Tool calling, streaming, JSON mode, and grammar constraints are supported. The Assistants API, Fine-tuning API, and Batch API are not implemented.

Q: Can I fine-tune models with llama.cpp?

Not directly — llama.cpp is an inference library, not a training framework. For fine-tuning, use tools like unsloth, axolotl, or llama-factory. You can then export LoRA adapters and use them with llama.cpp at inference time via --lora or by merging with llama-export-lora.

Q: Can I run multiple models simultaneously?

Run multiple separate llama-server instances on different ports. There is no built-in model multiplexer, but you can put a reverse proxy (nginx, Caddy, Traefik) in front to route requests to the correct instance based on model name or URL path.

Q: How do I set a persistent system prompt?

# Via CLI flag
llama-cli -m model.gguf --system-prompt "You are a helpful assistant."

# Via API messages array
# Include {"role": "system", "content": "..."} as the first message

Q: Does llama.cpp support function calling / tool use?

Yes, for models trained with tool-use support (Llama 3.1+, Qwen 2.5, Mistral, Command-R, etc.). Pass tools in the API request body. The server handles tool call formatting and parsing according to the model's chat template.

Q: Can I stream output?

Yes. Both the CLI (default behavior) and server support streaming. In the API, set "stream": true in the request body; the server responds with server-sent events (SSE) in the OpenAI data: format.

Technical

Q: What is GGUF?

GGUF (GGML Universal Format) is the binary model format used by llama.cpp. It is self-describing: a single .gguf file contains model architecture metadata, hyperparameters, tokenizer vocabulary (with special tokens, chat template, BOS/EOS), and quantized weight tensors. GGUF superseded the older GGML format in mid-2023 and is the standard for community model distribution.

Q: What is the KV cache and why does it matter?

During inference, the model computes key and value tensors for every token in the context. These are cached (the KV cache) so they do not need to be recomputed on every generation step. The KV cache size equals: context_length × n_layers × n_heads × head_dim × 2 × dtype_bytes. For a 7B model at 4096 context with F16 dtype, this is roughly 512 MB. Quantized KV cache (--cache-type-k q8_0 --cache-type-v q8_0) reduces this by ~50%.

Q: What is the difference between K-quants and I-quants?

K-quants (e.g., Q4_K_M, Q5_K_M) use block-wise quantization with per-block scales. The "K" refers to the k-means clustering approach used. They are fast to decode and optimized across all backends.

I-quants (e.g., IQ4_XS, IQ3_S) use importance-matrix-aware quantization. They require calibration data (imatrix) to determine which weights to quantize more carefully. At the same bits-per-weight, I-quants typically achieve better quality than K-quants — the imatrix step is worth the extra effort for aggressive quantization.

Q: What vocabulary types are supported?

llama.cpp supports: SPM (SentencePiece Model), BPE (Byte-Pair Encoding), WPM (WordPiece), UGM (Unigram), RWKV (RWKV world tokenizer), and PLaMo2. These are embedded in the GGUF file — no separate tokenizer files are needed.

Q: What is speculative decoding?

Speculative decoding uses a small "draft" model to propose multiple tokens at once, then verifies them with the large "target" model in a single forward pass. When the draft model is correct (frequently), multiple tokens are accepted simultaneously, effectively increasing throughput without quality loss. Works best with a draft model distilled from the target model.

Q: What is FlashAttention and should I enable it?

FlashAttention is a memory-efficient attention computation algorithm that reduces VRAM usage and increases throughput by computing attention in tiles without materializing the full attention matrix. Enable with --flash-attn. The flash attention mode can be AUTO (detect and use when safe — the default), DISABLED, or ENABLED.

Q: What are the multi-GPU split modes?

layer split is recommended for most multi-GPU setups. Combine with --main-gpu N to set which GPU handles non-offloaded work.

Q: What KV cache data types are available?

F32, F16, BF16, Q8_0, Q4_0, Q4_1, IQ4_NL, Q5_0, Q5_1. Quantized KV cache (Q8_0 or lower) reduces VRAM/RAM with minimal quality impact on most tasks. F16 is the default.

Q: What pooling types does the embeddings endpoint support?

none (all token embeddings), mean (average pool), cls (first token), last (last token), rank (reranking score). The correct type depends on the embedding model — check the model card.

Platform

Q: Does llama.cpp work on ARM processors?

Yes. llama.cpp has excellent ARM64 support with NEON SIMD optimizations. This covers Apple Silicon (M1/M2/M3/M4 with Metal GPU), Raspberry Pi, Android phones, AWS Graviton, and Ampere Altra servers. ARM32 is also supported but slower.

Q: Does it work on Windows?

Yes. Build with MSVC 2019+ or MinGW-w64. CUDA is supported on Windows. Vulkan works. Pre-built Windows binaries are available on the GitHub Releases page.

Q: Can I run llama.cpp on a Raspberry Pi?

Yes, with limitations. The Raspberry Pi 4/5 can run small models (1B–3B at Q4) at a few tokens per second. Use Q4_K_M or smaller quantization. The Pi 5 achieves roughly 3–5 tok/s for 1B models. 8 GB RAM variants are strongly recommended.

Q: Does it work in WSL (Windows Subsystem for Linux)?

Yes. WSL2 with GPU support (CUDA via WDDM passthrough) works with the CUDA backend. Metal is not available in WSL. Performance may be slightly lower than native Windows.

Q: Is there a Python binding?

The official community Python binding is llama-cpp-python (third-party, widely used). The gguf Python package (pip install gguf) handles GGUF file I/O (reading and writing) but is not an inference binding.

Q: Does llama.cpp support ONNX or other model formats?

No. llama.cpp is GGUF-native. Use the conversion scripts to convert from HuggingFace/PyTorch format to GGUF first.

Q: Can I use llama.cpp with LangChain or LlamaIndex?

Yes. Both frameworks support llama-cpp-python as a local model backend, and llama-server's OpenAI-compatible API works with any LangChain or LlamaIndex component that accepts an OpenAI-compatible endpoint (just set base_url).

Q: Is there support for continuous batching?

Yes. llama-server uses continuous batching when --parallel > 1, allowing new requests to start processing in available slots without waiting for all current requests to complete. This maximizes GPU utilization under concurrent load.

Q: What is the maximum context length supported?

Physically limited by available RAM/VRAM for the KV cache. Modern models support 4K–128K+ context natively in their GGUF. With RoPE scaling (--rope-scale, --rope-freq-base), context can be extended beyond the training length with some quality degradation. Pass -c N to set the context window up to the model's support limit.

Q: Does llama.cpp support mixture-of-experts (MoE) models?

Yes. Mixtral (8x7B, 8x22B), DeepSeek MoE, Qwen MoE, and other MoE architectures are supported. MoE models have a smaller active parameter count per forward pass, enabling large models to run with less memory than their total parameter count would suggest.

Q: Can I run llama.cpp without a GPU?

Yes. CPU-only builds work well and are the default. CPU inference is slower but fully functional. For small models (1B–7B) on modern CPUs (12+ cores, AVX2 or better), 5–30 tokens/second is achievable, which is comfortable for interactive use.

Q: Is there support for structured output beyond JSON?

Yes. llama.cpp supports GBNF (GGML BNF) grammars, which can enforce any context-free grammar. JSON schema is automatically converted to GBNF internally. You can write custom GBNF grammars for any structured format (CSV, code, domain-specific DSLs) and pass them via --grammar or --grammar-file. The server's response_format field also accepts JSON schema directly.

Q: Does llama.cpp support reranking models?

Yes. Reranking models (cross-encoders) are supported via the rank pooling type. Use llama-embedding with --pooling rank and the server's /v1/rerank endpoint (on supported builds) for passage reranking in RAG pipelines.