# llama.cpp — Administrator & Developer Guide > [!NOTE] > This guide targets administrators deploying llama.cpp in production environments, developers building on the library, and contributors working on the codebase. For end-user installation and usage, see the companion User Guide. --- ## Table of Contents 1. [Architecture Overview](#1-architecture-overview) 2. [System Requirements](#2-system-requirements) 3. [Configuration Reference](#3-configuration-reference) 4. [Deployment & Scaling](#4-deployment--scaling) 5. [Operations](#5-operations) 6. [Performance](#6-performance) 7. [Security](#7-security) 8. [Developer Internals](#8-developer-internals) 9. [Integration Points](#9-integration-points) --- ## 1. Architecture Overview ### What llama.cpp Is llama.cpp is a high-performance inference engine for large language models, written in C/C++ with minimal dependencies. Its primary goal is to run modern LLM architectures efficiently across the widest possible range of hardware — from laptop CPUs to multi-GPU server clusters to mobile devices — without requiring cloud infrastructure. The project is the canonical runtime for the GGUF model format and is the upstream source for many derivative projects (llama-python, ollama, LM Studio, Jan, and dozens more). The library is structured as a clean C API with C++ internals. Applications link against `libllama` (or consume it via the OpenAI-compatible server), while the underlying compute is handled by GGML — a separate tensor-operation library embedded in the repository at `ggml/`. The separation is intentional: GGML is usable as a standalone library for other inference engines (Whisper.cpp, Stable Diffusion, etc.) and has its own backend abstraction that allows new hardware targets to be added without touching llama-level code. ### What llama.cpp Does At runtime, llama.cpp loads a GGUF model file into memory (optionally memory-mapping it for zero-copy access), constructs an inference context that owns the KV cache and execution graph, and runs the transformer forward pass for each token generation step. The core decode path (`llama_decode()`) tokenizes prompts, schedules batch computation across available backends, performs attention with optional flash attention acceleration, applies the sampler chain to select the next token, and returns logits or sampled tokens to the caller. The server wraps this in an HTTP interface; the CLI wraps it in an interactive terminal session. Model support covers 100+ architectures (LLaMA, Mistral, Gemma, Phi, Qwen, DeepSeek, GPT-NeoX, Falcon, StarCoder, Command-R, BLOOM, and many more), implemented as separate files in `src/models/`. Quantization support spans 40+ types from full F32 down to sub-1-bit representations, all defined in the GGML quantization type registry. ### Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────────┐ │ Applications │ │ llama-cli llama-server llama-quantize llama-bench │ │ llama-perplexity llama-imatrix llama-tts llama-mtmd │ ├─────────────────────────────────────────────────────────────────┤ │ Common Layer │ │ arg.cpp chat.cpp sampling.cpp download.cpp │ │ log.cpp ngram-cache.cpp jinja/ (template engine) │ ├─────────────────────────────────────────────────────────────────┤ │ llama Library │ │ llama.h (public C API) llama-cpp.h (C++ RAII wrappers) │ │ llama-model llama-context llama-sampling llama-memory │ │ llama-kv-cache llama-graph llama-adapter llama-mmap │ │ src/models/ (114 architecture implementations) │ ├─────────────────────────────────────────────────────────────────┤ │ GGML Engine │ │ ggml.h ggml-backend (interface + registry) ggml-alloc │ │ gguf (reader/writer) ggml-threading ggml-quants │ ├─────────────────────────────────────────────────────────────────┤ │ Hardware Backends │ │ CPU │ CUDA │ Metal │ Vulkan │ OpenCL │ SYCL │ CANN │ Hexagon │ │ RPC │ HIP │ MUSA │ ZenDNN │ zDNN │ VirtGPU │ BLAS │ OpenVINO│ │ WebGPU │ KleidiAI (ARM) │ llamafile SGEMM │ └─────────────────────────────────────────────────────────────────┘ ``` ### Component Inventory | Component | Path | Role | |-----------|------|------| | llama library | `src/`, `include/llama.h` | Core inference engine (model loading, context, decoding) | | GGML | `ggml/` | Low-level tensor operations, backend abstraction | | Common | `common/` | Shared utilities (arg parsing, sampling, chat templates, download) | | CLI | `tools/cli/` | Interactive chat interface | | Server | `tools/server/` | HTTP server with OpenAI-compatible API + web UI | | Quantize | `tools/quantize/` | Model quantization tool | | Bench | `tools/llama-bench/` | Performance benchmarking | | iMatrix | `tools/imatrix/` | Importance matrix computation | | Perplexity | `tools/perplexity/` | Model quality measurement | | TTS | `tools/tts/` | Text-to-speech | | Multimodal | `tools/mtmd/` | Vision and audio model support | | Tokenizer | `tools/tokenize/` | Tokenization utility | | GGUF Split | `tools/gguf-split/` | Split/merge large GGUF files | | Export LoRA | `tools/export-lora/` | LoRA adapter export | | CVector | `tools/cvector-generator/` | Control vector generation | | Parser | `tools/parser/` | Grammar/constrained output parsing (PEG-based) | | Results | `tools/results/` | Benchmark result processing | | RPC | `tools/rpc/` | Remote procedure call backend for distributed inference | | Batched Bench | `tools/batched-bench/` | Batched inference benchmarking | | Fit Params | `tools/fit-params/` | Parameter fitting utility | | Completion | `tools/completion/` | Non-interactive text completion | | GGUF Python | `gguf-py/` | Python library for GGUF format read/write | | Conversion | `convert_hf_to_gguf.py` | HuggingFace → GGUF conversion | | Web UI | `tools/server/webui/` | SvelteKit web interface for the server | | Examples | `examples/` | Sample applications and integrations | | Mobile Android | `examples/llama.android/` | Android native app (JNI/NDK) | | Mobile iOS | `examples/llama.swiftui/` | iOS/macOS SwiftUI app (XCFramework) | | Jinja Engine | `common/jinja/` | Chat template rendering engine (Jinja2-compatible) | | Grammar files | `grammars/` | GBNF grammar definitions for constrained output | | Docs | `docs/` | Build guides, backend docs, ops CSV files | | CI | `ci/` | CI runner script for multi-backend validation | | CMake modules | `cmake/` | Toolchain files, find modules, utilities | ### Technology Stack | Layer | Technology | |-------|------------| | Core library | C17 / C++17 | | Build system | CMake 3.14+ | | GPU backends | CUDA (NVIDIA), Metal (Apple), Vulkan (cross-vendor), OpenCL, SYCL (Intel), HIP (AMD), MUSA (Moore Threads), CANN (Huawei Ascend) | | Accelerators | Hexagon NPU (Qualcomm), KleidiAI (ARM), llamafile SGEMM | | Python tooling | Python 3.8+ (conversion scripts, gguf-py library) | | Web UI | SvelteKit, TypeScript, Tailwind CSS | | Mobile — Apple | Swift, SwiftUI, XCFramework | | Mobile — Android | Kotlin, Java, JNI, NDK | | Testing | C++ (tests/), Python aiohttp (server tests), Vitest + Playwright + Storybook (web UI) | | Code quality | clang-format, clang-tidy, flake8, pre-commit | ### Data Flow ``` Model File (GGUF on disk) │ ▼ llama_model_load_from_file() ├── Parse GGUF header & metadata ├── Build tensor map ├── mmap or read tensors into memory └── Assign tensors to backend buffers (GPU/CPU) │ ▼ llama_init_from_model() ├── Allocate KV cache (type_k / type_v, quantized optional) ├── Build computation graph skeleton └── Allocate scratch buffers per backend │ ▼ Prompt Processing ├── llama_tokenize() → token IDs ├── llama_batch_init() → populate batch └── llama_decode() → fill KV cache │ ▼ Token Generation Loop ├── llama_decode() → compute logits ├── llama_sampler_chain_sample() → next token ├── llama_token_to_piece() → detokenize └── Stream to caller / HTTP client ``` --- ## 2. System Requirements ### Hardware Requirements by Model Size | Model Size | Parameters | Q4_K_M VRAM | Min RAM (CPU) | Recommended RAM | Notes | |------------|-----------|-------------|---------------|-----------------|-------| | Tiny | 1–3B | ~1–2 GB | 4 GB | 8 GB | Runs on most modern laptops | | Small | 7–8B | ~4–5 GB | 8 GB | 16 GB | Single consumer GPU (RTX 3060+) | | Medium | 13–14B | ~8–9 GB | 16 GB | 32 GB | RTX 3090 / 4090 or 2× mid-range | | Large | 30–34B | ~20 GB | 32 GB | 64 GB | A100 40GB or 2× RTX 3090 | | XL | 65–70B | ~40 GB | 64 GB | 128 GB | A100 80GB or 4× 24GB GPUs | | XXL | 180–405B | ~100–200 GB | 256 GB+ | 512 GB+ | Multi-GPU or multi-node RPC | > [!NOTE] > VRAM figures assume Q4_K_M quantization. Full F16 weights require roughly 2× the parameter count in bytes (e.g., 7B F16 ≈ 14 GB). The KV cache adds additional memory proportional to `n_ctx × n_layers × n_heads × head_dim × 2 × sizeof(type_k)`. ### Memory Estimation Formulas **Model weights** (approximate): ``` bytes ≈ parameters × bits_per_weight / 8 ``` | Quant Type | Bits/Weight | 7B Model | 70B Model | |------------|------------|---------|----------| | F32 | 32 | 28 GB | 280 GB | | F16 / BF16 | 16 | 14 GB | 140 GB | | Q8_0 | 8.5 | ~7.4 GB | ~74 GB | | Q6_K | 6.56 | ~5.7 GB | ~57 GB | | Q5_K_M | 5.68 | ~4.9 GB | ~49 GB | | Q4_K_M | 4.84 | ~4.2 GB | ~42 GB | | Q3_K_M | 3.91 | ~3.4 GB | ~34 GB | | Q2_K | 3.35 | ~2.9 GB | ~29 GB | | IQ1_S | ~1.56 | ~1.4 GB | ~14 GB | **KV cache** (per token, approximate): ``` kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × sizeof(kv_type) total_kv = kv_bytes_per_token × n_ctx ``` For Llama 3 8B at F16 KV, 4096 context: ``` 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes/token × 4096 = 512 MB ``` Quantized KV (Q8_0) halves this; Q4_0 quarters it at some quality cost. ### Software Requirements | Dependency | Minimum Version | Notes | |------------|----------------|-------| | C compiler | C17-capable (GCC 8+, Clang 7+, MSVC 2019+) | Required | | C++ compiler | C++17-capable | Required | | CMake | 3.14+ | Required | | Python | 3.8+ | Conversion scripts, gguf-py | | Git | Any recent | For source checkout | | CUDA Toolkit | 11.8+ (12.x recommended) | `GGML_CUDA=ON` | | Xcode / Metal SDK | macOS 12.3+ (Monterey) | `GGML_METAL=ON` (default on macOS) | | Vulkan SDK | 1.3+ | `GGML_VULKAN=ON` | | ROCm / HIP | 5.6+ | `GGML_HIP=ON` (AMD GPU) | | Intel oneAPI | 2024.0+ | `GGML_SYCL=ON` | | OpenCL | 3.0+ | `GGML_OPENCL=ON` | | CANN (Ascend) | 8.0+ | `GGML_CANN=ON` | | Node.js | 18+ | Web UI development only | ### Backend-Specific Requirements **CUDA:** - NVIDIA driver 520+ (for CUDA 11.8); 545+ (for CUDA 12.x) - cuBLAS is used automatically; no separate install needed with toolkit - Compute capability 5.0+ (Maxwell); 7.5+ recommended for best performance **Metal (Apple Silicon / Intel Mac):** - macOS 12.3 (Monterey) minimum - M1/M2/M3/M4 chips use unified memory — full model fits in shared GPU/CPU pool - `GGML_METAL_EMBED_LIBRARY=ON` bundles `.metallib` into the binary for portable distribution **Vulkan:** - Windows: Install [LunarG Vulkan SDK](https://vulkan.lunarg.com/) - Linux: `apt install libvulkan-dev vulkan-tools` (Ubuntu) or equivalent - Works on NVIDIA, AMD, Intel, and ARM Mali GPUs **Hexagon (Qualcomm Snapdragon):** - Qualcomm AI Engine Direct SDK - Android NDK r26+ - Device must have Hexagon DSP (Snapdragon 8 Gen 1+) - Set env vars: `GGML_HEXAGON_NDEV`, `GGML_HEXAGON_NHVX` to configure NPU units **CANN (Huawei Ascend):** - CANN 8.0+ toolkit installed on host - Ascend NPU driver - Tested on Atlas 300I/310/910 series --- ## 3. Configuration Reference ### CMake Build Options (Complete) #### Backend Selection | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_CUDA` | BOOL | OFF | Enable NVIDIA CUDA backend | | `GGML_METAL` | BOOL | ON (macOS) | Enable Apple Metal backend | | `GGML_VULKAN` | BOOL | OFF | Enable Vulkan backend (cross-vendor GPU) | | `GGML_OPENCL` | BOOL | OFF | Enable OpenCL backend | | `GGML_SYCL` | BOOL | OFF | Enable Intel SYCL backend (oneAPI) | | `GGML_HIP` | BOOL | OFF | Enable AMD ROCm/HIP backend | | `GGML_MUSA` | BOOL | OFF | Enable Moore Threads MUSA backend | | `GGML_CANN` | BOOL | OFF | Enable Huawei Ascend CANN backend | | `GGML_HEXAGON` | BOOL | OFF | Enable Qualcomm Hexagon NPU backend | | `GGML_BLAS` | BOOL | OFF | Enable BLAS backend for matrix ops | | `GGML_BLAS_VENDOR` | STRING | — | BLAS vendor: OpenBLAS, Apple, Intel, FLAME, FlexiBLAS, NVHPC, NVPL, AOCL | | `GGML_ACCELERATE` | BOOL | ON (macOS) | Enable Apple Accelerate framework (vDSP, BLAS) | | `GGML_BACKEND_DL` | BOOL | OFF | Enable dynamic backend loading at runtime | #### CPU Feature Flags | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_NATIVE` | BOOL | ON | Enable host-native CPU optimizations (`-march=native`) | | `GGML_OPENMP` | BOOL | ON | Enable OpenMP multi-threading | | `GGML_LLAMAFILE` | BOOL | ON | Enable llamafile SGEMM kernels (fast x86 matrix multiply) | | `GGML_CPU_HBM` | BOOL | OFF | Enable High Bandwidth Memory support | | `GGML_CPU_ALL_VARIANTS` | BOOL | OFF | Build all CPU SIMD variants in one binary | | `GGML_CPU_KLEIDIAI` | BOOL | OFF | Enable Arm KleidiAI optimized kernels | #### x86 SIMD | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_AVX` | BOOL | ON | Enable AVX instructions | | `GGML_AVX2` | BOOL | ON | Enable AVX2 instructions | | `GGML_AVX512` | BOOL | OFF | Enable AVX-512 foundation | | `GGML_AVX512_VBMI` | BOOL | OFF | Enable AVX-512 VBMI (byte manipulation) | | `GGML_AVX512_VNNI` | BOOL | OFF | Enable AVX-512 VNNI (neural net integer) | | `GGML_AVX512_BF16` | BOOL | OFF | Enable AVX-512 BF16 support | | `GGML_FMA` | BOOL | ON | Enable FMA (Fused Multiply-Add) | | `GGML_F16C` | BOOL | ON | Enable F16C (half-precision conversion) | #### Intel AMX (Advanced Matrix Extensions) | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_AMX_TILE` | BOOL | OFF | Enable AMX TILE instructions (Sapphire Rapids+) | | `GGML_AMX_INT8` | BOOL | OFF | Enable AMX INT8 operations | | `GGML_AMX_BF16` | BOOL | OFF | Enable AMX BF16 operations | #### RISC-V and LoongArch | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_RVV` | BOOL | OFF | Enable RISC-V Vector extension | | `GGML_LASX` | BOOL | OFF | Enable LoongArch Advanced SIMD Extension (LASX) | | `GGML_LSX` | BOOL | OFF | Enable LoongArch SIMD Extension (LSX) | #### Metal-Specific | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_METAL_EMBED_LIBRARY` | BOOL | OFF | Embed Metal shader library in binary | | `GGML_METAL_USE_BF16` | BOOL | OFF | Use BF16 precision on Metal backend | #### Library Build | Option | Type | Default | Description | |--------|------|---------|-------------| | `GGML_STATIC` | BOOL | OFF | Build static libraries instead of shared | | `LLAMA_BUILD_SERVER` | BOOL | ON | Build llama-server | | `LLAMA_BUILD_EXAMPLES` | BOOL | ON | Build example programs | | `LLAMA_BUILD_TOOLS` | BOOL | ON | Build tools (quantize, bench, etc.) | | `LLAMA_BUILD_TESTS` | BOOL | ON | Build test suite | #### Example Build Commands ```bash # CPU-only, release build cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --config Release -j$(nproc) # CUDA build cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build --config Release -j$(nproc) # Metal build (macOS, usually automatic) cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON cmake --build build --config Release # Vulkan build cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON cmake --build build -j$(nproc) # ROCm/HIP build (AMD GPU) cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1100" cmake --build build -j$(nproc) # Static library, no server or examples cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_STATIC=ON \ -DLLAMA_BUILD_SERVER=OFF -DLLAMA_BUILD_EXAMPLES=OFF cmake --build build -j$(nproc) # All CPU SIMD variants (fat binary for distribution) cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CPU_ALL_VARIANTS=ON cmake --build build -j$(nproc) ``` --- ### Model Parameters (`llama_model_params`) Populated by `llama_model_default_params()` and passed to `llama_model_load_from_file()`. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `n_gpu_layers` | int32_t | 0 | Number of transformer layers to offload to GPU. `-1` offloads all layers including embeddings | | `split_mode` | llama_split_mode | LAYER | Multi-GPU split strategy: `NONE` (single GPU), `LAYER` (layers across GPUs), `ROW` (tensor rows), `TENSOR` (full tensor parallelism) | | `main_gpu` | int32_t | 0 | Index of primary GPU for operations that cannot be split | | `tensor_split` | const float* | NULL | Array of proportions for distributing tensors across GPUs (length = number of GPUs, values sum to 1.0). NULL = equal distribution | | `vocab_only` | bool | false | Load only the tokenizer vocabulary, not weights. Useful for tokenization tools | | `use_mmap` | bool | true | Memory-map the model file. Reduces RAM usage but may increase page-fault latency | | `use_direct_io` | bool | false | Use O_DIRECT / FILE_FLAG_NO_BUFFERING. Bypasses OS page cache; useful when model doesn't fit in RAM | | `use_mlock` | bool | false | Lock model memory pages to prevent swapping. Requires elevated permissions on some systems | | `check_tensors` | bool | false | Validate tensor data during load (detects corrupt GGUF files at cost of load time) | | `use_extra_bufts` | bool | false | Enable extra buffer types for tensor repacking between backends | | `no_host` | bool | false | Bypass host (CPU) buffer allocation — advanced, backend-specific | | `no_alloc` | bool | false | Load metadata only, do not allocate tensor memory. For inspection tools | --- ### Context Parameters (`llama_context_params`) Populated by `llama_context_default_params()` and passed to `llama_init_from_model()`. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `n_ctx` | uint32_t | 0 | Context window size in tokens. 0 = use model's trained context length | | `n_batch` | uint32_t | 2048 | Maximum logical batch size (tokens submitted to `llama_decode()` at once) | | `n_ubatch` | uint32_t | 512 | Maximum physical (micro) batch size for GPU kernel launches | | `n_seq_max` | uint32_t | 1 | Maximum concurrent sequences in the KV cache | | `n_threads` | int32_t | -1 | Number of threads for token generation. -1 = hardware concurrency | | `n_threads_batch` | int32_t | -1 | Number of threads for prompt processing batches. -1 = hardware concurrency | | `rope_scaling_type` | enum | UNSPECIFIED | RoPE scaling: `NONE`, `LINEAR`, `YARN`, `LONGROPE`. UNSPECIFIED = auto from model | | `pooling_type` | enum | UNSPECIFIED | Embedding pooling: `NONE`, `MEAN`, `CLS`, `LAST`, `RANK`. For embedding models | | `attention_type` | enum | UNSPECIFIED | `CAUSAL` (decoder) or `NON_CAUSAL` (encoder). UNSPECIFIED = auto | | `flash_attn_type` | enum | AUTO | Flash attention: `AUTO` (enable if supported), `DISABLED`, `ENABLED` | | `rope_freq_base` | float | 0.0 | RoPE base frequency. 0 = use model default | | `rope_freq_scale` | float | 0.0 | RoPE frequency scale factor. 0 = use model default | | `yarn_ext_factor` | float | -1.0 | YaRN extrapolation mixing factor. -1 = use model default | | `yarn_attn_factor` | float | 1.0 | YaRN attention scaling factor | | `yarn_beta_fast` | float | 32.0 | YaRN low-frequency correction dimension | | `yarn_beta_slow` | float | 1.0 | YaRN high-frequency correction dimension | | `yarn_orig_ctx` | uint32_t | 0 | YaRN original training context length. 0 = use model default | | `defrag_thold` | float | -1.0 | KV cache defragmentation threshold (0.0–1.0). -1 = disabled. **DEPRECATED** | | `type_k` | ggml_type | F16 | KV cache key quantization type. **EXPERIMENTAL** — see KV Cache Types table | | `type_v` | ggml_type | F16 | KV cache value quantization type. **EXPERIMENTAL** | | `embeddings` | bool | false | Enable embedding extraction mode (returns embeddings instead of logits) | | `offload_kqv` | bool | true | Offload KQV operations to GPU when available | | `no_perf` | bool | false | Disable performance timing collection | | `op_offload` | bool | true | Offload host-side operations to GPU device | | `swa_full` | bool | true | Allocate full-size Sliding Window Attention cache | | `kv_unified` | bool | true | Use unified KV buffer shared across all sequences | --- ### Quantization Parameters (`llama_model_quantize_params`) | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `nthread` | int32_t | 0 | Number of threads for quantization. 0 = all hardware threads | | `ftype` | llama_ftype | Q4_K_M | Target quantization format (see quantization types table) | | `output_tensor_type` | ggml_type | GGML_TYPE_COUNT | Override type for output tensor. COUNT = use ftype | | `token_embedding_type` | ggml_type | GGML_TYPE_COUNT | Override type for token embedding tensor. COUNT = use ftype | | `allow_requantize` | bool | false | Allow requantizing already-quantized tensors | | `quantize_output_tensor` | bool | true | Quantize the output (lm_head) tensor | | `only_copy` | bool | false | Copy tensors without quantization (type conversion only) | | `pure` | bool | false | Quantize all tensors uniformly (disable mixed-precision heuristics) | | `keep_split` | bool | false | Preserve original tensor split when quantizing multi-file models | | `dry_run` | bool | false | Simulate quantization without writing output | | `imatrix` | void* | NULL | Pointer to importance matrix data for imatrix-guided quantization | | `kv_overrides` | llama_model_kv_override* | NULL | Array of GGUF key-value overrides to apply during quantization | | `tt_overrides` | llama_tensor_type_override* | NULL | Per-tensor type overrides | | `prune_layers` | llama_layer_range* | NULL | Ranges of layers to prune (remove) from the model | --- ### Sampling Parameters Sampling controls how the next token is selected from the model's output distribution. | Parameter | CLI Flag | Default | Description | |-----------|---------|---------|-------------| | `temp` | `--temp` | 0.8 | Temperature: higher = more random, 0 = greedy | | `top_k` | `--top-k` | 40 | Keep only top-K tokens by probability. 0 = disabled | | `top_p` | `--top-p` | 0.95 | Nucleus sampling: keep smallest set with cumulative prob ≥ top_p | | `min_p` | `--min-p` | 0.05 | Minimum probability relative to top token | | `tfs_z` | `--tfs` | 1.0 | Tail-free sampling z. 1.0 = disabled | | `typical_p` | `--typical` | 1.0 | Locally typical sampling. 1.0 = disabled | | `repeat_penalty` | `--repeat-penalty` | 1.0 | Penalty for recently seen tokens. 1.0 = disabled | | `repeat_last_n` | `--repeat-last-n` | 64 | Context window for repeat penalty. -1 = full context | | `frequency_penalty` | `--frequency-penalty` | 0.0 | Frequency-based repetition penalty | | `presence_penalty` | `--presence-penalty` | 0.0 | Presence-based repetition penalty | | `mirostat` | `--mirostat` | 0 | Mirostat algorithm version: 0 = off, 1 = Mirostat, 2 = Mirostat 2.0 | | `mirostat_tau` | `--mirostat-lr` | 5.0 | Mirostat target entropy (tau) | | `mirostat_eta` | `--mirostat-ent` | 0.1 | Mirostat learning rate (eta) | | `xtc_probability` | `--xtc-probability` | 0.0 | XTC (Exclude Top Choices) probability. 0 = disabled | | `xtc_threshold` | `--xtc-threshold` | 0.1 | Minimum probability for XTC exclusion | | `seed` | `--seed` | -1 | RNG seed. -1 = random | | `grammar` | `--grammar` | — | GBNF grammar string for constrained output | | `grammar_file` | `--grammar-file` | — | Path to GBNF grammar file | | `json_schema` | `--json-schema` | — | JSON Schema string (converted to grammar automatically) | --- ### KV Cache Quantization Types > [!IMPORTANT] > Quantized KV cache types are **EXPERIMENTAL**. Quality impact varies by model and quantization level. Always validate perplexity when using quantized KV cache in production. | Type | Description | Memory vs F16 | Quality Impact | |------|-------------|---------------|----------------| | `F32` | Full 32-bit float | 2× | None (reference) | | `F16` | 16-bit float (default) | 1× | Negligible | | `BF16` | Brain float 16 | 1× | Negligible | | `Q8_0` | 8-bit symmetric quantization | 0.5× | Minimal | | `Q4_0` | 4-bit symmetric quantization | 0.25× | Moderate | | `Q4_1` | 4-bit with separate scale+min | 0.3× | Moderate | | `IQ4_NL` | 4-bit non-linear quantization | 0.25× | Low | | `Q5_0` | 5-bit symmetric | 0.31× | Low | | `Q5_1` | 5-bit with separate scale+min | 0.37× | Low | --- ### Environment Variables #### General | Variable | Description | |----------|-------------| | `LLAMA_ARG_*` | Override any CLI flag by converting to uppercase with `LLAMA_ARG_` prefix and replacing `-` with `_`. E.g., `LLAMA_ARG_N_GPU_LAYERS=35` | | `LLAMA_TRACE` | Set to `1` to enable verbose trace-level debug logging | | `LLAMA_MEDIA_MARKER` | Override the token used as image/audio placeholder in multimodal prompts | | `HF_TOKEN` | HuggingFace API token for downloading gated models | | `HUGGINGFACE_HUB_TOKEN` | Alternative to `HF_TOKEN` for HuggingFace authentication | | `MODEL_PATH` | Default model path for conversion scripts | | `LLAMA_CACHE` | Override the cache directory for downloaded models | #### Hexagon NPU (Qualcomm) | Variable | Description | |----------|-------------| | `GGML_HEXAGON_NDEV` | Number of Hexagon DSP devices to use | | `GGML_HEXAGON_NHVX` | Number of HVX (Hexagon Vector eXtensions) units | | `GGML_HEXAGON_HOSTBUF` | Host buffer size for Hexagon DMA transfers | | `GGML_HEXAGON_VERBOSE` | Enable verbose Hexagon backend logging | | `GGML_HEXAGON_PROFILE` | Enable Hexagon performance profiling | | `GGML_HEXAGON_OPMASK` | Bitmask of operation types to execute on Hexagon | | `GGML_HEXAGON_OPFILTER` | Comma-separated list of op names to filter to Hexagon | #### CUDA | Variable | Description | |----------|-------------| | `CUDA_VISIBLE_DEVICES` | Restrict visible CUDA GPUs (e.g., `0,2` to use only GPUs 0 and 2) | | `GGML_CUDA_NO_PEER_COPY` | Set to `1` to disable peer-to-peer GPU memory copies | | `GGML_CUDA_FORCE_MMQ` | Force matrix-multiply quantized kernels instead of cuBLAS | | `GGML_CUDA_FORCE_CUBLAS` | Force cuBLAS for all matrix operations | --- ## 4. Deployment & Scaling ### Single-Machine CPU Deployment CPU inference is suitable for smaller models (≤13B parameters) or when GPU hardware is unavailable. OpenMP threading provides parallel computation across all cores. ```bash # Recommended CPU settings for generation ./llama-cli \ --model /models/llama-3-8b-q4_k_m.gguf \ --threads $(( $(nproc) / 2 )) \ --threads-batch $(nproc) \ --ctx-size 4096 \ --batch-size 512 \ -p "Your prompt here" ``` > [!TIP] > For CPU inference, setting `n_threads` to half the physical core count (not hyperthreads) often outperforms using all threads, since memory bandwidth becomes the bottleneck. Experiment with `llama-bench` to find the optimal value for your hardware. Thread tuning guidance: - `--threads` (generation): Set to physical core count, sometimes half performs better - `--threads-batch` (prompt): Set to all logical cores for maximum throughput - Disable hyperthreading in BIOS for pure inference workloads on server CPUs ### Single-GPU Deployment ```bash # Full GPU offload (all layers on GPU) ./llama-server \ --model /models/llama-3-8b-q4_k_m.gguf \ --n-gpu-layers -1 \ --ctx-size 8192 \ --batch-size 2048 \ --ubatch-size 512 \ --port 8080 # Partial GPU offload (model too large for VRAM) # Offload 20 of 32 layers to GPU ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers 20 \ --threads 8 \ --ctx-size 4096 \ --port 8080 ``` ### Multi-GPU Deployment Three split modes are available: **LAYER split (default, recommended):** Consecutive transformer layers are assigned to different GPUs. GPU 0 handles layers 0–N, GPU 1 handles N+1–M, etc. ```bash ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --split-mode layer \ --tensor-split 0.5,0.5 \ --ctx-size 8192 \ --port 8080 ``` **ROW split:** Tensor rows distributed across GPUs. Good for models where layer boundaries create load imbalance. ```bash ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --split-mode row \ --port 8080 ``` **TENSOR split (experimental):** Full tensor parallelism across GPUs. Higher communication overhead but potentially better VRAM utilization. **Uneven GPU configurations:** ```bash # 70B model on 3× GPUs with different VRAM (24GB, 24GB, 48GB) ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --split-mode layer \ --tensor-split 0.25,0.25,0.5 \ --main-gpu 2 \ --port 8080 ``` ### RPC Backend (Distributed Inference) The RPC backend allows distributing model layers across multiple machines connected over a network. Each machine runs an `llama-rpc-server` process, and the primary machine connects to all of them. **On each worker machine:** ```bash # Start the RPC server (listens on all interfaces, port 50052) ./llama-rpc-server --host 0.0.0.0 --port 50052 # Or with specific memory limit ./llama-rpc-server --host 0.0.0.0 --port 50052 --mem 24000 ``` **On the primary machine:** ```bash ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --rpc 192.168.1.10:50052,192.168.1.11:50052 \ --port 8080 ``` > [!WARNING] > The RPC backend has no built-in authentication or encryption. Operate only on trusted private networks or VPNs. Never expose RPC ports to the public internet. ### Docker Deployment **CPU-only:** ```dockerfile FROM ubuntu:22.04 RUN apt-get update && apt-get install -y \ build-essential cmake git python3 python3-pip \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY . . RUN cmake -B build -DCMAKE_BUILD_TYPE=Release && \ cmake --build build -j$(nproc) EXPOSE 8080 CMD ["./build/bin/llama-server", "--model", "/models/model.gguf", \ "--host", "0.0.0.0", "--port", "8080"] ``` **CUDA (NVIDIA GPU passthrough):** ```dockerfile FROM nvidia/cuda:12.2.0-devel-ubuntu22.04 RUN apt-get update && apt-get install -y \ build-essential cmake git \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY . . RUN cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON && \ cmake --build build -j$(nproc) EXPOSE 8080 CMD ["./build/bin/llama-server", "--model", "/models/model.gguf", \ "--host", "0.0.0.0", "--port", "8080", "--n-gpu-layers", "-1"] ``` Run with GPU access: ```bash docker run --gpus all \ -v /path/to/models:/models \ -p 8080:8080 \ llama-cuda ``` **Docker Compose example:** ```yaml version: '3.8' services: llama-server: image: llama-cuda:latest runtime: nvidia environment: - NVIDIA_VISIBLE_DEVICES=all volumes: - /models:/models:ro ports: - "8080:8080" command: > ./build/bin/llama-server --model /models/llama-3-8b-q4_k_m.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers -1 --ctx-size 8192 --parallel 4 --cont-batching restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 ``` ### Server Deployment for Multiple Concurrent Users ```bash ./llama-server \ --model /models/llama-3-8b-q4_k_m.gguf \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers -1 \ --ctx-size 32768 \ --parallel 8 \ --cont-batching \ --batch-size 2048 \ --ubatch-size 512 \ --type-k q8_0 \ --type-v q8_0 \ --api-key "your-secret-key" \ --log-file /var/log/llama-server.log ``` > [!IMPORTANT] > With `--parallel N`, the total KV cache size becomes `N × n_ctx × kv_per_token`. For 8 parallel slots at 32K context with F16 KV on Llama 3 8B, this can exceed 16 GB. Use `--type-k q8_0 --type-v q8_0` or `--n-ctx-cache-auto` to stay within VRAM limits. ### iOS/macOS Deployment (XCFramework) llama.cpp provides a pre-built XCFramework for Apple platform integration. ```bash # Build XCFramework from source cd examples/llama.swiftui ./build-frameworks.sh # Or download pre-built from GitHub releases # Integrate in Xcode: File → Add Package Dependencies # Use local path or GitHub URL ``` In Swift: ```swift import LlamaFramework let model = try LlamaModel(path: "/path/to/model.gguf") let context = try LlamaContext(model: model, params: .default) let result = try await context.complete(prompt: "Hello") ``` The `examples/llama.swiftui/` directory contains a full SwiftUI app demonstrating model loading, chat UI, and background inference. ### Android Deployment (NDK/JNI) ```bash # Build for Android ARM64 cmake -B build-android \ -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-26 \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_OPENMP=OFF \ -DGGML_NATIVE=OFF cmake --build build-android -j$(nproc) ``` For Hexagon NPU acceleration on Snapdragon devices: ```bash cmake -B build-android \ -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-26 \ -DGGML_HEXAGON=ON \ -DGGML_NATIVE=OFF cmake --build build-android -j$(nproc) ``` The `examples/llama.android/` directory contains a complete Android Studio project with JNI bindings, model download UI, and chat interface. ### CANN (Huawei Ascend) Deployment ```bash # Source CANN environment source /usr/local/Ascend/ascend-toolkit/set_env.sh cmake -B build \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_CANN=ON cmake --build build -j$(nproc) # Run with Ascend NPU ./build/bin/llama-server \ --model /models/model.gguf \ --n-gpu-layers -1 \ --port 8080 ``` ### WebGPU Deployment WebGPU support enables browser-based inference (experimental). Build the WASM target: ```bash # Requires Emscripten emcmake cmake -B build-wasm \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_BACKEND_DL=OFF cmake --build build-wasm -j$(nproc) ``` --- ## 5. Operations ### Server Lifecycle Management **Starting the server:** ```bash # Foreground (development) ./llama-server --model /models/model.gguf --host 0.0.0.0 --port 8080 # Background with logging nohup ./llama-server \ --model /models/model.gguf \ --host 0.0.0.0 \ --port 8080 \ --log-file /var/log/llama-server.log \ 2>&1 & ``` **systemd unit file (`/etc/systemd/system/llama-server.service`):** ```ini [Unit] Description=llama.cpp Inference Server After=network.target [Service] Type=simple User=llama Group=llama WorkingDirectory=/opt/llama ExecStart=/opt/llama/llama-server \ --model /models/llama-3-8b-q4_k_m.gguf \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers -1 \ --parallel 4 \ --cont-batching \ --api-key "${LLAMA_API_KEY}" EnvironmentFile=/etc/llama/env Restart=always RestartSec=5 StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable llama-server sudo systemctl start llama-server sudo systemctl status llama-server ``` ### Health Checks ```bash # Basic health check curl http://localhost:8080/health # Response when ready: # {"status":"ok","slots_idle":4,"slots_processing":0} # Response when loading: # {"status":"loading model"} # Response when all slots busy: # {"status":"no slot available","slots_idle":0,"slots_processing":4} ``` The `/health` endpoint returns HTTP 200 when slots are available, HTTP 503 when all slots are busy or the model is loading. ### Prometheus Metrics Enable with `--metrics` flag: ```bash ./llama-server --model /models/model.gguf --metrics --port 8080 ``` Metrics endpoint: `GET http://localhost:8080/metrics` Key metrics exposed: | Metric | Type | Description | |--------|------|-------------| | `llamacpp_prompt_tokens_total` | Counter | Total prompt tokens processed | | `llamacpp_completion_tokens_total` | Counter | Total completion tokens generated | | `llamacpp_tokens_seconds` | Gauge | Current tokens/second rate | | `llamacpp_prompt_processing_seconds` | Histogram | Prompt processing latency | | `llamacpp_completion_seconds` | Histogram | Time-to-first-token latency | | `llamacpp_kv_cache_usage_ratio` | Gauge | KV cache utilization (0–1) | | `llamacpp_kv_cache_tokens` | Gauge | Tokens in KV cache | | `llamacpp_requests_total` | Counter | Total requests by status | | `llamacpp_slots_idle` | Gauge | Available inference slots | | `llamacpp_slots_processing` | Gauge | Slots currently handling requests | **Prometheus scrape config:** ```yaml scrape_configs: - job_name: 'llama-server' static_configs: - targets: ['localhost:8080'] metrics_path: '/metrics' ``` ### Slot Management Each slot represents one concurrent inference stream. The server manages slots automatically for `--parallel N` concurrent requests. ```bash # View slot status curl http://localhost:8080/slots # Response shows each slot's state, current prompt, tokens generated, etc. ``` Slot states: `idle`, `processing` ### LoRA Adapter Management LoRA adapters can be loaded at startup or hot-swapped at runtime. ```bash # Load LoRA at startup ./llama-server \ --model /models/llama-3-8b.gguf \ --lora /adapters/custom-adapter.gguf \ --lora-scale 0.8 \ --port 8080 # Multiple LoRA adapters ./llama-server \ --model /models/llama-3-8b.gguf \ --lora /adapters/adapter1.gguf \ --lora /adapters/adapter2.gguf \ --port 8080 ``` Hot-swap via API: ```bash # Update LoRA adapters at runtime curl -X POST http://localhost:8080/lora-adapters \ -H "Content-Type: application/json" \ -d '{ "adapters": [ {"id": 0, "scale": 1.0} ] }' # Remove all LoRA adapters curl -X DELETE http://localhost:8080/lora-adapters ``` ### Memory Monitoring ```bash # NVIDIA GPU memory watch -n 1 nvidia-smi --query-gpu=memory.used,memory.free,memory.total \ --format=csv,noheader # Apple Silicon unified memory sudo powermetrics --samplers memory_pressure -i 1000 -n 1 # Linux system memory watch -n 1 free -h # KV cache usage via API curl http://localhost:8080/slots | python3 -c " import json, sys slots = json.load(sys.stdin) for s in slots: print(f'Slot {s[\"id\"]}: {s[\"n_ctx\"]} tokens context, state={s[\"state\"]}') " ``` ### Model Hot-Reloading The server does not support in-process model hot-reload. To swap models: ```bash # Graceful restart with new model kill -SIGTERM $(pgrep llama-server) # Wait for in-flight requests to complete sleep 5 ./llama-server --model /models/new-model.gguf --port 8080 & ``` For zero-downtime model swaps, use a load balancer (nginx, HAProxy) with two server instances and perform a blue-green swap. ### Session State (GGSN Format) The CLI supports saving and loading conversation state: ```bash # Save session ./llama-cli --model /models/model.gguf \ --prompt "..." \ --prompt-cache /tmp/session.ggsn # Resume session ./llama-cli --model /models/model.gguf \ --prompt-cache /tmp/session.ggsn \ --prompt-cache-ro ``` Session files use the GGSN format: a binary dump of the KV cache and sampler state. ### Backup and Recovery **Model files:** GGUF files are immutable after quantization. Back up originals (HuggingFace F16) before quantization; regenerating quantized files from originals is always possible. **Session caches:** GGSN files are transient; back up only if long-running conversation state is critical. **LoRA adapters:** Treat like model files — immutable binary artifacts to be version-controlled. **Configuration:** Store startup scripts and configuration in version control. The server has no mutable runtime configuration state beyond what's in loaded model weights. --- ## 6. Performance ### Benchmarking with llama-bench `llama-bench` provides standardized performance measurement for comparing configurations. ```bash # Standard benchmark (pp128 = prompt processing 128 tokens, tg128 = token generation 128 tokens) ./llama-bench \ --model /models/llama-3-8b-q4_k_m.gguf \ --n-gpu-layers -1 \ -pp 128 -tg 128 # Multiple test sizes ./llama-bench \ --model /models/llama-3-8b-q4_k_m.gguf \ --n-gpu-layers -1 \ -pp 128,512,2048 -tg 128,512 # JSON output for automated processing ./llama-bench \ --model /models/llama-3-8b-q4_k_m.gguf \ --n-gpu-layers -1 \ --output json \ > results.json # Multi-GPU benchmark ./llama-bench \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --split-mode layer \ --tensor-split 0.5,0.5 \ -pp 512 -tg 128 ``` Key metrics reported: - `t/s` (prompt): Tokens per second during prompt processing (prefill) - `t/s` (generation): Tokens per second during token generation (decode) - `ms/token`: Latency per generated token ### Perplexity Measurement Perplexity quantifies model quality — lower is better. Use to validate quantization doesn't degrade model quality beyond acceptable thresholds. ```bash # Measure perplexity on standard corpus ./llama-perplexity \ --model /models/llama-3-8b-q4_k_m.gguf \ --n-gpu-layers -1 \ --file wikitext-2-raw/wiki.test.raw \ --ctx-size 512 \ --batch-size 512 # Compare quantization levels for quant in q2_k q3_k_m q4_k_m q5_k_m q6_k q8_0; do echo "=== $quant ===" ./llama-perplexity \ --model /models/llama-3-8b-${quant}.gguf \ --n-gpu-layers -1 \ --file wiki.test.raw \ --ctx-size 512 done ``` ### Importance Matrix Generation The imatrix improves quantization quality by weighting quantization error by activation importance. Always generate before quantizing below Q4. ```bash # Generate importance matrix ./llama-imatrix \ --model /models/llama-3-8b-f16.gguf \ --n-gpu-layers -1 \ --training-data groups_merged.txt \ --output /models/llama-3-8b-imatrix.dat \ --ctx-size 512 \ --batch-size 512 \ -ngl 99 # Quantize using importance matrix ./llama-quantize \ --imatrix /models/llama-3-8b-imatrix.dat \ /models/llama-3-8b-f16.gguf \ /models/llama-3-8b-iq4_xs.gguf \ IQ4_XS ``` ### KV Cache Optimization **Quantized KV cache** reduces memory usage allowing larger context windows: ```bash # Q8_0 KV cache: 50% memory savings, minimal quality loss ./llama-server \ --model /models/model.gguf \ --n-gpu-layers -1 \ --type-k q8_0 --type-v q8_0 \ --ctx-size 32768 \ --port 8080 # Q4_0 KV cache: 75% savings, moderate quality impact ./llama-server \ --model /models/model.gguf \ --n-gpu-layers -1 \ --type-k q4_0 --type-v q4_0 \ --ctx-size 65536 \ --port 8080 ``` > [!CAUTION] > KV cache quantization below Q8_0 can visibly degrade coherence on long contexts. Always validate with perplexity measurements and qualitative testing before deploying Q4 KV cache in production. ### Flash Attention Flash attention reduces memory usage for long contexts by computing attention in tiles without materializing the full attention matrix. ```bash # Enable flash attention (reduces VRAM for long contexts) ./llama-server \ --model /models/model.gguf \ --n-gpu-layers -1 \ --flash-attn \ --ctx-size 32768 \ --port 8080 # Flash attention is AUTO by default on supported backends # Explicitly disable if you suspect compatibility issues: ./llama-server --no-flash-attn ... ``` Flash attention is supported on CUDA, Metal, and some Vulkan configurations. CPU flash attention is available for completeness but provides no memory benefit. ### Thread Tuning Guide | Scenario | `--threads` | `--threads-batch` | Rationale | |----------|------------|-------------------|-----------| | Consumer CPU (8–16 cores) | `nproc/2` | `nproc` | Memory BW limited during generation | | Server CPU (32+ cores) | 8–16 | `nproc` | Diminishing returns past 16 for generation | | Hybrid (CPU+GPU) | 4–8 | 4–8 | GPU does most work; CPU threads for sampling | | Pure CPU, embedded | `nproc` | `nproc` | Use all available resources | ```bash # Benchmark thread configurations for t in 1 2 4 8 16; do echo -n "threads=$t: " ./llama-bench --model /models/model.gguf --threads $t -tg 128 \ | grep "tg" | awk '{print $NF, "t/s"}' done ``` ### Batch Size Optimization | Parameter | Effect | Tuning Guidance | |-----------|--------|-----------------| | `--batch-size` (`n_batch`) | Max tokens per `llama_decode()` call | Larger = better GPU utilization during prefill; limited by VRAM | | `--ubatch-size` (`n_ubatch`) | Physical GPU batch size | Keep at 512 unless GPU kernel limits require adjustment | For multi-user server mode with continuous batching, larger `n_batch` allows the server to group more user requests together, improving GPU utilization: ```bash # High-throughput server ./llama-server \ --batch-size 4096 \ --ubatch-size 1024 \ --parallel 8 \ --cont-batching \ ... ``` ### Speculative Decoding Speculative decoding uses a small "draft" model to propose multiple tokens, which the large "target" model verifies in a single forward pass. Provides 2–4× speedup on generation-bound workloads. ```bash ./llama-server \ --model /models/llama-3-70b-q4_k_m.gguf \ --n-gpu-layers -1 \ --draft-model /models/llama-3-8b-q4_k_m.gguf \ --draft-n-gpu-layers -1 \ --draft-max 8 \ --port 8080 ``` > [!TIP] > Speculative decoding works best when the draft and target models share a tokenizer and training distribution. Using a smaller model from the same model family (e.g., Llama 3 8B drafting for Llama 3 70B) gives the highest acceptance rates. ### Lookahead Decoding Lookahead decoding provides speculative acceleration without a draft model, using n-gram caching from the current context. ```bash ./llama-cli \ --model /models/model.gguf \ --n-gpu-layers -1 \ --lookup-cache-static /path/to/ngram-cache.bin \ -p "Your prompt" ``` ### GPU Offloading Strategy Optimal layer distribution when model doesn't fully fit in VRAM: 1. First determine how many layers fit: run with `-ngl 999` and watch for OOM 2. If OOM, reduce by 5 layers at a time 3. The output tensor (`lm_head`) and input embeddings are included in the layer count with `-ngl -1` ```bash # Find maximum layers for your VRAM ./llama-bench \ --model /models/model.gguf \ --n-gpu-layers 20 -tg 32 # Increment until OOM, then back off by 2-3 layers for safety margin ``` For generation-latency-sensitive workloads (chatbots), prefer fewer GPU layers with full model in VRAM over partial offload. For throughput-sensitive workloads (batch processing), partial offload with high batch size may be acceptable. ### Performance Tuning Checklist - [ ] Use quantized model appropriate for available VRAM (Q4_K_M is the general-purpose recommendation) - [ ] Set `--n-gpu-layers -1` if model fits in VRAM - [ ] Enable `--flash-attn` for contexts > 8K tokens - [ ] Set `--cont-batching` for multi-user server deployments - [ ] Tune `--threads` / `--threads-batch` with `llama-bench` - [ ] Use `--type-k q8_0 --type-v q8_0` if context window is VRAM-limited - [ ] Generate imatrix before quantizing below Q4 quantization level - [ ] Enable `--mlock` on servers with sufficient RAM to prevent swapping - [ ] Set `--parallel` based on expected concurrent users and available VRAM - [ ] For CPU inference: disable hyperthreading, enable performance governor - [ ] Validate with `llama-bench` before and after configuration changes --- ## 7. Security ### API Key Authentication The server supports a single static API key for access control. ```bash ./llama-server \ --model /models/model.gguf \ --api-key "your-secret-key-here" \ --port 8080 ``` Clients must include the key in requests: ```bash # Authorization header (Bearer) curl http://localhost:8080/v1/completions \ -H "Authorization: Bearer your-secret-key-here" \ -H "Content-Type: application/json" \ -d '{"prompt": "Hello", "max_tokens": 50}' # X-API-Key header (alternative) curl http://localhost:8080/v1/completions \ -H "X-API-Key: your-secret-key-here" \ -H "Content-Type: application/json" \ -d '{"prompt": "Hello", "max_tokens": 50}' ``` > [!WARNING] > The server does not support multiple API keys or per-key rate limiting natively. For production deployments requiring multi-tenant access control, place a reverse proxy (nginx, Caddy, Kong) in front of the server. ### TLS/HTTPS llama-server has no built-in TLS. **Always use a reverse proxy for production.** **nginx reverse proxy example:** ```nginx server { listen 443 ssl; server_name llama.example.com; ssl_certificate /etc/letsencrypt/live/llama.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/llama.example.com/privkey.pem; ssl_protocols TLSv1.2 TLSv1.3; location / { proxy_pass http://localhost:8080; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 300s; proxy_send_timeout 300s; } } ``` ### Input Validation The server validates: - `max_tokens` and `n_predict`: capped at context window size - Prompt length: rejected if longer than `n_ctx` - JSON schema: malformed requests return 400 Administrators should additionally: - Set `--ctx-size` to the minimum needed (limits resource use per request) - Use `--parallel` to limit concurrent load - Implement request rate limiting at the reverse proxy layer ### Model File Integrity ```bash # Enable tensor validation during model load ./llama-server \ --model /models/model.gguf \ --check-tensors \ --port 8080 # Verify GGUF file is not corrupt before deployment ./llama-cli \ --model /models/model.gguf \ --check-tensors \ --vocab-only \ -p "" ``` > [!NOTE] > `--check-tensors` increases model load time significantly for large models (adds minutes for 70B models). Use for initial validation after download, not in routine startup. ### Security Policy The project maintains a `SECURITY.md` file at the repository root documenting the vulnerability disclosure process. Security issues should be reported privately to the maintainers rather than via public GitHub issues. Key security considerations for operators: - **No prompt sanitization**: the server passes prompts directly to the model; jailbreak attempts are a model-level concern, not a server-level one - **File system access**: the server reads model files with the permissions of its process user; run as a dedicated non-root user - **GPU memory isolation**: GPU memory is not zeroed between requests; on multi-user systems, one user's context is theoretically accessible to another through GPU memory forensics (not a practical concern for most deployments) - **Log files**: avoid logging full prompt content if handling sensitive user data ### MCP (Model Context Protocol) Security Considerations The web UI supports connecting to MCP servers, which allows the LLM to execute tools and access external data sources. This significantly expands the attack surface: > [!CAUTION] > MCP tool execution means the model can interact with external systems. In multi-user deployments, an adversarial user could craft prompts to abuse MCP tools. Only enable MCP with trusted MCP servers on trusted networks. Implement MCP server-side authorization and audit logging. - Restrict MCP server access to specific tool names where possible - Audit all MCP tool calls in server logs - Never run MCP servers with elevated privileges - Consider disabling MCP in the web UI for public-facing deployments --- ## 8. Developer Internals ### Repository Structure ``` llama.cpp/ ├── include/ # Public API headers │ ├── llama.h # Primary C API (stable public interface) │ └── llama-cpp.h # C++ RAII wrappers and smart pointers ├── src/ # Core llama library implementation │ ├── llama.cpp # Main library implementation file │ ├── llama-model.cpp/.h # Model loading and architecture dispatch │ ├── llama-context.cpp/.h # Inference context management │ ├── llama-sampling.cpp/.h # Sampler chain implementation │ ├── llama-memory.cpp/.h # Memory management (KV cache, buffers) │ ├── llama-kv-cache.cpp/.h # KV cache implementation │ ├── llama-graph.cpp/.h # Computation graph construction │ ├── llama-adapter.cpp/.h # LoRA adapter support │ ├── llama-mmap.cpp/.h # Memory-mapped file I/O │ ├── llama-batch.cpp/.h # Batch management │ ├── llama-vocab.cpp/.h # Vocabulary and tokenization │ ├── llama-chat.cpp/.h # Chat template utilities │ ├── llama-arch.cpp/.h # Architecture definitions and registry │ └── models/ # Model architecture implementations (114 files) │ ├── llama.cpp # LLaMA / LLaMA 2 / LLaMA 3 │ ├── mistral.cpp # Mistral / Mixtral │ ├── gemma.cpp # Google Gemma │ ├── phi.cpp # Microsoft Phi │ ├── qwen.cpp # Alibaba Qwen │ ├── deepseek.cpp # DeepSeek │ ├── command-r.cpp # Cohere Command R │ ├── bloom.cpp # BigScience BLOOM │ ├── falcon.cpp # Technology Innovation Institute Falcon │ ├── starcoder.cpp # BigCode StarCoder │ └── ... (100+ more) ├── ggml/ # GGML tensor library │ ├── include/ │ │ ├── ggml.h # Core tensor operations API │ │ ├── ggml-backend.h # Backend abstraction interface │ │ ├── ggml-alloc.h # Memory allocator │ │ └── gguf.h # GGUF format reader/writer │ └── src/ │ ├── ggml.c # Core implementation (quants, ops) │ ├── ggml-backend.cpp # Backend registry and routing │ ├── ggml-alloc.c # Allocator implementation │ ├── gguf.cpp # GGUF reader/writer implementation │ ├── ggml-quants.c # Quantization type implementations │ ├── ggml-threading.cpp # Thread pool │ ├── ggml-cpu/ # CPU backend (SIMD kernels) │ ├── ggml-cuda/ # CUDA backend │ ├── ggml-metal/ # Metal backend (+ .metal shaders) │ ├── ggml-vulkan/ # Vulkan backend (+ .comp shaders) │ ├── ggml-opencl/ # OpenCL backend │ ├── ggml-sycl/ # SYCL/oneAPI backend │ ├── ggml-hip/ # AMD ROCm/HIP backend │ ├── ggml-cann/ # Huawei CANN backend │ ├── ggml-hexagon/ # Qualcomm Hexagon NPU backend │ ├── ggml-musa/ # Moore Threads MUSA backend │ ├── ggml-blas/ # BLAS backend │ ├── ggml-rpc/ # RPC backend │ └── ggml-kompute/ # Vulkan compute (Kompute) ├── common/ # Shared utilities │ ├── arg.cpp/.h # Command-line argument parsing │ ├── chat.cpp/.h # Chat session management │ ├── sampling.cpp/.h # Sampler chain construction │ ├── download.cpp/.h # Model download from HuggingFace │ ├── log.cpp/.h # Logging subsystem │ ├── ngram-cache.cpp/.h # N-gram cache for lookahead decoding │ ├── speculative.cpp/.h # Speculative decoding utilities │ └── jinja/ # Jinja2-compatible template engine │ └── jinja.hpp # Single-header implementation ├── tools/ # Executable tools │ ├── cli/ # llama-cli (interactive chat) │ ├── server/ # llama-server (HTTP API) │ │ ├── server.cpp # HTTP server implementation │ │ ├── utils.hpp # Server utilities │ │ └── webui/ # SvelteKit web interface │ │ ├── src/ # TypeScript/Svelte source │ │ ├── tests/ # Vitest + Playwright tests │ │ └── package.json │ ├── quantize/ # llama-quantize │ ├── llama-bench/ # Performance benchmarking │ ├── perplexity/ # Perplexity measurement │ ├── imatrix/ # Importance matrix │ ├── tts/ # Text-to-speech │ ├── mtmd/ # Multimodal (vision/audio) │ ├── tokenize/ # Tokenization utility │ ├── gguf-split/ # GGUF file split/merge │ ├── export-lora/ # LoRA export │ ├── cvector-generator/ # Control vector generation │ ├── parser/ # PEG-based grammar parser │ ├── rpc/ # RPC server │ ├── batched-bench/ # Batched benchmarking │ ├── completion/ # Non-interactive completion │ └── results/ # Benchmark result processing ├── examples/ # Integration examples │ ├── llama.android/ # Android app (Kotlin/JNI) │ ├── llama.swiftui/ # iOS/macOS app (Swift/SwiftUI) │ └── ... ├── gguf-py/ # Python GGUF library │ └── gguf/ │ ├── gguf_reader.py # GGUF file reading │ ├── gguf_writer.py # GGUF file writing │ └── constants.py # Type definitions, enum mappings ├── tests/ # Test suite │ ├── test-*.cpp # C++ unit tests │ └── peg-parser/ # PEG parser-specific tests ├── cmake/ # CMake modules │ ├── FindSIMD.cmake # CPU feature detection │ └── llama-config.cmake.in # Package config template ├── grammars/ # GBNF grammar files │ ├── json.gbnf # JSON output constraint │ ├── json_arr.gbnf # JSON array output │ └── ... ├── docs/ # Documentation │ ├── build/ # Build guides per backend │ └── ops/ # Supported ops CSV per backend ├── ci/ │ └── run.sh # CI runner script ├── convert_hf_to_gguf.py # HuggingFace → GGUF converter ├── convert_hf_to_gguf_update.py # Update architecture support ├── .clang-format # C/C++ formatter config ├── .clang-tidy # C/C++ linter config ├── .editorconfig # Editor settings ├── .pre-commit-config.yaml # Pre-commit hooks ├── CODEOWNERS # Review routing ├── CMakeLists.txt # Root build file └── SECURITY.md # Vulnerability disclosure policy ``` ### Key Internal APIs #### Model Lifecycle ```c // Load a model from a GGUF file struct llama_model * llama_model_load_from_file( const char * path_model, const struct llama_model_params params); // Free model and all associated resources void llama_model_free(struct llama_model * model); // Create inference context from loaded model struct llama_context * llama_init_from_model( struct llama_model * model, struct llama_context_params params); // Free context (does not free the model) void llama_free(struct llama_context * ctx); ``` #### Core Inference ```c // Run the transformer forward pass for a batch of tokens // Returns 0 on success, negative on error int32_t llama_decode( struct llama_context * ctx, struct llama_batch batch); // For encoder-decoder models (T5, BART) int32_t llama_encode( struct llama_context * ctx, struct llama_batch batch); // Get logits for the last decoded token float * llama_get_logits(struct llama_context * ctx); // Get logits for a specific batch position float * llama_get_logits_ith(struct llama_context * ctx, int32_t i); // Get embeddings (when context has embeddings=true) float * llama_get_embeddings(struct llama_context * ctx); float * llama_get_embeddings_ith(struct llama_context * ctx, int32_t i); float * llama_get_embeddings_seq(struct llama_context * ctx, llama_seq_id seq_id); ``` #### Batch Management ```c // Allocate a batch (n_tokens capacity, n_seq_max sequences) struct llama_batch llama_batch_init(int32_t n_tokens, int32_t embd, int32_t n_seq_max); // Free batch resources void llama_batch_free(struct llama_batch batch); // llama_batch structure: struct llama_batch { int32_t n_tokens; // number of tokens in this batch llama_token * token; // token IDs (NULL if using embeddings) float * embd; // embeddings input (NULL if using tokens) llama_pos * pos; // position for each token int32_t * n_seq_id; // number of sequences for each token llama_seq_id ** seq_id; // sequence IDs for each token int8_t * logits; // 1 = compute logits for this position }; ``` #### Sampler Chain ```c // Initialize a sampler chain struct llama_sampler * llama_sampler_chain_init( struct llama_sampler_chain_params params); // Add a sampler to the chain (executed in order) void llama_sampler_chain_add( struct llama_sampler * chain, struct llama_sampler * smpl); // Sample next token from logits llama_token llama_sampler_chain_sample( struct llama_sampler * chain, struct llama_context * ctx, int32_t idx); // Built-in sampler constructors: struct llama_sampler * llama_sampler_init_top_k(int32_t k); struct llama_sampler * llama_sampler_init_top_p(float p, size_t min_keep); struct llama_sampler * llama_sampler_init_min_p(float p, size_t min_keep); struct llama_sampler * llama_sampler_init_typical(float p, size_t min_keep); struct llama_sampler * llama_sampler_init_temp(float t); struct llama_sampler * llama_sampler_init_dist(uint32_t seed); struct llama_sampler * llama_sampler_init_mirostat(int32_t n_vocab, uint32_t seed, float tau, float eta, int32_t m); struct llama_sampler * llama_sampler_init_mirostat_v2(uint32_t seed, float tau, float eta); struct llama_sampler * llama_sampler_init_grammar(const struct llama_vocab * vocab, const char * grammar_str, const char * grammar_root); struct llama_sampler * llama_sampler_init_penalties(int32_t penalty_last_n, float penalty_repeat, float penalty_freq, float penalty_present); struct llama_sampler * llama_sampler_init_xtc(float p, float t, size_t min_keep, uint32_t seed); // Free individual sampler void llama_sampler_free(struct llama_sampler * smpl); ``` #### Memory Management ```c // Copy sequence state (for forking conversations) bool llama_memory_seq_cp( struct llama_memory_state * ms, llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1); // Remove tokens from a sequence bool llama_memory_seq_rm( struct llama_memory_state * ms, llama_seq_id seq_id, llama_pos p0, llama_pos p1); // Get memory state from context struct llama_memory_state * llama_get_memory(struct llama_context * ctx); ``` #### C++ Smart Pointers (llama-cpp.h) ```cpp #include "llama-cpp.h" // RAII wrapper types using llama_model_ptr = std::unique_ptr; using llama_context_ptr = std::unique_ptr; using llama_sampler_ptr = std::unique_ptr; // Usage auto model = llama_model_ptr(llama_model_load_from_file(path, params)); auto context = llama_context_ptr(llama_init_from_model(model.get(), ctx_params)); // Automatically freed when going out of scope ``` #### Tokenization ```c // Tokenize text to token IDs int32_t llama_tokenize( const struct llama_vocab * vocab, const char * text, int32_t text_len, llama_token * tokens, int32_t n_tokens_max, bool add_special, bool parse_special); // Convert token ID back to text piece int32_t llama_token_to_piece( const struct llama_vocab * vocab, llama_token token, char * buf, int32_t length, int32_t lstrip, bool special); // Get vocabulary from model const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model); // Special tokens llama_token llama_vocab_bos(const struct llama_vocab * vocab); llama_token llama_vocab_eos(const struct llama_vocab * vocab); llama_token llama_vocab_pad(const struct llama_vocab * vocab); ``` ### Model Architecture Support The `src/models/` directory contains 114 files implementing model architectures. Each file implements the forward pass computation graph for that architecture using the GGML graph builder. Architectures are registered in `llama-arch.cpp` via an architecture registry keyed by the `general.architecture` GGUF metadata field. **Architecture dispatch flow:** ``` llama_model_load() → read "general.architecture" from GGUF metadata → look up architecture in registry (llama-arch.cpp) → call architecture-specific model builder → llama_model_build_graph() called at inference time → returns ggml_cgraph* for GGML to execute ``` Supported architecture families include (non-exhaustive): | Family | Architectures | |--------|--------------| | LLaMA | LLaMA 1/2/3/3.1/3.2/3.3, CodeLlama | | Mistral | Mistral, Mixtral (MoE), Mistral Nemo | | Google | Gemma, Gemma 2, Gemma 3, PaLiGemma (multimodal) | | Microsoft | Phi-2, Phi-3, Phi-3.5, Phi-4 | | Alibaba | Qwen 1/1.5/2/2.5, Qwen 2.5 VL (multimodal), QwQ | | DeepSeek | DeepSeek V1/V2/V3/R1, DeepSeek Coder | | Cohere | Command R, Command R+ | | xAI | Grok | | Encoder-only | BERT, RoBERTa, DistilBERT (for embeddings) | | Encoder-decoder | T5, FLAN-T5, BART, mBART | | Code models | StarCoder, StarCoder2, CodeGemma | | Multimodal | LLaVA, BakLLaVA, moondream, InternVL, SmolVLM | ### GGUF Format Internals GGUF (GGML Universal File) is the binary format used for all models in llama.cpp. **File structure:** ``` ┌─────────────────────────────────────┐ │ Magic: "GGUF" (4 bytes) │ │ Version: uint32 (current = 3) │ │ n_tensors: uint64 │ │ n_kv: uint64 │ ├─────────────────────────────────────┤ │ Key-Value Metadata │ │ key: string │ │ type: gguf_metadata_value_type │ │ value: (type-dependent) │ │ ... × n_kv │ ├─────────────────────────────────────┤ │ Tensor Descriptors │ │ name: string │ │ n_dims: uint32 │ │ dims: uint64 × n_dims │ │ type: ggml_type │ │ offset: uint64 (from data start) │ │ ... × n_tensors │ ├─────────────────────────────────────┤ │ Alignment padding │ ├─────────────────────────────────────┤ │ Tensor Data │ │ (binary, aligned to │ │ GGUF_DEFAULT_ALIGNMENT = 32 bytes)│ └─────────────────────────────────────┘ ``` **Standard GGUF metadata keys:** | Key | Type | Description | |-----|------|-------------| | `general.architecture` | string | Model architecture name | | `general.name` | string | Human-readable model name | | `general.quantization_version` | uint32 | Quantization scheme version | | `general.file_type` | uint32 | GGML file type enum | | `{arch}.context_length` | uint32 | Maximum context window | | `{arch}.embedding_length` | uint32 | Model hidden dimension | | `{arch}.block_count` | uint32 | Number of transformer layers | | `{arch}.attention.head_count` | uint32 | Number of attention heads | | `{arch}.attention.head_count_kv` | uint32 | Number of KV heads (GQA) | | `{arch}.rope.dimension_count` | uint32 | RoPE embedding dimension | | `{arch}.rope.freq_base` | float32 | RoPE base frequency | | `tokenizer.ggml.model` | string | Tokenizer type (llama, gpt2, etc.) | | `tokenizer.ggml.tokens` | string[] | Vocabulary tokens | | `tokenizer.ggml.scores` | float32[] | Token scores | | `tokenizer.ggml.token_type` | int32[] | Token type flags | | `tokenizer.chat_template` | string | Jinja2 chat template | **Python library usage:** ```python from gguf import GGUFReader, GGUFWriter, GGMLQuantizationType # Read a GGUF file reader = GGUFReader("model.gguf") for key, field in reader.fields.items(): print(f"{key}: {field.data}") for tensor in reader.tensors: print(f"{tensor.name}: shape={tensor.shape}, type={tensor.tensor_type}") # Write a GGUF file writer = GGUFWriter("output.gguf", arch="llama") writer.add_name("My Model") writer.add_context_length(4096) writer.add_embedding_length(4096) writer.add_tensor("token_embd.weight", tensor_data, raw_dtype=GGMLQuantizationType.Q4_K) writer.write_header_to_file() writer.write_kv_data_to_file() writer.write_tensors_to_file(progress=True) writer.close() ``` **Quantization type registry (GGMLQuantizationType enum, 40+ types):** | Type | Description | Block Size | Bits/Weight | |------|-------------|-----------|------------| | `F32` | 32-bit float | 1 | 32 | | `F16` | 16-bit float | 1 | 16 | | `BF16` | Brain float 16 | 1 | 16 | | `Q4_0` | 4-bit, no offset | 32 | 4.5 | | `Q4_1` | 4-bit, with offset | 32 | 5.0 | | `Q5_0` | 5-bit, no offset | 32 | 5.5 | | `Q5_1` | 5-bit, with offset | 32 | 6.0 | | `Q8_0` | 8-bit, no offset | 32 | 8.5 | | `Q8_1` | 8-bit, with offset | 32 | 9.0 | | `Q2_K` | 2-bit super-quantized | 256 | 3.35 | | `Q3_K_S` | 3-bit S variant | 256 | 3.5 | | `Q3_K_M` | 3-bit M variant | 256 | 3.91 | | `Q3_K_L` | 3-bit L variant | 256 | 4.27 | | `Q4_K_S` | 4-bit S variant | 256 | 4.58 | | `Q4_K_M` | 4-bit M variant (recommended) | 256 | 4.84 | | `Q5_K_S` | 5-bit S variant | 256 | 5.54 | | `Q5_K_M` | 5-bit M variant | 256 | 5.68 | | `Q6_K` | 6-bit super-quantized | 256 | 6.56 | | `Q8_K` | 8-bit super-quantized | 256 | 8.5 | | `IQ1_S` | 1-bit importance-matrix | 256 | ~1.56 | | `IQ1_M` | 1-bit M variant | 256 | ~1.75 | | `IQ2_XXS` | 2-bit extra-extra-small | 256 | 2.06 | | `IQ2_XS` | 2-bit extra-small | 256 | 2.31 | | `IQ2_S` | 2-bit small | 256 | 2.5 | | `IQ2_M` | 2-bit medium | 256 | 2.7 | | `IQ3_XXS` | 3-bit extra-extra-small | 256 | 3.06 | | `IQ3_XS` | 3-bit extra-small | 256 | 3.3 | | `IQ3_S` | 3-bit small | 256 | 3.44 | | `IQ3_M` | 3-bit medium | 256 | 3.66 | | `IQ4_NL` | 4-bit non-linear | 32 | 4.5 | | `IQ4_XS` | 4-bit extra-small | 256 | 4.25 | `IQ*` types use importance-matrix-guided quantization and require an imatrix file to generate but can be loaded and run without it. ### Chat Template System The chat template system renders conversation history into the exact prompt format a model expects. Templates are stored in GGUF metadata under `tokenizer.chat_template` as Jinja2-format strings. **Template auto-detection flow:** ``` llama_chat_apply_template() → if explicit template string: use it → else: read "tokenizer.chat_template" from model GGUF → if not found: detect from known model architectures → render messages array using Jinja2-compatible engine ``` **Jinja2 engine in `common/jinja/`:** The engine supports the subset of Jinja2 used in chat templates: - Variable substitution: `{{ variable }}` - Control flow: `{% if %}`, `{% for %}`, `{% set %}` - Filters: `| lower`, `| upper`, `| trim`, `| join`, `| tojson` - Special variables: `messages`, `tools`, `add_generation_prompt` **Tool calling support:** Models supporting tool calling use the template system to serialize tool definitions and parse tool call responses. The `tools` variable in templates contains the JSON-serialized tool definitions. **Reasoning format support:** Models with chain-of-thought reasoning (DeepSeek R1, QwQ) use templates that include `` block handling. The template engine correctly handles these extended formats. **Structured output (PEG parser):** `tools/parser/` implements a PEG (Parsing Expression Grammar) based parser for constrained output generation. This is distinct from GBNF grammars — it provides higher-level structured parsing for JSON extraction, tool call parsing, and similar tasks. ### Backend System Architecture The GGML backend system provides hardware abstraction. Every hardware backend implements the same interface, allowing the scheduler to dispatch operations to the appropriate hardware. **Backend interface (simplified):** ```c struct ggml_backend_i { const char * (*get_name)(ggml_backend_t backend); void (*free)(ggml_backend_t backend); void (*set_tensor_async)(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, size_t offset, size_t size); void (*get_tensor_async)(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, size_t offset, size_t size); bool (*cpy_tensor_async)(ggml_backend_t backend_src, ggml_backend_t backend_dst, const struct ggml_tensor * src, struct ggml_tensor * dst); void (*synchronize)(ggml_backend_t backend); enum ggml_status (*graph_compute)(ggml_backend_t backend, struct ggml_cgraph * cgraph); bool (*supports_op)(ggml_backend_dev_t dev, const struct ggml_tensor * op); bool (*supports_buft)(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft); bool (*offload_op)(ggml_backend_dev_t dev, const struct ggml_tensor * op); }; ``` **Backend registration:** ```c // Static registration (compile-time) ggml_backend_t ggml_backend_cuda_init(int device); ggml_backend_t ggml_backend_metal_init(void); ggml_backend_t ggml_backend_vulkan_init(size_t dev_num); // Dynamic loading (GGML_BACKEND_DL=ON) void ggml_add_backend_library(const char * path); ``` **Backend scheduler:** The `ggml_backend_sched` distributes computation graph nodes across available backends. Nodes are assigned based on: 1. Which backend owns the output buffer 2. Which backends support the operation 3. Whether peer-to-peer data transfer is available **Supported ops per backend:** Documented in `docs/ops/` as CSV files listing which GGML operations each backend supports. This is the authoritative reference for backend capability limitations. **CPU feature detection:** The CPU backend uses `CPUID` instruction probing at startup to detect available SIMD features and selects optimal kernel variants for each operation. When `GGML_CPU_ALL_VARIANTS=ON`, multiple kernel variants are compiled in and selected at runtime. ### Code Patterns and Conventions **Opaque pointer pattern:** Public headers declare structs without definitions: ```c // In llama.h (public): struct llama_model; struct llama_context; // In src/llama.cpp (private): struct llama_model { /* full definition */ }; struct llama_context { /* full definition */ }; ``` This provides ABI stability — the struct layout can change without breaking client code compiled against old headers. **C API with C++ internals:** The public API is pure C (`llama.h`), making it linkable from any language. Internally, the implementation is C++17. The `llama-cpp.h` header provides idiomatic C++ RAII wrappers for applications that prefer them. **Atomic file operations:** Model loading uses atomic operations to prevent partial reads from a concurrent write. The GGUF reader validates the magic bytes and file structure before committing to loading. **Thread pool:** `ggml-threading.cpp` implements a work-stealing thread pool. Each backend can request compute work be distributed across the pool. The pool size is controlled by `n_threads` / `n_threads_batch` context parameters. ### Testing Infrastructure #### C++ Unit Tests Located in `tests/`. Built when `LLAMA_BUILD_TESTS=ON` (default). ```bash # Build and run all tests cmake -B build -DLLAMA_BUILD_TESTS=ON cmake --build build --target test # or cd build && ctest --output-on-failure # Run specific test ./build/tests/test-tokenizer-0 /path/to/model.gguf ./build/tests/test-sampling ./build/tests/test-chat-template ``` Key test files: - `test-tokenizer-0.cpp`: BPE tokenizer correctness - `test-tokenizer-1-bpe.cpp`: Additional BPE tests - `test-sampling.cpp`: Sampler chain behavior - `test-chat-template.cpp`: Jinja template rendering - `test-gguf.cpp`: GGUF read/write correctness - `test-backend-ops.cpp`: Backend operation correctness (requires built backend) #### PEG Parser Tests ```bash cd tests/peg-parser npm install npm test ``` #### Server Integration Tests Python test suite using `aiohttp`: ```bash cd tools/server/tests pip install -r requirements.txt # Run tests (requires running server) ./run_tests.sh # Or manually python -m pytest test_completion.py -v python -m pytest test_chat_completion.py -v python -m pytest test_embedding.py -v ``` #### Web UI Tests ```bash cd tools/server/webui npm install # Unit tests (Vitest) npm run test # E2E tests (Playwright) npm run test:e2e # Component tests (Storybook) npm run storybook npm run test:storybook ``` #### CI Pipeline `ci/run.sh` orchestrates multi-backend test runs. CI validates: - CPU build and test suite - CUDA build (on CUDA-equipped runners) - Metal build (on macOS runners) - Vulkan build - Python conversion scripts - Server integration tests - Tokenizer correctness ### Contributing **Code style:** ```bash # Install pre-commit hooks pip install pre-commit pre-commit install # Run manually pre-commit run --all-files # Format C/C++ code clang-format -i src/*.cpp include/*.h clang-tidy src/*.cpp -- -I include ``` Configuration files: - `.clang-format`: Google style base with modifications - `.clang-tidy`: Warning set for C++ best practices - `.editorconfig`: 4-space indent, LF line endings, trim trailing whitespace **CODEOWNERS** routes PR reviews to backend-specific maintainers: - `ggml/src/ggml-cuda/` → CUDA backend maintainers - `ggml/src/ggml-metal/` → Metal maintainers - `ggml/src/ggml-vulkan/` → Vulkan maintainers - `examples/llama.android/` → Android maintainers - `examples/llama.swiftui/` → iOS/macOS maintainers **Adding a new model architecture:** 1. Add architecture name to `llama_arch` enum in `src/llama-arch.h` 2. Register architecture string mapping in `src/llama-arch.cpp` 3. Create `src/models/{arch}.cpp` implementing `llm_build_{arch}()` graph builder 4. Add model-specific metadata keys in `src/llama-arch.h` 5. Add conversion support in `convert_hf_to_gguf.py` (add model class) 6. Add test in `tests/` for tokenizer and/or architecture **Adding a new backend:** 1. Create `ggml/src/ggml-{backend}/` directory 2. Implement `ggml_backend_i` interface 3. Add CMake option `GGML_{BACKEND}` in `ggml/CMakeLists.txt` 4. Register backend in `ggml/src/ggml-backend-reg.cpp` 5. Document supported ops in `docs/ops/{backend}.csv` 6. Add build documentation in `docs/build/` --- ## 9. Integration Points ### C API Integration The stable public C API is the recommended integration point for all external consumers. ```c #include "llama.h" // Minimal working example int main() { llama_backend_init(); struct llama_model_params model_params = llama_model_default_params(); model_params.n_gpu_layers = 99; struct llama_model * model = llama_model_load_from_file("model.gguf", model_params); struct llama_context_params ctx_params = llama_context_default_params(); ctx_params.n_ctx = 4096; struct llama_context * ctx = llama_init_from_model(model, ctx_params); // Tokenize const char * prompt = "<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n"; int n_tokens = llama_tokenize(llama_model_get_vocab(model), prompt, strlen(prompt), NULL, 0, true, true); llama_token * tokens = malloc(n_tokens * sizeof(llama_token)); llama_tokenize(llama_model_get_vocab(model), prompt, strlen(prompt), tokens, n_tokens, true, true); // Create batch and decode struct llama_batch batch = llama_batch_init(n_tokens, 0, 1); for (int i = 0; i < n_tokens; i++) { batch.token[i] = tokens[i]; batch.pos[i] = i; batch.n_seq_id[i] = 1; batch.seq_id[i][0] = 0; batch.logits[i] = (i == n_tokens - 1); } batch.n_tokens = n_tokens; llama_decode(ctx, batch); // Sample struct llama_sampler * sampler = llama_sampler_chain_init( llama_sampler_chain_default_params()); llama_sampler_chain_add(sampler, llama_sampler_init_temp(0.8)); llama_sampler_chain_add(sampler, llama_sampler_init_dist(42)); llama_token next = llama_sampler_chain_sample(sampler, ctx, n_tokens - 1); // Cleanup llama_sampler_free(sampler); llama_batch_free(batch); llama_free(ctx); llama_model_free(model); llama_backend_free(); free(tokens); return 0; } ``` **Linking with CMake:** ```cmake find_package(llama REQUIRED) target_link_libraries(myapp PRIVATE llama) ``` Or with pkg-config: ```bash gcc -o myapp myapp.c $(pkg-config --cflags --libs llama) ``` ### C++ RAII API ```cpp #include "llama-cpp.h" #include "llama.h" int main() { llama_backend_init(); auto model_params = llama_model_default_params(); model_params.n_gpu_layers = -1; auto model = llama_model_ptr( llama_model_load_from_file("model.gguf", model_params)); auto ctx_params = llama_context_default_params(); ctx_params.n_ctx = 4096; auto ctx = llama_context_ptr( llama_init_from_model(model.get(), ctx_params)); auto sampler = llama_sampler_ptr( llama_sampler_chain_init(llama_sampler_chain_default_params())); llama_sampler_chain_add(sampler.get(), llama_sampler_init_temp(0.8f)); llama_sampler_chain_add(sampler.get(), llama_sampler_init_dist(42)); // model, ctx, sampler all freed automatically at scope exit llama_backend_free(); } ``` ### OpenAI-Compatible REST API The server implements the OpenAI API specification for drop-in compatibility with existing OpenAI API clients. **Base URL:** `http://localhost:8080` **Endpoints:** | Method | Path | OpenAI Equivalent | Description | |--------|------|-------------------|-------------| | POST | `/v1/chat/completions` | `POST /v1/chat/completions` | Chat completion (streaming and non-streaming) | | POST | `/v1/completions` | `POST /v1/completions` | Text completion | | POST | `/v1/embeddings` | `POST /v1/embeddings` | Text embeddings | | GET | `/v1/models` | `GET /v1/models` | List available models | | POST | `/v1/audio/speech` | `POST /v1/audio/speech` | TTS audio generation | | GET | `/health` | — | Server health check | | GET | `/metrics` | — | Prometheus metrics | | GET | `/slots` | — | Slot status | | POST | `/tokenize` | — | Tokenize text | | POST | `/detokenize` | — | Detokenize tokens | | POST | `/lora-adapters` | — | Manage LoRA adapters | | GET | `/props` | — | Server properties | **Chat completions example:** ```python import openai client = openai.OpenAI( base_url="http://localhost:8080/v1", api_key="your-api-key" ) response = client.chat.completions.create( model="local-model", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"} ], temperature=0.7, max_tokens=200, stream=True ) for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` **llama.cpp-specific request extensions:** ```json { "prompt": "...", "n_predict": 200, "temperature": 0.8, "top_k": 40, "top_p": 0.95, "min_p": 0.05, "repeat_penalty": 1.1, "repeat_last_n": 64, "grammar": "root ::= [a-z]+", "json_schema": {"type": "object", "properties": {}}, "cache_prompt": true, "slot_id": -1, "mirostat": 0, "mirostat_tau": 5.0, "mirostat_eta": 0.1, "seed": -1, "ignore_eos": false, "stream": true } ``` ### Python / gguf-py ```python # Install pip install gguf # Inspect a GGUF file from gguf import GGUFReader reader = GGUFReader("model.gguf", "r") print(f"Architecture: {reader.fields['general.architecture'].data}") print(f"Context length: {reader.fields['llama.context_length'].data}") for tensor in reader.tensors: print(f" {tensor.name}: {tensor.shape} ({tensor.tensor_type.name})") # Convert a custom model to GGUF from gguf import GGUFWriter, GGMLQuantizationType import numpy as np writer = GGUFWriter("output.gguf", arch="llama") writer.add_name("My Custom Model") writer.add_context_length(2048) writer.add_embedding_length(4096) writer.add_block_count(32) weights = np.random.randn(4096, 4096).astype(np.float32) writer.add_tensor("blk.0.attn_q.weight", weights) writer.write_header_to_file() writer.write_kv_data_to_file() writer.write_tensors_to_file(progress=True) writer.close() ``` ### Grammar / Constrained Output **GBNF (GGML BNF) grammar:** ```bash # Use built-in JSON grammar ./llama-cli --model model.gguf --grammar-file grammars/json.gbnf \ -p "List 3 fruits as JSON" # Custom grammar for specific output format ./llama-cli --model model.gguf \ --grammar 'root ::= "Yes" | "No" | "Maybe"' \ -p "Will it rain tomorrow?" ``` **JSON Schema to grammar conversion:** ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [{"role": "user", "content": "Describe a person"}], "response_format": { "type": "json_schema", "json_schema": { "name": "person", "schema": { "type": "object", "properties": { "name": {"type": "string"}, "age": {"type": "integer"}, "occupation": {"type": "string"} }, "required": ["name", "age"] } } } }' ``` ### Embedding Extraction ```bash # Start server in embedding mode ./llama-server \ --model /models/nomic-embed-text-v1.5-q8_0.gguf \ --port 8080 \ --embeddings \ --pooling mean # Generate embeddings curl http://localhost:8080/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "input": ["Hello world", "How are you?"], "model": "local-embed", "encoding_format": "float" }' ``` For encoder-only models, use `--pooling cls` or `--pooling mean` as appropriate for the model's training. ### CMake Package Integration ```cmake cmake_minimum_required(VERSION 3.14) project(my_llama_app) find_package(llama REQUIRED) add_executable(my_app main.cpp) target_link_libraries(my_app PRIVATE llama) target_include_directories(my_app PRIVATE ${LLAMA_INCLUDE_DIRS}) ``` ```bash cmake -B build \ -Dllama_DIR=/path/to/llama.cpp/build/lib/cmake/llama cmake --build build ``` ### pkg-config Integration ```bash # After installing llama.cpp pkg-config --cflags llama pkg-config --libs llama # Compile a single file gcc -o myapp myapp.c \ $(pkg-config --cflags llama) \ $(pkg-config --libs llama) ``` ### XCFramework (Apple) ```bash # Build XCFramework for distribution (iOS + macOS + simulators) cd examples/llama.swiftui ./build-xcframework.sh # Produces: llama.xcframework # Contains: # ios-arm64/ → Physical iPhone/iPad # ios-arm64_x86_64-simulator/ → Simulator # macos-arm64_x86_64/ → macOS (Apple Silicon + Intel) ``` Integrate in Xcode: 1. Drag `llama.xcframework` into your project 2. Set `Embed & Sign` for the framework 3. `import llama` in Swift files ### Android JNI/NDK Integration The `examples/llama.android/` directory provides a complete Android integration: ```kotlin // LlamaAndroid.kt (JNI wrapper) class LlamaAndroid { companion object { init { System.loadLibrary("llama-android") } } external fun loadModel(modelPath: String, nGpuLayers: Int): Long external fun createContext(modelPtr: Long, nCtx: Int): Long external fun completion(contextPtr: Long, prompt: String): String external fun freeContext(contextPtr: Long) external fun freeModel(modelPtr: Long) } ``` Build for multiple ABIs: ```bash cmake -B build-arm64 \ -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-26 \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_NATIVE=OFF cmake -B build-x86_64 \ -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=x86_64 \ -DANDROID_PLATFORM=android-26 \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_NATIVE=OFF ``` ### MCP (Model Context Protocol) The web UI supports MCP servers, enabling the LLM to call external tools: ```json { "servers": [ { "name": "filesystem", "transport": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/allowed/path"] }, { "name": "web-search", "transport": "sse", "url": "http://localhost:3001/sse" } ] } ``` The MCP client in the web UI translates tool calls from the LLM's function calling format into MCP protocol messages and returns results as tool response messages. > [!NOTE] > MCP support requires a model with function calling capability (tool use). Not all models support this — check the model's documentation. Models must be loaded with a chat template that includes tool definition rendering. ### Nix Flake ```bash # Run with Nix (CPU) nix run github:ggml-org/llama.cpp -- --model model.gguf -p "Hello" # Build with CUDA in Nix nix build github:ggml-org/llama.cpp#cuda # Enter development shell nix develop github:ggml-org/llama.cpp ``` ### Docker Hub / GHCR Images Official Docker images are available at `ghcr.io/ggml-org/llama.cpp`: ```bash # CPU docker pull ghcr.io/ggml-org/llama.cpp:latest # CUDA docker pull ghcr.io/ggml-org/llama.cpp:cuda # Run docker run --gpus all \ -v /models:/models \ -p 8080:8080 \ ghcr.io/ggml-org/llama.cpp:cuda \ --server \ --model /models/model.gguf \ --n-gpu-layers -1 \ --host 0.0.0.0 \ --port 8080 ``` --- _Generated May 2026 from ggml-org/llama.cpp at a279d0f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._