# bitnet.cpp User Guide

bitnet.cpp is the official inference framework for **1-bit large language models** — models whose weights are stored in roughly 1.58 bits each (the ternary values −1, 0, +1) instead of the usual 16 or 32 bits. It lets you run models such as **BitNet b1.58** fast and losslessly on an ordinary CPU, and (through a separate code path) on an NVIDIA GPU. A 100-billion-parameter BitNet model can run on a single CPU at roughly human reading speed (5–7 tokens per second), which makes capable local language models practical on everyday hardware.

This guide shows you how to install bitnet.cpp, build a model, run inference from the command line and as a server, benchmark performance, and convert your own models. Everything here is drawn directly from the framework's scripts and configuration as they exist at the documented commit.

> [!NOTE]
> bitnet.cpp is built on top of the [llama.cpp](https://github.com/ggerganov/llama.cpp) inference engine and uses lookup-table kernel techniques pioneered by [T-MAC](https://github.com/microsoft/T-MAC/). When you run inference, the Python scripts in this project drive a compiled `llama-cli` / `llama-server` binary under the hood. You do not need to understand llama.cpp to use bitnet.cpp, but it explains why the build step compiles a large C++ project.

---

## 1. Quick Start

The fastest path from nothing to a running chat model is three commands after installing the prerequisites.

```bash
# 1. Clone the repository (the --recursive flag pulls in the bundled llama.cpp)
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

# 2. Create the environment and install Python dependencies
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp
pip install -r requirements.txt

# 3. Download a model, build the kernels, and quantize — all in one command
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
```

Once `setup_env.py` finishes, run the model in chat mode:

```bash
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
```

```text
> You are a helpful assistant
Hello! How can I help you today?
```

> [!IMPORTANT]
> The `--recursive` flag on `git clone` is mandatory. bitnet.cpp embeds llama.cpp as a Git submodule under `3rdparty/llama.cpp`, and the build will fail without it. If you already cloned without `--recursive`, run `git submodule update --init --recursive` inside the repository.

> [!TIP]
> The `-cnv` (conversation) flag turns on chat mode and is intended for **instruct** models. When `-cnv` is on, the text you pass to `-p` becomes the system prompt rather than a one-shot completion prompt.

---

## 2. Installation

### 2.1 Requirements

| Requirement | Minimum version | Notes |
|---|---|---|
| Python | 3.9 | The orchestration scripts and converters are Python. |
| CMake | 3.22 | Drives the C++ build. |
| Clang | 18 | bitnet.cpp compiles with Clang/Clang++, not GCC. |
| conda | — | Strongly recommended for an isolated environment. |

> [!NOTE]
> A C++ toolchain is required because building a model compiles the bundled llama.cpp engine and the BitNet ternary kernels. This is a one-time step per machine/model, not something you do on every run.

### 2.2 Installing the compiler

=== "Debian / Ubuntu"

    Use the official LLVM installer script to get a recent Clang:

    ```bash
    bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
    ```

=== "Windows"

    Install [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/). In the installer, enable at least:

    - **Desktop development with C++**
    - **C++ CMake Tools for Windows**
    - **Git for Windows**
    - **C++ Clang Compiler for Windows**
    - **MS-Build Support for LLVM (clang) Toolset**

    These options also install CMake and the other tools you need.

=== "macOS"

    Install the Xcode command-line tools and a recent Clang via Homebrew (`brew install llvm cmake`). The Apple Silicon (ARM) path uses the I2_S and TL1 kernels.

> [!WARNING]
> On Windows you must run every build command from a **Developer Command Prompt** or **Developer PowerShell for VS 2022**, not a plain terminal. Otherwise CMake cannot find `clang` / `clang++`. See [Troubleshooting](#9-troubleshooting) for the exact environment-initialization commands.

### 2.3 Install Python dependencies

```bash
# (Recommended) isolated environment
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp

pip install -r requirements.txt
```

> [!NOTE]
> `requirements.txt` pulls in the conversion dependencies from the bundled llama.cpp (the various `convert_*` requirement files). You do not add packages to it directly; the model-conversion utilities rely on those exact pinned versions.

---

## 3. Preparing a Model with `setup_env.py`

`setup_env.py` is the one-stop build-and-prepare tool. A single invocation performs four steps in order: installs the GGUF Python helper, generates or copies the compute kernels, compiles the C++ project, and downloads + converts + quantizes your model into a ready-to-run `.gguf` file.

### 3.1 Two ways to point it at a model

You can either let it download a known model from Hugging Face, or point it at a directory you already have.

=== "Download by Hugging Face repo"

    ```bash
    python setup_env.py --hf-repo microsoft/BitNet-b1.58-2B-4T -q i2_s
    ```

=== "Use a local model directory"

    ```bash
    huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
    python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
    ```

### 3.2 Options

| Option | Short | Default | Description |
|---|---|---|---|
| `--hf-repo` | `-hr` | — | Hugging Face model to download and use. Must be one of the supported repos (see [Supported Models](#4-supported-models)). |
| `--model-dir` | `-md` | `models` | Directory to save/load the model. |
| `--log-dir` | `-ld` | `logs` | Directory where per-step build logs are written. |
| `--quant-type` | `-q` | `i2_s` | Quantization kernel to build for. Valid values depend on your CPU architecture (see below). |
| `--quant-embd` | — | off | Also quantize the token-embedding layer to a compact format (Q6_K) for extra speed. |
| `--use-pretuned` | `-p` | off | Use the pre-tuned kernel parameters shipped in `preset_kernels/` instead of regenerating them. |

> [!IMPORTANT]
> The valid `--quant-type` values depend on your CPU:
>
> | Architecture | Allowed quant types |
> |---|---|
> | ARM (arm64 / aarch64 / Apple Silicon) | `i2_s`, `tl1` |
> | x86-64 (Intel / AMD) | `i2_s`, `tl2` |
>
> Passing a type that doesn't match your architecture is rejected. `i2_s` works everywhere and is the default.

### 3.3 What the build produces

After `setup_env.py` completes you will have:

- A compiled engine under `build/bin/` (`llama-cli`, `llama-server`, `llama-quantize`, and the other llama.cpp tools).
- A quantized model file named `ggml-model-<quant-type>.gguf` inside your model directory (for example `models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf`).
- Per-step logs in your `--log-dir` (default `logs/`) — `download_model.log`, `convert_to_f32_gguf.log`, `quantize_to_i2s.log`, `codegen.log`, `compile.log`, and so on. These are where to look first if a step fails.

> [!TIP]
> If `setup_env.py` reports that a step failed, open the matching log file named after that step in your log directory. The error message tells you which command failed and why — this is far more informative than the one-line console summary.

> [!NOTE]
> If the target `.gguf` already exists and is non-empty, the model-preparation step is skipped and reused. To force a fresh conversion, delete the existing `ggml-model-*.gguf` first.

---

## 4. Supported Models

bitnet.cpp demonstrates inference using existing 1-bit LLMs published on Hugging Face. The table below lists the models the framework officially recognizes, with the kernel support per CPU type. ✅ = supported, ❌ = not built for that combination.

| Model | Parameters | x86 (I2_S) | x86 (TL2) | ARM (I2_S) | ARM (TL1) |
|---|---|:---:|:---:|:---:|:---:|
| [BitNet-b1.58-2B-4T](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) | 2.4B | ✅ | ✅ | ✅ | ✅ |
| [bitnet_b1_58-large](https://huggingface.co/1bitLLM/bitnet_b1_58-large) | 0.7B | ✅ | ✅ | ✅ | ✅ |
| [bitnet_b1_58-3B](https://huggingface.co/1bitLLM/bitnet_b1_58-3B) | 3.3B | ❌ | ✅ | ❌ | ✅ |
| [Llama3-8B-1.58-100B-tokens](https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens) | 8.0B | ✅ | ✅ | ✅ | ✅ |
| [Falcon3 family](https://huggingface.co/collections/tiiuae/falcon3-67605ae03578be86e4e87026) | 1B–10B | ✅ | ✅ | ✅ | ✅ |
| [Falcon-E family](https://huggingface.co/collections/tiiuae/falcon-edge-series-6804fd13344d6d8a8fa71130) | 1B–3B | ✅ | ✅ | ✅ | ✅ |

The full set of Hugging Face repositories accepted by `--hf-repo` is:

- `microsoft/BitNet-b1.58-2B-4T`
- `1bitLLM/bitnet_b1_58-large`
- `1bitLLM/bitnet_b1_58-3B`
- `HF1BitLLM/Llama3-8B-1.58-100B-tokens`
- `tiiuae/Falcon3-1B-Instruct-1.58bit`, `Falcon3-1B-1.58bit`
- `tiiuae/Falcon3-3B-Instruct-1.58bit`, `Falcon3-3B-1.58bit`
- `tiiuae/Falcon3-7B-Instruct-1.58bit`, `Falcon3-7B-1.58bit`
- `tiiuae/Falcon3-10B-Instruct-1.58bit`, `Falcon3-10B-1.58bit`
- `tiiuae/Falcon-E-1B-Instruct`, `Falcon-E-1B-Base`
- `tiiuae/Falcon-E-3B-Instruct`, `Falcon-E-3B-Base`

> [!NOTE]
> These are demonstration models, not the limit of what is possible. The framework's purpose is to show that 1-bit inference is fast and accurate; the maintainers hope it encourages more 1-bit models at larger scales.

---

## 5. Running Inference

### 5.1 Single prompt and chat mode

`run_inference.py` is the command-line inference tool. It locates the compiled `llama-cli` binary in `build/bin/` and runs your prompt through it.

```bash
# One-shot completion
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "Microsoft Corporation is an American multinational corporation and technology company headquartered in"

# Interactive chat (instruct models)
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "You are a helpful assistant" -cnv
```

#### Options

| Option | Short | Default | Description |
|---|---|---|---|
| `--model` | `-m` | `models/bitnet_b1_58-3B/ggml-model-i2_s.gguf` | Path to the `.gguf` model file. |
| `--prompt` | `-p` | *(required)* | Prompt text. In `-cnv` mode this is the system prompt. |
| `--n-predict` | `-n` | `128` | Number of tokens to generate. |
| `--threads` | `-t` | `2` | Number of CPU threads to use. |
| `--ctx-size` | `-c` | `2048` | Size of the prompt context window. |
| `--temperature` | `-temp` | `0.8` | Sampling temperature; higher = more random output. |
| `--conversation` | `-cnv` | off | Enable chat mode (for instruct models). |

> [!TIP]
> Set `--threads` to roughly the number of **physical** cores on your machine for best throughput. The default of `2` is conservative; on an 8-core desktop, `-t 8` is usually faster.

> [!NOTE]
> bitnet.cpp runs entirely on the CPU here — `run_inference.py` always passes `-ngl 0` (zero GPU layers) to the engine. GPU acceleration is a separate code path covered in [Section 8](#8-gpu-inference-experimental).

### 5.2 Running as an HTTP server

`run_inference_server.py` starts a long-running HTTP server (the llama.cpp `llama-server`) so you can send requests over the network with continuous batching enabled.

```bash
python run_inference_server.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  --host 127.0.0.1 --port 8080
```

```text
Starting server on 127.0.0.1:8080
```

#### Options

| Option | Short | Default | Description |
|---|---|---|---|
| `--model` | `-m` | `models/bitnet_b1_58-3B/ggml-model-i2_s.gguf` | Path to the `.gguf` model file. |
| `--prompt` | `-p` | — | Optional system prompt for the model. |
| `--n-predict` | `-n` | `4096` | Maximum tokens to predict per request. |
| `--threads` | `-t` | `2` | Number of CPU threads. |
| `--ctx-size` | `-c` | `2048` | Context window size. |
| `--temperature` | — | `0.8` | Sampling temperature. |
| `--host` | — | `127.0.0.1` | IP address to listen on. |
| `--port` | — | `8080` | Port to listen on. |

> [!WARNING]
> The server binds to `127.0.0.1` (localhost only) by default. Setting `--host 0.0.0.0` exposes it to your network — only do this on a trusted network, and put authentication/TLS in front of it before exposing it publicly. bitnet.cpp's server has no built-in authentication.

> [!NOTE]
> Chat mode (`-cnv`) is **not** available on the server — that flag is deliberately omitted because `llama-server` does not support it. Use the server's HTTP API (inherited from llama.cpp) for conversational requests instead.

---

## 6. Benchmarking Performance

bitnet.cpp ships an end-to-end benchmark so you can measure throughput on your own hardware.

```bash
python utils/e2e_benchmark.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 200 -p 256 -t 4
```

| Option | Short | Default | Description |
|---|---|---|---|
| `--model` | `-m` | *(required)* | Path to the model file. |
| `--n-token` | `-n` | `128` | Number of tokens to generate. |
| `--n-prompt` | `-p` | `512` | Number of prompt tokens to process. |
| `--threads` | `-t` | `2` | Number of threads. |

### 6.1 Benchmarking without a real model

If you want to measure kernel performance for a layout that no public model uses, generate a dummy model of the right shape and benchmark it:

```bash
# Generate a dummy 125M model in the TL1 layout
python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large \
  --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M

# Benchmark it: process 512 prompt tokens, generate 128
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128
```

> [!TIP]
> Use the dummy-model workflow to estimate how a hypothetical model size would perform on your machine before committing to downloading and converting real weights.

### 6.2 What to expect

Published results give a sense of the speedups over standard full-precision inference (your numbers will vary with hardware and threads):

- **ARM CPUs:** 1.37×–5.07× faster, with energy use cut by 55.4%–70.0%.
- **x86 CPUs:** 2.37×–6.17× faster, with energy use cut by 71.9%–82.2%.
- Larger models see the bigger speedups.

---

## 7. Converting Your Own Models

If you have BitNet-style weights that aren't already in GGUF form, the `utils/` directory has conversion tools. The most common path is converting a `.safetensors` checkpoint.

```bash
# 1. Download the bf16 safetensors weights
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 \
  --local-dir ./models/bitnet-b1.58-2B-4T-bf16

# 2. Convert to a GGUF model
python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16
```

### 7.1 Conversion and tuning tools at a glance

| Script | Purpose |
|---|---|
| `utils/convert-helper-bitnet.py` | One-shot helper that converts a downloaded BitNet checkpoint to GGUF. |
| `utils/convert-hf-to-gguf-bitnet.py` | Convert a Hugging Face model directory to GGUF (`--outtype f32/f16/tl1/tl2`). |
| `utils/convert-ms-to-gguf-bitnet.py` | Convert Microsoft-format checkpoints (`.pth`/`.pt`/`.bin`) to GGUF. |
| `utils/quantize_embeddings.py` | Quantize and benchmark the embedding layer across formats. |
| `utils/test_perplexity.py` | Measure model accuracy (perplexity) across datasets and quant types. |
| `utils/tune_gemm_config.py` | Auto-search the best kernel tiling parameters for your machine. |
| `utils/codegen_tl1.py`, `utils/codegen_tl2.py` | Regenerate the lookup-table kernels for a given model shape (advanced). |
| `utils/generate-dummy-bitnet-model.py` | Create a dummy model of a given size/layout for benchmarking. |

> [!NOTE]
> Most users never call these directly — `setup_env.py` invokes the right converter for you. Reach for them when you're bringing a new model or tuning performance. The complete options for each are in the Admin Guide.

### 7.2 Quantizing embeddings for extra speed

The token-embedding layer can be stored in a smaller format to save memory and gain speed. The easiest way is to pass `--quant-embd` to `setup_env.py`, which converts embeddings to **Q6_K** automatically:

```bash
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s --quant-embd
```

> [!TIP]
> Q6_K is the recommended embedding format — it preserves accuracy almost perfectly while reducing memory and improving throughput. Quantizing the embeddings all the way down to I2_S is **not** supported (it causes the model to fail), so stick with the default `--quant-embd` behavior.

---

## 8. GPU Inference (Experimental)

In addition to the primary CPU framework, bitnet.cpp includes a separate **GPU inference kernel** in the `gpu/` directory. It implements a custom W2A8 (2-bit weight × 8-bit activation) CUDA kernel for NVIDIA GPUs and is tuned for the BitNet-b1.58-2B-4T model.

> [!IMPORTANT]
> The GPU path is independent of the CPU path described above. It has its own dependencies, its own build step, and its own conversion scripts. It targets NVIDIA Ampere-class GPUs (compute capability 8.0, e.g. A100) and requires a CUDA toolchain plus PyTorch ≥ 2.2 with xformers.

### 8.1 Build and test the kernel

```bash
# Separate environment for the GPU path
conda create --name bitnet-gpu "python<3.13"
conda activate bitnet-gpu
pip install -r gpu/requirements.txt

# Compile the CUDA kernel into a shared library
cd gpu/bitnet_kernels
bash compile.sh
cd ..

# Run the kernel performance tests
python test.py
```

### 8.2 End-to-end GPU generation

```bash
# Download and convert the 2B model into the int2 + fp16 checkpoint pair
mkdir checkpoints
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 \
  --local-dir ./checkpoints/bitnet-b1.58-2B-4T-bf16
python ./convert_safetensors.py \
  --safetensors_file ./checkpoints/bitnet-b1.58-2B-4T-bf16/model.safetensors \
  --output checkpoints/model_state.pt --model_name 2B
python ./convert_checkpoint.py --input ./checkpoints/model_state.pt
rm ./checkpoints/model_state.pt

# Interactive chat on the GPU
python3 ./generate.py ./checkpoints/ --interactive --chat_format
```

`generate.py` accepts: a checkpoint directory (positional), `--interactive` for a prompt loop, `--chat_format` for chat-style turns, and `--sampling` to use top-p sampling instead of greedy decoding.

> [!NOTE]
> The GPU kernel reports performance like `Shape(2560, 2560), W2A8: 13.32us, torch BF16: 18.32us` from `test.py`. On an A100, the custom kernel is roughly 1.3×–3.6× faster than the equivalent BF16 matrix multiply, and end-to-end generation is around 3× faster than a comparable BF16 2B model.

---

## 9. Troubleshooting

### Error: `'clang' is not recognized as an internal or external command`

**Cause:** On Windows, your terminal isn't initialized with the Visual Studio tools, so CMake cannot find the Clang compiler.

**Solution:** Initialize the VS environment before building. First verify with `clang -v`. If it errors, run the appropriate command for your shell:

```bat
:: Command Prompt
"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64
```

```powershell
# PowerShell
Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"
```

**Prevention:** Always build from a Developer Command Prompt / Developer PowerShell for VS 2022 on Windows.

### Error: build fails in `log.cpp` with `std::chrono` errors

**Cause:** A known incompatibility introduced in a recent version of the bundled llama.cpp.

**Solution:** Apply the upstream fix referenced in the project FAQ (the `tinglou/llama.cpp` commit `4e3db1e3` that corrects the `std::chrono` usage in `log.cpp`).

**Prevention:** Use the commit of bitnet.cpp documented here, and avoid manually advancing the `3rdparty/llama.cpp` submodule past a tested point.

### Error: model produces garbage / random characters on x86

**Cause:** On x86 CPUs **without AVX2**, the I2_S quantization path can produce incorrect output because there is no scalar fallback for the missing instruction set (reported as community issue #547; see also #348).

**Solution:** Run on a CPU with AVX2 support, or use a different quantization type that is supported for your architecture (`tl2` on x86).

**Prevention:** Confirm your CPU advertises AVX2 (`cat /proc/cpuinfo | grep avx2` on Linux) before relying on the I2_S kernel.

### Error: `Cmake is not available. Please install CMake and try again.`

**Cause:** `setup_env.py` could not find a `cmake` executable on your `PATH`.

**Solution:** Install CMake ≥ 3.22 and ensure it is on your `PATH`. On Windows, installing the "C++ CMake Tools for Windows" Visual Studio component provides it.

**Prevention:** Verify with `cmake --version` before running `setup_env.py`.

### Error: `Arch <name> is not supported yet`

**Cause:** bitnet.cpp recognizes `x86_64` and `arm64` architectures only; the build step bails out on anything else.

**Solution:** Run on a supported architecture. The recognized aliases are `AMD64`/`x86`/`x86_64` → x86-64 and `aarch64`/`arm64`/`ARM64` → ARM.

### Error: `Pretuned kernels not found for model <name>`

**Cause:** You passed `--use-pretuned` but `preset_kernels/` has no pre-tuned kernel for that model.

**Solution:** Drop `--use-pretuned` so the kernels are generated for your model, or choose a model that has pre-tuned kernels shipped (`bitnet_b1_58-large`, `bitnet_b1_58-3B`, `Llama3-8B-1.58-100B-tokens`).

### Error: `llama-cli.exe` not produced on Windows

**Cause:** A historical issue (#461) where `LLAMA_BUILD_EXAMPLES` defaulted off, so the CLI executable wasn't built.

**Solution:** Rebuild after a clean checkout at the documented commit; verify `build/bin/Release/llama-cli.exe` exists after `setup_env.py` finishes.

### Error: server crashes after the first response (Falcon3-10B)

**Cause:** A reported instability (#532) generating with `Falcon3-10B-Instruct-1.58bit` in server mode.

**Solution:** If you hit this, try a smaller Falcon3 variant or the `BitNet-b1.58-2B-4T` model, and report your configuration upstream.

---

## 10. Frequently Asked Questions

**Q: What does "1-bit" or "1.58-bit" actually mean?**
The model's weights are ternary — each is one of −1, 0, or +1. Three such values carry log₂(3) ≈ 1.58 bits of information, hence "1.58-bit." Storing weights this way is what makes the model small and fast.

**Q: Do I need a GPU?**
No. The primary framework runs entirely on the CPU and is its main strength. A GPU path exists separately for NVIDIA Ampere cards but is experimental and tuned for one model.

**Q: Which quantization type should I pick?**
Use the default `i2_s` unless you have a reason not to — it works on both x86 and ARM. On ARM you may also use `tl1`; on x86 you may also use `tl2`. The framework rejects a type that doesn't match your CPU.

**Q: Why does the build take so long?**
Because `setup_env.py` compiles the bundled llama.cpp engine and the BitNet kernels from C++ source. This happens once per machine/model; afterward, running inference is instant.

**Q: Can I run a model I downloaded myself?**
Yes. Point `setup_env.py` at the directory with `-md`, or convert the weights with the `utils/convert-*` scripts and then run `run_inference.py` on the resulting `.gguf` file.

**Q: How do I make generation faster?**
Increase `--threads` toward your physical core count, enable embedding quantization with `--quant-embd`, and (advanced) tune the kernel block sizes with `utils/tune_gemm_config.py`.

**Q: How big a model can I run?**
bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU at 5–7 tokens per second. Your practical ceiling is system RAM — the model must fit in memory.

**Q: Is the output identical to the full-precision model?**
The framework is designed for fast and *lossless* inference of the ternary model — it faithfully runs the 1.58-bit weights. It does not magically recover precision that was removed during the model's training/quantization; accuracy is that of the underlying 1-bit model.

**Q: The server won't accept `-cnv`. Why?**
Chat mode is a CLI-only feature. `run_inference_server.py` intentionally omits it because `llama-server` doesn't support it. Use the HTTP API for chat-style interaction.

**Q: Where do I see what went wrong during a build?**
In your log directory (default `logs/`). Each build step writes its own log file named after the step (`compile.log`, `quantize_to_i2s.log`, etc.).

**Q: Does bitnet.cpp support NPUs?**
Not yet. The project notes NPU support is planned for a future release. Current targets are CPU (primary) and NVIDIA GPU (experimental).

---

## 11. Reference

### 11.1 Command summary

| Task | Command |
|---|---|
| Clone with submodules | `git clone --recursive https://github.com/microsoft/BitNet.git` |
| Build + prepare a model | `python setup_env.py -md <dir> -q i2_s` |
| Run inference | `python run_inference.py -m <model.gguf> -p "<prompt>" [-cnv]` |
| Start a server | `python run_inference_server.py -m <model.gguf> --port 8080` |
| Benchmark | `python utils/e2e_benchmark.py -m <model.gguf> -n 128 -p 512 -t 4` |
| Convert a checkpoint | `python utils/convert-helper-bitnet.py <model-dir>` |
| Measure accuracy | `python utils/test_perplexity.py -m <model.gguf>` |

### 11.2 Default file locations

| Path | Contents |
|---|---|
| `build/bin/` | Compiled engine binaries (`llama-cli`, `llama-server`, `llama-quantize`). |
| `models/` | Downloaded and converted models (default `--model-dir`). |
| `logs/` | Per-step build logs (default `--log-dir`). |
| `include/bitnet-lut-kernels.h`, `include/kernel_config.ini` | Generated/selected kernel files used by the build. |
| `preset_kernels/` | Pre-tuned kernels for specific models (used with `--use-pretuned`). |

### 11.3 Further reading

- Technical report: [1-bit AI Infra: Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144)
- [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880)
- [The Era of 1-bit LLMs](https://arxiv.org/abs/2402.17764)

---

_Generated June 2026 from microsoft/BitNet at 01eb415. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._