# Unsloth — Administrator & Developer Guide

This guide covers the architecture, configuration, deployment, security, and internals of Unsloth — a high-performance library for LLM fine-tuning with custom Triton kernels. It is written for system administrators, ML engineers, and contributors who need to understand how Unsloth works beneath the surface, how to configure and deploy it, and how to extend it. See the User Guide for getting started with basic training workflows.

---

## 1. Architecture Overview

Unsloth consists of three main components that work together to provide optimized LLM fine-tuning and local inference.

> [!NOTE]
> Unsloth monkey-patches HuggingFace transformers classes at import time. This means `import unsloth` must occur **before** importing `trl`, `transformers`, or `peft`. The `UNSLOTH_IS_PRESENT` environment variable prevents double-patching when multiple entry points are active.

### 1.1 Core Library (`unsloth/`)

The Python package containing custom Triton kernels and model-loading logic. Key subpackages:

| Subpackage | Responsibility |
|---|---|
| `models/` | Architecture-specific model loading and LoRA application (22 files covering Llama, Mistral, Gemma, Qwen, Phi, DeepSeek, and more) |
| `kernels/` | Custom Triton kernels for cross-entropy loss, RMSNorm, RoPE, SwiGLU, GEGLU, LoRA, and FP8 |
| `kernels/moe/` | Grouped GEMM kernels for Mixture-of-Experts architectures (Qwen3 MoE, Llama4). Licensed AGPL-3.0 |
| `optimizers/` | Q-GaLore optimizer (quantized gradient low-rank projection) |
| `dataprep/` | Raw text chunking and synthetic data generation via vLLM |
| `registry/` | Model metadata registry for DeepSeek, Gemma, Llama, Mistral, Phi, and Qwen families |
| `utils/` | Attention dispatch, sequence packing, HuggingFace Hub integration |

### 1.2 CLI (`unsloth_cli/`)

A Typer-based command-line tool exposing commands: `train`, `inference`, `export`, `list-checkpoints`, and `studio` (with subcommands `run`, `stop`, `setup`, `update`, `reset-password`). Configuration uses Pydantic models loaded from YAML or JSON files, with CLI flags as overrides.

### 1.3 Unsloth Studio (`studio/`)

A local web application for interactive fine-tuning, inference, data recipe creation, and model export.

- **Backend** (`studio/backend/`): FastAPI on uvicorn, default port 8888. Routes cover auth, training, models, inference, datasets, data-recipe, export, and training-history. Mounts OpenAI-compatible endpoints at `/v1`. Auth via JWT + API keys. Storage: local filesystem + SQLite (WAL mode).
- **Frontend** (`studio/frontend/`): React 19 + Vite 8 + TanStack Router + Tailwind CSS 4 + Zustand 5 state management. Features include a training studio, chat interface, export wizard, Recipe Studio (data recipes), settings, onboarding wizard, and guided tours.
- **llama.cpp integration**: Prebuilt binaries downloaded via `install_llama_prebuilt.py` (5400+ lines). Validates checksums, falls back to source build. Used for GGUF inference via `llama-server` subprocess.

### 1.4 Data Flow — Training

```text
User → CLI/Studio UI → FastLanguageModel.from_pretrained() → HuggingFace model
     → get_peft_model() → LoRA adapters applied
     → UnslothTrainer → Custom Triton kernels (CE loss, RMSNorm, RoPE, SwiGLU, LoRA)
     → save_pretrained_merged/gguf/lora → Output (HF format, GGUF, LoRA adapter)
```

### 1.5 Data Flow — Studio

```text
Browser → FastAPI backend → Training worker (subprocess) → Model training
                          → Inference (llama-server subprocess)
                          → Export worker (subprocess)
                          → Data Recipe engine (data-designer-engine)
```

### 1.6 Component Inventory

| Component | Path | Role | Technology |
|---|---|---|---|
| Core library | `unsloth/` | Model loading, LoRA, training optimization | Python, Triton, PyTorch |
| Models | `unsloth/models/` | Architecture-specific model patching (22 files) | Python |
| Kernels | `unsloth/kernels/` | Custom Triton kernels for CE loss, RMSNorm, RoPE, SwiGLU, GEGLU, LoRA, FP8 | Triton |
| MoE kernels | `unsloth/kernels/moe/` | Grouped GEMM for Mixture-of-Experts (Qwen3 MoE, Llama4) | Triton, AGPL-3.0 |
| Registry | `unsloth/registry/` | Model metadata registry (DeepSeek, Gemma, Llama, Mistral, Phi, Qwen families) | Python |
| Optimizers | `unsloth/optimizers/` | Q-GaLore optimizer (quantized gradient low-rank projection) | Python, BitsAndBytes |
| Data prep | `unsloth/dataprep/` | Raw text chunking, synthetic data generation via vLLM | Python |
| CLI | `unsloth_cli/` | Command-line interface | Typer, Pydantic |
| Studio backend | `studio/backend/` | FastAPI server, auth, training, inference, export | FastAPI, uvicorn, SQLite |
| Studio frontend | `studio/frontend/` | Web UI | React 19, Vite 8, TanStack Router, Tailwind 4 |
| Studio installer | `studio/install_llama_prebuilt.py` | llama.cpp binary installer (5400+ lines) | Python |
| Studio Python stack | `studio/install_python_stack.py` | Python dependency installer for Studio | Python, uv |
| Tests | `tests/` | Unit, integration, e2e tests | pytest, shell scripts |

### 1.7 Technology Stack

| Layer | Technology | Version/Notes |
|---|---|---|
| Language | Python | 3.11-3.13 |
| ML Framework | PyTorch | 2.1.0-2.10.x |
| Model Hub | HuggingFace transformers | 4.38+ (varies by model architecture) |
| LoRA | PEFT | 0.7.2+ |
| Training | TRL (SFTTrainer) | Patched at runtime |
| Quantization | BitsAndBytes | NF4 double quantization |
| Kernels | Triton | JIT compilation |
| FP8 | torchao | Optional, H100+ only |
| Fast inference | vLLM | Optional |
| GGUF inference | llama.cpp (llama-server) | Prebuilt or source-compiled |
| Web framework | FastAPI | Studio backend |
| Frontend | React 19 + Vite 8 | TypeScript |
| CSS | Tailwind CSS 4 | Studio frontend |
| State management | Zustand 5 | Studio frontend |
| Routing | TanStack Router | Studio frontend |
| UI components | Radix UI, shadcn, Lucide icons | Studio frontend |
| Data recipes | data-designer-engine | >=0.5.4 |
| Auth | JWT + API keys | bcrypt hashing |
| Database | SQLite | WAL mode |
| Package manager | uv | For Studio venv |
| Build | setuptools + wheel | `build.sh` |

---

## 2. System Requirements

### 2.1 Hardware

#### GPU

| GPU Vendor | Compute Capability | Features Available |
|---|---|---|
| NVIDIA >= 7.0 | V100 and above | Core Triton kernels, 4-bit training |
| NVIDIA >= 8.0 | A100 and above | bfloat16 support |
| NVIDIA >= 8.9 | Ada Lovelace (RTX 4090, L40S) | FP8 quantization |
| NVIDIA >= 9.0 | Hopper (H100) | FBGEMM FP8, full FP8 pipeline |
| AMD ROCm | Supported GPUs | ROCm backend via HIP |
| Intel XPU | Supported devices | XPU backend |

#### Memory

| Resource | Minimum | Recommended |
|---|---|---|
| VRAM | 8 GB (4-bit 7B models) | 24 GB+ for larger models, 80 GB for H100 FP8 |
| System RAM | 16 GB | 32 GB+ |
| Disk | Model-dependent | 2x model size temp space for GGUF conversion; Kaggle environments limited to 20 GB |

### 2.2 Software

| Dependency | Supported Versions |
|---|---|
| OS | Linux, macOS (arm64 + x64), Windows (native + WSL2) |
| Python | 3.11, 3.12, 3.13 |
| CUDA | 11.8, 12.1, 12.4, 12.6, 12.8, 13.0 |
| Node.js | >=20.19.0 or >=22.12.0 (frontend build only) |

---

## 3. Configuration Reference

> [!IMPORTANT]
> Environment variables are the primary configuration mechanism for Unsloth. They are read at import time (core library) or at startup (Studio). Changing them after import has no effect on the core library.

### 3.1 Core Library Environment Variables

| Variable | Default | Description | Source File |
|---|---|---|---|
| `UNSLOTH_USE_MODELSCOPE` | `False` | Use ModelScope hub instead of HuggingFace | `unsloth-cli.py:117` |
| `UNSLOTH_ENABLE_LOGGING` | `0` | Enable verbose Unsloth logging | `unsloth/import_fixes.py:30` |
| `UNSLOTH_SKIP_TORCHVISION_CHECK` | `0` | Skip torchvision compatibility check | `unsloth/import_fixes.py:683` |
| `UNSLOTH_STRICT_CHAT_TEMPLATE` | `0` | Enforce strict chat template validation | `unsloth/tokenizer_utils.py:942` |
| `UNSLOTH_RETURN_LOGITS` | `0` | Return logits during training | `unsloth/trainer.py:506` |
| `UNSLOTH_DISABLE_AUTO_PADDING_FREE` | unset | Disable auto padding-free training | `unsloth/trainer.py:54` |
| `UNSLOTH_COMPILE_DEBUG` | `0` | Debug torch.compile | `unsloth/models/_utils.py:1411` |
| `UNSLOTH_COMPILE_MAXIMUM` | `0` | Maximum torch.compile optimization | `unsloth/models/_utils.py:1412` |
| `UNSLOTH_COMPILE_IGNORE_ERRORS` | `1` | Ignore torch.compile errors (default on) | `unsloth/models/_utils.py:1414` |
| `UNSLOTH_COMPILE_DISABLE` | `0` | Disable torch.compile entirely | `unsloth/models/sentence_transformer.py:1427` |
| `UNSLOTH_COMPILE_LOCATION` | unset | Custom compile cache location | `unsloth/models/_utils.py:2356` |
| `TRANSFORMERS_OFFLINE` | `0` | Run transformers in offline mode | `unsloth/models/_utils.py:1580` |
| `HF_HUB_OFFLINE` | unset | Run HF Hub in offline mode | `unsloth/models/_utils.py:1585` |
| `WORLD_SIZE` | `1` | Distributed training world size | `unsloth/models/_utils.py:1789` |
| `UNSLOTH_WARN_UNINITIALIZED` | `1` | Warn on uninitialized weights | `unsloth/models/_utils.py:883` |
| `UNSLOTH_ENABLE_FLEX_ATTENTION` | `1` | Enable flex attention dispatch | `unsloth/models/_utils.py:467` |
| `UNSLOTH_RETURN_HIDDEN_STATES` | `0` | Return hidden states from model | `unsloth/models/llama.py:1509` |
| `UNSLOTH_USE_NEW_MODEL` | `0` | Use new model architecture path | `unsloth/models/llama.py:2829` |
| `UNSLOTH_ENABLE_FULL_FINETUNING` | `0` | Enable full fine-tuning mode (not LoRA) | `unsloth/models/llama.py:2860` |
| `UNSLOTH_VLLM_STANDBY` | `0` | Keep vLLM in standby mode | `unsloth/models/llama.py:2242` |
| `UNSLOTH_TILED_MLP` | unset | Enable tiled MLP optimization | `unsloth/models/loader.py:807` |
| `UNSLOTH_MODEL_NAME` | unset | Override model name detection | `unsloth/models/loader.py:1122` |
| `UNSLOTH_HAS_FBGEMM` | `0` | Enable FBGEMM FP8 quantization | `unsloth/models/loader_utils.py:117` |
| `UNSLOTH_FORCE_FLOAT32` | `0` | Force float32 computation | `unsloth/models/rl_replacements.py:671` |
| `ACCELERATE_MIXED_PRECISION` | `fp16` | Accelerate mixed precision setting | `unsloth/models/rl_replacements.py:668` |
| `UNSLOTH_BFLOAT16_MIXED_PRECISION` | `0` | Enable bfloat16 mixed precision for vision models | `unsloth/models/vision.py:309` |
| `UNSLOTH_DISABLE_STATIC_GENERATION` | `0` | Disable static generation cache | `unsloth/models/vision.py:394` |
| `UNSLOTH_FORCE_CUSTOM_DTYPE` | unset | Force custom dtype for vision models | `unsloth/models/vision.py:720` |
| `UNSLOTH_HIGH_PRECISION_LAYERNORM` | `0` | Use high precision layer norm in vision | `unsloth/models/vision.py:1064` |
| `UNSLOTH_DISABLE_FAST_GENERATION` | `0` | Disable fast generation path | `unsloth/models/vision.py:1284` |
| `HF_TOKEN` | unset | HuggingFace Hub authentication token | `studio/backend/utils/models/model_config.py:870` |
| `CUDA_VISIBLE_DEVICES` | unset | GPU device visibility | `studio/install_llama_prebuilt.py:2492` |

### 3.2 Studio Environment Variables

| Variable | Default | Description |
|---|---|---|
| `ENVIRONMENT_TYPE` | `production` | Studio environment type |
| `LOG_LEVEL` | `INFO` | Backend log level |
| `OLLAMA_MODELS` | unset | Path to Ollama model directory |
| `UNSLOTH_LLAMA_TAG` | `latest` | llama.cpp release tag |
| `UNSLOTH_LLAMA_RELEASE_TAG` | unset | Pin specific llama.cpp release |
| `UNSLOTH_VERBOSE` | `0` | Verbose installer output |
| `UV_TORCH_BACKEND` | unset | Override torch backend for uv |
| `STUDIO_PACKAGE_NAME` | `unsloth` | Package name for Studio install |
| `STUDIO_LOCAL_REPO` | unset | Local repo path for editable install |
| `UNSLOTH_PYTORCH_MIRROR` | unset | PyTorch download mirror URL |
| `UNSLOTH_STUDIO_SKIP_FLASHATTN_INSTALL` | unset | Skip Flash Attention install |
| `SKIP_STUDIO_BASE` | `0` | Skip base Studio install |
| `GH_TOKEN` / `GITHUB_TOKEN` | unset | GitHub API token for llama.cpp downloads (avoids rate limits) |
| `ROCM_PATH` | unset | AMD ROCm installation path |

### 3.3 Attention Implementation Resolution

Unsloth resolves the attention backend at model load time using this priority order:

1. `flash_attention_2` (if installed and GPU supports it)
2. `sdpa` (PyTorch scaled dot-product attention)
3. `flex_attention` (if `UNSLOTH_ENABLE_FLEX_ATTENTION=1`, the default)
4. `eager` (fallback, slowest)

Override by setting `attn_implementation` in `from_pretrained()`.

### 3.4 Quantization Modes

| Mode | How to Enable | Requirements |
|---|---|---|
| 4-bit NF4 | `load_in_4bit=True` (default) | BitsAndBytes |
| 8-bit | `load_in_8bit=True` | BitsAndBytes |
| 16-bit | `load_in_4bit=False` | Sufficient VRAM |
| FP8 | `load_in_4bit="fp8"` | torchao, compute capability >= 8.9 |
| FBGEMM FP8 | `UNSLOTH_HAS_FBGEMM=1` | compute capability >= 9.0 (H100) |
| QAT | Via torchao integration | torchao |

---

## 4. Deployment & Scaling

### 4.1 Installation Methods

**pip (recommended for training):**

```bash
pip install unsloth
```

**Studio (local web application):**

```bash
unsloth studio setup
unsloth studio run
```

The Studio setup command:
1. Creates a Python virtual environment using `uv`
2. Installs the Python dependency stack (`install_python_stack.py`)
3. Downloads llama.cpp prebuilt binaries (`install_llama_prebuilt.py`)
4. Builds the frontend (Node.js required for development builds only; production uses pre-built assets)

**Editable install (development):**

```bash
pip install -e .
```

Or for Studio development with a local repo:

```bash
STUDIO_LOCAL_REPO=/path/to/unsloth unsloth studio setup
```

### 4.2 Studio Deployment

Studio runs as a local service on port 8888. It is designed for single-user, single-machine use.

```bash
unsloth studio run          # Start Studio
unsloth studio stop         # Stop Studio
unsloth studio reset-password  # Reset admin password
unsloth studio update       # Update Studio
```

The backend binds to `0.0.0.0:8888` by default. For production or shared environments, place a reverse proxy (nginx, Traefik) in front to handle TLS termination and access control.

### 4.3 Distributed Training

Unsloth checks the `WORLD_SIZE` environment variable to detect distributed training setups. When `WORLD_SIZE > 1`, certain behaviors change:

- Gradient checkpointing offload to disk is disabled
- Certain monkey-patches may behave differently

Multi-GPU training is handled via HuggingFace Accelerate and DeepSpeed integration rather than custom Unsloth infrastructure.

### 4.4 Offline Deployment

For air-gapped environments:

```bash
export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1
```

Pre-download models and cache them locally. Point `UNSLOTH_COMPILE_LOCATION` to a persistent directory to cache Triton compiled kernels.

### 4.5 llama.cpp Binary Management

The `install_llama_prebuilt.py` installer (5400+ lines) handles multi-platform binary acquisition:

| Platform | Variant |
|---|---|
| Linux | CUDA 11.8, 12.1, 12.4, 12.6, 12.8, 13.0 |
| macOS | arm64 (Apple Silicon), x64 (Intel) |
| Windows | CUDA variants, CPU fallback |
| AMD | ROCm/HIP |

The installer uses a published bundle system with SHA-256 verification. If prebuilt binaries are unavailable, it falls back to source compilation. An install lock mechanism prevents concurrent installs from corrupting the binary directory.

Pin a specific release:

```bash
export UNSLOTH_LLAMA_RELEASE_TAG=b5678
```

---

## 5. Operations

> [!NOTE]
> Studio uses file-based state for authentication (JSON files) and SQLite in WAL mode for training history. There is no external database dependency.

> [!TIP]
> Set `UNSLOTH_ENABLE_LOGGING=1` for verbose output when diagnosing training issues. For Studio, set `LOG_LEVEL=DEBUG` for detailed backend logs.

### 5.1 Studio Authentication

Default credentials on first launch:

- **Username:** `admin`
- **Password:** auto-generated, displayed in terminal output and injected into the HTML page

The `must_change_password` flag forces a password change on first login. The bootstrap password is removed from the HTML once the password is changed.

Token lifecycle:

| Token Type | Lifetime |
|---|---|
| Access token (JWT) | 30 minutes |
| Refresh token | 7 days |

API keys are an alternative for programmatic access. They are returned exactly once on creation, stored as SHA-256 hashes with prefix-based lookup, and cannot be retrieved after creation.

### 5.2 Training Runs

Training jobs run as subprocesses managed by the Studio backend. Operators can:

- Start training via `POST /api/train/start`
- Monitor progress via the SSE stream at `GET /api/train/progress`
- Stop training via `POST /api/train/stop`
- Reset state via `POST /api/train/reset`
- Review history via `GET /api/train/runs`

### 5.3 Inference Management

Inference runs via `llama-server` as a subprocess. The backend manages the lifecycle:

1. Load model (`POST /api/inference/load`)
2. Validate model can run (`POST /api/inference/validate`)
3. Generate responses (`POST /api/inference/generate/stream`)
4. Unload when done (`POST /api/inference/unload`)

The OpenAI-compatible endpoints at `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and `/v1/models` allow drop-in replacement for OpenAI API clients. An Anthropic Messages API-compatible endpoint is also available at `/v1/messages`.

### 5.4 Export Operations

Export runs as a subprocess with SSE log streaming:

1. Load a checkpoint (`POST /api/export/load-checkpoint`)
2. Export in the desired format (`POST /api/export/export/merged`, `/gguf`, `/lora`, `/base`)
3. Stream logs via `GET /api/export/logs/stream`
4. Clean up memory via `POST /api/export/cleanup`

### 5.5 Data Recipes

The Recipe Studio uses `data-designer-engine` (>=0.5.4) for synthetic data generation:

1. Validate recipe (`POST /api/data-recipe/validate`)
2. Create job (`POST /api/data-recipe/jobs`)
3. Monitor via SSE (`GET /api/data-recipe/jobs/{id}/events`)
4. Retrieve results (`GET /api/data-recipe/jobs/{id}/dataset`)
5. Publish to HuggingFace (`POST /api/data-recipe/jobs/{id}/publish`)

### 5.6 Dataset Upload

Dataset upload supports CSV, JSON, JSONL, and Parquet formats. Files are streamed in 1 MB chunks. Format detection uses a 2-tier approach: quick detection first, then streaming analysis if needed. An LLM-assisted mapping feature (`POST /api/datasets/ai-assist-mapping`) uses a 3-pass process (classify, generate, validate) with a 7B GGUF model.

### 5.7 Health Monitoring

```bash
curl http://localhost:8888/api/health
curl http://localhost:8888/api/system
curl http://localhost:8888/api/system/hardware
```

The `/api/system` endpoint returns platform, Python version, GPU info, and memory. The `/api/system/hardware` endpoint returns GPU name, VRAM, and installed ML package versions.

---

## 6. Performance

### 6.1 Triton Kernel Optimizations

Unsloth replaces standard PyTorch operations with custom Triton kernels that are JIT-compiled with heuristic decorators for feature flags:

| Kernel | File | Optimization |
|---|---|---|
| Cross-entropy loss | `kernels/cross_entropy_loss.py` | Fused computation with softcapping (Gemma 2) and logit scaling (Cohere). Handles vocab >65536 via chunking (MAX_FUSED_SIZE = 65536) |
| RMSNorm | `kernels/rms_layernorm.py` | Triton replacement for RMSNorm |
| LayerNorm | `kernels/layernorm.py` | Triton replacement for LayerNorm |
| RoPE | `kernels/rope_embedding.py` | Triton rotary position embedding |
| SwiGLU | `kernels/swiglu.py` | Fused SwiGLU activation |
| GEGLU | `kernels/geglu.py` | Fused GEGLU activation |
| Fast LoRA | `kernels/fast_lora.py` | Fused LoRA forward passes (qkv, o, mlp_swiglu, mlp_geglu) |
| FP8 | `kernels/fp8.py` | FP8 quantization support |
| Utilities | `kernels/utils.py` | fast_dequantize, fast_gemv, fast_linear_forward, matmul_lora |
| MoE GEMM | `kernels/moe/` | Grouped GEMM for Mixture-of-Experts |

### 6.2 Gradient Checkpointing

Unsloth implements a custom gradient checkpointing mode (`"unsloth"`) that offloads activations to system RAM during the backward pass, reducing VRAM usage at the cost of slightly increased training time. This mode is disabled when `WORLD_SIZE > 1`.

### 6.3 torch.compile Integration

| Variable | Effect |
|---|---|
| `UNSLOTH_COMPILE_DISABLE=1` | Disable torch.compile entirely |
| `UNSLOTH_COMPILE_MAXIMUM=1` | Maximum optimization level |
| `UNSLOTH_COMPILE_DEBUG=1` | Debug compilation |
| `UNSLOTH_COMPILE_IGNORE_ERRORS=1` | Silently fall back on compile errors (default) |
| `UNSLOTH_COMPILE_LOCATION=/path` | Persistent compile cache |

### 6.4 Padding-Free Training

Auto padding-free training removes padding tokens from sequences, improving throughput. It is enabled by default but blocked for `gemma2` and `gpt_oss` model architectures due to compatibility issues. Disable globally with `UNSLOTH_DISABLE_AUTO_PADDING_FREE=1`.

### 6.5 vLLM Fast Inference

When vLLM is installed, `fast_generate()` uses vLLM for high-throughput inference. Set `UNSLOTH_VLLM_STANDBY=1` to keep vLLM loaded between generation calls.

---

## 7. Security

> [!WARNING]
> Studio's CORS configuration allows all origins (`allow_origins=["*"]`). This must be restricted in any deployment accessible beyond localhost. Place Studio behind a reverse proxy that enforces origin restrictions.

> [!CAUTION]
> The default admin password is auto-generated and injected into the HTML page on first launch. The `must_change_password` flag ensures it must be changed, and the bootstrap password is removed from the HTML afterward. However, anyone with access to the terminal output or network traffic during first launch can capture the initial password.

### 7.1 Authentication Model

| Mechanism | Details |
|---|---|
| JWT access tokens | 30-minute lifetime, required for most endpoints |
| JWT refresh tokens | 7-day lifetime, used to obtain new access tokens |
| Password hashing | bcrypt |
| API keys | SHA-256 hashed, prefix-based lookup, returned once on creation |
| Default credentials | Username: `admin`, password: auto-generated |

### 7.2 Endpoint Authentication

| Endpoint Category | Auth Required |
|---|---|
| `/api/health`, `/api/system`, `/api/system/hardware` | No |
| `/api/auth/status`, `/api/auth/login` | No |
| `/v1/*` (OpenAI-compatible) | JWT or API key |
| `/api/inference/sandbox/{session_id}/{filename}` | No |
| All other endpoints | JWT |

### 7.3 Path Traversal Protection

Frontend static file serving includes `is_relative_to` checks to prevent path traversal attacks.

### 7.4 Shell Injection Prevention

The `save.py` module invokes llama.cpp via `subprocess` for GGUF conversion. Shell injection prevention is implemented and validated by dedicated tests (`tests/saving/test_save_shell_injection.py`).

### 7.5 Network Exposure

| Concern | Status |
|---|---|
| Default bind address | `0.0.0.0` (all interfaces) |
| HTTPS | Not enforced (relies on reverse proxy) |
| CORS | `allow_origins=["*"]` (permissive) |

For production deployments:

1. Bind to `127.0.0.1` or use a reverse proxy
2. Enable TLS at the proxy layer
3. Restrict CORS origins
4. Use API keys for programmatic access instead of sharing JWT credentials

### 7.6 Secret Management

- `HF_TOKEN` — store in environment, not in code or config files
- `GH_TOKEN` / `GITHUB_TOKEN` — used for llama.cpp binary downloads to avoid GitHub rate limits
- API keys — stored as SHA-256 hashes; raw key returned only once
- JWT signing key — generated at runtime by the auth module

---

## 8. Developer Internals

### 8.1 Module Reference

#### 8.1.1 `unsloth/models/_utils.py` — Core Utilities

Approximately 3000 lines. The foundational module containing:

- **Version checking**: Extensive compatibility checks for transformers, xformers, peft, torch, and vLLM. Error messages always include fix instructions (e.g., `pip install` commands, links to documentation).
- **torch.compile configuration**: Reads `UNSLOTH_COMPILE_*` environment variables and configures the compilation backend.
- **Model initialization safety**: Patches for safe weight initialization and uninitialized weight warnings.
- **BitsAndBytes 4-bit config**: NF4 with double quantization setup.
- **Attention implementation resolution**: Priority chain: flash_attention_2 > sdpa > flex_attention > eager.
- **Gradient checkpointing**: Custom `"unsloth"` mode that offloads to disk.
- **vLLM fast inference setup**: Configuration and initialization.
- **FP8 quantization config**: Via torchao integration.
- **QAT (Quantization-Aware Training)**: Via torchao.
- **Library version**: `__version__ = "2026.4.6"`.

#### 8.1.2 `unsloth/models/loader.py` — Model Loading Dispatcher

Routes to architecture-specific loaders based on the `model_type` field in the model config:

| model_type | Loader Class |
|---|---|
| `llama` | `FastLlamaModel` |
| `mistral` | `FastMistralModel` |
| `gemma` | `FastGemmaModel` |
| `gemma2` | `FastGemma2Model` |
| `qwen2` | `FastQwen2Model` |
| `phi3` | `FastPhi3Model` |
| (others) | Architecture-specific FastXModel |

Handles 4-bit/8-bit/16-bit/FP8 loading, RoPE scaling configuration, `trust_remote_code`, vLLM integration, and ModelScope support.

#### 8.1.3 `unsloth/models/llama.py` — Base Model Implementation

Approximately 3500 lines. Implements `FastLlamaModel` with:

- **`pre_patch()`**: Monkey-patches attention and MLP forward methods with Unsloth's optimized Triton-backed implementations.
- **`from_pretrained()`**: Model loading with quantization, dtype selection, and device mapping.
- **`get_peft_model()`**: LoRA adapter application with rank, alpha, target module configuration.
- **`for_inference()` / `for_training()`**: Mode switching that reconfigures the model for the appropriate use case.
- **`fast_generate()`**: vLLM-backed generation when available.
- **`inner_training_loop` patching**: Overrides the HuggingFace trainer's inner loop.

#### 8.1.4 `unsloth/models/vision.py` — Vision & Universal Model Support

Implements `FastBaseModel`, `FastModel`, and `FastVisionModel`. Supports:

- All transformers models (universal fallback)
- Vision-language models (LLaVA, Gemma3-VL, Qwen2.5-VL)
- Whisper (audio)
- BERT, sentence transformers
- Fast inference limited to language models and Qwen2.5-VL/Gemma3 vision models

#### 8.1.5 `unsloth/save.py` — Model Saving

Approximately 3200 lines. Three save methods:

| Method | Description |
|---|---|
| `lora` | Save LoRA adapter weights only |
| `merged_16bit` | Merge LoRA into base model at 16-bit precision |
| `merged_4bit` | Merge and force 4-bit quantization |

GGUF conversion via llama.cpp (auto-clones and builds if not present). Supports 20+ quantization methods. Push to HuggingFace Hub. Merge validation checks for infinity values in merged weights.

#### 8.1.6 `unsloth/chat_templates.py` — Chat Template Library

Contains 26+ chat templates with Jinja templates, EOS tokens, and Ollama modelfile definitions. Functions for template application, ShareGPT format conversion, and custom template construction.

#### 8.1.7 `unsloth/trainer.py` — UnslothTrainer

Extends `SFTTrainer` from TRL. Adds:

- Q-GaLore optimizer support
- Auto padding-free training (with blocklist for `gemma2` and `gpt_oss`)
- Per-embedding learning rate configuration
- Gradient accumulation fix

#### 8.1.8 `unsloth/kernels/` — Triton Kernel Suite

Each kernel file contains Triton-decorated functions that are JIT-compiled for the target GPU:

- **`cross_entropy_loss.py`**: `Fast_CrossEntropyLoss` with softcapping for Gemma 2 and logit scaling for Cohere models. Chunks vocabularies exceeding 65536 tokens (MAX_FUSED_SIZE).
- **`rms_layernorm.py` / `layernorm.py`**: Drop-in Triton replacements.
- **`rope_embedding.py`**: Fused RoPE computation.
- **`swiglu.py` / `geglu.py`**: Fused activation functions.
- **`fast_lora.py`**: Fused LoRA forward passes for qkv, output, and MLP layers.
- **`fp8.py`**: FP8 quantization operations.
- **`utils.py`**: `fast_dequantize`, `fast_gemv`, `fast_linear_forward`, `matmul_lora`.
- **`moe/`**: Grouped GEMM for Mixture-of-Experts architectures. Licensed AGPL-3.0.

#### 8.1.9 `studio/backend/auth/` — Authentication Module

- JWT token management: access (30 min) and refresh (7 day) tokens
- Password hashing via bcrypt
- API key creation, listing (masked), and revocation
- SHA-256 hashed storage with prefix-based lookup
- Default admin account with auto-generated password and `must_change_password` flag
- Token revocation on password change
- Storage via JSON files

#### 8.1.10 `studio/backend/routes/inference.py` — Inference Router

Manages `llama-server` as a subprocess for inference. Provides:

- OpenAI-compatible endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`
- Anthropic Messages API compatible endpoint: `/v1/messages`
- OpenAI Responses API compatible endpoint: `/v1/responses`
- Audio generation for TTS models
- Sandbox file serving for code execution outputs
- Model loading progress tracking

#### 8.1.11 `studio/install_llama_prebuilt.py` — Binary Installer

5400+ line installer with:

- Multi-platform support (Linux CUDA, macOS arm64/x64, Windows CUDA/CPU, AMD ROCm/HIP)
- Published bundle system with SHA-256 verification
- Fallback to source compilation
- Install lock mechanism to prevent concurrent installs
- Checksum validation of all downloaded binaries

### 8.2 Code Patterns

#### Monkey-Patching

Unsloth replaces HuggingFace transformers methods at import time. The `UNSLOTH_IS_PRESENT` environment variable is set to prevent double-patching when multiple modules attempt the same operation.

```python
# Import order matters
import unsloth  # Must come first — patches transformers
from trl import SFTTrainer  # Now uses patched methods
```

#### Import Guards

The library performs extensive version checks at import time. If incompatible versions are detected, error messages include exact fix commands:

```text
Unsloth requires transformers>=4.38. Please run: pip install --upgrade transformers
```

#### Error Message Convention

Every error and warning includes actionable fix instructions — a `pip install` command, a link to documentation, or a specific configuration change.

#### File-Based State (Studio)

- Authentication: JSON files for user records and API key hashes
- Training history: SQLite database in WAL mode
- No external database or cache dependencies

---

## 9. Integration Points

### 9.1 API Endpoint Reference

All 71 endpoints served by the Studio backend.

#### System Endpoints

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/api/health` | No | Health check |
| POST | `/api/shutdown` | JWT | Gracefully shut down server |
| GET | `/api/system` | No | System info (platform, Python, GPU, memory) |
| GET | `/api/system/gpu-visibility` | JWT | GPU device visibility |
| GET | `/api/system/hardware` | No | GPU name, VRAM, ML package versions |

#### Auth Endpoints (prefix: `/api/auth`)

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/status` | No | Auth status, password change required flag |
| POST | `/login` | No | Username/password login, returns JWT tokens |
| POST | `/refresh` | Refresh token | Exchange refresh token for new access token (7-day) |
| POST | `/change-password` | JWT | Change password, revokes old tokens |
| POST | `/api-keys` | JWT | Create API key (returned once, never retrievable) |
| GET | `/api-keys` | JWT | List API keys (keys masked) |
| DELETE | `/api-keys/{key_id}` | JWT | Revoke API key |

#### Training Endpoints (prefix: `/api/train`)

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/hardware` | JWT | Hardware utilization |
| GET | `/hardware/visible` | JWT | Visible hardware utilization |
| POST | `/start` | JWT | Start training job |
| POST | `/stop` | JWT | Stop training |
| POST | `/reset` | JWT | Reset training state |
| GET | `/status` | JWT | Training status |
| GET | `/metrics` | JWT | Training metrics |
| GET | `/progress` | JWT | SSE stream of training progress |
| GET | `/runs` | JWT | List training run history |
| GET | `/runs/{run_id}` | JWT | Training run detail |
| DELETE | `/runs/{run_id}` | JWT | Delete training run |

#### Model Endpoints (prefix: `/api/models`)

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/local` | JWT | List locally cached models |
| GET | `/scan-folders` | JWT | Get model scan folders |
| POST | `/scan-folders` | JWT | Add scan folder |
| DELETE | `/scan-folders/{folder_id}` | JWT | Remove scan folder |
| GET | `/recommended-folders` | JWT | Get recommended model folders |
| GET | `/browse-folders` | JWT | Browse filesystem folders |
| GET | `/list` | JWT | List available models |
| GET | `/config/{model_name}` | JWT | Get model configuration |
| GET | `/loras` | JWT | Scan for LoRA adapters |
| GET | `/loras/{lora_path}/base-model` | JWT | Get base model for a LoRA adapter |
| GET | `/check-vision/{model_name}` | JWT | Check if model is a vision model |
| GET | `/check-embedding/{model_name}` | JWT | Check if model is an embedding model |
| GET | `/gguf-variants` | JWT | Get GGUF quantization variants |
| GET | `/gguf-download-progress` | JWT | GGUF download progress |
| GET | `/download-progress` | JWT | Model download progress |
| GET | `/cached-gguf` | JWT | List cached GGUF models |
| GET | `/cached-models` | JWT | List all cached models |
| DELETE | `/delete-cached` | JWT | Delete cached model |
| GET | `/checkpoints` | JWT | List training checkpoints |

#### Inference Endpoints (prefix: `/api/inference` and `/v1`)

| Method | Path | Auth | Description |
|---|---|---|---|
| POST | `/api/inference/load` | JWT | Load model for inference |
| POST | `/api/inference/validate` | JWT | Validate model can be loaded |
| POST | `/api/inference/unload` | JWT | Unload current model |
| POST | `/api/inference/generate/stream` | JWT | Stream generation |
| GET | `/api/inference/status` | JWT | Inference status |
| GET | `/api/inference/load-progress` | JWT | Model loading progress |
| POST | `/api/inference/audio/generate` | JWT | Generate audio (TTS models) |
| POST | `/v1/chat/completions` | JWT/API-Key | OpenAI-compatible chat completions |
| GET | `/v1/models` | JWT/API-Key | OpenAI-compatible model list |
| POST | `/v1/completions` | JWT/API-Key | OpenAI-compatible completions |
| POST | `/v1/embeddings` | JWT/API-Key | OpenAI-compatible embeddings |
| POST | `/v1/responses` | JWT/API-Key | OpenAI Responses API compatible |
| POST | `/v1/messages` | JWT/API-Key | Anthropic Messages API compatible |
| GET | `/v1/sandbox/{session_id}/{filename}` | No | Serve sandbox file |

#### Dataset Endpoints (prefix: `/api/datasets`)

| Method | Path | Auth | Description |
|---|---|---|---|
| POST | `/upload` | JWT | Upload dataset (csv/json/jsonl/parquet, max streamed 1 MB chunks) |
| GET | `/local` | JWT | List local datasets |
| GET | `/download-progress` | JWT | Dataset download progress |
| POST | `/check-format` | JWT | Detect dataset format (2-tier: quick then streaming) |
| POST | `/ai-assist-mapping` | JWT | LLM-assisted dataset mapping (3-pass: classify, generate, validate using 7B GGUF) |

#### Export Endpoints (prefix: `/api/export`)

| Method | Path | Auth | Description |
|---|---|---|---|
| POST | `/load-checkpoint` | JWT | Load checkpoint for export |
| POST | `/cleanup` | JWT | Clean up export memory |
| GET | `/status` | JWT | Export status |
| POST | `/export/merged` | JWT | Export merged model |
| POST | `/export/base` | JWT | Export base model |
| POST | `/export/gguf` | JWT | Export GGUF |
| POST | `/export/lora` | JWT | Export LoRA adapter |
| GET | `/logs/stream` | JWT | SSE stream export logs |

#### Data Recipe Endpoints (prefix: `/api/data-recipe`)

| Method | Path | Auth | Description |
|---|---|---|---|
| POST | `/validate` | JWT | Validate recipe |
| POST | `/jobs` | JWT | Create recipe job |
| GET | `/jobs/{id}/status` | JWT | Job status |
| GET | `/jobs/{id}/analysis` | JWT | Job analysis |
| GET | `/jobs/{id}/dataset` | JWT | Job dataset results |
| POST | `/jobs/{id}/cancel` | JWT | Cancel job |
| POST | `/jobs/{id}/publish` | JWT | Publish to HuggingFace |
| GET | `/jobs/{id}/events` | JWT | SSE event stream |
| POST | `/seed/inspect` | JWT | Inspect seed dataset |
| POST | `/seed/inspect-upload` | JWT | Inspect uploaded seed |
| POST | `/seed/upload-unstructured-file` | JWT | Upload unstructured file |
| DELETE | `/seed/unstructured-file/{block_id}/{file_id}` | JWT | Remove uploaded file |
| POST | `/mcp/tools` | JWT | List MCP tools |

### 9.2 OpenAI Compatibility

The `/v1` endpoints provide drop-in compatibility with OpenAI client libraries:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8888/v1",
    api_key="your-api-key"  # Created via /api/auth/api-keys
)

response = client.chat.completions.create(
    model="loaded-model-name",
    messages=[{"role": "user", "content": "Hello"}]
)
```

### 9.3 Anthropic Compatibility

The `/v1/messages` endpoint supports Anthropic Messages API format, enabling Claude client libraries to communicate with locally-loaded models.

### 9.4 HuggingFace Hub Integration

- Model download: `FastLanguageModel.from_pretrained()` pulls from HuggingFace Hub (or ModelScope with `UNSLOTH_USE_MODELSCOPE=1`)
- Model upload: `save_pretrained_merged()` with `push_to_hub=True`
- Authentication: `HF_TOKEN` environment variable
- Offline mode: `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`

### 9.5 Known Issues and Limitations

| Issue | Status | Detail |
|---|---|---|
| MoE TMA store disabled | Active | Non-determinism from Triton pipelining bug |
| Hopper warp specialization | Active | Not enabled on Triton 3.3x with PyTorch 2.7 |
| Llama4 reference | Active | Under-optimized routing and shared expert calculation |
| GGUF not supported for vision models | Active | Language models only |
| iq2 quantization types | Not supported | Excluded from GGUF quantization options |
| Auto padding-free blocked for gemma2/gpt_oss | By design | Compatibility issues |
| Studio model management (#5098) | Feature request | Delete/update models from UI |
| Gemma-4 checkpoint resume (#5082) | Bug | Cannot resume from checkpoint |
| Gemma 4-26B-A4B-it vision LoRA (#5039) | Bug | Zero gradients on vision LoRA |

### 9.6 Testing Strategy

> [!TIP]
> Run the full test suite with `pytest tests/`. Shell-based tests in `tests/sh/` must be run separately via bash.

| Test Category | Location | Coverage |
|---|---|---|
| QLoRA training | `tests/` | HF vs Unsloth training comparison + merge validation |
| Saving | `tests/saving/` | Merge perplexity validation, push to hub, vision, TTS, non-PEFT, shell injection |
| Studio installer | `tests/` | llama prebuilt logic, ROCm, selection logic, PR bugfix regression |
| Python stack | `tests/` | no-torch filtering, cross-platform parity, flash attention wheel selection, import chain |
| Utilities | `tests/` | Attention masks, packing, Q-GaLore, QAT, trunc normal patching |
| Backend routes | `tests/` | API routes, Anthropic messages, GPU selection, KV cache estimation, VRAM estimation, llama.cpp context fit |
| Shell tests | `tests/sh/` | torch index URL, Mac Intel compat, torch constraint |

### 9.7 Troubleshooting

> [!NOTE]
> Unsloth's error messages are designed to be self-documenting. Most errors include the exact `pip install` command or configuration change needed to resolve the issue.

> [!TIP]
> For kernel compilation issues, try setting `UNSLOTH_COMPILE_DISABLE=1` to bypass torch.compile entirely and isolate whether the problem is in Triton JIT compilation or elsewhere.

**Triton kernel compilation failures**

Triton kernels are JIT-compiled on first use. If compilation fails:

1. Verify CUDA toolkit version matches PyTorch's CUDA version
2. Check that the GPU compute capability meets minimum requirements (>=7.0)
3. Set `UNSLOTH_COMPILE_LOCATION` to a writable directory for persistent cache
4. Set `UNSLOTH_COMPILE_IGNORE_ERRORS=1` (default) to fall back gracefully

**Import order errors**

If patching fails or behaves unexpectedly:

```python
# Correct order
import unsloth
from trl import SFTTrainer

# Incorrect — transformers loaded before unsloth can patch
from transformers import AutoModelForCausalLM
import unsloth  # Too late
```

**Studio fails to start**

1. Check port 8888 is not in use
2. Verify llama.cpp binaries installed: `unsloth studio setup`
3. Check `LOG_LEVEL=DEBUG` output for specific errors
4. Verify Python version is 3.11-3.13

**GGUF conversion fails**

1. Ensure sufficient disk space (2x model size for temp files)
2. Check llama.cpp is installed and accessible
3. Verify the model is not a vision model (GGUF not supported for vision)
4. Check for infinity values in merged weights (validation runs automatically)

**4-bit model loading OOM**

1. Verify `load_in_4bit=True` is set
2. Check `CUDA_VISIBLE_DEVICES` targets the correct GPU
3. Reduce `max_seq_length`
4. Enable gradient checkpointing with `gradient_checkpointing="unsloth"` for disk offload

**vLLM integration issues**

1. Ensure vLLM is installed and compatible with the current PyTorch version
2. Set `UNSLOTH_VLLM_STANDBY=0` to disable standby mode if memory is constrained
3. Check that the model architecture supports fast inference (language models and select vision models only)

---

_Generated May 2026 from unslothai/unsloth at d1f9ab6. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._