Unsloth — Administrator & Developer Guide

This guide covers the architecture, configuration, deployment, security, and internals of Unsloth — a high-performance library for LLM fine-tuning with custom Triton kernels. It is written for system administrators, ML engineers, and contributors who need to understand how Unsloth works beneath the surface, how to configure and deploy it, and how to extend it. See the User Guide for getting started with basic training workflows.

1. Architecture Overview

Unsloth consists of three main components that work together to provide optimized LLM fine-tuning and local inference.

Note

Unsloth monkey-patches HuggingFace transformers classes at import time. This means import unsloth must occur before importing trl, transformers, or peft. The UNSLOTH_IS_PRESENT environment variable prevents double-patching when multiple entry points are active.

1.1 Core Library (unsloth/)

The Python package containing custom Triton kernels and model-loading logic. Key subpackages:

1.2 CLI (unsloth_cli/)

A Typer-based command-line tool exposing commands: train, inference, export, list-checkpoints, and studio (with subcommands run, stop, setup, update, reset-password). Configuration uses Pydantic models loaded from YAML or JSON files, with CLI flags as overrides.

1.3 Unsloth Studio (studio/)

A local web application for interactive fine-tuning, inference, data recipe creation, and model export.

• Backend (studio/backend/): FastAPI on uvicorn, default port 8888. Routes cover auth, training, models, inference, datasets, data-recipe, export, and training-history. Mounts OpenAI-compatible endpoints at /v1. Auth via JWT + API keys. Storage: local filesystem + SQLite (WAL mode).
• Frontend (studio/frontend/): React 19 + Vite 8 + TanStack Router + Tailwind CSS 4 + Zustand 5 state management. Features include a training studio, chat interface, export wizard, Recipe Studio (data recipes), settings, onboarding wizard, and guided tours.
• llama.cpp integration: Prebuilt binaries downloaded via install_llama_prebuilt.py (5400+ lines). Validates checksums, falls back to source build. Used for GGUF inference via llama-server subprocess.

1.4 Data Flow — Training

User → CLI/Studio UI → FastLanguageModel.from_pretrained() → HuggingFace model
     → get_peft_model() → LoRA adapters applied
     → UnslothTrainer → Custom Triton kernels (CE loss, RMSNorm, RoPE, SwiGLU, LoRA)
     → save_pretrained_merged/gguf/lora → Output (HF format, GGUF, LoRA adapter)

1.5 Data Flow — Studio

Browser → FastAPI backend → Training worker (subprocess) → Model training
                          → Inference (llama-server subprocess)
                          → Export worker (subprocess)
                          → Data Recipe engine (data-designer-engine)

1.6 Component Inventory

1.7 Technology Stack

2. System Requirements

2.1 Hardware

GPU

Memory

2.2 Software

3. Configuration Reference

Important

Environment variables are the primary configuration mechanism for Unsloth. They are read at import time (core library) or at startup (Studio). Changing them after import has no effect on the core library.

3.1 Core Library Environment Variables

3.2 Studio Environment Variables

3.3 Attention Implementation Resolution

Unsloth resolves the attention backend at model load time using this priority order:

1. flash_attention_2 (if installed and GPU supports it)
2. sdpa (PyTorch scaled dot-product attention)
3. flex_attention (if UNSLOTH_ENABLE_FLEX_ATTENTION=1, the default)
4. eager (fallback, slowest)

Override by setting attn_implementation in from_pretrained().

3.4 Quantization Modes

4. Deployment & Scaling

4.1 Installation Methods

pip (recommended for training):

pip install unsloth

Studio (local web application):

unsloth studio setup
unsloth studio run

The Studio setup command:

1. Creates a Python virtual environment using uv
2. Installs the Python dependency stack (install_python_stack.py)
3. Downloads llama.cpp prebuilt binaries (install_llama_prebuilt.py)
4. Builds the frontend (Node.js required for development builds only; production uses pre-built assets)

Editable install (development):

pip install -e .

Or for Studio development with a local repo:

STUDIO_LOCAL_REPO=/path/to/unsloth unsloth studio setup

4.2 Studio Deployment

Studio runs as a local service on port 8888. It is designed for single-user, single-machine use.

unsloth studio run          # Start Studio
unsloth studio stop         # Stop Studio
unsloth studio reset-password  # Reset admin password
unsloth studio update       # Update Studio

The backend binds to 0.0.0.0:8888 by default. For production or shared environments, place a reverse proxy (nginx, Traefik) in front to handle TLS termination and access control.

4.3 Distributed Training

Unsloth checks the WORLD_SIZE environment variable to detect distributed training setups. When WORLD_SIZE > 1, certain behaviors change:

• Gradient checkpointing offload to disk is disabled
• Certain monkey-patches may behave differently

Multi-GPU training is handled via HuggingFace Accelerate and DeepSpeed integration rather than custom Unsloth infrastructure.

4.4 Offline Deployment

For air-gapped environments:

export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1

Pre-download models and cache them locally. Point UNSLOTH_COMPILE_LOCATION to a persistent directory to cache Triton compiled kernels.

4.5 llama.cpp Binary Management

The install_llama_prebuilt.py installer (5400+ lines) handles multi-platform binary acquisition:

The installer uses a published bundle system with SHA-256 verification. If prebuilt binaries are unavailable, it falls back to source compilation. An install lock mechanism prevents concurrent installs from corrupting the binary directory.

Pin a specific release:

export UNSLOTH_LLAMA_RELEASE_TAG=b5678

5. Operations

Note

Studio uses file-based state for authentication (JSON files) and SQLite in WAL mode for training history. There is no external database dependency.

Tip

Set UNSLOTH_ENABLE_LOGGING=1 for verbose output when diagnosing training issues. For Studio, set LOG_LEVEL=DEBUG for detailed backend logs.

5.1 Studio Authentication

Default credentials on first launch:

• Username: admin
• Password: auto-generated, displayed in terminal output and injected into the HTML page

The must_change_password flag forces a password change on first login. The bootstrap password is removed from the HTML once the password is changed.

Token lifecycle:

API keys are an alternative for programmatic access. They are returned exactly once on creation, stored as SHA-256 hashes with prefix-based lookup, and cannot be retrieved after creation.

5.2 Training Runs

Training jobs run as subprocesses managed by the Studio backend. Operators can:

• Start training via POST /api/train/start
• Monitor progress via the SSE stream at GET /api/train/progress
• Stop training via POST /api/train/stop
• Reset state via POST /api/train/reset
• Review history via GET /api/train/runs

5.3 Inference Management

Inference runs via llama-server as a subprocess. The backend manages the lifecycle:

1. Load model (POST /api/inference/load)
2. Validate model can run (POST /api/inference/validate)
3. Generate responses (POST /api/inference/generate/stream)
4. Unload when done (POST /api/inference/unload)

The OpenAI-compatible endpoints at /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models allow drop-in replacement for OpenAI API clients. An Anthropic Messages API-compatible endpoint is also available at /v1/messages.

5.4 Export Operations

Export runs as a subprocess with SSE log streaming:

1. Load a checkpoint (POST /api/export/load-checkpoint)
2. Export in the desired format (POST /api/export/export/merged, /gguf, /lora, /base)
3. Stream logs via GET /api/export/logs/stream
4. Clean up memory via POST /api/export/cleanup

5.5 Data Recipes

The Recipe Studio uses data-designer-engine (>=0.5.4) for synthetic data generation:

1. Validate recipe (POST /api/data-recipe/validate)
2. Create job (POST /api/data-recipe/jobs)
3. Monitor via SSE (GET /api/data-recipe/jobs/{id}/events)
4. Retrieve results (GET /api/data-recipe/jobs/{id}/dataset)
5. Publish to HuggingFace (POST /api/data-recipe/jobs/{id}/publish)

5.6 Dataset Upload

Dataset upload supports CSV, JSON, JSONL, and Parquet formats. Files are streamed in 1 MB chunks. Format detection uses a 2-tier approach: quick detection first, then streaming analysis if needed. An LLM-assisted mapping feature (POST /api/datasets/ai-assist-mapping) uses a 3-pass process (classify, generate, validate) with a 7B GGUF model.

5.7 Health Monitoring

curl http://localhost:8888/api/health
curl http://localhost:8888/api/system
curl http://localhost:8888/api/system/hardware

The /api/system endpoint returns platform, Python version, GPU info, and memory. The /api/system/hardware endpoint returns GPU name, VRAM, and installed ML package versions.

6. Performance

6.1 Triton Kernel Optimizations

Unsloth replaces standard PyTorch operations with custom Triton kernels that are JIT-compiled with heuristic decorators for feature flags:

6.2 Gradient Checkpointing

Unsloth implements a custom gradient checkpointing mode ("unsloth") that offloads activations to system RAM during the backward pass, reducing VRAM usage at the cost of slightly increased training time. This mode is disabled when WORLD_SIZE > 1.

6.3 torch.compile Integration

6.4 Padding-Free Training

Auto padding-free training removes padding tokens from sequences, improving throughput. It is enabled by default but blocked for gemma2 and gpt_oss model architectures due to compatibility issues. Disable globally with UNSLOTH_DISABLE_AUTO_PADDING_FREE=1.

6.5 vLLM Fast Inference

When vLLM is installed, fast_generate() uses vLLM for high-throughput inference. Set UNSLOTH_VLLM_STANDBY=1 to keep vLLM loaded between generation calls.

7. Security

Warning

Studio's CORS configuration allows all origins (allow_origins=["*"]). This must be restricted in any deployment accessible beyond localhost. Place Studio behind a reverse proxy that enforces origin restrictions.

Caution

The default admin password is auto-generated and injected into the HTML page on first launch. The must_change_password flag ensures it must be changed, and the bootstrap password is removed from the HTML afterward. However, anyone with access to the terminal output or network traffic during first launch can capture the initial password.

7.1 Authentication Model

7.2 Endpoint Authentication

7.3 Path Traversal Protection

Frontend static file serving includes is_relative_to checks to prevent path traversal attacks.

7.4 Shell Injection Prevention

The save.py module invokes llama.cpp via subprocess for GGUF conversion. Shell injection prevention is implemented and validated by dedicated tests (tests/saving/test_save_shell_injection.py).

7.5 Network Exposure

For production deployments:

1. Bind to 127.0.0.1 or use a reverse proxy
2. Enable TLS at the proxy layer
3. Restrict CORS origins
4. Use API keys for programmatic access instead of sharing JWT credentials

7.6 Secret Management

• HF_TOKEN — store in environment, not in code or config files
• GH_TOKEN / GITHUB_TOKEN — used for llama.cpp binary downloads to avoid GitHub rate limits
• API keys — stored as SHA-256 hashes; raw key returned only once
• JWT signing key — generated at runtime by the auth module

8. Developer Internals

8.1 Module Reference

8.1.1 unsloth/models/_utils.py — Core Utilities

Approximately 3000 lines. The foundational module containing:

• Version checking: Extensive compatibility checks for transformers, xformers, peft, torch, and vLLM. Error messages always include fix instructions (e.g., pip install commands, links to documentation).
• torch.compile configuration: Reads UNSLOTH_COMPILE_* environment variables and configures the compilation backend.
• Model initialization safety: Patches for safe weight initialization and uninitialized weight warnings.
• BitsAndBytes 4-bit config: NF4 with double quantization setup.
• Attention implementation resolution: Priority chain: flash_attention_2 > sdpa > flex_attention > eager.
• Gradient checkpointing: Custom "unsloth" mode that offloads to disk.
• vLLM fast inference setup: Configuration and initialization.
• FP8 quantization config: Via torchao integration.
• QAT (Quantization-Aware Training): Via torchao.
• Library version: __version__ = "2026.4.6".

8.1.2 unsloth/models/loader.py — Model Loading Dispatcher

Routes to architecture-specific loaders based on the model_type field in the model config:

Handles 4-bit/8-bit/16-bit/FP8 loading, RoPE scaling configuration, trust_remote_code, vLLM integration, and ModelScope support.

8.1.3 unsloth/models/llama.py — Base Model Implementation

Approximately 3500 lines. Implements FastLlamaModel with:

• pre_patch(): Monkey-patches attention and MLP forward methods with Unsloth's optimized Triton-backed implementations.
• from_pretrained(): Model loading with quantization, dtype selection, and device mapping.
• get_peft_model(): LoRA adapter application with rank, alpha, target module configuration.
• for_inference() / for_training(): Mode switching that reconfigures the model for the appropriate use case.
• fast_generate(): vLLM-backed generation when available.
• inner_training_loop patching: Overrides the HuggingFace trainer's inner loop.

8.1.4 unsloth/models/vision.py — Vision & Universal Model Support

Implements FastBaseModel, FastModel, and FastVisionModel. Supports:

• All transformers models (universal fallback)
• Vision-language models (LLaVA, Gemma3-VL, Qwen2.5-VL)
• Whisper (audio)
• BERT, sentence transformers
• Fast inference limited to language models and Qwen2.5-VL/Gemma3 vision models

8.1.5 unsloth/save.py — Model Saving

Approximately 3200 lines. Three save methods:

GGUF conversion via llama.cpp (auto-clones and builds if not present). Supports 20+ quantization methods. Push to HuggingFace Hub. Merge validation checks for infinity values in merged weights.

8.1.6 unsloth/chat_templates.py — Chat Template Library

Contains 26+ chat templates with Jinja templates, EOS tokens, and Ollama modelfile definitions. Functions for template application, ShareGPT format conversion, and custom template construction.

8.1.7 unsloth/trainer.py — UnslothTrainer

Extends SFTTrainer from TRL. Adds:

• Q-GaLore optimizer support
• Auto padding-free training (with blocklist for gemma2 and gpt_oss)
• Per-embedding learning rate configuration
• Gradient accumulation fix

8.1.8 unsloth/kernels/ — Triton Kernel Suite

Each kernel file contains Triton-decorated functions that are JIT-compiled for the target GPU:

• cross_entropy_loss.py: Fast_CrossEntropyLoss with softcapping for Gemma 2 and logit scaling for Cohere models. Chunks vocabularies exceeding 65536 tokens (MAX_FUSED_SIZE).
• rms_layernorm.py / layernorm.py: Drop-in Triton replacements.
• rope_embedding.py: Fused RoPE computation.
• swiglu.py / geglu.py: Fused activation functions.
• fast_lora.py: Fused LoRA forward passes for qkv, output, and MLP layers.
• fp8.py: FP8 quantization operations.
• utils.py: fast_dequantize, fast_gemv, fast_linear_forward, matmul_lora.
• moe/: Grouped GEMM for Mixture-of-Experts architectures. Licensed AGPL-3.0.

8.1.9 studio/backend/auth/ — Authentication Module

• JWT token management: access (30 min) and refresh (7 day) tokens
• Password hashing via bcrypt
• API key creation, listing (masked), and revocation
• SHA-256 hashed storage with prefix-based lookup
• Default admin account with auto-generated password and must_change_password flag
• Token revocation on password change
• Storage via JSON files

8.1.10 studio/backend/routes/inference.py — Inference Router

Manages llama-server as a subprocess for inference. Provides:

• OpenAI-compatible endpoints: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models
• Anthropic Messages API compatible endpoint: /v1/messages
• OpenAI Responses API compatible endpoint: /v1/responses
• Audio generation for TTS models
• Sandbox file serving for code execution outputs
• Model loading progress tracking

8.1.11 studio/install_llama_prebuilt.py — Binary Installer

5400+ line installer with:

• Multi-platform support (Linux CUDA, macOS arm64/x64, Windows CUDA/CPU, AMD ROCm/HIP)
• Published bundle system with SHA-256 verification
• Fallback to source compilation
• Install lock mechanism to prevent concurrent installs
• Checksum validation of all downloaded binaries

8.2 Code Patterns

Monkey-Patching

Unsloth replaces HuggingFace transformers methods at import time. The UNSLOTH_IS_PRESENT environment variable is set to prevent double-patching when multiple modules attempt the same operation.

# Import order matters
import unsloth  # Must come first — patches transformers
from trl import SFTTrainer  # Now uses patched methods

Import Guards

The library performs extensive version checks at import time. If incompatible versions are detected, error messages include exact fix commands:

Unsloth requires transformers>=4.38. Please run: pip install --upgrade transformers

Error Message Convention

Every error and warning includes actionable fix instructions — a pip install command, a link to documentation, or a specific configuration change.

File-Based State (Studio)

• Authentication: JSON files for user records and API key hashes
• Training history: SQLite database in WAL mode
• No external database or cache dependencies

9. Integration Points

9.1 API Endpoint Reference

All 71 endpoints served by the Studio backend.

System Endpoints

Auth Endpoints (prefix: /api/auth)

Training Endpoints (prefix: /api/train)

Model Endpoints (prefix: /api/models)

Inference Endpoints (prefix: /api/inference and /v1)

Dataset Endpoints (prefix: /api/datasets)

Export Endpoints (prefix: /api/export)

Data Recipe Endpoints (prefix: /api/data-recipe)

9.2 OpenAI Compatibility

The /v1 endpoints provide drop-in compatibility with OpenAI client libraries:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8888/v1",
    api_key="your-api-key"  # Created via /api/auth/api-keys
)

response = client.chat.completions.create(
    model="loaded-model-name",
    messages=[{"role": "user", "content": "Hello"}]
)

9.3 Anthropic Compatibility

The /v1/messages endpoint supports Anthropic Messages API format, enabling Claude client libraries to communicate with locally-loaded models.

9.4 HuggingFace Hub Integration

• Model download: FastLanguageModel.from_pretrained() pulls from HuggingFace Hub (or ModelScope with UNSLOTH_USE_MODELSCOPE=1)
• Model upload: save_pretrained_merged() with push_to_hub=True
• Authentication: HF_TOKEN environment variable
• Offline mode: HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1

9.5 Known Issues and Limitations

9.6 Testing Strategy

Tip

Run the full test suite with pytest tests/. Shell-based tests in tests/sh/ must be run separately via bash.

9.7 Troubleshooting

Note

Unsloth's error messages are designed to be self-documenting. Most errors include the exact pip install command or configuration change needed to resolve the issue.

Tip

For kernel compilation issues, try setting UNSLOTH_COMPILE_DISABLE=1 to bypass torch.compile entirely and isolate whether the problem is in Triton JIT compilation or elsewhere.

Triton kernel compilation failures

Triton kernels are JIT-compiled on first use. If compilation fails:

1. Verify CUDA toolkit version matches PyTorch's CUDA version
2. Check that the GPU compute capability meets minimum requirements (>=7.0)
3. Set UNSLOTH_COMPILE_LOCATION to a writable directory for persistent cache
4. Set UNSLOTH_COMPILE_IGNORE_ERRORS=1 (default) to fall back gracefully

Import order errors

If patching fails or behaves unexpectedly:

# Correct order
import unsloth
from trl import SFTTrainer

# Incorrect — transformers loaded before unsloth can patch
from transformers import AutoModelForCausalLM
import unsloth  # Too late

Studio fails to start

1. Check port 8888 is not in use
2. Verify llama.cpp binaries installed: unsloth studio setup
3. Check LOG_LEVEL=DEBUG output for specific errors
4. Verify Python version is 3.11-3.13

GGUF conversion fails

1. Ensure sufficient disk space (2x model size for temp files)
2. Check llama.cpp is installed and accessible
3. Verify the model is not a vision model (GGUF not supported for vision)
4. Check for infinity values in merged weights (validation runs automatically)

4-bit model loading OOM

1. Verify load_in_4bit=True is set
2. Check CUDA_VISIBLE_DEVICES targets the correct GPU
3. Reduce max_seq_length
4. Enable gradient checkpointing with gradient_checkpointing="unsloth" for disk offload

vLLM integration issues

1. Ensure vLLM is installed and compatible with the current PyTorch version
2. Set UNSLOTH_VLLM_STANDBY=0 to disable standby mode if memory is constrained
3. Check that the model architecture supports fast inference (language models and select vision models only)