LLaMA Factory User Guide

LLaMA Factory is a unified framework for efficiently fine-tuning over 100 large language models. Whether you need supervised fine-tuning, preference optimization, reward modeling, or continual pretraining, LLaMA Factory provides a single CLI and web interface that handles it all — including multimodal models with image, video, and audio support.

Table of Contents

• 1. Quick Start
• 2. Installation
• 3. Configuration
• 4. Core Usage
• 5. Common Tasks
• 6. Troubleshooting
• 7. FAQ

1. Quick Start

Fine-tune a model in under five minutes. This example runs supervised fine-tuning (SFT) with LoRA on a single GPU.

Step 1 — Install:

pip install llamafactory

Step 2 — Create a YAML config (my_sft.yaml):

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
dataset: alpaca_en
finetuning_type: lora
lora_rank: 8
lora_target: all
output_dir: ./output/llama2_sft
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 3
bf16: true
logging_steps: 10
save_steps: 500
plot_loss: true
report_to: none

Step 3 — Train:

llamafactory-cli train my_sft.yaml

Step 4 — Chat with your model:

llamafactory-cli chat my_sft.yaml

Tip

The shortcut lmf is a drop-in alias for llamafactory-cli. Every command shown in this guide works with either name: lmf train my_sft.yaml is identical to llamafactory-cli train my_sft.yaml.

2. Installation

2.1 Requirements

2.2 pip Install

pip install llamafactory

For the latest development version:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .

2.3 Docker

Pre-built Dockerfiles are provided in the docker/ directory for three hardware platforms:

Build and run example (CUDA):

cd docker/docker-cuda
docker build -t llamafactory:cuda .
docker run --gpus all -it llamafactory:cuda bash

2.4 Verify Installation

llamafactory-cli env

This prints your Python version, PyTorch version, CUDA availability, and installed optional dependencies. Use this output when filing bug reports.

llamafactory-cli version

Prints the installed LLaMA Factory version.

Note

Set DISABLE_VERSION_CHECK=1 to skip the automatic version check on startup. This is useful in air-gapped environments or CI pipelines.

3. Configuration

LLaMA Factory uses YAML configuration files (or JSON) for all training, inference, and export operations. Every parameter can also be overridden on the command line.

3.1 Configuration File Basics

A config file is a flat YAML dictionary. There is no nesting — every parameter is a top-level key:

# Model
model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
trust_remote_code: false

# Data
dataset: alpaca_en
cutoff_len: 1024
max_samples: 10000

# Training
stage: sft
finetuning_type: lora
output_dir: ./output/my_experiment
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

# Logging
logging_steps: 10
save_steps: 500
plot_loss: true
report_to: none

3.2 CLI Overrides

Any YAML parameter can be overridden directly on the command line:

llamafactory-cli train my_config.yaml --learning_rate 1e-4 --num_train_epochs 5

3.3 Key Parameters Reference

For the complete parameter reference and architecture details, see the LLaMA Factory Admin Guide.

3.4 LoRA Configuration

LoRA (Low-Rank Adaptation) is the most common fine-tuning method — it trains a small number of adapter parameters while freezing the base model.

finetuning_type: lora
lora_rank: 8
lora_target: all

QLoRA adds 4-bit quantization to dramatically reduce VRAM:

finetuning_type: lora
lora_rank: 8
lora_target: all
quantization_bit: 4
quantization_method: bnb

3.5 Training Stages

3.6 V1 Training Engine

LLaMA Factory includes a new v1 training engine with a plugin-based architecture (DataEngine, ModelEngine, BaseTrainer). Enable it with an environment variable:

USE_V1=1 llamafactory-cli train my_config.yaml

The v1 engine uses YAML config files parsed with OmegaConf and offers a more modular, extensible design. It is opt-in and does not affect existing workflows unless explicitly enabled.

3.7 Alternative Model Hubs

By default, models are downloaded from Hugging Face. Two alternative hubs are supported:

# Use ModelScope (popular in China)
USE_MODELSCOPE_HUB=1 llamafactory-cli train my_config.yaml

# Use OpenMind Hub
USE_OPENMIND_HUB=1 llamafactory-cli train my_config.yaml

4. Core Usage

4.1 CLI Commands

LLaMA Factory provides a single entry point with subcommands:

4.2 Training

Single-GPU Training

llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

LLaMA Factory auto-detects available GPUs. With one GPU, it runs standard single-process training.

Multi-GPU Training

When multiple GPUs are detected, LLaMA Factory automatically launches torchrun for distributed data parallel (DDP) training. No manual configuration is needed for the common case.

To force distributed mode even on a single GPU (useful for testing):

FORCE_TORCHRUN=1 llamafactory-cli train my_config.yaml

Control GPU allocation and distributed settings with environment variables:

For elastic launch (fault-tolerant multi-node training):

DeepSpeed Integration

For large models that exceed single-GPU memory, use DeepSpeed ZeRO:

# Add to your training config
deepspeed: examples/deepspeed/ds_z2_config.json

Pre-built DeepSpeed configs are included:

Important

DeepSpeed ZeRO Stage 3 changes model loading behavior significantly. Some adapter configurations may not be compatible. Test with a small max_samples value before committing to a full training run.

4.3 Inference and Chat

CLI Chat

llamafactory-cli chat my_config.yaml

Opens an interactive terminal session. Type messages and receive model responses. The config must specify model_name_or_path and template at minimum.

To chat with a LoRA-adapted model, include the adapter path in your config:

model_name_or_path: meta-llama/Llama-2-7b-hf
adapter_name_or_path: ./output/llama2_sft
template: llama2
finetuning_type: lora

Web Chat

llamafactory-cli webchat my_config.yaml

Launches a browser-based chat interface.

Inference Backends

LLaMA Factory supports multiple inference backends for serving and chat:

Specify the backend in your inference config:

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
infer_backend: vllm

4.4 API Server

Launch an OpenAI-compatible API server:

llamafactory-cli api my_inference_config.yaml

The server exposes these endpoints:

Configure the server with environment variables:

Warning

Setting ALLOW_LOCAL_FILES=1 permits the API to read local files referenced in requests. In production, always set SAFE_MEDIA_PATH to restrict file access to a specific directory to prevent path traversal attacks.

Example client call (using curl):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

4.5 LlamaBoard (Web UI)

LlamaBoard is a Gradio-based web UI that exposes the full training and inference pipeline in a browser:

llamafactory-cli webui

This launches a local Gradio server. Configure it with environment variables:

LlamaBoard provides a point-and-click interface for:

• Selecting models and datasets
• Configuring training parameters (stage, LoRA rank, learning rate, etc.)
• Starting, monitoring, and stopping training runs
• Chatting with trained models
• Exporting and merging adapters

Tip

LlamaBoard is the fastest way to experiment with different training configurations. Once you find settings that work, export them as a YAML file for reproducible CLI-based training runs.

5. Common Tasks

5.1 Supervised Fine-Tuning with LoRA

The most common use case: teach a model to follow instructions using your own data.

Config (sft_lora.yaml):

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: sft
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: alpaca_en
cutoff_len: 1024

output_dir: ./output/llama2_lora_sft
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

logging_steps: 10
save_steps: 500
plot_loss: true
report_to: none

llamafactory-cli train sft_lora.yaml

5.2 QLoRA Fine-Tuning (Low VRAM)

Run fine-tuning on consumer GPUs (8 GB+) by combining LoRA with 4-bit quantization:

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: sft
finetuning_type: lora
lora_rank: 8
lora_target: all
quantization_bit: 4
quantization_method: bnb

dataset: alpaca_en
cutoff_len: 1024
output_dir: ./output/llama2_qlora_sft
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3
bf16: true
logging_steps: 10
save_steps: 500
report_to: none

5.3 Full Fine-Tuning

For maximum quality when you have sufficient VRAM (40 GB+ per GPU, or use DeepSpeed):

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: sft
finetuning_type: full

dataset: alpaca_en
cutoff_len: 1024
output_dir: ./output/llama2_full_sft
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-5
num_train_epochs: 3
bf16: true
deepspeed: examples/deepspeed/ds_z2_config.json
logging_steps: 10
save_steps: 500
report_to: none

5.4 DPO (Direct Preference Optimization)

Align a model using human preference data (chosen/rejected pairs). Start from an SFT checkpoint:

model_name_or_path: ./output/llama2_lora_sft
template: llama2
stage: dpo
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid

dataset: dpo_en_demo
cutoff_len: 1024
output_dir: ./output/llama2_dpo
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 5.0e-6
num_train_epochs: 1
bf16: true
logging_steps: 10
save_steps: 500
report_to: none

DPO loss variants:

5.5 KTO (Kahneman-Tversky Optimization)

KTO uses binary feedback (good/bad) instead of preference pairs, which is often easier to collect:

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: kto
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: kto_en_demo
cutoff_len: 1024
output_dir: ./output/llama2_kto
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 5.0e-6
num_train_epochs: 1
bf16: true
logging_steps: 10
save_steps: 500
report_to: none

5.6 Reward Modeling

Train a reward model for use in RLHF or best-of-n sampling:

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: rm
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: dpo_en_demo
cutoff_len: 1024
output_dir: ./output/llama2_reward
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 1.0e-5
num_train_epochs: 1
bf16: true
logging_steps: 10
save_steps: 500
report_to: none

5.7 Continual Pretraining

Extend a model's knowledge with domain-specific text:

model_name_or_path: meta-llama/Llama-2-7b-hf
template: llama2
stage: pt
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: c4_demo
cutoff_len: 1024
output_dir: ./output/llama2_pretrain
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 1
bf16: true
logging_steps: 10
save_steps: 500
report_to: none

5.8 Merging LoRA Adapters and Exporting

After training with LoRA, merge the adapter weights back into the base model for deployment:

model_name_or_path: meta-llama/Llama-2-7b-hf
adapter_name_or_path: ./output/llama2_lora_sft
template: llama2
finetuning_type: lora
export_dir: ./output/llama2_merged
export_size: 5
export_device: auto

llamafactory-cli export merge_config.yaml

Caution

Do NOT use a quantized base model when merging LoRA adapters. The merge must operate on the full-precision base model. Using a quantized model will produce corrupted outputs.

GPTQ Quantized Export

Export a merged model with GPTQ quantization for efficient inference:

model_name_or_path: meta-llama/Llama-2-7b-hf
adapter_name_or_path: ./output/llama2_lora_sft
template: llama2
finetuning_type: lora
export_dir: ./output/llama2_gptq
export_quantization_bit: 4
export_quantization_dataset: alpaca_en

5.9 Using Custom Datasets

LLaMA Factory ships with 70+ pre-configured datasets in data/dataset_info.json. To add your own, create entries in that file or a custom dataset info JSON.

Alpaca Format

For instruction-following data (SFT, pretraining):

[
  {
    "instruction": "Summarize the following text.",
    "input": "LLaMA Factory is a unified framework...",
    "output": "LLaMA Factory unifies LLM fine-tuning across 100+ models."
  }
]

ShareGPT Format

For multi-turn conversation data:

[
  {
    "conversations": [
      {"from": "human", "value": "What is LLaMA Factory?"},
      {"from": "gpt", "value": "LLaMA Factory is a unified fine-tuning framework."},
      {"from": "human", "value": "What methods does it support?"},
      {"from": "gpt", "value": "It supports LoRA, QLoRA, full fine-tuning, DPO, KTO, and more."}
    ]
  }
]

Identity Dataset

For teaching a model its own identity, use the {{name}} and {{author}} template variables in your dataset. These are replaced at data loading time.

Registering a Custom Dataset

Add an entry to data/dataset_info.json:

{
  "my_custom_dataset": {
    "file_name": "my_data.json",
    "formatting": "alpaca"
  }
}

Then reference it by name in your config:

dataset: my_custom_dataset

Multiple datasets can be combined with commas:

dataset: alpaca_en,my_custom_dataset

5.10 Multimodal Fine-Tuning

LLaMA Factory supports fine-tuning multimodal models that handle images, video, and audio. Multimodal data uses special placeholders in messages:

Supported multimodal model families include Qwen3-VL, Gemma3/4, InternVL, Llama4, LLaVA, Pixtral, and PaliGemma, among others.

Example multimodal dataset entry:

{
  "conversations": [
    {"from": "human", "value": "<image>\nDescribe what you see in this image."},
    {"from": "gpt", "value": "The image shows a cat sitting on a windowsill."}
  ],
  "images": ["path/to/cat.jpg"]
}

5.11 Experiment Tracking

LLaMA Factory integrates with multiple experiment tracking platforms:

When plot_loss: true is set, a loss curve plot is automatically saved to output_dir after training completes.

5.12 Multi-Node Distributed Training

For training across multiple machines:

Node 0 (master):

NNODES=2 NODE_RANK=0 MASTER_ADDR=192.168.1.100 MASTER_PORT=29500 \
  llamafactory-cli train my_config.yaml

Node 1:

NNODES=2 NODE_RANK=1 MASTER_ADDR=192.168.1.100 MASTER_PORT=29500 \
  llamafactory-cli train my_config.yaml

For fault-tolerant elastic training:

RDZV_ID=my_job MIN_NNODES=2 MAX_NNODES=4 MAX_RESTARTS=3 \
  llamafactory-cli train my_config.yaml

6. Troubleshooting

6.1 Common Errors

Error: RuntimeError: can't start new thread

Cause: The system has hit its thread limit, typically from DataLoader workers combined with distributed training processes.

Solution:

• Reduce dataloader_num_workers in your config
• Increase the system thread limit: ulimit -u 65536
• Check for resource leaks from previous crashed training runs

Prevention: Set dataloader_num_workers conservatively (2–4) when running distributed training.

Error: RuntimeError: Trying to backward through the graph a second time

Cause: Reported with specific combinations of gradient checkpointing (e.g., unsloth_gc) and DeepSpeed ZeRO-3 on certain model architectures (e.g., Gemma4).

Solution:

• Disable gradient checkpointing or switch to standard gradient_checkpointing: true instead of unsloth_gc
• Try DeepSpeed ZeRO Stage 2 instead of Stage 3

Prevention: When combining advanced memory optimization techniques, test with a small max_samples first.

Error: loss is 0 during SFT

Cause: A mismatch between the chat template and the model, or a data formatting issue where the model's response tokens are being masked.

Solution:

• Verify that the template parameter matches your model
• Check that your dataset responses are non-empty
• Inspect a few training examples by adding max_samples: 10 and logging the tokenized output

Error: Garbled output after SFT when serving with vLLM

Cause: Mismatched tokenizer settings or template between training and inference, particularly for long text generation.

Solution:

• Ensure the inference config uses the exact same template as training
• If you merged LoRA adapters, verify the tokenizer was copied correctly to export_dir
• Try the huggingface backend to isolate whether the issue is vLLM-specific

Error: GPU 0 barrier warning during distributed training

Cause: In multi-GPU training, GPU 0 may print warnings about processes not reaching barriers in sync. This is typically a logging artifact.

Solution:

• If training completes successfully, the warning can be safely ignored
• If training hangs, check for OOM on non-zero GPUs (which may silently fail)
• Ensure all GPUs have comparable VRAM

Error: MLflow logging keeps failing

Cause: MLflow external logger integration may fail due to connection issues or incompatible MLflow versions.

Solution:

• Verify MLflow tracking server is reachable
• Check MLflow version compatibility
• Set report_to: none to disable and isolate whether MLflow is the cause of training issues

Error: Data packing error in SFT with multimodal models

Cause: Data packing (combining multiple short examples into one sequence) can conflict with multimodal inputs that contain special image/video/audio tokens.

Solution:

• Disable data packing when fine-tuning multimodal models
• Ensure multimodal placeholder tokens (<image>, <video>, <audio>) are properly handled by your template

6.2 General Debugging Tips

1. Check your environment first: llamafactory-cli env prints all relevant version info.
2. Start small: Set max_samples: 100 to validate your config before full training.
3. Increase verbosity: Set LLAMAFACTORY_VERBOSITY=DEBUG for detailed logging.
4. Record VRAM usage: Set RECORD_VRAM=1 to track GPU memory during training.
5. API debugging: Set API_VERBOSE=1 when running the API server to log all requests and responses.

Note

When filing issues on GitHub, always include the output of llamafactory-cli env and the full YAML config (with any sensitive values redacted).

7. FAQ

Q: What models does LLaMA Factory support?

LLaMA Factory supports 100+ models from the Hugging Face ecosystem, including LLaMA 2/3/4, Qwen, ChatGLM, Gemma, Mistral, InternLM, and many more. Any model compatible with the Hugging Face Transformers library can generally be used. Set trust_remote_code: true for models that require custom code.

Q: How much VRAM do I need?

It depends on the model size and fine-tuning method:

These are approximate values. Actual usage depends on batch size, sequence length, and other parameters.

Q: Can I resume training from a checkpoint?

Yes. Point model_name_or_path at your checkpoint directory (e.g., ./output/my_experiment/checkpoint-500) and training will resume.

Q: How do I use my own data?

Prepare your data in Alpaca format (instruction/input/output fields) or ShareGPT format (conversations array). Register it in data/dataset_info.json and reference it by name in your config. See Section 5.9 for details.

Q: What is the difference between llamafactory-cli and lmf?

They are identical. lmf is a short alias for llamafactory-cli installed alongside it.

Q: Can I fine-tune with multiple datasets at once?

Yes. List them as comma-separated names in the dataset parameter:

dataset: alpaca_en,my_dataset,another_dataset

Q: How do I serve my fine-tuned model as an API?

Create an inference config pointing to your model (or merged export) and run:

llamafactory-cli api my_inference_config.yaml

This starts an OpenAI-compatible server. Any client that works with the OpenAI API (including the official Python SDK) can connect by pointing to http://localhost:8000.

Q: What is the V1 engine and should I use it?

The V1 engine (USE_V1=1) is a newer, plugin-based training architecture. It is opt-in and experimental. Unless you specifically need its plugin system or have been directed to use it, the default engine is recommended for production workloads.

Q: Can I use LLaMA Factory with AMD GPUs?

Yes. LLaMA Factory provides a ROCm Docker image in docker/docker-rocm/. Install PyTorch with ROCm support and use the standard CLI commands.

Q: How do I track training progress?

Set report_to to your preferred platform (wandb, tensorboard, swanlab, or mlflow). For a quick visual, set plot_loss: true to generate a loss curve PNG after training finishes.

Tip

For architecture details, security configuration, advanced DeepSpeed tuning, and the complete parameter reference, see the LLaMA Factory Admin Guide at Admin Guide.