Stable Diffusion — User Guide

1. Quick Start

Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images from text prompts. It operates in the latent space of a pretrained autoencoder, making high-resolution image synthesis computationally efficient compared to pixel-space diffusion models.

Key features:

• Text-to-image generation from natural language prompts
• Image-to-image translation (style transfer, guided editing)
• Image inpainting with mask-based conditioning
• Unconditional image generation (faces, bedrooms, churches)
• Class-conditional ImageNet synthesis
• Super-resolution via latent diffusion
• Training and fine-tuning with PyTorch Lightning
• Multiple sampling methods: DDIM, PLMS, DPM-Solver

Prerequisites: Python 3.8+, NVIDIA GPU with CUDA 11.3+, ~10 GB VRAM for inference, ~24 GB VRAM for training

Minimal install:

conda env create -f environment.yaml
conda activate ldm

Generate your first image:

python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" \
    --plms --outdir outputs/txt2img-samples

Tip

Use --plms for faster, higher-quality sampling compared to the default DDIM sampler. For the quickest results with acceptable quality, try --ddim_steps 50.

What to try next:

• Image-to-image with scripts/img2img.py
• Inpainting with scripts/inpaint.py
• Training your own model with main.py

2. Installation

System Requirements

Installation via Conda (Recommended)

git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion
conda env create -f environment.yaml
conda activate ldm

Important

The environment.yaml installs two editable packages from Git: taming-transformers (from CompVis) and CLIP (from OpenAI). Both are cloned and installed in development mode (-e) during environment creation. These are required dependencies — the build will fail without network access to GitHub.

Core Dependencies

Downloading Model Weights

Pre-trained model checkpoints are not included in the repository. Download the Stable Diffusion v1 weights and place them in the project directory:

# Example: place the checkpoint at the project root
# The inference scripts expect a --ckpt flag pointing to the .ckpt file
ls -lh sd-v1-4.ckpt

-rw-r--r-- 1 user user 4.0G Oct  1 12:00 sd-v1-4.ckpt

Warning

Model checkpoints are typically 2-7 GB each. Ensure sufficient disk space and a stable network connection before downloading. The model card is available at Stable_Diffusion_v1_Model_Card.md in the repository root.

Verify Installation

conda activate ldm
python -c "import ldm; print('LDM package loaded successfully')"

LDM package loaded successfully

Installing the Package in Development Mode

The repository includes a setup.py that registers the latent-diffusion package (version 0.0.1):

pip install -e .

This makes the ldm package importable from anywhere in the conda environment. The minimal install requires only torch, numpy, and tqdm.

3. Configuration

Stable Diffusion uses YAML configuration files managed by OmegaConf. Configs define the model architecture, training parameters, data pipeline, and logging.

Configuration Structure

Every config file follows this top-level structure:

model:
  base_learning_rate: 1.0e-04
  target: ldm.models.diffusion.ddpm.LatentDiffusion
  params:
    # Diffusion parameters
    # UNet config
    # First-stage (autoencoder) config
    # Conditioning-stage config

data:
  target: main.DataModuleFromConfig
  params:
    batch_size: 4
    # Dataset configs

lightning:
  trainer:
    # PyTorch Lightning Trainer args
  callbacks:
    # Callback configs

Pre-built Configurations

Key Model Parameters (Stable Diffusion v1)

From configs/stable-diffusion/v1-inference.yaml:

Note

The image_size parameter refers to the latent resolution, not the output pixel resolution. With the default f8 autoencoder (8x downsampling), a latent size of 64 corresponds to 512x512 pixel output.

Overriding Config Values from CLI

Config parameters can be overridden from the command line using dot notation:

python main.py -b configs/stable-diffusion/v1-inference.yaml \
    model.base_learning_rate=2.0e-05 \
    data.params.batch_size=2

See the Admin Guide for the complete configuration reference.

4. Core Usage

Text-to-Image Generation

Generate images from text prompts using scripts/txt2img.py:

python scripts/txt2img.py \
    --prompt "a painting of a sunset over mountains, oil on canvas" \
    --outdir outputs/txt2img-samples \
    --plms \
    --ckpt sd-v1-4.ckpt

Key options for text-to-image generation include prompt text, output directory, sampler selection (PLMS or DDIM), step count, guidance scale, image dimensions, batch size, and random seed for reproducibility.

Tip

Higher guidance scale values (e.g., 7.5-15.0) produce images that more closely match the prompt but with less diversity. Lower values (e.g., 1.0-5.0) produce more varied but less prompt-faithful results.

Image-to-Image Translation

Transform existing images with text guidance using scripts/img2img.py:

python scripts/img2img.py \
    --prompt "A fantasy landscape, trending on artstation" \
    --init-img assets/stable-samples/img2img/sketch-mountains-input.jpg \
    --outdir outputs/img2img-samples \
    --ckpt sd-v1-4.ckpt

The strength parameter controls how much the output deviates from the input image — higher values produce more dramatic transformations while lower values preserve more of the original structure.

Inpainting

Fill masked regions of an image using scripts/inpaint.py:

python scripts/inpaint.py \
    --indir data/inpainting_examples \
    --outdir outputs/inpainting-samples \
    --ckpt sd-v1-4.ckpt

The inpainting script expects pairs of files in the input directory: original images and corresponding mask images (with _mask suffix). White regions in the mask indicate areas to be inpainted. Example pairs are provided in data/inpainting_examples/.

Important

Inpainting masks must be the same dimensions as the source image. White pixels (255) mark regions to regenerate; black pixels (0) mark regions to preserve.

Unconditional Sampling

Generate samples from unconditional models (faces, bedrooms, churches):

python scripts/sample_diffusion.py \
    -r models/ldm/lsun_churches256 \
    -n 10 \
    --batch_size 5

Retrieval-Augmented Diffusion

Generate high-resolution (768x768) images using retrieval-augmented diffusion with the config at configs/retrieval-augmented-diffusion/768x768.yaml. This model conditions on retrieved nearest-neighbor images using scripts/knn2img.py.

Training

Train or fine-tune models using main.py with a YAML config:

python main.py \
    --base configs/latent-diffusion/lsun_churches-ldm-kl-8.yaml \
    -t --gpus 0,1 \
    -n my_experiment

Warning

Training requires significant GPU resources. The default configuration uses DDP (Distributed Data Parallel) when GPUs are specified. Training on a single consumer GPU may require reducing batch size and enabling gradient accumulation.

Training CLI Reference (main.py)

Note

In addition to these flags, main.py accepts all PyTorch Lightning Trainer arguments (e.g., --gpus, --max_epochs, --precision). Unknown --key value arguments are parsed as CLI config parameters; nested.key=value arguments override config file values.

5. Common Tasks

Generating Multiple Images with Different Seeds

for seed in 42 123 456 789; do
    python scripts/txt2img.py \
        --prompt "a cyberpunk cityscape at night" \
        --seed $seed \
        --outdir outputs/seed-comparison \
        --ckpt sd-v1-4.ckpt \
        --plms
done

Resuming Training from a Checkpoint

python main.py \
    --resume logs/2024-01-15T12-30-00_my_experiment \
    -t --gpus 0,

The --resume flag accepts either a log directory path or a direct checkpoint file. When given a directory, it automatically loads checkpoints/last.ckpt and merges the saved config files from the configs/ subdirectory.

Note

You cannot combine --name (-n) and --resume (-r) simultaneously. To resume training with a new experiment name, use --resume_from_checkpoint instead.

Running on CPU (No GPU)

Omit the --gpus flag when calling main.py — the trainer automatically falls back to CPU mode when no GPUs are configured:

python main.py --base configs/latent-diffusion/celebahq-ldm-vq-4.yaml -t

Adjusting Learning Rate Scaling

By default, the effective learning rate is automatically scaled:

effective_lr = accumulate_grad_batches * num_gpus * batch_size * base_learning_rate

Disable this with --scale_lr False to use the raw base_learning_rate from the config.

Using Debug Mode

Enable post-mortem debugging for development:

python main.py --base configs/stable-diffusion/v1-inference.yaml -t -d

Debug mode enables pudb (or pdb as fallback) post-mortem debugging on exceptions and moves the log directory to debug_runs/ after completion.

Saving Checkpoints via Signal

During training, send SIGUSR1 to trigger an immediate checkpoint save:

kill -USR1 <pid>

This saves last.ckpt in the checkpoint directory without interrupting training. Send SIGUSR2 to drop into an interactive pudb debugger.

Using the Jupyter Notebook for Super-Resolution

The repository includes scripts/latent_imagenet_diffusion.ipynb and notebook_helpers.py for interactive super-resolution experiments in Google Colab:

from notebook_helpers import get_model, run

model = get_model("superresolution")
logs = run(model, selected_path, task="superresolution", custom_steps=100)

Note

The notebook helper download_models() currently only supports "superresolution" mode. Other modes raise NotImplementedError.

Training Autoencoders

Train first-stage autoencoder models (VQ-VAE or KL-VAE) using the configs in configs/autoencoder/:

python main.py \
    --base configs/autoencoder/autoencoder_kl_32x32x4.yaml \
    -t --gpus 0,

Available autoencoder configurations:

6. Troubleshooting

Error: ModuleNotFoundError: No module named 'ldm.util'; 'ldm' is not a package

Cause: A file or directory named ldm in the current working directory shadows the ldm package. This commonly happens when running scripts from a directory that contains an ldm file that isn't the package, or when the package isn't installed.

Solution:

1. Ensure you are in the repository root directory
2. Install the package in development mode:

   pip install -e .
   

3. Verify no conflicting ldm files exist in your working directory

Prevention: Always run scripts from the repository root and install the package via pip install -e ..

Related: GitHub issues #818, #865

Error: CUDA driver initialization failed, you might not have a CUDA gpu

Cause: CUDA toolkit version mismatch, missing NVIDIA drivers, or no compatible GPU available.

Solution:

1. Verify GPU visibility:

   nvidia-smi
   

2. Check CUDA toolkit version matches PyTorch build:

   python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
   

3. Reinstall PyTorch with the correct CUDA version:

   conda install pytorch=1.11.0 torchvision=0.12.0 cudatoolkit=11.3 -c pytorch
   

Prevention: Always create the environment using conda env create -f environment.yaml to ensure version consistency.

Related: GitHub issue #94

Error: ImportError: cannot import name 'VectorQuantizer2'

Cause: The taming-transformers package is not installed or an incompatible version is present.

Solution:

pip install -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers

Prevention: Use conda env create -f environment.yaml for a clean setup — it includes the correct taming-transformers version as an editable install.

Related: GitHub issue #72

Error: AttributeError: module 'ldm.modules.encoders.modules' has no attribute 'FrozenCLIPEmbedder'

Cause: The CLIP package or transformers library version is incompatible, or the ldm package import path is broken.

Solution:

1. Reinstall CLIP:

   pip install -e git+https://github.com/openai/CLIP.git@main#egg=clip
   

2. Verify transformers version:

   pip install transformers==4.19.2
   

Prevention: Do not upgrade transformers beyond 4.19.2 within this environment.

Related: GitHub issue #190

Caution

Upgrading transformers to a newer version may break compatibility with the CLIP encoder. Pin the version at 4.19.2 as specified in environment.yaml.

Error: OSError: Error no file named model_index.json found in directory

Cause: Attempting to load a checkpoint using the diffusers library format when the model is in the original CompVis .ckpt format, or vice versa.

Solution: Use the correct loading method for the checkpoint format:

• For .ckpt files: Use --ckpt path/to/model.ckpt with the provided scripts
• For diffusers format: Use the diffusers library directly

Related: GitHub issue #394

Error: KeyError: "filename 'storages' not found"

Cause: Corrupted or incompletely downloaded checkpoint file.

Solution:

1. Delete the corrupted checkpoint
2. Re-download the complete checkpoint file
3. Verify file integrity (check file size matches expected values)

Related: GitHub issue #395

Warning

Partially downloaded checkpoint files will cause cryptic PyTorch deserialization errors. Always verify the file size after download — Stable Diffusion v1 checkpoints are approximately 4 GB.

Error: RuntimeError: Sizes of tensors must match except in dimension 1

Cause: Input image dimensions are not divisible by 64, or a mismatch between the model config and checkpoint.

Solution:

1. Ensure input images have dimensions divisible by 64
2. Verify the config file matches the checkpoint being loaded
3. Check that image_size and channels in the config match the checkpoint's expected values

Related: GitHub issue #841

Wrong GPU Selected / Out of Memory

Cause: PyTorch defaults to GPU 0. On multi-GPU systems, GPU 0 may already be in use.

Solution:

CUDA_VISIBLE_DEVICES=1 python scripts/txt2img.py --prompt "test" --ckpt sd-v1-4.ckpt

Or specify GPUs in the training command:

python main.py --base config.yaml -t --gpus 1,2

Related: GitHub issue #349

Tip

Monitor GPU memory usage with nvidia-smi -l 1 in a separate terminal during generation to identify memory bottlenecks.

Error: RuntimeError: One of the differentiated Tensors does not require grad

Cause: Selectively freezing UNet blocks while training causes gradient computation errors when PyTorch Lightning's gradient checkpointing expects gradients from all parameters.

Solution: Ensure all parameters in the forward path either all require grad or none do. When fine-tuning specific layers, freeze via param.requires_grad = False and verify no checkpointed layers reference frozen params.

Related: GitHub issue #784

7. FAQ

Q: What image resolutions does Stable Diffusion v1 support?

Stable Diffusion v1 was trained on 512x512 images. The model can generate at other resolutions, but quality degrades significantly outside the 512x512 range. Input dimensions should be divisible by 64 due to the autoencoder's downsampling factor.

Q: How much VRAM do I need?

Inference with default settings requires approximately 8-10 GB VRAM. Training requires 24+ GB VRAM depending on batch size and model size. Reduce VRAM usage by lowering output dimensions or reducing batch size.

Q: Can I run Stable Diffusion on macOS / Apple Silicon?

The original CompVis implementation targets NVIDIA CUDA GPUs. Community forks and the diffusers library provide MPS (Metal) support for Apple Silicon. See GitHub issue #25 for community setup instructions.

Q: What is the difference between DDIM and PLMS sampling?

DDIM (Denoising Diffusion Implicit Models) is the default sampler. PLMS (Pseudo Linear Multi-Step) typically produces higher-quality results in fewer steps. Both are implemented in ldm/models/diffusion/ as ddim.py and plms.py. A DPM-Solver implementation is also available at ldm/models/diffusion/dpm_solver/.

Q: How do I use my own dataset for training?

Create a custom dataset class inheriting from torch.utils.data.Dataset or Txt2ImgIterableBaseDataset (for text-image pairs). Reference it in your YAML config under data.params.train.target using the full Python import path. The DataModuleFromConfig class in main.py handles instantiation via OmegaConf's target resolution.

Q: What are the taming-transformers and CLIP dependencies?

taming-transformers provides the VQ-VAE codebook implementation (VectorQuantizer2) used by the VQ-based autoencoders. CLIP provides the frozen text encoder (FrozenCLIPEmbedder) that converts text prompts into conditioning vectors for text-to-image generation.

Q: Can I fine-tune on my own images?

Yes. Prepare a dataset class, write a YAML config pointing to it, and run python main.py --base your_config.yaml -t --gpus 0,. Use --resume to initialize from a pre-trained checkpoint.

Q: What does scale_factor: 0.18215 mean in the config?

This scaling factor normalizes the latent space values output by the autoencoder before they enter the diffusion process. It was computed empirically during training to ensure the latent distribution has approximately unit variance.

Q: What is the difference between VQ-based and KL-based autoencoders?

VQ (Vector Quantized) autoencoders discretize the latent space using a codebook (e.g., VQModel with n_embed=8192). KL autoencoders (AutoencoderKL) use a continuous Gaussian latent space regularized with a KL divergence term. Stable Diffusion v1 uses the KL autoencoder (kl-f8, 4-channel, embed_dim=4).

Q: What datasets are supported out of the box?

The data loaders support ImageNet (train/validation via ImageNetTrain/ImageNetValidation), ImageNet super-resolution (ImageNetSR), LSUN scenes (churches, bedrooms, cats via LSUNChurchesTrain, etc.), CelebA-HQ and FFHQ faces (via taming-transformers), and custom text-image datasets via Txt2ImgIterableBaseDataset.