Segment Anything Model (SAM) — Admin Guide

1. Architecture Overview

SAM follows a three-component encoder–decoder architecture designed around prompt-based mask prediction. Every inference pass flows through these stages:

Input Image → ImageEncoderViT → Image Embedding (256-dim, 64×64)
                                        ↓
Prompts → PromptEncoder → Sparse + Dense Embeddings
                                        ↓
                              MaskDecoder (TwoWayTransformer)
                                        ↓
                              Masks + IoU Scores

Note

The image encoder runs once per image. Multiple prompts can be evaluated against the same embedding without re-encoding, making interactive use efficient.

1.1 Component Responsibilities

1.2 Image Encoding Pipeline

The Sam model preprocesses images before encoding:

1. Resize — ResizeLongestSide(1024) scales the longest dimension to 1024 pixels, preserving aspect ratio
2. Normalize — subtract pixel_mean=[123.675, 116.28, 103.53], divide by pixel_std=[58.395, 57.12, 57.375] (ImageNet statistics)
3. Pad — zero-pad to a square 1024×1024 tensor
4. Encode — ImageEncoderViT produces a (1, 256, 64, 64) feature map

ImageEncoderViT uses a Vision Transformer with:

• Patch embedding via PatchEmbed (kernel 16×16, stride 16) converting 3×1024×1024 → (64×64) × embed_dim
• Stacked Block modules with multi-head self-attention (Attention) and MLPBlock
• Windowed attention with configurable window_size (14 for global, applied every few blocks)
• Decomposed relative positional encoding via add_decomposed_rel_pos
• Final neck: Conv2d(embed_dim→256) → LayerNorm2d → Conv2d(256→256) → LayerNorm2d

1.3 Prompt Encoding

PromptEncoder converts three prompt types into embeddings:

When no mask prompt is provided, a learned no_mask_embed tensor is used as the dense embedding.

1.4 Mask Decoding

MaskDecoder combines prompt and image information through a TwoWayTransformer:

1. Token preparation — concatenate IoU output token + mask tokens (4 for multi-mask, 1 for single) + sparse prompt tokens
2. Image embedding — sum image embedding with dense prompt embedding and positional encoding
3. TwoWayTransformer — 2 layers of TwoWayAttentionBlock:
   • Self-attention on token sequence
   • Cross-attention: tokens → image
   • MLP on tokens
   • Cross-attention: image → tokens
   • Final token-to-image cross-attention (no skip-first-layer-PE trick on layer 0)
4. Upscale — ConvTranspose2d(256→64, k=2, s=2) → GELU → ConvTranspose2d(64→32, k=2, s=2) → GELU producing (32, 4H, 4W)
5. Hypernetwork — each mask token passes through a 3-layer MLP to produce a (32,) vector
6. Dot product — hypernetwork_output @ upscaled_image → per-mask spatial prediction
7. IoU head — separate MLP predicts quality score per mask

Important

When multimask_output=True, SAM returns 3 masks + 3 IoU scores (slicing from index 1:). When multimask_output=False, it returns 1 mask + 1 score (index 0 only). The first token is always the single-mask token.

1.5 Module Dependency Graph

Sam
├── ImageEncoderViT
│   ├── PatchEmbed
│   ├── Block (×depth)
│   │   ├── Attention (with optional relative positional encoding)
│   │   └── MLPBlock
│   └── Neck (Conv2d → LayerNorm2d → Conv2d → LayerNorm2d)
├── PromptEncoder
│   ├── PositionEmbeddingRandom
│   ├── Learned point embeddings (4 labels)
│   ├── Learned mask downscaling (Conv2d ×3)
│   └── no_mask_embed
└── MaskDecoder
    ├── TwoWayTransformer
    │   └── TwoWayAttentionBlock (×2)
    │       ├── Attention (self + cross ×2)
    │       └── MLPBlock
    ├── output_upscaling (ConvTranspose2d ×2)
    ├── output_hypernetworks_mlps (MLP ×4)
    └── iou_prediction_head (MLP)

2. System Requirements

2.1 Hardware

Tip

For CPU-only inference, all model variants work but are significantly slower. The ONNX-exported decoder can run on CPU efficiently since only the prompt encoder and mask decoder are exported (the expensive image encoder runs separately).

2.2 Software Dependencies

Core:

Python >= 3.8
PyTorch >= 1.7
torchvision >= 0.8

Required for full functionality:

opencv-python    # Image I/O and preprocessing
pycocotools      # COCO RLE encoding for automatic mask generation

Optional — ONNX export:

onnx             # Model export
onnxruntime      # Python ONNX inference

Optional — development (from setup.py extras_require["dev"]):

black, flake8, isort, mypy

Demo app (Node.js):

Node.js + npm
onnxruntime-web  # Browser-based ONNX inference
npyjs            # Load .npy embeddings in browser
react, react-dom

2.3 Installation Paths

Standard install (from GitHub):

pip install git+https://github.com/facebookresearch/segment-anything.git

Editable install (for development):

git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

With all optional dependencies:

pip install -e ".[all]"

Note

The setup.py declares version="1.0" with no install_requires — PyTorch and torchvision are assumed pre-installed. The extras_require has "all": ["opencv-python", "pycocotools", "matplotlib", "onnxruntime", "onnx"] and "dev": ["black", "flake8", "isort", "mypy"].

3. Configuration Reference

SAM has no configuration files. All behavior is controlled through Python constructor parameters and method arguments.

3.1 Model Registry

sam_model_registry maps string identifiers to builder functions:

All builders accept a single checkpoint parameter (path to .pth file). The internal _build_sam() function constructs the full Sam model with these fixed architecture constants:

• image_size=1024
• patch_size=16
• prompt_embed_dim=256
• mask_decoder depth=2, num_heads=8, mlp_dim=2048
• window_size=14 for windowed attention blocks
• Global attention block indices vary by model: ViT-H uses [7, 15, 23, 31], ViT-L uses [5, 11, 17, 23], ViT-B uses [2, 5, 8, 11]

3.2 SamPredictor Parameters

Constructor:

set_image():

predict():

Caution

Calling predict() or predict_torch() before set_image() raises RuntimeError: 'An image must be set with .set_image(...) before mask prediction.' Similarly, get_image_embedding() without a set image raises RuntimeError: 'An image must be set with .set_image(...) to generate an embedding.'

3.3 SamAutomaticMaskGenerator Parameters

Warning

Setting both points_per_side and point_grids to None raises ValueError: "Can't have both points_per_side and point_grid be None." At least one must be specified.

3.4 ONNX Export Parameters (scripts/export_onnx_model.py)

3.5 Automatic Mask Generation CLI (scripts/amg.py)

All SamAutomaticMaskGenerator constructor parameters are also exposed as CLI flags with matching names and defaults.

4. Deployment & Scaling

4.1 ONNX Deployment Architecture

The ONNX export strategy separates the model into two stages:

Stage 1 (Python/PyTorch):
  Image → ImageEncoderViT → image_embedding.npy (saved to disk)

Stage 2 (ONNX Runtime, any platform):
  image_embedding + prompts → SamOnnxModel → masks + scores

Important

SamOnnxModel wraps only the prompt encoder and mask decoder. The image encoder is NOT exported to ONNX. This is by design — the encoder runs once per image and is the most expensive component. Pre-computing embeddings allows the lightweight decoder to run interactively.

Export workflow:

python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_onnx.onnx \
  --model-type default

# Optional: quantize for smaller model + faster CPU inference
python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_onnx.onnx \
  --quantize-out sam_onnx_quantized.onnx \
  --model-type default

ONNX model inputs:

4.2 Demo App Deployment

The React demo (demo/) runs SAM's decoder entirely in-browser via onnxruntime-web:

cd demo
npm install
npm start        # Development server (webpack-dev-server)
npm run build    # Production build

Tip

The demo loads pre-computed image embeddings as .npy files via npyjs. To deploy for custom images: (1) run the image encoder in Python to produce .npy embedding files, (2) serve them alongside the ONNX decoder model, (3) the browser handles prompting and mask decoding.

Build configuration (webpack):

• Development: webpack-dev-server with hot reload on port 8081
• Production: minification, source maps, output to build/
• TypeScript via ts-loader with tsconfig.json targeting ES2017

4.3 Batch Processing

For processing many images, use the scripts/amg.py CLI:

python scripts/amg.py \
  --input ./images/ \
  --output ./masks/ \
  --checkpoint sam_vit_h_4b8939.pth \
  --model-type default \
  --device cuda \
  --convert-to-rle

This processes all images in a directory, writing per-image JSON mask files. Each JSON contains a list of mask annotations with segmentation, bounding box, area, predicted IoU, point coordinates, stability score, and crop box.

5. Operations

5.1 Model Checkpoint Management

SAM checkpoints are standard PyTorch state_dict files. In _build_sam() the Sam model is constructed first, then the checkpoint is opened in binary mode and loaded with torch.load(f) (no map_location is passed — weights load onto their saved device, so move the model with sam.to(device=...) after building). Finally sam.load_state_dict(state_dict) applies the weights.

Three official checkpoints:

5.2 Memory Management

• SamPredictor stores the image embedding, original image size, and input size as instance state after set_image(). Call reset_image() to free this memory
• SamAutomaticMaskGenerator processes points in batches of points_per_batch (default 64) to control peak GPU memory
• Crop layers (crop_n_layers > 0) multiply memory usage since each crop is processed independently

5.3 Output Formats

SamAutomaticMaskGenerator.generate() returns a list of dicts per mask:

6. Performance

6.1 Computational Profile

6.2 Tuning Automatic Mask Generation

Note

Automatic mask generation performance is primarily controlled by three parameters: points_per_side (grid density), points_per_batch (GPU batching), and crop_n_layers (multi-scale crops).

Speed vs. quality tradeoffs:

Stability score (calculate_stability_score): measures mask consistency under threshold perturbation. For each mask, it compares the binary mask at mask_threshold vs. mask_threshold ± stability_score_offset and computes IoU. Higher thresholds filter less-confident masks.

6.3 Multi-GPU and Batched Inference

Sam.forward() accepts batched_input — a list of dicts, each containing an image tensor and optional prompts. This supports batch inference when all images share the same dimensions (after preprocessing). For heterogeneous image sizes, process sequentially via SamPredictor.

7. Security Considerations

Caution

SAM processes arbitrary image inputs and loads model checkpoints via torch.load(), which uses Python's pickle module. Only load checkpoints from trusted sources — malicious checkpoint files can execute arbitrary code during loading.

7.1 Input Validation

SAM performs minimal input validation:

• SamPredictor.set_image() does not validate image dimensions or dtype — malformed inputs will propagate to the encoder and may produce garbage output or CUDA errors
• SamAutomaticMaskGenerator.__init__() validates that at least one of points_per_side or point_grids is not None
• No bounds checking on point coordinates or box dimensions — out-of-bounds prompts may produce unexpected masks

7.2 ONNX Model Security

When deploying ONNX models:

• The exported ONNX file contains only the prompt encoder and mask decoder weights — no image encoder weights are included
• ONNX Runtime provides a sandboxed execution environment compared to torch.load()
• Pre-computed embeddings (.npy files) should be treated as opaque tensors — validate array shapes before passing to the model

7.3 Demo App

The React demo uses onnxruntime-web for client-side inference. All computation happens in the browser:

• No server-side model execution
• Image data stays on the client
• Pre-computed embeddings must be served as static assets

8. Developer Internals

8.1 Class Reference

19 classes across 4 packages:

8.2 Key Utility Functions (segment_anything/utils/)

amg.py — mask generation utilities:

transforms.py — ResizeLongestSide:

onnx.py — SamOnnxModel.forward() inputs and processing:

• select_masks() — choose best mask by IoU score (single mask mode) or return all (multi-mask mode)
• resize_longest_image_size() — compute resize dimensions for mask postprocessing
• mask_postprocessing() — bilinear interpolation to input size, then crop to original size

8.3 Code Patterns

State management: SamPredictor uses instance variables (self.features, self.orig_size, self.input_size, self.is_image_set) to cache the image embedding between calls. The @torch.no_grad() decorator is applied to set_image, predict, and predict_torch.

Atomic model construction: _build_sam() constructs all three components, assembles Sam, loads weights, and calls .eval() in one function — no partial initialization possible.

Coordinate conventions: all external coordinates are (x, y) order. Internal ViT processing uses (H, W). ResizeLongestSide handles the conversion.

Mask threshold: Sam.mask_threshold = 0.0 — logits > 0 are foreground. This constant is referenced by SamPredictor.predict() when return_logits=False.

8.4 Error Conditions

8.5 Known Issues (from GitHub)

Warning

Issue #277: batch size > 1 with SamPredictor.predict_torch() may cause RuntimeError due to shape mismatches. Process prompts individually or use the Sam.forward() batched interface instead.

• #585: quantize_dynamic() got an unexpected keyword argument 'optimize_model' — the export script calls onnxruntime.quantization.quantize.quantize_dynamic(..., optimize_model=True, ...); newer onnxruntime releases removed that argument. Pin a compatible onnxruntime or drop the argument
• #446: torch.compile with TensorRT backend incompatibilities
• #678: Demo ModuleNotFoundError — ensure dependencies are installed via npm install
• #849: Notebook execution errors — verify checkpoint path and Python environment match

9. Quick Reference

9.1 Minimal Inference

from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

predictor = SamPredictor(sam)
predictor.set_image(image_rgb)  # HWC uint8 RGB

masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[np.argmax(scores)]

9.2 Full Automatic Segmentation

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,
    pred_iou_thresh=0.88,
    stability_score_thresh=0.95,
    crop_n_layers=1,
    min_mask_region_area=100,
)
masks = generator.generate(image_rgb)

9.3 ONNX Export + Inference

python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_decoder.onnx \
  --model-type vit_h

import onnxruntime

# Pre-compute embedding with PyTorch
predictor.set_image(image)
embedding = predictor.get_image_embedding().cpu().numpy()

# Run decoder with ONNX Runtime
session = onnxruntime.InferenceSession("sam_decoder.onnx")
masks, scores, low_res = session.run(None, {
    "image_embeddings": embedding,
    "point_coords": point_coords,
    "point_labels": point_labels,
    "mask_input": np.zeros((1, 1, 256, 256), dtype=np.float32),
    "has_mask_input": np.array([0.0], dtype=np.float32),
    "orig_im_size": np.array([H, W], dtype=np.float32),
})