# Segment Anything Model (SAM) — Admin Guide

## 1. Architecture Overview

SAM follows a three-component encoder–decoder architecture designed around prompt-based mask prediction. Every inference pass flows through these stages:

```
Input Image → ImageEncoderViT → Image Embedding (256-dim, 64×64)
                                        ↓
Prompts → PromptEncoder → Sparse + Dense Embeddings
                                        ↓
                              MaskDecoder (TwoWayTransformer)
                                        ↓
                              Masks + IoU Scores
```

> [!NOTE]
> The image encoder runs once per image. Multiple prompts can be evaluated against the same embedding without re-encoding, making interactive use efficient.

### 1.1 Component Responsibilities

| Component | Class | Purpose |
|---|---|---|
| Image Encoder | `ImageEncoderViT` | ViT backbone with windowed attention; produces 256-channel spatial embeddings |
| Prompt Encoder | `PromptEncoder` | Encodes points, boxes, and masks into sparse/dense embedding vectors |
| Mask Decoder | `MaskDecoder` | Cross-attends prompt tokens to image embeddings; produces masks and quality scores |
| Orchestrator | `Sam` | Composes the three components; handles preprocessing, batching, and postprocessing |

### 1.2 Image Encoding Pipeline

The `Sam` model preprocesses images before encoding:

1. **Resize** — `ResizeLongestSide(1024)` scales the longest dimension to 1024 pixels, preserving aspect ratio
2. **Normalize** — subtract `pixel_mean=[123.675, 116.28, 103.53]`, divide by `pixel_std=[58.395, 57.12, 57.375]` (ImageNet statistics)
3. **Pad** — zero-pad to a square `1024×1024` tensor
4. **Encode** — `ImageEncoderViT` produces a `(1, 256, 64, 64)` feature map

`ImageEncoderViT` uses a Vision Transformer with:

- Patch embedding via `PatchEmbed` (kernel 16×16, stride 16) converting `3×1024×1024` → `(64×64) × embed_dim`
- Stacked `Block` modules with multi-head self-attention (`Attention`) and `MLPBlock`
- Windowed attention with configurable `window_size` (14 for global, applied every few blocks)
- Decomposed relative positional encoding via `add_decomposed_rel_pos`
- Final neck: `Conv2d(embed_dim→256) → LayerNorm2d → Conv2d(256→256) → LayerNorm2d`

### 1.3 Prompt Encoding

`PromptEncoder` converts three prompt types into embeddings:

| Prompt Type | Encoding Method | Output |
|---|---|---|
| Points | `PositionEmbeddingRandom` + learned label embeddings (foreground=1, background=0, padding=-1) | Sparse embeddings |
| Boxes | Two corner points (top-left + bottom-right) with learned corner embeddings | Sparse embeddings (2 tokens per box) |
| Masks | `Conv2d(1→4, k=2, s=2) → GELU → Conv2d(4→16, k=2, s=2) → GELU → Conv2d(16→256, k=1)` | Dense embedding (256×64×64) |

When no mask prompt is provided, a learned `no_mask_embed` tensor is used as the dense embedding.

### 1.4 Mask Decoding

`MaskDecoder` combines prompt and image information through a `TwoWayTransformer`:

1. **Token preparation** — concatenate IoU output token + mask tokens (4 for multi-mask, 1 for single) + sparse prompt tokens
2. **Image embedding** — sum image embedding with dense prompt embedding and positional encoding
3. **TwoWayTransformer** — 2 layers of `TwoWayAttentionBlock`:
   - Self-attention on token sequence
   - Cross-attention: tokens → image
   - MLP on tokens
   - Cross-attention: image → tokens
   - Final token-to-image cross-attention (no skip-first-layer-PE trick on layer 0)
4. **Upscale** — `ConvTranspose2d(256→64, k=2, s=2) → GELU → ConvTranspose2d(64→32, k=2, s=2) → GELU` producing `(32, 4H, 4W)`
5. **Hypernetwork** — each mask token passes through a 3-layer `MLP` to produce a `(32,)` vector
6. **Dot product** — `hypernetwork_output @ upscaled_image` → per-mask spatial prediction
7. **IoU head** — separate `MLP` predicts quality score per mask

> [!IMPORTANT]
> When `multimask_output=True`, SAM returns 3 masks + 3 IoU scores (slicing from index 1:). When `multimask_output=False`, it returns 1 mask + 1 score (index 0 only). The first token is always the single-mask token.

### 1.5 Module Dependency Graph

```
Sam
├── ImageEncoderViT
│   ├── PatchEmbed
│   ├── Block (×depth)
│   │   ├── Attention (with optional relative positional encoding)
│   │   └── MLPBlock
│   └── Neck (Conv2d → LayerNorm2d → Conv2d → LayerNorm2d)
├── PromptEncoder
│   ├── PositionEmbeddingRandom
│   ├── Learned point embeddings (4 labels)
│   ├── Learned mask downscaling (Conv2d ×3)
│   └── no_mask_embed
└── MaskDecoder
    ├── TwoWayTransformer
    │   └── TwoWayAttentionBlock (×2)
    │       ├── Attention (self + cross ×2)
    │       └── MLPBlock
    ├── output_upscaling (ConvTranspose2d ×2)
    ├── output_hypernetworks_mlps (MLP ×4)
    └── iou_prediction_head (MLP)
```

---

## 2. System Requirements

### 2.1 Hardware

| Component | ViT-B | ViT-L | ViT-H |
|---|---|---|---|
| Parameters | ~91M | ~308M | ~636M |
| Checkpoint size | ~375 MB | ~1.25 GB | ~2.56 GB |
| GPU VRAM (inference) | ~2 GB | ~4 GB | ~8 GB |
| Recommended GPU | Any CUDA GPU | 8+ GB VRAM | 16+ GB VRAM |

> [!TIP]
> For CPU-only inference, all model variants work but are significantly slower. The ONNX-exported decoder can run on CPU efficiently since only the prompt encoder and mask decoder are exported (the expensive image encoder runs separately).

### 2.2 Software Dependencies

**Core:**

```
Python >= 3.8
PyTorch >= 1.7
torchvision >= 0.8
```

**Required for full functionality:**

```
opencv-python    # Image I/O and preprocessing
pycocotools      # COCO RLE encoding for automatic mask generation
```

**Optional — ONNX export:**

```
onnx             # Model export
onnxruntime      # Python ONNX inference
```

**Optional — development (from setup.py extras_require["dev"]):**

```
black, flake8, isort, mypy
```

**Demo app (Node.js):**

```
Node.js + npm
onnxruntime-web  # Browser-based ONNX inference
npyjs            # Load .npy embeddings in browser
react, react-dom
```

### 2.3 Installation Paths

**Standard install (from GitHub):**

```bash
pip install git+https://github.com/facebookresearch/segment-anything.git
```

**Editable install (for development):**

```bash
git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .
```

**With all optional dependencies:**

```bash
pip install -e ".[all]"
```

> [!NOTE]
> The `setup.py` declares `version="1.0"` with no `install_requires` — PyTorch and torchvision are assumed pre-installed. The `extras_require` has `"all": ["opencv-python", "pycocotools", "matplotlib", "onnxruntime", "onnx"]` and `"dev": ["black", "flake8", "isort", "mypy"]`.

---

## 3. Configuration Reference

SAM has no configuration files. All behavior is controlled through Python constructor parameters and method arguments.

### 3.1 Model Registry

`sam_model_registry` maps string identifiers to builder functions:

| Key | Builder | Backbone | embed_dim | depth | num_heads |
|---|---|---|---|---|---|
| `"default"` | `build_sam_vit_h` | ViT-H | 1280 | 32 | 16 |
| `"vit_h"` | `build_sam_vit_h` | ViT-H | 1280 | 32 | 16 |
| `"vit_l"` | `build_sam_vit_l` | ViT-L | 1024 | 24 | 16 |
| `"vit_b"` | `build_sam_vit_b` | ViT-B | 768 | 12 | 12 |

All builders accept a single `checkpoint` parameter (path to `.pth` file). The internal `_build_sam()` function constructs the full `Sam` model with these fixed architecture constants:

- `image_size=1024`
- `patch_size=16`
- `prompt_embed_dim=256`
- `mask_decoder depth=2`, `num_heads=8`, `mlp_dim=2048`
- `window_size=14` for windowed attention blocks
- Global attention block indices vary by model: ViT-H uses `[7, 15, 23, 31]`, ViT-L uses `[5, 11, 17, 23]`, ViT-B uses `[2, 5, 8, 11]`

### 3.2 SamPredictor Parameters

**Constructor:**

| Parameter | Type | Description |
|---|---|---|
| `sam_model` | `Sam` | Loaded SAM model instance |

**`set_image()`:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `image` | `np.ndarray` | — | HWC uint8 image |
| `image_format` | `str` | `"RGB"` | `"RGB"` or `"BGR"`. If BGR, channels are reversed before processing |

**`predict()`:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `point_coords` | `np.ndarray` or `None` | `None` | Nx2 array of point coordinates (x, y) |
| `point_labels` | `np.ndarray` or `None` | `None` | N-length array: 1=foreground, 0=background |
| `box` | `np.ndarray` or `None` | `None` | Length-4 array in XYXY format |
| `mask_input` | `np.ndarray` or `None` | `None` | 1×256×256 low-resolution mask (logits from prior prediction) |
| `multimask_output` | `bool` | `True` | Return 3 masks or 1 |
| `return_logits` | `bool` | `False` | Return raw logits instead of binary masks |

> [!CAUTION]
> Calling `predict()` or `predict_torch()` before `set_image()` raises `RuntimeError: 'An image must be set with .set_image(...) before mask prediction.'` Similarly, `get_image_embedding()` without a set image raises `RuntimeError: 'An image must be set with .set_image(...) to generate an embedding.'`

### 3.3 SamAutomaticMaskGenerator Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `model` | `Sam` | — | SAM model instance |
| `points_per_side` | `int` or `None` | `32` | Grid density per side. 32 → 1024 points. Mutually exclusive with `point_grids` |
| `points_per_batch` | `int` | `64` | Points processed per batch |
| `pred_iou_thresh` | `float` | `0.88` | Minimum predicted IoU to keep a mask |
| `stability_score_thresh` | `float` | `0.95` | Minimum stability score |
| `stability_score_offset` | `float` | `1.0` | Offset for stability score calculation |
| `box_nms_thresh` | `float` | `0.7` | NMS IoU threshold for deduplication |
| `crop_n_layers` | `int` | `0` | Number of additional crop layers (0 = whole image only) |
| `crop_nms_thresh` | `float` | `0.7` | NMS threshold across crops |
| `crop_overlap_ratio` | `float` | `512/1500` | Overlap fraction between crops |
| `crop_n_points_downscale_factor` | `int` | `1` | Point density reduction per crop layer |
| `point_grids` | `List[np.ndarray]` or `None` | `None` | Custom point grids (overrides `points_per_side`) |
| `min_mask_region_area` | `int` | `0` | Post-processing: remove connected regions smaller than this (requires OpenCV) |
| `output_mode` | `str` | `"binary_mask"` | `"binary_mask"`, `"uncompressed_rle"`, or `"coco_rle"` |

> [!WARNING]
> Setting both `points_per_side` and `point_grids` to `None` raises `ValueError: "Can't have both points_per_side and point_grid be None."` At least one must be specified.

### 3.4 ONNX Export Parameters (scripts/export_onnx_model.py)

| Flag | Default | Description |
|---|---|---|
| `--checkpoint` | required | Path to SAM checkpoint |
| `--output` | required | Output ONNX file path |
| `--model-type` | `"default"` | `vit_h`, `vit_l`, or `vit_b` |
| `--return-single-mask` | `False` | Export for single mask output |
| `--opset` | `17` | ONNX opset version (must be >= 11) |
| `--quantize-out` | `None` | Path for quantized model output |
| `--gelu-approximate` | `False` | Use approximate GELU (for compatibility) |
| `--use-stability-score` | `False` | Replace IoU prediction with stability score |
| `--return-extra-metrics` | `False` | Return additional metrics (stability score, IoU) |

### 3.5 Automatic Mask Generation CLI (scripts/amg.py)

| Flag | Default | Description |
|---|---|---|
| `--input` | required | Input image or directory |
| `--output` | required | Output directory |
| `--model-type` | `"default"` | `vit_h`, `vit_l`, or `vit_b` |
| `--checkpoint` | required | Path to SAM checkpoint |
| `--device` | `"cuda"` | `cuda` or `cpu` |
| `--convert-to-rle` | `False` | Save masks as COCO RLE instead of binary |

All `SamAutomaticMaskGenerator` constructor parameters are also exposed as CLI flags with matching names and defaults.

---

## 4. Deployment & Scaling

### 4.1 ONNX Deployment Architecture

The ONNX export strategy separates the model into two stages:

```
Stage 1 (Python/PyTorch):
  Image → ImageEncoderViT → image_embedding.npy (saved to disk)

Stage 2 (ONNX Runtime, any platform):
  image_embedding + prompts → SamOnnxModel → masks + scores
```

> [!IMPORTANT]
> `SamOnnxModel` wraps only the prompt encoder and mask decoder. The image encoder is NOT exported to ONNX. This is by design — the encoder runs once per image and is the most expensive component. Pre-computing embeddings allows the lightweight decoder to run interactively.

**Export workflow:**

```bash
python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_onnx.onnx \
  --model-type default

# Optional: quantize for smaller model + faster CPU inference
python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_onnx.onnx \
  --quantize-out sam_onnx_quantized.onnx \
  --model-type default
```

**ONNX model inputs:**

| Input | Shape | Description |
|---|---|---|
| `image_embeddings` | `(1, 256, 64, 64)` | Pre-computed image encoder output |
| `point_coords` | `(1, N, 2)` | Point coordinates (x, y) |
| `point_labels` | `(1, N)` | Point labels (1=fg, 0=bg) |
| `mask_input` | `(1, 1, 256, 256)` | Prior mask logits (zeros if none) |
| `has_mask_input` | `(1,)` | 1.0 if mask_input is provided, 0.0 otherwise |
| `orig_im_size` | `(2,)` | Original image (H, W) for output scaling |

### 4.2 Demo App Deployment

The React demo (`demo/`) runs SAM's decoder entirely in-browser via `onnxruntime-web`:

```bash
cd demo
npm install
npm start        # Development server (webpack-dev-server)
npm run build    # Production build
```

> [!TIP]
> The demo loads pre-computed image embeddings as `.npy` files via `npyjs`. To deploy for custom images: (1) run the image encoder in Python to produce `.npy` embedding files, (2) serve them alongside the ONNX decoder model, (3) the browser handles prompting and mask decoding.

**Build configuration** (webpack):

- Development: `webpack-dev-server` with hot reload on port 8081
- Production: minification, source maps, output to `build/`
- TypeScript via `ts-loader` with `tsconfig.json` targeting `ES2017`

### 4.3 Batch Processing

For processing many images, use the `scripts/amg.py` CLI:

```bash
python scripts/amg.py \
  --input ./images/ \
  --output ./masks/ \
  --checkpoint sam_vit_h_4b8939.pth \
  --model-type default \
  --device cuda \
  --convert-to-rle
```

This processes all images in a directory, writing per-image JSON mask files. Each JSON contains a list of mask annotations with segmentation, bounding box, area, predicted IoU, point coordinates, stability score, and crop box.

---

## 5. Operations

### 5.1 Model Checkpoint Management

SAM checkpoints are standard PyTorch `state_dict` files. In `_build_sam()` the `Sam` model is constructed first, then the checkpoint is opened in binary mode and loaded with `torch.load(f)` (no `map_location` is passed — weights load onto their saved device, so move the model with `sam.to(device=...)` after building). Finally `sam.load_state_dict(state_dict)` applies the weights.

Three official checkpoints:

| Model | Filename | Approximate Size |
|---|---|---|
| ViT-H | `sam_vit_h_4b8939.pth` | 2.56 GB |
| ViT-L | `sam_vit_l_0b3195.pth` | 1.25 GB |
| ViT-B | `sam_vit_b_01ec64.pth` | 375 MB |

### 5.2 Memory Management

- `SamPredictor` stores the image embedding, original image size, and input size as instance state after `set_image()`. Call `reset_image()` to free this memory
- `SamAutomaticMaskGenerator` processes points in batches of `points_per_batch` (default 64) to control peak GPU memory
- Crop layers (`crop_n_layers > 0`) multiply memory usage since each crop is processed independently

### 5.3 Output Formats

`SamAutomaticMaskGenerator.generate()` returns a list of dicts per mask:

| Key | Type | Description |
|---|---|---|
| `segmentation` | `np.ndarray` or RLE dict | Binary mask or encoded mask |
| `bbox` | `list[int]` | `[x, y, w, h]` bounding box |
| `area` | `int` | Mask area in pixels |
| `predicted_iou` | `float` | Model's predicted IoU quality score |
| `point_coords` | `list[list[float]]` | Input point that generated this mask |
| `stability_score` | `float` | Stability score from threshold perturbation |
| `crop_box` | `list[int]` | `[x, y, w, h]` of the crop used |

---

## 6. Performance

### 6.1 Computational Profile

| Phase | Cost | Notes |
|---|---|---|
| Image encoding | ~150ms (ViT-H, A100) | Runs once per image; dominates single-image latency |
| Prompt encoding | <1ms | Lightweight embedding lookup + small convolutions |
| Mask decoding | ~5ms per prompt set | 2-layer transformer + upscaling |
| Automatic generation | Seconds to minutes | Depends on `points_per_side` (default 32 → 1024 points) |

### 6.2 Tuning Automatic Mask Generation

> [!NOTE]
> Automatic mask generation performance is primarily controlled by three parameters: `points_per_side` (grid density), `points_per_batch` (GPU batching), and `crop_n_layers` (multi-scale crops).

**Speed vs. quality tradeoffs:**

| Scenario | Recommended Settings |
|---|---|
| Fast preview | `points_per_side=16`, `crop_n_layers=0` |
| Balanced | `points_per_side=32`, `crop_n_layers=0` (defaults) |
| High quality | `points_per_side=64`, `crop_n_layers=1` |
| Maximum coverage | `points_per_side=64`, `crop_n_layers=2`, `crop_n_points_downscale_factor=2` |

**Stability score** (`calculate_stability_score`): measures mask consistency under threshold perturbation. For each mask, it compares the binary mask at `mask_threshold` vs. `mask_threshold ± stability_score_offset` and computes IoU. Higher thresholds filter less-confident masks.

### 6.3 Multi-GPU and Batched Inference

`Sam.forward()` accepts `batched_input` — a list of dicts, each containing an image tensor and optional prompts. This supports batch inference when all images share the same dimensions (after preprocessing). For heterogeneous image sizes, process sequentially via `SamPredictor`.

---

## 7. Security Considerations

> [!CAUTION]
> SAM processes arbitrary image inputs and loads model checkpoints via `torch.load()`, which uses Python's `pickle` module. Only load checkpoints from trusted sources — malicious checkpoint files can execute arbitrary code during loading.

### 7.1 Input Validation

SAM performs minimal input validation:

- `SamPredictor.set_image()` does not validate image dimensions or dtype — malformed inputs will propagate to the encoder and may produce garbage output or CUDA errors
- `SamAutomaticMaskGenerator.__init__()` validates that at least one of `points_per_side` or `point_grids` is not `None`
- No bounds checking on point coordinates or box dimensions — out-of-bounds prompts may produce unexpected masks

### 7.2 ONNX Model Security

When deploying ONNX models:

- The exported ONNX file contains only the prompt encoder and mask decoder weights — no image encoder weights are included
- ONNX Runtime provides a sandboxed execution environment compared to `torch.load()`
- Pre-computed embeddings (`.npy` files) should be treated as opaque tensors — validate array shapes before passing to the model

### 7.3 Demo App

The React demo uses `onnxruntime-web` for client-side inference. All computation happens in the browser:

- No server-side model execution
- Image data stays on the client
- Pre-computed embeddings must be served as static assets

---

## 8. Developer Internals

### 8.1 Class Reference

**19 classes** across 4 packages:

| Class | Module | Base | Purpose |
|---|---|---|---|
| `Sam` | `modeling.sam` | `nn.Module` | Top-level model orchestrator |
| `ImageEncoderViT` | `modeling.image_encoder` | `nn.Module` | ViT image backbone |
| `Block` | `modeling.image_encoder` | `nn.Module` | Transformer block with windowed attention |
| `Attention` | `modeling.image_encoder` | `nn.Module` | Multi-head attention with optional relative positional encoding |
| `PatchEmbed` | `modeling.image_encoder` | `nn.Module` | Convolutional patch embedding |
| `PromptEncoder` | `modeling.prompt_encoder` | `nn.Module` | Multi-type prompt embedding |
| `PositionEmbeddingRandom` | `modeling.prompt_encoder` | `nn.Module` | Fourier positional encoding |
| `MaskDecoder` | `modeling.mask_decoder` | `nn.Module` | Mask prediction head |
| `MLP` | `modeling.mask_decoder` | `nn.Module` | Multi-layer perceptron (hypernetwork) |
| `TwoWayTransformer` | `modeling.transformer` | `nn.Module` | Bidirectional cross-attention transformer |
| `TwoWayAttentionBlock` | `modeling.transformer` | `nn.Module` | Single bidirectional attention layer |
| `Attention` | `modeling.transformer` | `nn.Module` | Attention with optional downsampling |
| `MLPBlock` | `modeling.common` | `nn.Module` | Feed-forward block |
| `LayerNorm2d` | `modeling.common` | `nn.Module` | Channel-wise layer normalization for 2D feature maps |
| `SamPredictor` | `predictor` | — | High-level single-image prediction interface |
| `SamAutomaticMaskGenerator` | `automatic_mask_generator` | — | Grid-based whole-image segmentation |
| `SamOnnxModel` | `utils.onnx` | `nn.Module` | ONNX export wrapper for decoder |
| `ResizeLongestSide` | `utils.transforms` | — | Aspect-preserving image/coordinate transforms |
| `MaskData` | `utils.amg` | — | Dict-like container for mask metadata with batch filtering |

### 8.2 Key Utility Functions (segment_anything/utils/)

**amg.py** — mask generation utilities:

| Function | Purpose |
|---|---|
| `build_point_grid(n_per_side)` | Generate uniform `n×n` grid of normalized coordinates |
| `build_all_layer_point_grids(n_per_side, n_layers, scale_per_layer)` | Grids for base + crop layers |
| `generate_crop_boxes(im_size, n_layers, overlap_ratio)` | Compute overlapping crop regions |
| `calculate_stability_score(masks, mask_threshold, threshold_offset)` | Threshold perturbation stability |
| `batched_mask_to_box(masks)` | Convert binary masks to XYXY bounding boxes |
| `mask_to_rle_pytorch(tensor)` | PyTorch-based RLE encoding |
| `rle_to_mask(rle)` | Decode RLE to binary NumPy mask |
| `coco_encode_rle(uncompressed_rle)` | Convert to pycocotools-compatible COCO RLE |
| `remove_small_regions(mask, area_thresh, mode)` | OpenCV-based connected component filtering |
| `batch_iterator(batch_size, *args)` | Yield batched slices of input arrays |
| `is_box_near_crop_edge(boxes, crop_box, orig_box, atol)` | Filter masks near crop boundaries |
| `uncrop_boxes_xyxy / uncrop_points / uncrop_masks` | Transform coordinates from crop space to image space |

**transforms.py** — `ResizeLongestSide`:

| Method | Purpose |
|---|---|
| `apply_image(image)` | Resize NumPy image (preserves aspect ratio) |
| `apply_coords(coords, original_size)` | Scale coordinates to match resized image |
| `apply_boxes(boxes, original_size)` | Scale boxes (delegates to `apply_coords`) |
| `apply_image_torch / apply_coords_torch / apply_boxes_torch` | Tensor variants |
| `get_preprocess_shape(oldh, oldw, long_side_length)` | Static: compute target dimensions |

**onnx.py** — `SamOnnxModel.forward()` inputs and processing:

- `select_masks()` — choose best mask by IoU score (single mask mode) or return all (multi-mask mode)
- `resize_longest_image_size()` — compute resize dimensions for mask postprocessing
- `mask_postprocessing()` — bilinear interpolation to input size, then crop to original size

### 8.3 Code Patterns

**State management:** `SamPredictor` uses instance variables (`self.features`, `self.orig_size`, `self.input_size`, `self.is_image_set`) to cache the image embedding between calls. The `@torch.no_grad()` decorator is applied to `set_image`, `predict`, and `predict_torch`.

**Atomic model construction:** `_build_sam()` constructs all three components, assembles `Sam`, loads weights, and calls `.eval()` in one function — no partial initialization possible.

**Coordinate conventions:** all external coordinates are `(x, y)` order. Internal ViT processing uses `(H, W)`. `ResizeLongestSide` handles the conversion.

**Mask threshold:** `Sam.mask_threshold = 0.0` — logits > 0 are foreground. This constant is referenced by `SamPredictor.predict()` when `return_logits=False`.

### 8.4 Error Conditions

| Error | Location | Trigger |
|---|---|---|
| `ValueError: "Can't have both points_per_side and point_grid be None."` | `automatic_mask_generator.py:110` | Neither grid source specified |
| `RuntimeError: 'An image must be set with .set_image(...)...'` | `predictor.py:134, 214` | `predict()` / `predict_torch()` without `set_image()` |
| `RuntimeError: 'An image must be set with .set_image(...)...'` | `predictor.py:252` | `get_image_embedding()` without `set_image()` |
| `TypeError: f'MaskData key {k} has an unsupported type {type(v)}.'` | `amg.py:57, 70` | Unsupported value type in `MaskData.cat()` or `filter()` |

### 8.5 Known Issues (from GitHub)

> [!WARNING]
> Issue [#277](https://github.com/facebookresearch/segment-anything/issues/277): batch size > 1 with `SamPredictor.predict_torch()` may cause `RuntimeError` due to shape mismatches. Process prompts individually or use the `Sam.forward()` batched interface instead.

- **[#585](https://github.com/facebookresearch/segment-anything/issues/585):** `quantize_dynamic() got an unexpected keyword argument 'optimize_model'` — the export script calls `onnxruntime.quantization.quantize.quantize_dynamic(..., optimize_model=True, ...)`; newer `onnxruntime` releases removed that argument. Pin a compatible `onnxruntime` or drop the argument
- **[#446](https://github.com/facebookresearch/segment-anything/issues/446):** `torch.compile` with TensorRT backend incompatibilities
- **[#678](https://github.com/facebookresearch/segment-anything/issues/678):** Demo `ModuleNotFoundError` — ensure dependencies are installed via `npm install`
- **[#849](https://github.com/facebookresearch/segment-anything/issues/849):** Notebook execution errors — verify checkpoint path and Python environment match

---

## 9. Quick Reference

### 9.1 Minimal Inference

```python
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

predictor = SamPredictor(sam)
predictor.set_image(image_rgb)  # HWC uint8 RGB

masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[np.argmax(scores)]
```

### 9.2 Full Automatic Segmentation

```python
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,
    pred_iou_thresh=0.88,
    stability_score_thresh=0.95,
    crop_n_layers=1,
    min_mask_region_area=100,
)
masks = generator.generate(image_rgb)
```

### 9.3 ONNX Export + Inference

```bash
python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h_4b8939.pth \
  --output sam_decoder.onnx \
  --model-type vit_h
```

```python
import onnxruntime

# Pre-compute embedding with PyTorch
predictor.set_image(image)
embedding = predictor.get_image_embedding().cpu().numpy()

# Run decoder with ONNX Runtime
session = onnxruntime.InferenceSession("sam_decoder.onnx")
masks, scores, low_res = session.run(None, {
    "image_embeddings": embedding,
    "point_coords": point_coords,
    "point_labels": point_labels,
    "mask_input": np.zeros((1, 1, 256, 256), dtype=np.float32),
    "has_mask_input": np.array([0.0], dtype=np.float32),
    "orig_im_size": np.array([H, W], dtype=np.float32),
})
```

---

_Generated June 2026 from facebookresearch/segment-anything at dca509f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._