Segment Anything Model (SAM) — User Guide

1. Quick Start

Segment Anything Model (SAM) is a promptable image segmentation model from Meta AI Research. Given an image and a prompt — a point, a bounding box, or a low-resolution mask — SAM produces high-quality object masks. SAM can also automatically generate masks for every object in an image without any prompt at all.

Note

SAM (this repository) accepts point, box, and mask prompts. It does not accept text prompts — there is no text encoder in the released model. A successor, Segment Anything Model 2 (SAM 2), extends SAM to video and lives at facebookresearch/segment-anything-2. This guide documents the original image-only SAM.

Key features:

• Promptable segmentation with points, boxes, or masks as input
• Automatic mask generation for entire images without prompts
• Three model sizes: ViT-H (default, highest quality), ViT-L, and ViT-B (fastest)
• ONNX export for deployment in non-Python environments
• Batch inference support for processing multiple prompts simultaneously
• Multi-mask output with quality scores for ambiguous prompts
• Crop-based generation for higher resolution mask coverage
• Interactive browser demo using ONNX Runtime Web

Prerequisites: Python 3.8+, PyTorch 1.7+, torchvision

Install and run in 60 seconds:

pip install git+https://github.com/facebookresearch/segment-anything.git
pip install opencv-python pycocotools

from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["default"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
)

Tip

Start with vit_b for experimentation — it loads fastest and uses the least GPU memory. Switch to vit_h when you need maximum mask quality.

What to try next: See the Prompted Segmentation section for point, box, and mask prompts. See Automatic Mask Generation to segment every object in an image.

2. Installation

System Requirements

Install from Source

pip install git+https://github.com/facebookresearch/segment-anything.git

Install with Optional Dependencies

The base install has no mandatory dependencies beyond PyTorch. Optional extras are available:

# All optional dependencies (matplotlib, pycocotools, opencv-python, onnx, onnxruntime)
pip install 'segment-anything[all] @ git+https://github.com/facebookresearch/segment-anything.git'

# Development tools only (flake8, isort, black, mypy)
pip install 'segment-anything[dev] @ git+https://github.com/facebookresearch/segment-anything.git'

Important

The pycocotools package is required if you use output_mode="coco_rle" in SamAutomaticMaskGenerator. The opencv-python package is required if you set min_mask_region_area > 0 for postprocessing small regions.

Download Model Checkpoints

Three model checkpoints are available, corresponding to the three ViT backbone sizes:

Verify Installation

import segment_anything
from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator
print("SAM installed successfully")

SAM installed successfully

Install Troubleshooting

Error: ModuleNotFoundError: No module named 'segment_anything'

Ensure you installed from the git repository, not from PyPI. The package is not published on PyPI.

Error: torch.cuda.OutOfMemoryError when loading the model

Use a smaller model variant. Switch from vit_h to vit_l or vit_b, or move the model to CPU with sam.to(device="cpu").

Error: ImportError: No module named 'pycocotools'

Install pycocotools: pip install pycocotools. On Windows, you may need pip install pycocotools-windows instead.

Error: ImportError: No module named 'cv2'

Install OpenCV: pip install opencv-python. Required only when using min_mask_region_area > 0.

Error: quantize_dynamic() got an unexpected keyword argument 'optimize_model'

This occurs with newer versions of onnxruntime. Downgrade to a compatible version: pip install onnxruntime==1.14.0. (Community issue #585)

Error: RuntimeError when using batch size > 1

Batched inference via Sam.forward() requires all images in the batch to be pre-processed to the same size. Use SamPredictor for single-image workflows. (Community issue #277)

3. Configuration

SAM does not use configuration files. All behavior is controlled through constructor parameters and method arguments passed in Python code.

For the complete parameter reference for SamAutomaticMaskGenerator and SamPredictor, see the Admin Guide Configuration Reference.

4. Core Usage

Prompted Segmentation with SamPredictor

SamPredictor is the primary interface for interactive segmentation. It computes an image embedding once, then produces masks efficiently for any number of prompts.

Step 1: Load a Model and Create a Predictor

from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

predictor = SamPredictor(sam)

The sam_model_registry maps model type strings to builder functions:

Step 2: Set an Image

import numpy as np
from PIL import Image

image = np.array(Image.open("photo.jpg"))
predictor.set_image(image)

The image must be a NumPy array in HWC uint8 format with pixel values in [0, 255]. The default expected color format is RGB. If your image is in BGR format (e.g., from OpenCV's cv2.imread):

predictor.set_image(image, image_format="BGR")

Note

set_image computes the image embedding using the image encoder. This is the most expensive step. Once set, you can run predict many times with different prompts efficiently without re-encoding.

Step 3: Predict with Point Prompts

point_coords = np.array([[500, 375]])   # (X, Y) pixel coordinates
point_labels = np.array([1])            # 1 = foreground, 0 = background

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True,
)

Returns:

Tip

For ambiguous prompts like a single point, use multimask_output=True and pick the mask with the highest score. For unambiguous prompts like multiple points or a bounding box, use multimask_output=False for better results.

Step 4: Predict with Box Prompts

box = np.array([100, 200, 400, 500])   # [x1, y1, x2, y2] in XYXY format

masks, scores, logits = predictor.predict(
    box=box,
    multimask_output=False,
)

Step 5: Combine Point and Box Prompts

masks, scores, logits = predictor.predict(
    point_coords=np.array([[250, 350]]),
    point_labels=np.array([1]),
    box=np.array([100, 200, 400, 500]),
    multimask_output=False,
)

Step 6: Iterative Refinement with Mask Input

Feed the low-resolution logits from a previous prediction as input to refine the mask:

masks, scores, logits = predictor.predict(
    point_coords=np.array([[250, 350]]),
    point_labels=np.array([1]),
    mask_input=logits[np.argmax(scores), :, :][None, :, :],  # shape 1x256x256
    multimask_output=False,
)

Warning

You must call set_image before predict. Calling predict without first setting an image raises RuntimeError: An image must be set with .set_image(...) before mask prediction.

Step 7: Retrieve the Image Embedding

To reuse the embedding (e.g., save it for ONNX inference):

embedding = predictor.get_image_embedding()
# Shape: 1xCxHxW (typically 1x256x64x64)

Step 8: Reset Between Images

predictor.reset_image()

This clears the stored embedding and internal state. Always call reset_image() or set_image() with a new image before switching to a different input.

Automatic Mask Generation with SamAutomaticMaskGenerator

SamAutomaticMaskGenerator segments every object in an image without any manual prompts. It generates a grid of point prompts, runs SAM on each, and applies filtering and NMS to produce clean masks.

Basic Usage

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)

The generate method accepts a NumPy array in HWC uint8 format and returns a list of mask dictionaries:

Tuning Generation Quality

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,                 # Grid density (default: 32 → 1024 points)
    pred_iou_thresh=0.88,               # Min predicted IoU to keep (default: 0.88)
    stability_score_thresh=0.95,        # Min stability score to keep (default: 0.95)
    box_nms_thresh=0.7,                 # NMS threshold for deduplication (default: 0.7)
    min_mask_region_area=100,           # Remove tiny regions (requires opencv)
)

Note

Increasing points_per_side produces more masks at the cost of runtime. The total number of initial point prompts is points_per_side ** 2 (e.g., 32 → 1024 points, 64 → 4096 points).

Multi-Crop Generation for Dense Scenes

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    crop_n_layers=1,                     # Run on crops at 1 additional scale
    crop_nms_thresh=0.7,                 # NMS between crops
    crop_overlap_ratio=512/1500,         # Overlap between adjacent crops
    crop_n_points_downscale_factor=1,    # Point density reduction per layer
)

When crop_n_layers > 0, the generator runs on the full image plus cropped sub-regions. Layer n produces (2^n)^2 crops. Masks from different crops are deduplicated via NMS, preferring masks from smaller crops.

Output Modes

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    output_mode="coco_rle",
)

Warning

Using "binary_mask" output mode on very high-resolution images can consume large amounts of memory since each mask is stored as a full-resolution boolean array. Use "coco_rle" or "uncompressed_rle" for large images.

Batch Inference with the Sam Model Directly

For advanced use cases, call the Sam model's forward method directly with batched inputs:

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

batched_input = [
    {
        "image": preprocessed_image_tensor,          # 3xHxW float tensor
        "original_size": (orig_h, orig_w),
        "point_coords": point_coords_tensor,         # BxNx2
        "point_labels": point_labels_tensor,          # BxN
    },
]

outputs = sam(batched_input, multimask_output=False)

Each output dictionary contains:

Important

When calling Sam.forward() directly, images must already be transformed and normalized. Use SamPredictor for simpler single-image workflows.

5. Common Tasks

Generate Masks for All Objects in an Image

Goal: Automatically segment every visible object.

Steps:

1. Load the model and create a mask generator:

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator
import cv2
import numpy as np

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
generator = SamAutomaticMaskGenerator(sam)

2. Load and generate masks:

image = cv2.imread("photo.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
masks = generator.generate(image)
print(f"Found {len(masks)} masks")

3. Sort by area and visualize:

sorted_masks = sorted(masks, key=lambda x: x["area"], reverse=True)
for i, mask in enumerate(sorted_masks[:5]):
    print(f"Mask {i}: area={mask['area']}, iou={mask['predicted_iou']:.3f}")

Segment a Specific Object with a Click

Goal: Select an object by clicking on it.

from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
predictor = SamPredictor(sam)

predictor.set_image(image)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[x, y]]),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[np.argmax(scores)]

Exclude a Region with Background Points

Goal: Mark an area as background to refine the mask.

point_coords = np.array([[500, 375], [600, 200]])
point_labels = np.array([1, 0])  # 1=foreground, 0=background

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,
)

Process a Directory of Images with the AMG Script

Goal: Batch-process images using the command-line script.

python scripts/amg.py \
    --input ./images/ \
    --output ./masks/ \
    --model-type vit_h \
    --checkpoint sam_vit_h_4b8939.pth \
    --device cuda

This produces a folder of PNG masks and a metadata.csv per input image.

To save masks in COCO RLE format as JSON instead:

python scripts/amg.py \
    --input ./images/ \
    --output ./masks/ \
    --model-type vit_h \
    --checkpoint sam_vit_h_4b8939.pth \
    --convert-to-rle

Export a Model to ONNX

Goal: Export the prompt encoder and mask decoder for deployment.

python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_onnx.onnx

To export a quantized model for smaller size:

python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_onnx.onnx \
    --quantize-out sam_onnx_quantized.onnx

Tip

The ONNX model exports only the prompt encoder and mask decoder — not the image encoder. You still need to compute image embeddings in PyTorch (or pre-compute and save them as .npy files) before running the ONNX model.

Export with GELU Approximation for Compatibility

For runtimes with slow or missing erf operations (used in GELU):

python scripts/export_onnx_model.py \
    --checkpoint sam_vit_b_01ec64.pth \
    --model-type vit_b \
    --output sam_onnx.onnx \
    --gelu-approximate

Run the Interactive Browser Demo

Goal: Try SAM in a browser with click-based prompting.

Prerequisites: Node.js, Yarn, a pre-computed image embedding .npy file, and a quantized ONNX model.

cd demo
yarn install

Place the ONNX model at demo/model/sam_onnx_quantized_example.onnx and the image embedding at demo/src/assets/data/dogs_embedding.npy.

yarn start

The demo opens a browser window where hovering over the image generates masks in real time using ONNX Runtime Web.

Use Pre-Transformed Torch Tensors

For workflows that already have images as tensors:

import torch

# Image must be 1x3xHxW with longest side resized to 1024
transformed_image = torch.randn(1, 3, 1024, 768).to("cuda")

predictor.set_torch_image(
    transformed_image,
    original_image_size=(2048, 1536),
)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
)

Predict with Torch Tensors Directly

For batched GPU workflows, use predict_torch instead of predict:

masks, scores, logits = predictor.predict_torch(
    point_coords=torch.tensor([[[500.0, 375.0]]], device="cuda"),
    point_labels=torch.tensor([[1]], device="cuda"),
    multimask_output=True,
)
# masks shape: Bx3xHxW (torch.Tensor on GPU)

6. Troubleshooting

Error: RuntimeError: An image must be set with .set_image(...) before mask prediction.

Cause: You called predict(), predict_torch(), or get_image_embedding() without first calling set_image() or set_torch_image().

Solution:

1. Call predictor.set_image(image) before calling predict()

Prevention: Always set an image immediately after creating the predictor or after calling reset_image().

Error: ValueError: Can't have both points_per_side and point_grid be None.

Cause: SamAutomaticMaskGenerator was initialized with both points_per_side=None and point_grids=None.

Solution:

1. Provide points_per_side (integer) OR point_grids (list of NumPy arrays)
2. Exactly one of these two parameters must be non-None

Error: AssertionError: Exactly one of points_per_side or point_grid must be provided.

Cause: Both points_per_side and point_grids were given non-None values.

Solution: Set one to None. Use points_per_side for a regular grid or point_grids for a custom point layout.

Error: torch.cuda.OutOfMemoryError during mask generation

Cause: The image is too large or the model too big for available GPU memory.

Solution:

1. Reduce points_per_batch (default: 64) to process fewer points at a time
2. Use a smaller model (vit_b instead of vit_h)
3. Move to CPU: sam.to(device="cpu") (much slower but works)
4. Resize the input image to a smaller resolution

Prevention: Monitor GPU memory with torch.cuda.memory_summary() before processing large images.

Error: Module not found: Error: Can't resolve 'process/browser' in the demo

Cause: The webpack configuration expects the process polyfill.

Solution:

1. Ensure process is listed in demo/package.json dependencies
2. Run yarn install to install all dependencies

Community issue #678 provides additional context.

Masks look low quality or noisy

Cause: Default thresholds may not suit all images.

Solution:

1. Increase pred_iou_thresh (e.g., from 0.88 to 0.92) to keep only high-confidence masks
2. Increase stability_score_thresh (e.g., from 0.95 to 0.98) to keep only stable masks
3. Set min_mask_region_area to a positive value (e.g., 100) to remove small disconnected regions

Caution

Setting min_mask_region_area > 0 requires opencv-python. If OpenCV is not installed, SamAutomaticMaskGenerator.__init__ will raise an ImportError.

Demo shows incorrect or no mask overlay

Cause: The ONNX model or image embedding file is not in the expected location.

Solution:

1. Verify demo/model/sam_onnx_quantized_example.onnx exists
2. Verify demo/src/assets/data/dogs_embedding.npy exists
3. Check the browser console for loading errors

Warning

The ONNX Runtime Web demo requires Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: credentialless headers. The dev server sets these automatically. If deploying the built demo, configure your web server to include these headers.

7. FAQ

Q: Which model variant should I use?

Use vit_h for the best mask quality — it is the default. Use vit_b for the fastest inference and lowest memory usage. vit_l is a middle ground.

Q: Can I run SAM on CPU?

Yes. Pass device="cpu" when moving the model: sam.to(device="cpu"). Inference will be significantly slower but functional.

Q: What image formats does SAM accept?

SAM accepts NumPy arrays in HWC uint8 format (height x width x channels, values 0-255). Both RGB and BGR formats are supported via the image_format parameter in set_image.

Q: Does SAM support video segmentation?

SAM v1 (this repository) is designed for single-image segmentation. It does not have built-in video or temporal tracking capabilities. For video and streaming-memory tracking, use the successor model SAM 2 at facebookresearch/segment-anything-2.

Q: How do I get the best mask for a single point prompt?

Use multimask_output=True and select the mask with the highest score: best_mask = masks[np.argmax(scores)]. For unambiguous prompts (multiple points or a box), use multimask_output=False.

Q: Can I combine points and boxes in one prompt?

Yes. Pass both point_coords/point_labels and box to predictor.predict(). The model treats them as complementary prompts.

Q: What is stability_score?

The stability score measures how much the predicted mask changes when you slightly vary the threshold used to binarize the mask logits. A high stability score means the boundary is well-defined. It is computed as the IoU between masks obtained at threshold + offset and threshold - offset.

Q: How do I save masks to disk?

For binary masks, use OpenCV or PIL:

import cv2
cv2.imwrite("mask.png", masks[0].astype(np.uint8) * 255)

For COCO RLE format, use output_mode="coco_rle" in the automatic generator and save the result as JSON.

Q: What does the ONNX export include?

The ONNX export includes the prompt encoder and mask decoder only. The image encoder (ViT backbone) is not exported. You must pre-compute image embeddings in PyTorch and provide them as input to the ONNX model.

Q: Is the SA-1B dataset still available?

Check the project's official pages for current dataset availability. (Community issue #850)