Supervision Admin & Architecture Guide

This guide is for the engineers, architects, and contributors who integrate, deploy, extend, or maintain Supervision inside larger systems. It documents the library's internal architecture, dependency surface, configuration, security posture, module layout, and engineering conventions — everything you need to operate it in production or contribute to it confidently.

For task-oriented usage (annotating, tracking, counting, dataset I/O, metrics), see the Supervision User Guide; this document assumes that material and goes deeper.

1. Executive Summary

Supervision is a pure-Python computer-vision utility library published by Roboflow under the MIT license. It is not a model, a service, or an application — it is a toolkit that sits downstream of inference, normalizing model output into one container type and providing the post-processing every CV pipeline repeats: visualization, filtering, tracking, zone logic, dataset conversion, and metric computation.

Architectural keystone: a single dataclass, sv.Detections, plus a matrix of ~16 from_* connector classmethods. The connectors absorb every model's idiosyncratic output shape; everything else in the library consumes the normalized object. This "unified container + adapter matrix" design is what makes Supervision model-agnostic and is the first thing to understand before reading any other module.

2. Architecture

2.1 The data-flow model

   ┌─────────────┐   from_<model>()   ┌──────────────┐
   │  Any model  │ ─────────────────▶ │ sv.Detections │  (xyxy, mask, confidence,
   │  (external) │                    │  (dataclass)  │   class_id, tracker_id, data)
   └─────────────┘                    └──────┬───────┘
                                             │
        ┌────────────────────────────────────┼────────────────────────────────────┐
        ▼                  ▼                  ▼                  ▼                  ▼
  ┌───────────┐     ┌────────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
  │ annotators│     │  filtering │     │  trackers │     │   zones   │     │  metrics  │
  │ (24 draw  │     │ (numpy     │     │ (ByteTrack│     │ (Polygon/ │     │ (mAP, F1, │
  │  classes) │     │  indexing) │     │  + Kalman)│     │  Line)    │     │  CM, …)   │
  └───────────┘     └────────────┘     └───────────┘     └───────────┘     └───────────┘

Everything is stateless and functional except the few objects that legitimately accumulate state across video frames: ByteTrack (track history), DetectionsSmoother (per-track box deques), LineZone (crossing history), TraceAnnotator/HeatMapAnnotator (visual history). There is no global state, no singletons, and no I/O at import time.

2.2 Package layout

src/supervision/
├── __init__.py            # The public API surface — the __all__ list IS the contract
├── config.py              # Two module-level constants (see §4)
├── annotators/            # 24 visual annotators (core.py) + utils (ColorLookup)
├── detection/
│   ├── core.py            # sv.Detections — the central dataclass + all connectors
│   ├── compact_mask.py    # RLE-encoded masks (memory-efficient segmentation)
│   ├── line_zone.py       # LineZone + LineZoneAnnotator(Multiclass)
│   ├── vlm.py             # VLM/LMM enums + per-model parsers (PaliGemma, Gemini, …)
│   ├── tools/             # InferenceSlicer, PolygonZone, smoother, CSV/JSON sinks
│   └── utils/             # boxes, converters, iou_and_nms, masks, polygons, vlms
├── tracker/byte_tracker/  # ByteTrack: core, kalman_filter, matching, single_object_track
├── dataset/               # DetectionDataset + formats/ (coco, yolo, pascal_voc)
├── geometry/              # Point, Vector, Rect, Position enum
├── draw/                  # Color, ColorPalette, drawing primitives
├── metrics/               # mAP, mAR, F1, Precision, Recall, ConfusionMatrix
├── key_points/            # KeyPoints + keypoint annotators (Edge/Vertex)
├── keypoint/              # DEPRECATED alias of key_points (removal 0.30.0)
├── classification/        # Classifications container
├── utils/                 # video, image, conversion, file, iterables, notebook, logger
├── assets/                # Sample video/image downloader
└── validators/            # Internal field validators (DEPRECATED public helpers)

Note

The public API is defined entirely by src/supervision/__init__.py's __all__ (~130 names re-exported as sv.*). Anything not in that list — and anything under a module whose name starts with _ — is internal and offers no stability guarantee.

2.3 The Detections contract

Detections is a @dataclass of seven aligned fields: xyxy (required), and the optional mask, confidence, class_id, tracker_id, data (a dict of extra per-detection arrays such as class_name), and metadata (collection-level info). __post_init__ runs _validate_detections_fields to enforce array-length alignment.

The class implements Python container and NumPy-friendly protocols: __len__, __iter__, __eq__, __getitem__ (boolean-mask and integer indexing, plus detections["key"] for data fields), and __setitem__. This is why filtering reads like detections[detections.confidence > 0.5] — the dataclass is a thin, indexable view over its arrays.

2.4 Connector matrix

Adding support for a new model means adding one classmethod that maps that model's output into the container's arrays — no other module needs to change. This is the extension seam to know when contributing.

2.5 Tracking subsystem

ByteTrack (tracker/byte_tracker/) implements the ByteTrack algorithm: a constant- velocity Kalman filter (kalman_filter.py) predicts each track's next box, and the matching module (matching.py) associates detections to tracks via IoU cost and linear assignment, using a two-stage high/low confidence association. STrack (single_object_track.py) holds per-object state. The wrapper translates between Supervision's Detections and the algorithm's internal tensors, writing results back into detections.tracker_id.

Important

The built-in ByteTrack is deprecated since 0.28.0 and scheduled for removal in 0.30.0. New tracking work (ByteTrack, SORT, DeepSORT, and re-identification) lives in the standalone trackers package. Plan migrations now: the replacement is ByteTrackTracker, and its update method is update() rather than update_with_detections().

2.6 Memory architecture for masks

High-resolution segmentation can allocate enormous (N, H, W) boolean arrays. compact_mask.py provides CompactMask, a run-length-encoded representation, and InferenceSlicer(compact_masks=True) keeps masks RLE-encoded through the entire merge/ NMS/annotate pipeline, computing IoU directly on the RLE crops. Use it when slicing 4K+ images with many objects to avoid out-of-memory failures (see issue #27677-class OOM reports).

3. Requirements & Compatibility

3.1 Runtime dependencies

Optional: pandas>=2 via the metrics extra, for tabular metric results.

Note

Notably absent: PyTorch, Ultralytics, Transformers, and every model framework. Supervision deliberately does not depend on any inference engine. Connector classmethods import nothing at module load and accept duck-typed result objects, so the library stays lightweight and framework-neutral.

3.2 Platform support

Officially classified for macOS, Windows, and Linux, Python 3.9 through 3.14, Python-3- only. The package is fully type-annotated and ships py.typed, so downstream mypy users get inline types.

4. Configuration

Supervision has no configuration files and no required environment variables. Its behavior is configured entirely through constructor arguments on annotators, zones, trackers, and metrics (documented in the User Guide). Two categories of global settings exist:

4.1 Module constants (config.py)

4.2 Environment variables

These are the only runtime-tunable globals; everything else is per-call state.

5. Security

Supervision is an embedded library that runs in-process with your application's privileges. It has no auth surface, no network listeners, and no persistence — so the threat model centers on untrusted input rather than network exposure.

5.1 Untrusted file parsing

Caution

Loading a dataset or media file executes the underlying parsers (OpenCV, Pillow, defusedxml, PyYAML). Treat dataset directories and media from third parties as untrusted input: validate paths, cap image dimensions, and keep the image/XML dependencies current.

5.2 Resource exhaustion

Dense segmentation masks scale as O(N · H · W). A maliciously large image with many objects can exhaust memory. Mitigation: use InferenceSlicer(compact_masks=True) and bound input resolution before processing.

5.3 Asset downloads

supervision.assets.download_assets() fetches sample media over the network via requests. It is a convenience for tutorials only; do not call it in production paths, and verify integrity if you do.

5.4 What Supervision does not do

It does not open sockets, spawn subprocesses, evaluate code, read credentials, or write outside the paths you give it. There is no telemetry in the core library. (The documentation site uses analytics, but the PyPI package does not.)

6. Operations & Integration

6.1 Deployment patterns

Supervision is a library dependency — "deployment" means pinning it in your environment. Typical patterns:

• [x] Pin a known-good version (supervision==0.29.0) in requirements.txt/pyproject.toml.
• [x] Install only what you use; add [metrics] only if you compute tabular metrics.
• [x] Run headless (servers, containers) — the base package needs no display.
• [x] Co-locate with your inference engine (Ultralytics/Inference/Transformers), which you manage separately.

6.2 Real-time video pipelines

The canonical production shape is a per-frame callback driven by sv.process_video(source_path, target_path, callback), or a manual loop over sv.get_video_frames_generator(path) writing through sv.VideoSink. Recommended ordering inside the callback:

infer → from_<model>() → filter → track → smooth → zone.trigger → annotate

Use sv.FPSMonitor to measure live throughput and sv.VideoInfo.from_video_path to size buffers and configure ByteTrack(frame_rate=info.fps) correctly.

6.3 Performance tuning

Tip

The dominant cost in any Supervision pipeline is almost always model inference, not the library. Profile end-to-end before optimizing post-processing.

7. Code Quality & Conventions

Notable design idioms a contributor should recognize:

• Adapter pattern for model connectors (from_*) — the single extension seam.
• Strategy enums (ColorLookup, OverlapFilter, OverlapMetric, MetricTarget, AveragingMethod, Position) keep behavior switchable without subclassing.
• Base classes (BaseAnnotator, _BaseLabelAnnotator, BaseDataset, Metric) define small abstract contracts each concrete class fills.
• Dataclasses everywhere (Detections, KeyPoints, Color, Point, Rect) for cheap, equatable, indexable value objects.

7.1 Deprecation discipline

Supervision maintains a disciplined deprecation policy via pydeprecate and a public docs/deprecated.md. Features are warned for several releases before removal. Current deprecations to track:

8. Module Reference

Metrics API shape

All metric classes follow a uniform lifecycle: construct with a MetricTarget (BOXES, MASKS, ORIENTED_BOUNDING_BOXES), accumulate with update(predictions, targets), then compute() to get a result object that supports print() and .plot(). ConfusionMatrix additionally offers from_detections(...) and from_tensors(...) constructors and a benchmark(...) helper. AveragingMethod (MACRO, MICRO, WEIGHTED) controls per-class aggregation for F1/Precision/Recall.

9. Testing & Contributing

The test suite mirrors src/ under tests/ (annotators, detection, detection/utils, metrics, tracker, dataset, geometry, draw, key_points, utils, classification) and is the largest part of the repository by volume — a signal of the project's maturity. Doctests embedded in docstrings run as part of pytest (--doctest-modules), so every >>> example in the source is verified on each run.

To contribute:

git clone -b develop https://github.com/roboflow/supervision.git
cd supervision
uv venv && source .venv/bin/activate
uv pip install -r pyproject.toml -e . --all-extras
pre-commit install
pytest

• [x] Branch from develop (not main).
• [x] Add a from_* classmethod to support a new model — the established extension point.
• [x] Keep public symbols in __init__.__all__; everything else stays private.
• [x] Run pre-commit (ruff, mypy strict, codespell) and pytest before opening a PR.
• [x] Add doctest-able examples to new public methods.

10. Known Issues & Improvement Areas

Drawn from the project's open issue tracker at the analyzed commit; useful for risk assessment and contribution targeting.

Strategic direction: tracking is migrating out of the core into the dedicated trackers package; VLM support is consolidating under the VLM enum; and the keypoint module is unifying under key_points. Align new work with these directions rather than extending the deprecated paths.

11. Quick Reference

Architecture in one sentence

A unified Detections dataclass + a matrix of from_* model connectors, consumed by stateless annotator/filter/zone/metric modules and a few stateful video-history objects (tracker, smoother, line zone, trace/heatmap annotators).

Strategy enums

Operational checklist

• [ ] Pin supervision==<version>; avoid *.dev builds in production.
• [ ] Install [metrics] only if computing tabular metrics.
• [ ] Set ByteTrack(frame_rate=video_info.fps) for correct buffering.
• [ ] Migrate off sv.ByteTrack → trackers.ByteTrackTracker before 0.30.0.
• [ ] Use compact_masks=True for high-resolution segmentation slicing.
• [ ] Treat all dataset/media inputs as untrusted; keep OpenCV/Pillow patched.