# Supervision Admin & Architecture Guide

This guide is for the engineers, architects, and contributors who integrate, deploy,
extend, or maintain **Supervision** inside larger systems. It documents the library's
internal architecture, dependency surface, configuration, security posture, module
layout, and engineering conventions — everything you need to operate it in production
or contribute to it confidently.

For task-oriented usage (annotating, tracking, counting, dataset I/O, metrics), see the
**Supervision User Guide**; this document assumes that material and goes deeper.

---

## 1. Executive Summary

Supervision is a **pure-Python computer-vision utility library** published by Roboflow
under the MIT license. It is *not* a model, a service, or an application — it is a
toolkit that sits **downstream of inference**, normalizing model output into one
container type and providing the post-processing every CV pipeline repeats:
visualization, filtering, tracking, zone logic, dataset conversion, and metric
computation.

| Property | Value |
|----------|-------|
| Language | Python (3.9–3.14) |
| License | MIT |
| Distribution | PyPI (`supervision`), conda-forge |
| Package layout | `src/` layout, `py.typed` (fully typed) |
| Build backend | setuptools (`>=61`) |
| Runtime dependencies | NumPy, OpenCV, Pillow, SciPy, Matplotlib, PyYAML, Requests, tqdm, defusedxml, pydeprecate |
| Optional dependencies | pandas (`metrics` extra) |
| Hard runtime services | **None** — no server, no database, no network at rest |
| Analyzed commit | `b18a30e` |

**Architectural keystone:** a single dataclass, `sv.Detections`, plus a matrix of
~16 `from_*` connector classmethods. The connectors absorb every model's idiosyncratic
output shape; everything else in the library consumes the normalized object. This
"unified container + adapter matrix" design is what makes Supervision model-agnostic
and is the first thing to understand before reading any other module.

---

## 2. Architecture

### 2.1 The data-flow model

```
   ┌─────────────┐   from_<model>()   ┌──────────────┐
   │  Any model  │ ─────────────────▶ │ sv.Detections │  (xyxy, mask, confidence,
   │  (external) │                    │  (dataclass)  │   class_id, tracker_id, data)
   └─────────────┘                    └──────┬───────┘
                                             │
        ┌────────────────────────────────────┼────────────────────────────────────┐
        ▼                  ▼                  ▼                  ▼                  ▼
  ┌───────────┐     ┌────────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
  │ annotators│     │  filtering │     │  trackers │     │   zones   │     │  metrics  │
  │ (24 draw  │     │ (numpy     │     │ (ByteTrack│     │ (Polygon/ │     │ (mAP, F1, │
  │  classes) │     │  indexing) │     │  + Kalman)│     │  Line)    │     │  CM, …)   │
  └───────────┘     └────────────┘     └───────────┘     └───────────┘     └───────────┘
```

Everything is **stateless and functional** except the few objects that legitimately
accumulate state across video frames: `ByteTrack` (track history), `DetectionsSmoother`
(per-track box deques), `LineZone` (crossing history), `TraceAnnotator`/`HeatMapAnnotator`
(visual history). There is no global state, no singletons, and no I/O at import time.

### 2.2 Package layout

```
src/supervision/
├── __init__.py            # The public API surface — the __all__ list IS the contract
├── config.py              # Two module-level constants (see §4)
├── annotators/            # 24 visual annotators (core.py) + utils (ColorLookup)
├── detection/
│   ├── core.py            # sv.Detections — the central dataclass + all connectors
│   ├── compact_mask.py    # RLE-encoded masks (memory-efficient segmentation)
│   ├── line_zone.py       # LineZone + LineZoneAnnotator(Multiclass)
│   ├── vlm.py             # VLM/LMM enums + per-model parsers (PaliGemma, Gemini, …)
│   ├── tools/             # InferenceSlicer, PolygonZone, smoother, CSV/JSON sinks
│   └── utils/             # boxes, converters, iou_and_nms, masks, polygons, vlms
├── tracker/byte_tracker/  # ByteTrack: core, kalman_filter, matching, single_object_track
├── dataset/               # DetectionDataset + formats/ (coco, yolo, pascal_voc)
├── geometry/              # Point, Vector, Rect, Position enum
├── draw/                  # Color, ColorPalette, drawing primitives
├── metrics/               # mAP, mAR, F1, Precision, Recall, ConfusionMatrix
├── key_points/            # KeyPoints + keypoint annotators (Edge/Vertex)
├── keypoint/              # DEPRECATED alias of key_points (removal 0.30.0)
├── classification/        # Classifications container
├── utils/                 # video, image, conversion, file, iterables, notebook, logger
├── assets/                # Sample video/image downloader
└── validators/            # Internal field validators (DEPRECATED public helpers)
```

> [!NOTE]
> The public API is defined entirely by `src/supervision/__init__.py`'s `__all__`
> (~130 names re-exported as `sv.*`). Anything not in that list — and anything under a
> module whose name starts with `_` — is internal and offers no stability guarantee.

### 2.3 The `Detections` contract

`Detections` is a `@dataclass` of seven aligned fields: `xyxy` (required), and the
optional `mask`, `confidence`, `class_id`, `tracker_id`, `data` (a dict of extra
per-detection arrays such as `class_name`), and `metadata` (collection-level info).
`__post_init__` runs `_validate_detections_fields` to enforce array-length alignment.

The class implements Python container and NumPy-friendly protocols: `__len__`,
`__iter__`, `__eq__`, `__getitem__` (boolean-mask and integer indexing, plus
`detections["key"]` for data fields), and `__setitem__`. This is why filtering reads
like `detections[detections.confidence > 0.5]` — the dataclass *is* a thin, indexable
view over its arrays.

### 2.4 Connector matrix

| Domain | Container | Connectors |
|--------|-----------|-----------|
| Detection / segmentation | `Detections` | `from_inference`, `from_ultralytics`, `from_transformers`, `from_sam`, `from_sam3`, `from_detectron2`, `from_mmdetection`, `from_yolo_nas`, `from_yolov5`, `from_deepsparse`, `from_paddledet`, `from_tensorflow`, `from_ncnn`, `from_easyocr`, `from_azure_analyze_image`, `from_vlm` (+ deprecated `from_lmm`) |
| Keypoints | `KeyPoints` | `from_inference`, `from_ultralytics`, `from_mediapipe`, `from_yolo_nas`, `from_detectron2`, `from_transformers`, `from_rfdetr` |
| Classification | `Classifications` | `from_clip`, `from_ultralytics`, `from_timm` |

Adding support for a new model means adding one classmethod that maps that model's
output into the container's arrays — no other module needs to change. This is the
extension seam to know when contributing.

### 2.5 Tracking subsystem

`ByteTrack` (`tracker/byte_tracker/`) implements the ByteTrack algorithm: a constant-
velocity **Kalman filter** (`kalman_filter.py`) predicts each track's next box, and the
**matching** module (`matching.py`) associates detections to tracks via IoU cost and
linear assignment, using a two-stage high/low confidence association. `STrack`
(`single_object_track.py`) holds per-object state. The wrapper translates between
Supervision's `Detections` and the algorithm's internal tensors, writing results back
into `detections.tracker_id`.

> [!IMPORTANT]
> The built-in `ByteTrack` is **deprecated since 0.28.0** and scheduled for removal in
> `0.30.0`. New tracking work (ByteTrack, SORT, DeepSORT, and re-identification) lives
> in the standalone [`trackers`](https://pypi.org/project/trackers/) package. Plan
> migrations now: the replacement is `ByteTrackTracker`, and its update method is
> `update()` rather than `update_with_detections()`.

### 2.6 Memory architecture for masks

High-resolution segmentation can allocate enormous `(N, H, W)` boolean arrays.
`compact_mask.py` provides `CompactMask`, a run-length-encoded representation, and
`InferenceSlicer(compact_masks=True)` keeps masks RLE-encoded through the entire merge/
NMS/annotate pipeline, computing IoU directly on the RLE crops. Use it when slicing
4K+ images with many objects to avoid out-of-memory failures (see issue
[#27677-class](https://github.com/roboflow/supervision/issues) OOM reports).

---

## 3. Requirements & Compatibility

### 3.1 Runtime dependencies

| Package | Constraint | Why |
|---------|-----------|-----|
| `numpy` | `>=1.21.2` | Core array type for all detection fields |
| `opencv-python` | `>=4.5.5.64` | Drawing primitives, image I/O, video |
| `pillow` | `>=9.4` | `ImageType` interop (PIL ⇄ ndarray) |
| `scipy` | `>=1.10` | Linear assignment (tracking), spatial ops |
| `matplotlib` | `>=3.6` | `ColorPalette.from_matplotlib`, metric plots |
| `pyyaml` | `>=5.3` | YOLO `data.yaml` parsing |
| `requests` | `>=2.26` | Asset downloads |
| `tqdm` | `>=4.62.3` | Progress bars |
| `defusedxml` | `>=0.7.1` | **Safe** Pascal VOC XML parsing (see §5) |
| `pydeprecate` | `>=0.9,<0.10` | Deprecation machinery |

Optional: `pandas>=2` via the `metrics` extra, for tabular metric results.

> [!NOTE]
> Notably absent: **PyTorch, Ultralytics, Transformers, and every model framework**.
> Supervision deliberately does not depend on any inference engine. Connector
> classmethods import nothing at module load and accept duck-typed result objects, so
> the library stays lightweight and framework-neutral.

### 3.2 Platform support

Officially classified for macOS, Windows, and Linux, Python 3.9 through 3.14, Python-3-
only. The package is fully type-annotated and ships `py.typed`, so downstream `mypy`
users get inline types.

---

## 4. Configuration

Supervision has **no configuration files and no required environment variables**. Its
behavior is configured entirely through constructor arguments on annotators, zones,
trackers, and metrics (documented in the User Guide). Two categories of global settings
exist:

### 4.1 Module constants (`config.py`)

| Constant | Value | Purpose |
|----------|-------|---------|
| `CLASS_NAME_DATA_FIELD` | `"class_name"` | The `data` dict key under which connectors store class-name strings |
| `ORIENTED_BOX_COORDINATES` | `"xyxyxyxy"` | The `data` key holding OBB corner coordinates; its presence switches `InferenceSlicer` into sequential (thread-safe) mode |

### 4.2 Environment variables

| Variable | Default | Effect |
|----------|---------|--------|
| `SUPERVISON_DEPRECATION_WARNING` | (enabled) | Set to `0` to suppress deprecation warnings. *(The variable name carries an upstream spelling — note the missing "i".)* |
| `LOG_LEVEL` | `INFO` | Logging verbosity for `supervision.utils.logger` |

These are the **only** runtime-tunable globals; everything else is per-call state.

---

## 5. Security

Supervision is an embedded library that runs **in-process with your application's
privileges**. It has no auth surface, no network listeners, and no persistence — so the
threat model centers on **untrusted input** rather than network exposure.

### 5.1 Untrusted file parsing

| Input | Risk | Mitigation in Supervision |
|-------|------|---------------------------|
| Pascal VOC XML | XML entity-expansion / billion-laughs attacks | Parsed with **`defusedxml`** (a hard dependency), which blocks entity-expansion and external-entity attacks |
| COCO / YOLO annotations (JSON/YAML) | Malformed or hostile structure | Standard `json`/`pyyaml` parsing; validate provenance of dataset files |
| Images & video | Decoder vulnerabilities, hostile dimensions | Delegated to OpenCV/Pillow — keep them patched; bound resolution before slicing |

> [!CAUTION]
> Loading a dataset or media file executes the underlying parsers (OpenCV, Pillow,
> defusedxml, PyYAML). Treat dataset directories and media from third parties as
> untrusted input: validate paths, cap image dimensions, and keep the image/XML
> dependencies current.

### 5.2 Resource exhaustion

Dense segmentation masks scale as `O(N · H · W)`. A maliciously large image with many
objects can exhaust memory. **Mitigation:** use `InferenceSlicer(compact_masks=True)`
and bound input resolution before processing.

### 5.3 Asset downloads

`supervision.assets.download_assets()` fetches sample media over the network via
`requests`. It is a convenience for tutorials only; do not call it in production paths,
and verify integrity if you do.

### 5.4 What Supervision does *not* do

It does not open sockets, spawn subprocesses, evaluate code, read credentials, or write
outside the paths you give it. There is no telemetry in the core library. (The
documentation *site* uses analytics, but the PyPI package does not.)

---

## 6. Operations & Integration

### 6.1 Deployment patterns

Supervision is a library dependency — "deployment" means pinning it in your
environment. Typical patterns:

- [x] Pin a known-good version (`supervision==0.29.0`) in `requirements.txt`/`pyproject.toml`.
- [x] Install only what you use; add `[metrics]` only if you compute tabular metrics.
- [x] Run headless (servers, containers) — the base package needs no display.
- [x] Co-locate with your inference engine (Ultralytics/Inference/Transformers), which you manage separately.

### 6.2 Real-time video pipelines

The canonical production shape is a per-frame callback driven by
`sv.process_video(source_path, target_path, callback)`, or a manual loop over
`sv.get_video_frames_generator(path)` writing through `sv.VideoSink`. Recommended
ordering inside the callback:

```text
infer → from_<model>() → filter → track → smooth → zone.trigger → annotate
```

Use `sv.FPSMonitor` to measure live throughput and `sv.VideoInfo.from_video_path` to
size buffers and configure `ByteTrack(frame_rate=info.fps)` correctly.

### 6.3 Performance tuning

| Lever | Guidance |
|-------|----------|
| `InferenceSlicer.thread_workers` | Increase for many tiles; falls back to sequential automatically for OBB |
| `InferenceSlicer.compact_masks` | Enable for high-res segmentation to cut memory |
| `ByteTrack.lost_track_buffer` | Raise to survive longer occlusions, at some memory cost |
| `DetectionsSmoother.length` | Longer windows = smoother but laggier boxes |
| `TraceAnnotator.trace_length` | Bound to control per-frame draw cost |

> [!TIP]
> The dominant cost in any Supervision pipeline is almost always **model inference**,
> not the library. Profile end-to-end before optimizing post-processing.

---

## 7. Code Quality & Conventions

| Aspect | Convention |
|--------|-----------|
| Layout | `src/` layout; public surface is `__init__.__all__` only |
| Typing | `strict` mypy, `py.typed` shipped, `from __future__ import annotations` throughout |
| Linting/format | Ruff (`line-length = 88`, double quotes, isort, bandit `S`, pyupgrade) |
| Docstrings | Google style, surfaced via `mkdocstrings` |
| Pre-commit | ruff, mypy, codespell, mdformat, prettier, pyproject-fmt, validate-pyproject |
| Tests | `pytest`, doctests enabled (`--doctest-modules`), coverage target ~95% (codecov) |
| Docs | MkDocs Material; `docs/how_to/*.md` examples are executable and CI-checked |

Notable design idioms a contributor should recognize:

- **Adapter pattern** for model connectors (`from_*`) — the single extension seam.
- **Strategy enums** (`ColorLookup`, `OverlapFilter`, `OverlapMetric`, `MetricTarget`,
  `AveragingMethod`, `Position`) keep behavior switchable without subclassing.
- **Base classes** (`BaseAnnotator`, `_BaseLabelAnnotator`, `BaseDataset`, `Metric`)
  define small abstract contracts each concrete class fills.
- **Dataclasses everywhere** (`Detections`, `KeyPoints`, `Color`, `Point`, `Rect`) for
  cheap, equatable, indexable value objects.

### 7.1 Deprecation discipline

Supervision maintains a disciplined deprecation policy via `pydeprecate` and a public
`docs/deprecated.md`. Features are warned for several releases before removal. Current
deprecations to track:

| Deprecated | Replacement | Removal |
|------------|-------------|---------|
| `sv.ByteTrack` | `ByteTrackTracker` (`trackers` pkg) | 0.30.0 |
| `supervision.keypoint` | `supervision.key_points` | 0.30.0 |
| `sv.LMM` / `Detections.from_lmm` | `sv.VLM` / `Detections.from_vlm` | 0.31.0 |
| `rle_to_mask`/`mask_to_rle` in `dataset.utils` | `detection.utils.converters` | 0.30.0 |
| `normalized_xyxy=` in `denormalize_boxes` | `xyxy=` | 0.30.0 |
| `create_tiles`, `ensure_cv2_image_for_processing` | (n/a) | 0.31.0 |
| Public `validate_*` helpers | private `_validate_*` | 0.31.0 |

---

## 8. Module Reference

| Module | Key exports | Responsibility |
|--------|-------------|----------------|
| `detection.core` | `Detections` | Central container + all detection connectors |
| `detection.compact_mask` | `CompactMask` | RLE masks for memory efficiency |
| `detection.line_zone` | `LineZone`, `LineZoneAnnotator`, `LineZoneAnnotatorMulticlass` | Line-crossing counts |
| `detection.vlm` | `VLM`, `LMM`(dep) | VLM output parsers |
| `detection.tools.inference_slicer` | `InferenceSlicer` | SAHI tiled inference |
| `detection.tools.polygon_zone` | `PolygonZone`, `PolygonZoneAnnotator` | Region membership/counting |
| `detection.tools.smoother` | `DetectionsSmoother` | Temporal box smoothing |
| `detection.tools.csv_sink` / `json_sink` | `CSVSink`, `JSONSink` | Streaming export |
| `detection.utils.*` | `box_iou_batch`, `mask_non_max_suppression`, converters | Numeric primitives |
| `annotators.core` | 24 annotator classes | Visualization |
| `annotators.utils` | `ColorLookup`, hex helpers | Color mapping |
| `tracker.byte_tracker.core` | `ByteTrack` | Multi-object tracking |
| `dataset.core` | `DetectionDataset`, `ClassificationDataset`, `BaseDataset` | Dataset I/O |
| `dataset.formats.*` | coco/yolo/pascal_voc readers & writers | Format adapters |
| `geometry.core` | `Point`, `Vector`, `Rect`, `Position` | Geometry primitives |
| `draw.color` | `Color`, `ColorPalette` | Color model |
| `draw.utils` | `draw_text`, `draw_polygon`, … | Drawing primitives |
| `metrics.*` | `MeanAveragePrecision`, `MeanAverageRecall`, `F1Score`, `Precision`, `Recall`, `ConfusionMatrix`, `MetricTarget`, `AveragingMethod` | Evaluation |
| `key_points.core` | `KeyPoints` | Keypoint container |
| `key_points.annotators` | `EdgeAnnotator`, `VertexAnnotator`, `VertexLabelAnnotator`, `VertexEllipseAnnotator` | Pose visualization |
| `classification.core` | `Classifications` | Whole-image classification |
| `utils.video` | `VideoInfo`, `VideoSink`, `process_video`, `get_video_frames_generator`, `FPSMonitor` | Video I/O |
| `utils.image` | `crop_image`, `resize_image`, `letterbox_image`, `ImageSink`, … | Image ops |
| `assets` | `download_assets`, `VideoAssets` | Sample media |

### Metrics API shape

All metric classes follow a uniform lifecycle: construct with a `MetricTarget`
(`BOXES`, `MASKS`, `ORIENTED_BOUNDING_BOXES`), accumulate with
`update(predictions, targets)`, then `compute()` to get a result object that supports
`print()` and `.plot()`. `ConfusionMatrix` additionally offers `from_detections(...)`
and `from_tensors(...)` constructors and a `benchmark(...)` helper. `AveragingMethod`
(`MACRO`, `MICRO`, `WEIGHTED`) controls per-class aggregation for F1/Precision/Recall.

---

## 9. Testing & Contributing

The test suite mirrors `src/` under `tests/` (annotators, detection, detection/utils,
metrics, tracker, dataset, geometry, draw, key_points, utils, classification) and is
the largest part of the repository by volume — a signal of the project's maturity.
Doctests embedded in docstrings run as part of `pytest` (`--doctest-modules`), so every
`>>>` example in the source is verified on each run.

To contribute:

```bash
git clone -b develop https://github.com/roboflow/supervision.git
cd supervision
uv venv && source .venv/bin/activate
uv pip install -r pyproject.toml -e . --all-extras
pre-commit install
pytest
```

- [x] Branch from `develop` (not `main`).
- [x] Add a `from_*` classmethod to support a new model — the established extension point.
- [x] Keep public symbols in `__init__.__all__`; everything else stays private.
- [x] Run `pre-commit` (ruff, mypy strict, codespell) and `pytest` before opening a PR.
- [x] Add doctest-able examples to new public methods.

---

## 10. Known Issues & Improvement Areas

Drawn from the project's open issue tracker at the analyzed commit; useful for risk
assessment and contribution targeting.

| Area | Issue | Notes |
|------|-------|-------|
| COCO export | [#2285](https://github.com/roboflow/supervision/issues/2285) | `as_coco()` may drop segmentation masks |
| YOLO OBB export | [#2288](https://github.com/roboflow/supervision/issues/2288) | `as_yolo` can silently drop OBB rotation |
| OBB metrics | [#1679](https://github.com/roboflow/supervision/issues/1679), [#1760](https://github.com/roboflow/supervision/issues/1760) | OBB + SAHI small-object path; ConfusionMatrix OBB support requested |
| COCO IDs | [#768](https://github.com/roboflow/supervision/issues/768) | Annotation ID collisions across dataset splits |
| Tracking | [#1545](https://github.com/roboflow/supervision/issues/1545) | ReID — being addressed in the external `trackers` package |

**Strategic direction:** tracking is migrating out of the core into the dedicated
`trackers` package; VLM support is consolidating under the `VLM` enum; and the keypoint
module is unifying under `key_points`. Align new work with these directions rather than
extending the deprecated paths.

---

## 11. Quick Reference

### Architecture in one sentence

A unified `Detections` dataclass + a matrix of `from_*` model connectors, consumed by
stateless annotator/filter/zone/metric modules and a few stateful video-history objects
(tracker, smoother, line zone, trace/heatmap annotators).

### Strategy enums

| Enum | Values |
|------|--------|
| `Position` | CENTER, CENTER_LEFT/RIGHT, TOP_/BOTTOM_ LEFT/CENTER/RIGHT, CENTER_OF_MASS |
| `ColorLookup` | INDEX, CLASS, TRACK |
| `OverlapFilter` | NON_MAX_SUPPRESSION, NON_MAX_MERGE, NONE |
| `OverlapMetric` | IOU, IOS |
| `MetricTarget` | BOXES, MASKS, ORIENTED_BOUNDING_BOXES |
| `AveragingMethod` | MACRO, MICRO, WEIGHTED |
| `VLM` | PALIGEMMA, FLORENCE_2, QWEN_2_5_VL, QWEN_3_VL, GOOGLE_GEMINI_2_0/2_5, MOONDREAM, DEEPSEEK_VL_2 |

### Operational checklist

- [ ] Pin `supervision==<version>`; avoid `*.dev` builds in production.
- [ ] Install `[metrics]` only if computing tabular metrics.
- [ ] Set `ByteTrack(frame_rate=video_info.fps)` for correct buffering.
- [ ] Migrate off `sv.ByteTrack` → `trackers.ByteTrackTracker` before 0.30.0.
- [ ] Use `compact_masks=True` for high-resolution segmentation slicing.
- [ ] Treat all dataset/media inputs as untrusted; keep OpenCV/Pillow patched.

---

_Generated June 2026 from roboflow/supervision at b18a30e. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._