# Ultralytics YOLO Administrator & Developer Guide

Ultralytics YOLO (v8.4.38) is a unified Python framework for state-of-the-art computer vision: object detection, instance segmentation, pose estimation, image classification, oriented bounding boxes (OBB), and real-time tracking. The package exposes a single `YOLO` model class that transparently handles training, validation, prediction, export, tracking, benchmarking, and hyperparameter tuning across 18+ deployment formats and every major hardware target from cloud GPUs to edge TPUs.

This guide is the authoritative technical reference for administrators, DevOps engineers, ML engineers, and contributors who need to deploy, scale, secure, extend, or debug the Ultralytics YOLO stack. See the User Guide for basic usage examples and getting-started walkthroughs.

---

## 1. Architecture Overview

### 1.1 What the System Does

Ultralytics YOLO provides an end-to-end pipeline from raw images to deployed inference. A single `Model` object (instantiated as `YOLO("yolo11n.pt")`) lazy-loads the correct task map -- detection, segmentation, classification, pose, OBB, or world/YOLOE -- and delegates to the appropriate `Trainer`, `Predictor`, `Validator`, or `Exporter`. The framework handles dataset preparation, augmentation, distributed training across multiple GPUs, model export to any target format, and real-time inference with integrated object tracking.

The architecture follows a plugin-style pattern: each vision task registers a `task_map` dictionary mapping mode strings (`"train"`, `"predict"`, `"val"`, `"export"`) to concrete class implementations. This lets the same `model.train()` call dispatch to `DetectionTrainer`, `SegmentationTrainer`, `PoseTrainer`, etc. without the caller knowing the internals.

Beyond core model operations, the `solutions/` package provides 17+ pre-built computer vision applications (object counting, speed estimation, parking management, security alarms, heatmaps, analytics dashboards) that compose YOLO predictions with domain logic and visualization.

### 1.2 Package Structure Diagram

```
ultralytics/                          Python package root
|
+-- __init__.py                       Version (8.4.38), lazy imports, OMP_NUM_THREADS=1
|
+-- engine/                           Core engine (orchestration layer)
|   +-- model.py                      Model class -- unified entry point
|   +-- trainer.py                    BaseTrainer -- training loop, DDP, EMA, NaN recovery
|   +-- predictor.py                  BasePredictor -- streaming inference pipeline
|   +-- validator.py                  BaseValidator -- validation with metrics
|   +-- exporter.py                   Exporter -- 18+ format export, NMSModel wrapper
|   +-- results.py                    Results, Boxes, Masks, Keypoints, Probs, OBB
|   +-- tuner.py                      Tuner -- Ray Tune hyperparameter optimization
|
+-- cfg/                              Configuration system
|   +-- __init__.py                   CLI entrypoint(), config validation, MODES/TASKS
|   +-- default.yaml                  Master default settings
|   +-- datasets/                     40+ dataset YAML configs
|   +-- models/                       Model architecture YAMLs (v3-v12, v26, RT-DETR)
|   +-- trackers/                     bytetrack.yaml, botsort.yaml
|
+-- nn/                               Neural network definitions
|   +-- tasks.py                      All model architectures (Detection, Seg, Pose, OBB, ...)
|   +-- autobackend.py               AutoBackend -- unified multi-format inference
|   +-- text_model.py                TextModel, CLIP, MobileCLIP (YOLO-World/YOLOE)
|   +-- modules/                      Building blocks (conv, block, head, transformer, ...)
|   +-- backends/                     Per-format inference implementations
|
+-- data/                             Data loading and augmentation
|   +-- augment.py                    Mosaic, MixUp, CutMix, CopyPaste, Albumentations, ...
|   +-- base.py                       BaseDataset -- caching (RAM/disk), rectangle training
|   +-- dataset.py                    YOLODataset, YOLOMultiModalDataset, GroundingDataset, ...
|   +-- build.py                      InfiniteDataLoader, dataset/dataloader builders
|   +-- loaders.py                    LoadStreams, LoadScreenshots, LoadImagesAndVideos, ...
|   +-- converter.py                  COCO->YOLO, DOTA->YOLO format converters
|   +-- split.py                      Dataset splitting utilities
|   +-- annotator.py                  Auto-annotation with SAM
|   +-- utils.py                      HUBDatasetStats, dataset checking
|
+-- models/                           Task-specific model implementations
|   +-- yolo/                         YOLO variants (detect, segment, classify, pose, obb,
|   |                                 world, yoloe)
|   +-- sam/                          SAM, SAM2, SAM3 (segment anything)
|   +-- rtdetr/                       RT-DETR (real-time detection transformer)
|   +-- fastsam/                      FastSAM
|   +-- nas/                          YOLO-NAS
|   +-- utils/                        DETRLoss, HungarianMatcher
|
+-- solutions/                        Pre-built CV applications (17+ solutions)
|   +-- solutions.py                  BaseSolution, SolutionAnnotator, SolutionResults
|   +-- config.py                     SolutionConfig validation
|   +-- object_counter.py, heatmap.py, speed_estimator.py, ...
|
+-- trackers/                         Object tracking
|   +-- byte_tracker.py              BYTETracker (STrack)
|   +-- bot_sort.py                  BOTSORT (BOTrack, ReID)
|   +-- basetrack.py                 TrackState, BaseTrack
|   +-- track.py                     Integration with predict pipeline
|   +-- utils/                        GMC, Kalman filters, matching
|
+-- hub/                              Ultralytics HUB integration
|   +-- auth.py                      Auth (API key, browser, cookie)
|   +-- session.py                   HUBTrainingSession
|   +-- utils.py                     Request helpers
|
+-- utils/                            Utilities (26 files)
|   +-- __init__.py                   SimpleClass, SettingsManager, platform detection
|   +-- torch_utils.py               select_device(), ModelEMA, EarlyStopping, DDP
|   +-- loss.py                      All loss functions (v8Detection, Seg, Pose, OBB, ...)
|   +-- metrics.py                   ConfusionMatrix, DetMetrics, SegmentMetrics, ...
|   +-- ops.py                       NMS, coordinate transforms, Profile
|   +-- plotting.py                  Annotator, Colors
|   +-- checks.py                    Version/requirement/CUDA checks
|   +-- downloads.py                 Safe download with retry
|   +-- benchmarks.py               ProfileModels, RF100Benchmark
|   +-- autodevice.py               GPUInfo -- GPU monitoring, idle selection
|   +-- autobatch.py                Automatic batch size estimation
|   +-- nms.py                      TorchNMS -- fast NMS, batched NMS
|   +-- tal.py                      TaskAlignedAssigner, RotatedTaskAlignedAssigner
|   +-- callbacks/                    ClearML, Comet, DVC, MLflow, Neptune, Ray, TB, W&B, HUB
|   +-- export/                      Per-format export utilities
|
+-- optim/                            Custom optimizers
|   +-- muon.py                      MuSGD, Muon (Newton-Schulz orthogonalization)
|
docker/                               14 Dockerfiles (GPU, CPU, ARM64, Jetson, ...)
tests/                                Test suite (CLI, CUDA, engine, exports, integrations, ...)
```

### 1.3 Data Flow: Training Pipeline

```
                                    +-----------------+
                                    |   User Code     |
                                    | model.train()   |
                                    +--------+--------+
                                             |
                                    +--------v--------+
                                    |  engine/model.py |
                                    |  Model.train()   |
                                    |  task_map lookup |
                                    +--------+--------+
                                             |
                                    +--------v--------+
                                    | engine/trainer.py|
                                    | BaseTrainer      |
                                    | ._do_train()     |
                                    +--------+--------+
                                             |
                    +------------------------+------------------------+
                    |                        |                        |
           +--------v--------+     +--------v--------+     +--------v--------+
           | data/build.py   |     | nn/tasks.py     |     | utils/loss.py   |
           | build_dataloader|     | DetectionModel  |     | v8DetectionLoss |
           | build_dataset   |     | forward()       |     | compute loss    |
           +--------+--------+     +-----------------+     +-----------------+
                    |
           +--------v--------+
           | data/augment.py |
           | Mosaic, MixUp   |
           | LetterBox, HSV  |
           +-----------------+
```

### 1.4 Data Flow: Prediction Pipeline

```
  Input Source                  Engine                        Output
  +-----------+          +------------------+          +------------------+
  | Image     |          | engine/          |          | engine/results.py|
  | Video     +--------->| predictor.py     +--------->| Results          |
  | Stream    |          | BasePredictor    |          |   .boxes         |
  | Directory |          |   .stream_       |          |   .masks         |
  | URL       |          |    inference()   |          |   .keypoints     |
  | Tensor    |          +---------+--------+          |   .probs         |
  | PIL Image |                    |                   |   .obb           |
  | NumPy     |          +---------v--------+          |   .plot()        |
  | Screenshot|          | nn/autobackend.py|          |   .to_json()     |
  +-----------+          | AutoBackend      |          +------------------+
                         | (18+ formats)    |
                         +------------------+
```

### 1.5 Data Flow: Export Pipeline

```
  model.export(format="onnx")
        |
        v
  engine/exporter.py
  Exporter.__call__()
        |
        +-- Validates args, checks format
        +-- Calls format-specific export method
        |     (e.g., export_onnx, export_engine, export_openvino, ...)
        +-- Optional NMS embedding (NMSModel wrapper)
        +-- Metadata embedding (description, author, task, stride, names)
        v
  Exported model file (.onnx, .engine, .tflite, ...)
```

### 1.6 Component Inventory

| Component | Location | Lines (approx.) | Purpose |
|---|---|---|---|
| Model | `engine/model.py` | 1,200 | Unified API entry point |
| BaseTrainer | `engine/trainer.py` | 1,400 | Training orchestration |
| BasePredictor | `engine/predictor.py` | 600 | Inference pipeline |
| BaseValidator | `engine/validator.py` | 500 | Validation loop |
| Exporter | `engine/exporter.py` | 1,800 | Multi-format export |
| Results | `engine/results.py` | 1,000 | Result containers |
| AutoBackend | `nn/autobackend.py` | 1,200 | Multi-format inference |
| Model architectures | `nn/tasks.py` | 2,000 | All neural network models |
| Augmentations | `data/augment.py` | 1,500 | Data augmentation pipeline |
| Losses | `utils/loss.py` | 1,200 | All loss functions |
| Metrics | `utils/metrics.py` | 1,200 | Evaluation metrics |
| Operations | `utils/ops.py` | 800 | NMS, coordinate transforms |
| CLI/Config | `cfg/__init__.py` | 1,000 | CLI parsing, validation |
| Solutions | `solutions/` (20 files) | 3,000 | Pre-built applications |
| Trackers | `trackers/` | 1,500 | ByteTrack, BoT-SORT |

### 1.7 Technology Stack

| Layer | Technology |
|---|---|
| Language | Python >=3.8 |
| Deep Learning | PyTorch >=1.8 |
| GPU Acceleration | CUDA (NVIDIA), MPS (Apple Silicon) |
| Data Loading | PyTorch DataLoader, OpenCV, Pillow |
| Augmentation | Custom + Albumentations (optional) |
| Export Formats | ONNX, TensorRT, OpenVINO, CoreML, TFLite, PaddlePaddle, NCNN, MNN, RKNN, ExecuTorch, Axelera, Edge TPU, TF.js |
| Tracking | ByteTrack, BoT-SORT (SciPy, NumPy) |
| Experiment Tracking | TensorBoard, W&B, MLflow, ClearML, Comet, Neptune, DVC |
| Hyperparameter Tuning | Ray Tune |
| Visualization | Matplotlib, OpenCV |
| Web UI | Streamlit (Inference solution) |
| Hub | Ultralytics HUB (cloud training/deployment) |

> [!NOTE]
> The entire framework is built on a lazy-import pattern. Model classes (`YOLO`, `SAM`, `SAM2`, `RTDETR`, etc.) are not imported at package load time. Instead, `ultralytics/__init__.py` defines `__getattr__` to import them on first access. This keeps `import ultralytics` fast (~100ms) even though the package has extensive dependencies.

---

## 2. System Requirements

### 2.1 Hardware Requirements

| Component | Minimum | Recommended | Notes |
|---|---|---|---|
| CPU | 4 cores | 8+ cores | Data loading is CPU-bound; more cores = faster DataLoader |
| RAM | 8 GB | 32+ GB | Image caching (`cache="ram"`) can consume significant memory |
| GPU (training) | NVIDIA GPU, 4 GB VRAM | NVIDIA GPU, 8+ GB VRAM | Required for practical training speed |
| GPU (inference) | Optional (CPU works) | NVIDIA GPU or Apple M-series | TensorRT requires NVIDIA GPU |
| Disk | 10 GB free | 50+ GB | Datasets + model checkpoints + exported models |

**GPU memory by model size (typical, 640px images):**

| Model | Params | Training (bs=16) | Inference |
|---|---|---|---|
| YOLOv8n / YOLO11n | ~3M | ~4 GB VRAM | ~1 GB |
| YOLOv8s / YOLO11s | ~11M | ~6 GB VRAM | ~2 GB |
| YOLOv8m / YOLO11m | ~26M | ~8 GB VRAM | ~3 GB |
| YOLOv8l / YOLO11l | ~44M | ~12 GB VRAM | ~4 GB |
| YOLOv8x / YOLO11x | ~68M | ~16 GB VRAM | ~6 GB |

### 2.2 Software Requirements

| Software | Version | Required | Notes |
|---|---|---|---|
| Python | >=3.8 | Yes | 3.10+ recommended |
| PyTorch | >=1.8 | Yes | 2.0+ recommended for `torch.compile` |
| CUDA | 11.x or 12.x | For GPU | Must match PyTorch build |
| cuDNN | >=8.0 | For GPU | Bundled with CUDA toolkit |
| TensorRT | >=8.0 | For `.engine` export | NVIDIA only |
| OpenVINO | >=2023.0 | For OpenVINO export | Intel hardware |
| CoreML Tools | >=7.0 | For CoreML export | macOS / iOS |
| ONNX Runtime | >=1.15 | For ONNX inference | CPU or GPU |

**Python dependencies (core):**
- `torch`, `torchvision` -- deep learning framework
- `opencv-python` -- image/video I/O
- `numpy` -- array operations
- `pillow` -- image processing
- `pyyaml` -- YAML config parsing
- `requests` -- HTTP downloads
- `tqdm` -- progress bars
- `matplotlib` -- plotting
- `pandas` -- data export (results.to_df())
- `scipy` -- tracking (linear assignment)
- `seaborn` -- confusion matrix plotting
- `psutil` -- system resource monitoring
- `py-cpuinfo` -- CPU detection

### 2.3 Platform Support

| Platform | Training | Inference | Export | Notes |
|---|---|---|---|---|
| Linux x86_64 | Full | Full | Full | Primary platform |
| Linux ARM64 | Full | Full | Most | Jetson, Raspberry Pi |
| macOS x86_64 | Full | Full | Most | No TensorRT |
| macOS ARM64 (M-series) | Full (MPS) | Full (MPS) | Most | MPS acceleration |
| Windows x86_64 | Full | Full | Most | DDP requires `if __name__ == "__main__"` guard |
| Google Colab | Full | Full | Most | Free GPU tier available |
| Kaggle | Full | Full | Most | Free GPU tier available |

### 2.4 Docker Images

Ultralytics provides 14 purpose-built Docker images:

| Dockerfile | Base | Purpose | GPU |
|---|---|---|---|
| `Dockerfile` | `nvidia/cuda:12.x` | Standard GPU training/inference | Yes |
| `Dockerfile-cpu` | `ubuntu` | CPU-only inference | No |
| `Dockerfile-arm64` | `arm64v8/ubuntu` | ARM64 servers | No |
| `Dockerfile-conda` | `continuumio/miniconda3` | Conda environments | Yes |
| `Dockerfile-jupyter` | `nvidia/cuda:12.x` | JupyterLab with GPU | Yes |
| `Dockerfile-export` | `nvidia/cuda:12.x` | Export-optimized (all export deps) | Yes |
| `Dockerfile-jetson-jetpack4` | `nvcr.io/nvidia/l4t-pytorch` | Jetson Nano/TX2 (JetPack 4) | Yes |
| `Dockerfile-jetson-jetpack5` | `nvcr.io/nvidia/l4t-pytorch` | Jetson Xavier/Orin (JetPack 5) | Yes |
| `Dockerfile-jetson-jetpack6` | `nvcr.io/nvidia/l4t-pytorch` | Jetson Orin (JetPack 6) | Yes |
| `Dockerfile-nvidia-arm64` | `nvcr.io/nvidia/pytorch` | NVIDIA ARM64 | Yes |
| `Dockerfile-runner` | `ubuntu` | CI runner (CPU) | No |
| `Dockerfile-runner-cuda` | `nvidia/cuda` | CI runner (GPU) | Yes |
| `Dockerfile-runner-arm64` | `arm64v8/ubuntu` | CI runner (ARM64) | No |
| `Dockerfile-runner-jetson-jetpack6` | `nvcr.io/nvidia/l4t-pytorch` | CI runner (Jetson) | Yes |

---

## 3. Configuration Reference

### 3.1 Master Default Configuration (default.yaml)

All training, prediction, validation, and export settings are defined in `ultralytics/cfg/default.yaml` and validated by `cfg/__init__.py`. Settings can be overridden via CLI arguments, Python kwargs, or YAML files.

> [!IMPORTANT]
> Configuration validation is strict. Passing a float where an int is expected, a value outside [0.0, 1.0] for fraction keys, or an unrecognized key will raise a clear error. This prevents silent misconfigurations that could waste hours of GPU time.

#### 3.1.1 Training Configuration Keys

**String keys:**

| Key | Default | Description |
|---|---|---|
| `task` | `"detect"` | Vision task: `detect`, `segment`, `classify`, `pose`, `obb` |
| `mode` | `"train"` | Operation mode: `train`, `val`, `predict`, `export`, `track`, `benchmark` |
| `model` | `None` | Model file path (.pt, .yaml, or pretrained name) |
| `data` | `None` | Dataset config file path (.yaml) |
| `project` | `None` | Project directory name |
| `name` | `None` | Experiment name |
| `device` | `None` | Device: `""` (auto), `"cpu"`, `"0"`, `"0,1,2,3"`, `"mps"` |
| `optimizer` | `"auto"` | Optimizer: `SGD`, `Adam`, `AdamW`, `NAdam`, `RAdam`, `RMSProp`, `auto` |
| `resume` | `False` | Resume from last checkpoint (bool or path string) |

**Integer keys:**

| Key | Default | Description |
|---|---|---|
| `epochs` | `100` | Number of training epochs |
| `patience` | `100` | EarlyStopping patience (epochs without improvement) |
| `batch` | `16` | Batch size (-1 for auto-batch) |
| `imgsz` | `640` | Input image size (pixels) |
| `save_period` | `-1` | Save checkpoint every N epochs (-1 = disabled) |
| `workers` | `8` | DataLoader worker processes |
| `seed` | `0` | Random seed for reproducibility |
| `close_mosaic` | `10` | Disable mosaic augmentation for last N epochs |
| `nbs` | `64` | Nominal batch size for loss normalization |

**Float keys:**

| Key | Default | Description |
|---|---|---|
| `lr0` | `0.01` | Initial learning rate |
| `lrf` | `0.01` | Final learning rate factor (lr0 * lrf) |
| `momentum` | `0.937` | SGD momentum / Adam beta1 |
| `weight_decay` | `0.0005` | L2 regularization |
| `warmup_epochs` | `3.0` | Warmup epochs (can be fractional) |
| `warmup_momentum` | `0.8` | Warmup initial momentum |
| `warmup_bias_lr` | `0.1` | Warmup initial bias learning rate |
| `box` | `7.5` | Box loss weight |
| `cls` | `0.5` | Classification loss weight |
| `dfl` | `1.5` | Distribution focal loss weight |
| `pose` | `12.0` | Pose loss weight |
| `kobj` | `1.0` | Keypoint objectness loss weight |
| `label_smoothing` | `0.0` | Label smoothing epsilon |
| `conf` | `0.25` | Prediction confidence threshold |
| `iou` | `0.7` | NMS IoU threshold |
| `max_det` | `300` | Maximum detections per image |
| `line_width` | `None` | Bounding box line width (auto-scaled if None) |

**Fraction keys (must be 0.0-1.0):**

| Key | Default | Description |
|---|---|---|
| `dropout` | `0.0` | Dropout rate (classification only) |
| `fraction` | `1.0` | Fraction of dataset to use |

**Boolean keys:**

| Key | Default | Description |
|---|---|---|
| `save` | `True` | Save checkpoints and predictions |
| `save_txt` | `False` | Save results as .txt files |
| `save_conf` | `False` | Include confidence in saved .txt |
| `save_crop` | `False` | Save cropped detections |
| `save_json` | `False` | Save results as .json |
| `save_frames` | `False` | Save individual video frames |
| `show` | `False` | Display results in window |
| `show_labels` | `True` | Show class labels on plots |
| `show_conf` | `True` | Show confidence on plots |
| `show_boxes` | `True` | Show bounding boxes on plots |
| `verbose` | `True` | Verbose output |
| `deterministic` | `True` | Enable deterministic mode |
| `single_cls` | `False` | Treat all classes as one class |
| `rect` | `False` | Rectangular training/inference |
| `cos_lr` | `False` | Cosine learning rate scheduler |
| `overlap_mask` | `True` | Overlapping masks in segmentation |
| `retina_masks` | `False` | High-resolution segmentation masks |
| `plots` | `True` | Generate training plots |
| `amp` | `True` | Automatic mixed precision |
| `multi_scale` | `False` | Multi-scale training (vary imgsz +/- 50%) |
| `cache` | `False` | Dataset caching: `True`/`"ram"`, `"disk"`, or `False` |
| `exist_ok` | `False` | Allow overwriting existing project/name |
| `pretrained` | `True` | Use pretrained weights |
| `half` | `False` | FP16 inference |
| `int8` | `False` | INT8 quantization |
| `dynamic` | `False` | Dynamic input shapes (ONNX/TensorRT) |
| `simplify` | `True` | Simplify ONNX model |
| `nms` | `False` | Embed NMS in exported model |
| `profile` | `False` | Profile ONNX/TensorRT speed |
| `augment` | `False` | Test-time augmentation |
| `agnostic_nms` | `False` | Class-agnostic NMS |
| `visualize` | `False` | Visualize model features |

**List/None keys:**

| Key | Default | Description |
|---|---|---|
| `classes` | `None` | Filter predictions by class index (list of ints) |
| `embed` | `None` | Extract embeddings from specified layers (list of ints) |

#### 3.1.2 Augmentation Configuration Keys

| Key | Default | Type | Description |
|---|---|---|---|
| `hsv_h` | `0.015` | fraction | HSV hue augmentation range |
| `hsv_s` | `0.7` | fraction | HSV saturation augmentation range |
| `hsv_v` | `0.4` | fraction | HSV value (brightness) augmentation range |
| `degrees` | `0.0` | float | Rotation range (+/- degrees) |
| `translate` | `0.1` | fraction | Translation range (+/- fraction) |
| `scale` | `0.5` | float | Scale range (+/- gain) |
| `shear` | `0.0` | float | Shear range (+/- degrees) |
| `perspective` | `0.0` | float | Perspective transform range |
| `flipud` | `0.0` | fraction | Vertical flip probability |
| `fliplr` | `0.5` | fraction | Horizontal flip probability |
| `bgr` | `0.0` | fraction | BGR channel swap probability |
| `mosaic` | `1.0` | fraction | Mosaic augmentation probability |
| `mixup` | `0.0` | fraction | MixUp augmentation probability |
| `cutmix` | `0.0` | fraction | CutMix augmentation probability |
| `copy_paste` | `0.0` | fraction | Copy-Paste augmentation probability |
| `copy_paste_mode` | `"flip"` | string | Copy-paste mode: `"flip"` or `"mixup"` |
| `auto_augment` | `"randaugment"` | string | Auto augment policy: `"randaugment"`, `"augmix"`, `"autoaugment"` |
| `erasing` | `0.4` | fraction | Random erasing probability (classification) |
| `crop_fraction` | `1.0` | fraction | Classification crop fraction |

### 3.2 Dataset YAML Format

Dataset configuration files define data paths and class names:

```yaml
# Example: coco8.yaml
path: ../datasets/coco8  # dataset root (relative to datasets_dir or absolute)
train: images/train      # train images (relative to path)
val: images/val          # val images (relative to path)
test:                    # test images (optional)

# Class names
names:
  0: person
  1: bicycle
  2: car
  3: motorcycle
  # ... etc

# Optional: download script
download: https://github.com/ultralytics/assets/releases/download/v0.0.0/coco8.zip
```

**Keypoint dataset format (additional fields):**

```yaml
kpt_shape: [17, 3]    # [num_keypoints, dimensions] (2=xy, 3=xy+visibility)
flip_idx: [0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15]
```

**Segmentation dataset labels:** Labels use polygon format (space-separated `class x1 y1 x2 y2 ... xN yN`) with normalized coordinates.

**OBB dataset labels:** Labels use rotated bounding box format (`class x1 y1 x2 y2 x3 y3 x4 y4`).

### 3.3 Model Architecture YAML Format

Model YAML files define the network architecture:

```yaml
# Example: yolo11n.yaml
nc: 80         # number of classes
scales:
  n: [0.50, 0.25, 1024]   # [depth_multiple, width_multiple, max_channels]
  s: [0.50, 0.50, 1024]
  m: [0.50, 1.00, 512]
  l: [1.00, 1.00, 512]
  x: [1.00, 1.50, 512]

backbone:
  - [-1, 1, Conv, [64, 3, 2]]         # [from, repeats, module, args]
  - [-1, 1, Conv, [128, 3, 2]]
  - [-1, 2, C3k2, [256, False, 0.25]]
  # ... more layers

head:
  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 6], 1, Concat, [1]]
  - [-1, 2, C3k2, [512, False, 0.25]]
  # ... more layers
  - [[15, 18, 21], 1, Detect, [nc]]   # Detect head with feature maps
```

**Architecture notation:**
- `from`: Input layer index (-1 = previous, -2 = two back, or list for concatenation)
- `repeats`: Number of times to repeat the module (scaled by depth_multiple)
- `module`: Module class name from `nn/modules/`
- `args`: Constructor arguments (channels scaled by width_multiple)

### 3.4 Tracker Configuration

**ByteTrack (`cfg/trackers/bytetrack.yaml`):**

```yaml
tracker_type: bytetrack
track_high_thresh: 0.5     # High confidence detection threshold
track_low_thresh: 0.1      # Low confidence detection threshold
new_track_thresh: 0.6      # New track initialization threshold
track_buffer: 30           # Frames to keep lost tracks
match_thresh: 0.8          # Matching threshold (IoU)
fuse_score: True           # Fuse confidence score with IoU
```

**BoT-SORT (`cfg/trackers/botsort.yaml`):**

```yaml
tracker_type: botsort
track_high_thresh: 0.5
track_low_thresh: 0.1
new_track_thresh: 0.6
track_buffer: 30
match_thresh: 0.8
fuse_score: True
gmc_method: sparseOptFlow  # GMC method: sparseOptFlow, orb, ecc, none
proximity_thresh: 0.5      # ReID proximity threshold
appearance_thresh: 0.25    # ReID appearance threshold
with_reid: False           # Enable ReID (requires model)
```

### 3.5 Settings File

The global settings file lives at `~/.config/Ultralytics/settings.json` (or platform equivalent) and is managed by `SettingsManager`:

```json
{
  "settings_version": "0.0.6",
  "datasets_dir": "/path/to/datasets",
  "weights_dir": "/path/to/weights",
  "runs_dir": "/path/to/runs",
  "uuid": "auto-generated-uuid",
  "sync": true,
  "api_key": "",
  "hub_url": "https://hub.ultralytics.com",
  "openai_api_key": "",
  "vscode_msg": true,
  "dvc": true,
  "clearml": true,
  "comet": true,
  "mlflow": true,
  "neptune": true,
  "raytune": true,
  "tensorboard": true,
  "wandb": true
}
```

**Key settings:**
- `datasets_dir`: Root directory for auto-downloaded datasets (default: `~/datasets`)
- `weights_dir`: Root directory for auto-downloaded weights
- `runs_dir`: Root directory for training output (default: working_dir)
- `sync`: Enable anonymous usage analytics
- `api_key`: Ultralytics HUB API key
- Integration toggles (`clearml`, `comet`, `mlflow`, etc.): Enable/disable individual integrations

**Modifying settings:**

```python
from ultralytics import settings
settings.update({"datasets_dir": "/data/datasets", "wandb": False})
settings.reset()  # Reset to defaults
```

Or via CLI:

```bash
yolo settings datasets_dir=/data/datasets wandb=False
yolo settings reset
```

### 3.6 Environment Variables

| Variable | Purpose | Default |
|---|---|---|
| `OMP_NUM_THREADS` | OpenMP thread count | Set to `"1"` by ultralytics at import |
| `YOLO_VERBOSE` | Enable/disable verbose logging | `"true"` |
| `YOLO_AUTOINSTALL` | Auto-install missing packages | `"true"` |
| `YOLO_CONFIG_DIR` | Config directory path | `~/.config/Ultralytics` |
| `ULTRALYTICS_HUB_API` | HUB API endpoint | `https://api.ultralytics.com` |
| `ULTRALYTICS_HUB_WEB` | HUB web endpoint | `https://hub.ultralytics.com` |
| `CUDA_VISIBLE_DEVICES` | Limit visible GPUs | All GPUs |
| `WORLD_SIZE` | DDP world size | Auto-detected |
| `LOCAL_RANK` | DDP local rank | Auto-detected |
| `RANK` | DDP global rank | Auto-detected |
| `MASTER_ADDR` | DDP master address | `"127.0.0.1"` |
| `MASTER_PORT` | DDP master port | Auto-selected |
| `COMET_API_KEY` | Comet ML API key | None |
| `MLFLOW_TRACKING_URI` | MLflow server URI | `runs/mlflow` (local) |
| `MLFLOW_EXPERIMENT_NAME` | MLflow experiment name | From project name |
| `WANDB_API_KEY` | Weights & Biases API key | None |
| `CLEARML_WEB_HOST` | ClearML web UI URL | None |
| `NEPTUNE_API_TOKEN` | Neptune API token | None |

> [!TIP]
> Set `OMP_NUM_THREADS` before importing ultralytics if you need a different value. The package sets it to `"1"` at import time to prevent CPU over-subscription during multi-GPU training. For CPU-only inference with a single model, a higher value may improve throughput.

### 3.7 Annotated Configuration Walkthroughs

**High-accuracy detection training:**

```yaml
# train_high_accuracy.yaml
task: detect
mode: train
model: yolo11l.pt           # Large model for accuracy
data: coco.yaml
epochs: 300                  # Long training
patience: 50                 # Early stopping
batch: -1                    # Auto-batch (maximize GPU usage)
imgsz: 640
optimizer: auto              # Auto-selects AdamW for >100 epochs
cos_lr: true                 # Cosine learning rate decay
close_mosaic: 20             # Disable mosaic last 20 epochs
amp: true                    # Mixed precision
cache: ram                   # Cache images in RAM for speed
augment: false               # No TTA during validation
plots: true
```

**Fast edge deployment export:**

```yaml
# export_edge.yaml
task: detect
mode: export
model: yolo11n.pt            # Nano model for speed
format: tflite               # TensorFlow Lite
imgsz: 320                   # Smaller images for edge
half: true                   # FP16 quantization
int8: true                   # INT8 quantization (further compression)
nms: true                    # Embed NMS in model
```

**Multi-GPU training:**

```bash
# 4-GPU DDP training
yolo detect train data=coco.yaml model=yolo11x.pt \
  device=0,1,2,3 batch=64 epochs=300
```

---

## 4. Deployment & Scaling

### 4.1 Docker Deployment

**Standard GPU deployment:**

```bash
# Build
docker build -t ultralytics -f docker/Dockerfile .

# Run with GPU access
docker run --gpus all -it --rm \
  -v /data:/data \
  -v /models:/models \
  ultralytics

# Inside container
yolo detect train data=/data/coco.yaml model=yolo11n.pt
```

**CPU-only deployment:**

```bash
docker build -t ultralytics-cpu -f docker/Dockerfile-cpu .
docker run -it --rm -v /data:/data ultralytics-cpu
```

**Jupyter Lab with GPU:**

```bash
docker build -t ultralytics-jupyter -f docker/Dockerfile-jupyter .
docker run --gpus all -p 8888:8888 -it --rm ultralytics-jupyter
```

**Export-optimized (all export dependencies pre-installed):**

```bash
docker build -t ultralytics-export -f docker/Dockerfile-export .
docker run --gpus all -it --rm \
  -v /models:/models \
  ultralytics-export \
  yolo export model=/models/best.pt format=engine half=True
```

### 4.2 Multi-GPU Training (DDP)

Ultralytics uses PyTorch Distributed Data Parallel (DDP) for multi-GPU training. The `BaseTrainer` detects multiple devices and automatically spawns DDP processes.

```python
from ultralytics import YOLO

model = YOLO("yolo11n.pt")
# Automatically uses DDP when multiple devices specified
model.train(data="coco.yaml", device=[0, 1, 2, 3], batch=64)
```

```bash
# CLI equivalent
yolo detect train data=coco.yaml model=yolo11n.pt device=0,1,2,3 batch=64
```

**DDP internals:**
1. `BaseTrainer.__init__()` detects `len(device) > 1`
2. Spawns `torch.multiprocessing` processes via `ddp_cleanup()` / DDP setup
3. Each process gets `LOCAL_RANK`, `RANK`, `WORLD_SIZE` environment variables
4. Model wrapped with `torch.nn.parallel.DistributedDataParallel`
5. `DistributedSampler` ensures each GPU gets unique data
6. Gradient sync happens automatically via DDP
7. Only rank 0 saves checkpoints and logs

> [!NOTE]
> Auto-batch (`batch=-1`) is not supported with multi-GPU DDP training. You must specify an explicit batch size. The error message "AutoBatch with batch<1 not supported for Multi-GPU training" will be raised.

**DDP troubleshooting:**

| Issue | Solution |
|---|---|
| NCCL timeout | Set `NCCL_P2P_DISABLE=1` or increase `NCCL_SOCKET_TIMEOUT` |
| OOM on one GPU | Reduce batch size; DDP divides batch across GPUs |
| Windows DDP fails | Wrap training code in `if __name__ == "__main__":` |
| Port conflict | Set `MASTER_PORT` to an available port |

### 4.3 Edge Deployment

#### NVIDIA Jetson (JetPack 4/5/6)

```bash
# Use appropriate Dockerfile
docker build -t ultralytics-jetson -f docker/Dockerfile-jetson-jetpack6 .

# Export to TensorRT on the Jetson
yolo export model=yolo11n.pt format=engine half=True device=0

# Run inference with TensorRT
yolo detect predict model=yolo11n.engine source=image.jpg
```

**Key considerations:**
- Export on the target device (TensorRT engines are not portable across GPU architectures)
- Use FP16 (`half=True`) for Jetson -- significant speedup with minimal accuracy loss
- INT8 requires calibration data (`int8=True`, `data=calibration.yaml`)

#### Raspberry Pi / ARM64

```bash
# Export to NCNN (optimized for ARM)
yolo export model=yolo11n.pt format=ncnn

# Or TFLite for broader compatibility
yolo export model=yolo11n.pt format=tflite int8=True
```

#### Edge TPU (Google Coral)

```bash
# Export to Edge TPU format (requires edgetpu-compiler)
yolo export model=yolo11n.pt format=edgetpu
```

#### RKNN (Rockchip NPU)

```bash
yolo export model=yolo11n.pt format=rknn name=rk3588
```

#### Axelera AI

```bash
yolo export model=yolo11n.pt format=axelera int8=True data=coco8.yaml
```

#### IMX (NXP i.MX)

```bash
yolo export model=yolo11n.pt format=imx int8=True
```

### 4.4 Cloud Deployment

**AWS SageMaker:**

```python
# Export to ONNX, deploy as SageMaker endpoint
from ultralytics import YOLO

model = YOLO("yolo11n.pt")
model.export(format="onnx", dynamic=True)
# Deploy yolo11n.onnx via SageMaker ONNX Runtime container
```

**Triton Inference Server:**

```python
# Export to TensorRT engine
model.export(format="engine", dynamic=True, half=True, batch=8)
# Place in Triton model repository:
# model_repository/
#   yolo11n/
#     1/
#       model.engine
#     config.pbtxt
```

**ONNX Runtime (universal):**

```python
model.export(format="onnx", dynamic=True, simplify=True)
# Deploy with ONNX Runtime on any platform
# onnxruntime (CPU) or onnxruntime-gpu (NVIDIA)
```

### 4.5 Export Format Reference

| Format | Argument | Suffix | CPU | GPU | Key Args |
|---|---|---|---|---|---|
| PyTorch | - | `.pt` | Yes | Yes | - |
| TorchScript | `torchscript` | `.torchscript` | Yes | Yes | `batch`, `optimize`, `half`, `nms`, `dynamic` |
| ONNX | `onnx` | `.onnx` | Yes | Yes | `batch`, `dynamic`, `half`, `opset`, `simplify`, `nms` |
| OpenVINO | `openvino` | `_openvino_model` | Yes | No | `batch`, `dynamic`, `half`, `int8`, `nms`, `fraction` |
| TensorRT | `engine` | `.engine` | No | Yes | `batch`, `dynamic`, `half`, `int8`, `simplify`, `nms`, `fraction` |
| CoreML | `coreml` | `.mlpackage` | Yes | No | `batch`, `dynamic`, `half`, `int8`, `nms` |
| TF SavedModel | `saved_model` | `_saved_model` | Yes | Yes | `batch`, `int8`, `keras`, `nms` |
| TF GraphDef | `pb` | `.pb` | Yes | Yes | `batch` |
| TFLite | `tflite` | `.tflite` | Yes | No | `batch`, `half`, `int8`, `nms`, `fraction` |
| Edge TPU | `edgetpu` | `_edgetpu.tflite` | Yes | No | - |
| TF.js | `tfjs` | `_web_model` | Yes | No | `batch`, `half`, `int8`, `nms` |
| PaddlePaddle | `paddle` | `_paddle_model` | Yes | Yes | `batch` |
| MNN | `mnn` | `.mnn` | Yes | Yes | `batch`, `half`, `int8` |
| NCNN | `ncnn` | `_ncnn_model` | Yes | Yes | `batch`, `half` |
| IMX | `imx` | `_imx_model` | Yes | Yes | `int8`, `fraction`, `nms` |
| RKNN | `rknn` | `_rknn_model` | No | No | `batch`, `name` |
| ExecuTorch | `executorch` | `_executorch_model` | Yes | No | `batch` |
| Axelera AI | `axelera` | `_axelera_model` | No | No | `batch`, `int8`, `fraction`, `data` |

### 4.6 Model Optimization

**FP16 (Half Precision):**

```python
model.export(format="engine", half=True)  # TensorRT FP16
model.export(format="onnx", half=True)    # ONNX FP16
```

- ~2x inference speedup on modern GPUs
- Minimal accuracy loss (<0.5% mAP typically)
- Supported on all NVIDIA GPUs with compute capability >= 5.3

**INT8 Quantization:**

```python
# Post-training quantization with calibration
model.export(format="engine", int8=True, data="coco.yaml", fraction=0.5)
```

- ~4x inference speedup vs FP32
- 1-3% mAP loss typical
- `fraction` controls how much calibration data to use (0.5 = 50%)
- Calibration data should be representative of deployment data

**NMS Embedding:**

```python
model.export(format="onnx", nms=True)
```

- Wraps model with `NMSModel` that includes NMS post-processing
- Single-pass inference (no separate NMS step needed)
- Simplifies deployment pipelines

---

## 5. Operations

### 5.1 Training Monitoring

#### Built-in Logging

Training automatically logs to the console and generates plots:

```
Epoch  GPU_mem  box_loss  cls_loss  dfl_loss  Instances  Size
1/300    5.2G    1.234    2.345    1.456       42      640
...
```

Output directory (`runs/detect/train/`):
- `results.csv` -- per-epoch metrics
- `results.png` -- training curves plot
- `confusion_matrix.png` -- validation confusion matrix
- `val_batch0_pred.jpg` -- sample validation predictions
- `weights/best.pt` -- best model checkpoint
- `weights/last.pt` -- last epoch checkpoint
- `args.yaml` -- full configuration used

#### TensorBoard

```python
# Enabled by default (settings.tensorboard = True)
model.train(data="coco.yaml", epochs=100)
```

```bash
# View in separate terminal
tensorboard --logdir runs/detect/train
```

#### Weights & Biases

```bash
pip install wandb
wandb login
yolo detect train data=coco.yaml model=yolo11n.pt
# W&B automatically logs metrics, images, model artifacts
```

#### MLflow

```bash
pip install mlflow
export MLFLOW_TRACKING_URI=http://mlflow-server:5000
yolo detect train data=coco.yaml model=yolo11n.pt
```

#### ClearML

```bash
pip install clearml
clearml-init  # Configure server connection
yolo detect train data=coco.yaml model=yolo11n.pt
```

> [!TIP]
> Integration callbacks are loaded automatically when the corresponding package is installed and the settings toggle is enabled. To disable an integration without uninstalling: `yolo settings wandb=False`.

### 5.2 Model Management

#### Checkpointing

```python
# Save every 10 epochs
model.train(data="coco.yaml", save_period=10)

# Resume from checkpoint
model = YOLO("runs/detect/train/weights/last.pt")
model.train(resume=True)
```

**Checkpoint contents (`.pt` file):**
- `model` -- model state dict
- `optimizer` -- optimizer state dict
- `ema` -- EMA model state dict
- `updates` -- EMA update count
- `epoch` -- last completed epoch
- `best_fitness` -- best validation metric
- `train_args` -- full training configuration
- `date` -- training date
- `version` -- ultralytics version

#### Loading Models

```python
from ultralytics import YOLO

# From pretrained weights
model = YOLO("yolo11n.pt")

# From architecture YAML (random weights)
model = YOLO("yolo11n.yaml")

# From YAML + pretrained transfer
model = YOLO("yolo11n.yaml").load("yolo11n.pt")

# From exported format (inference only)
model = YOLO("yolo11n.onnx")
model = YOLO("yolo11n.engine")
model = YOLO("yolo11n_openvino_model")
```

#### NaN Recovery

The trainer automatically handles NaN loss during training:

1. Detects NaN in loss computation
2. Restores model from last valid checkpoint
3. Retries the training step
4. Attempts up to 3 recoveries before failing

This is controlled internally in `BaseTrainer._do_train()` and cannot be configured. If NaN persists after 3 attempts, training terminates with: `"Training failed: NaN persisted for {attempts} epochs"`.

> [!NOTE]
> NaN loss typically indicates a learning rate that is too high, corrupted data, or numerical instability. If NaN recovery triggers, investigate the root cause rather than relying on automatic recovery.

### 5.3 Ultralytics HUB Integration

```python
from ultralytics import HUB

# Login
HUB.login(api_key="your-api-key")
# Or set in settings
# yolo settings api_key=your-api-key

# Start a HUB training session
model = YOLO("https://hub.ultralytics.com/models/MODEL_ID")
model.train()
# Metrics and checkpoints automatically uploaded to HUB
```

**HUB capabilities:**
- Cloud training management
- Model versioning and comparison
- Dataset hosting and annotation
- One-click deployment
- Team collaboration

### 5.4 Batch Processing Patterns

```python
from ultralytics import YOLO

model = YOLO("yolo11n.pt")

# Process directory
results = model.predict(source="path/to/images/", save=True)

# Process video
results = model.predict(source="video.mp4", save=True, stream=True)
for result in results:
    # Process frame-by-frame (streaming, memory-efficient)
    boxes = result.boxes
    masks = result.masks

# Process webcam stream
results = model.predict(source=0, show=True, stream=True)

# Process RTSP stream
results = model.predict(source="rtsp://...", stream=True)
```

> [!TIP]
> Always use `stream=True` for video and stream sources. Without streaming, all frames are loaded into memory before processing, which causes OOM for long videos.

### 5.5 Logging Configuration

```python
import logging

# Ultralytics uses the "ultralytics" logger
logger = logging.getLogger("ultralytics")
logger.setLevel(logging.WARNING)  # Reduce verbosity
```

```bash
# Via environment variable
export YOLO_VERBOSE=false
yolo detect predict model=yolo11n.pt source=image.jpg
```

---

## 6. Performance

### 6.1 Benchmarking

```bash
# Benchmark across all export formats
yolo benchmark model=yolo11n.pt imgsz=640 half=True

# Benchmark specific format
yolo benchmark model=yolo11n.engine imgsz=640
```

```python
from ultralytics.utils.benchmarks import ProfileModels

# Profile and compare multiple models
profiler = ProfileModels(
    ["yolo11n.pt", "yolo11s.pt", "yolo11m.pt"],
    imgsz=640,
    half=True,
    device="0"
)
profiler.profile()
```

**RF100 Benchmark** (Roboflow 100):

```python
from ultralytics.utils.benchmarks import RF100Benchmark

benchmark = RF100Benchmark()
benchmark.setup_rfapi()
benchmark.evaluate(model="yolo11n.pt")
```

### 6.2 Auto-Batch Sizing

```python
# Automatically find maximum batch size for available GPU memory
model.train(data="coco.yaml", batch=-1)
```

**How it works (`utils/autobatch.py`):**
1. Measures baseline GPU memory usage
2. Runs forward passes with increasing batch sizes (1, 2, 4, 8, 16, ...)
3. Fits a linear model to batch_size vs. memory usage
4. Selects the largest batch that uses ~60% of available VRAM (safety margin)
5. Falls back to batch=1 if estimation fails

### 6.3 Rectangle Training

```python
model.train(data="coco.yaml", rect=True)
```

Standard training pads all images to square (`imgsz x imgsz`). Rectangle training groups images by aspect ratio into batches, then pads to the minimum rectangle that fits the batch. This reduces wasted computation on padding pixels, improving throughput by 10-30% for datasets with varied aspect ratios.

**Implementation details (`data/base.py`):**
- Images sorted by aspect ratio
- Batches formed from similar aspect ratios
- Each batch padded to the minimum enclosing rectangle (stride-aligned)
- Effective image size varies per batch

### 6.4 Image Caching

```python
# Cache images in RAM (fastest, highest memory usage)
model.train(data="coco.yaml", cache="ram")

# Cache images on disk (moderate speed, lower memory)
model.train(data="coco.yaml", cache="disk")

# No caching (default -- reads from disk each epoch)
model.train(data="coco.yaml", cache=False)
```

**RAM caching:**
- Stores decoded images as numpy arrays in memory
- Checks available system memory before caching
- Safety margin: requires additional free memory beyond dataset size
- Provides dramatic speedup for I/O-bound training

**Disk caching:**
- Stores decoded images as `.npy` files alongside originals
- First epoch is slower (writing cache), subsequent epochs benefit
- Cache persists across training runs

### 6.5 Multi-Worker Data Loading

```python
model.train(data="coco.yaml", workers=8)
```

The `InfiniteDataLoader` (in `data/build.py`) wraps PyTorch's `DataLoader` to reset workers indefinitely (no epoch boundaries cause worker restarts). This avoids the overhead of worker process creation every epoch.

**Guidelines:**
- Set `workers` to the number of CPU cores available
- On shared systems, use fewer workers to avoid contention
- Docker: ensure `--shm-size` is sufficient (default 64M is too low)
- Recommended: `--shm-size=8g` or higher for multi-worker loading

### 6.6 Mixed Precision Training

```python
# Enabled by default
model.train(data="coco.yaml", amp=True)
```

Uses `torch.amp.autocast` to run forward passes in FP16 while maintaining FP32 master weights. `GradScaler` prevents underflow during backward pass.

**Benefits:**
- ~1.5-2x training speedup on modern GPUs (V100, A100, H100, RTX 30xx+)
- Reduced GPU memory usage (~30-40%)
- No measurable accuracy impact for YOLO models

**When to disable AMP:**
- Debugging numerical issues
- Custom loss functions with FP16 overflow
- Older GPUs without FP16 tensor cores

### 6.7 Muon Optimizer

The `optim/muon.py` module provides `MuSGD` and `Muon` -- novel optimizers combining Nesterov momentum with Newton-Schulz orthogonalization:

```python
model.train(data="coco.yaml", optimizer="auto")
# 'auto' may select MuSGD for certain configurations
```

- `MuSGD`: SGD-like but applies Newton-Schulz orthogonalization to the gradient update
- Orthogonalization helps escape sharp minima, potentially finding flatter (more generalizable) optima
- Primarily a research feature; standard `SGD` and `AdamW` remain the defaults

---

## 7. Security

### 7.1 Safe Model Loading (SafeUnpickler)

> [!WARNING]
> PyTorch model files (`.pt`) use Python's `pickle` serialization, which can execute arbitrary code during deserialization. A malicious `.pt` file could compromise your system simply by loading it.

Ultralytics implements `SafeUnpickler` (in `nn/tasks.py`) as a defense:

```python
class SafeUnpickler(pickle.Unpickler):
    """Custom unpickler that restricts allowed classes during deserialization."""

    def find_class(self, module, name):
        # Only allows specific safe classes:
        # - torch.* (tensors, parameters, etc.)
        # - collections.OrderedDict
        # - numpy.* (arrays)
        # - ultralytics.* (model classes)
        # Raises UnpicklingError for everything else
        ...
```

**What it protects against:**
- Arbitrary code execution via `__reduce__` in pickle
- OS command injection via serialized `os.system` calls
- File system access via malicious deserialization
- Network exfiltration during model loading

**What to do:**
- Only load `.pt` files from trusted sources (Ultralytics official, your own training)
- Verify file hashes when downloading from third parties
- Use exported formats (ONNX, TensorRT, etc.) for untrusted models -- these are safer by design
- The `SafeUnpickler` is always active; there is no flag to disable it

### 7.2 AGPL-3.0 License Implications

> [!CAUTION]
> Ultralytics YOLO is licensed under AGPL-3.0. This has significant implications for commercial deployment:
>
> - Any software that uses ultralytics and is distributed or offered as a network service **must** release its complete source code under AGPL-3.0
> - This applies to server-side inference APIs, SaaS products, and embedded systems
> - Internal use (not distributed, not offered as a service) does not trigger AGPL obligations
> - Ultralytics offers commercial licenses for organizations that cannot comply with AGPL

**Options for commercial use:**
1. Comply with AGPL-3.0 (open-source your entire application)
2. Purchase a commercial license from Ultralytics
3. Use only exported models (ONNX, TensorRT, etc.) with your own inference code that does not import `ultralytics`

### 7.3 Model Provenance

Checkpoint files (`.pt`) embed metadata:

```python
import torch
ckpt = torch.load("best.pt", map_location="cpu")
print(ckpt.get("date"))          # Training date
print(ckpt.get("version"))       # Ultralytics version
print(ckpt.get("train_args"))    # Full training config
```

- Track which data and configuration produced each model
- Store checksums of production models
- Version-control model architecture YAMLs alongside training configs

### 7.4 Data Handling Security

> [!WARNING]
> The `downloads.py` module downloads files from URLs, including Google Drive. Auto-download is enabled for datasets and pretrained weights referenced by name. In air-gapped or security-sensitive environments, disable auto-download and pre-place all assets.

```python
# Disable auto-download by pre-placing all assets:
# 1. Pre-download all model weights to weights_dir
# 2. Pre-download all datasets to datasets_dir
# 3. Use absolute paths in data YAML configs
# 4. Set YOLO_AUTOINSTALL=false to prevent pip auto-installs
```

**Network access points:**
- Model weights: `https://github.com/ultralytics/assets/releases/`
- Datasets: Various sources defined in dataset YAML `download:` fields
- HUB: `https://api.ultralytics.com` (if HUB integration enabled)
- PyPI: Auto-installs missing packages unless `YOLO_AUTOINSTALL=false`
- Analytics: Usage telemetry sent if `settings.sync=True`

---

## 8. Developer Internals

### 8.1 Repository Structure (Complete)

```
ultralytics/
  __init__.py              Package init, version, lazy imports, OMP_NUM_THREADS
  assets/                  Sample images for testing/demos
  cfg/
    __init__.py            CLI entrypoint(), MODES, TASKS, SOLUTION_MAP, config validation
    default.yaml           Master default configuration
    datasets/              40+ dataset configs (coco.yaml, coco8.yaml, VOC.yaml, ...)
    models/                Architecture YAMLs by version:
      v3/                  YOLOv3 architectures
      v5/                  YOLOv5 architectures
      v6/                  YOLOv6 architectures
      v8/                  YOLOv8 architectures
      v9/                  YOLOv9 architectures
      v10/                 YOLOv10 architectures
      11/                  YOLO11 architectures
      12/                  YOLO12 architectures
      26/                  YOLO26 architectures
      rt-detr/             RT-DETR architectures
    trackers/              bytetrack.yaml, botsort.yaml
  data/
    augment.py             Augmentation classes: Compose, BaseMixTransform, Mosaic,
                           MixUp, CutMix, RandomPerspective, RandomHSV, RandomFlip,
                           LetterBox, CopyPaste, Albumentations, classify transforms
    base.py                BaseDataset: label loading, image caching, rect training
    build.py               InfiniteDataLoader, build_yolo_dataset(),
                           build_dataloader(), load_inference_source(), check_source()
    converter.py           convert_coco(), convert_dota_to_yolo(), merge_multi_segment(),
                           convert_segment_masks_to_yolo(), convert_ndjson_to_yolo()
    dataset.py             YOLODataset, YOLOMultiModalDataset, GroundingDataset,
                           ClassificationDataset, SemanticDataset
    loaders.py             LoadStreams, LoadScreenshots, LoadImagesAndVideos,
                           LoadPilAndNumpy, LoadTensor, SourceTypes, LOADERS
    split.py               Dataset splitting (train/val/test)
    annotator.py           auto_annotate() using SAM
    utils.py               HUBDatasetStats, check_det_dataset, check_cls_dataset,
                           compress_one_image, polygons2masks
  engine/
    model.py               Model: train(), val(), predict(), export(), track(),
                           benchmark(), embed(), tune(), task_map, smart_load()
    trainer.py             BaseTrainer: _do_train(), _setup_train(), build_optimizer(),
                           build_dataset(), get_dataloader(), EMA, DDP, NaN recovery
    predictor.py           BasePredictor: __call__(), stream_inference(),
                           setup_source(), preprocess(), postprocess()
    validator.py           BaseValidator: __call__(), get_stats(), check_stats(),
                           get_desc(), metrics property
    exporter.py            Exporter: __call__(), export_torchscript(), export_onnx(),
                           export_openvino(), export_engine(), export_coreml(), ...
                           NMSModel, export_formats()
    results.py             Results: plot(), to_json(), to_csv(), to_df(), to_xml(),
                           summary(), verbose(), save_txt(), save_crop()
                           Boxes, Masks, Keypoints, Probs, OBB
    tuner.py               Tuner: Ray Tune hyperparameter optimization
  hub/
    auth.py                Auth: api_key_auth(), browser_auth(), cookie_auth(),
                           request_api_key(), authenticate()
    session.py             HUBTrainingSession: model upload, metrics, heartbeat
    utils.py               requests_with_progress(), smart_request()
  models/
    yolo/
      detect/              DetectionTrainer, DetectionPredictor, DetectionValidator
      segment/             SegmentationTrainer, SegmentationPredictor, SegmentationValidator
      classify/            ClassificationTrainer, ClassificationPredictor,
                           ClassificationValidator
      pose/                PoseTrainer, PosePredictor, PoseValidator
      obb/                 OBBTrainer, OBBPredictor, OBBValidator
      world/               WorldTrainer, WorldPredictor, WorldValidator (YOLO-World)
      yoloe/               YOLOETrainer, YOLOEPredictor, YOLOEValidator (YOLOE)
      model.py             YOLO, YOLOWorld, YOLOE model classes
    sam/                   SAM (Segment Anything Model)
      model.py             SAM, SAM2, SAM3 model classes
      predict.py           SAMPredictor, SAM2Predictor
      modules/             blocks.py, decoders.py, encoders.py, memory_attention.py,
                           sam2_utils.py, transformer.py, utils.py
    rtdetr/                RT-DETR (Real-Time Detection Transformer)
      model.py, predict.py, train.py, val.py
    fastsam/               FastSAM
      model.py, predict.py, val.py
    nas/                   YOLO-NAS
      model.py, predict.py, val.py
    utils/                 DETRLoss, HungarianMatcher (DETR-family losses)
  nn/
    tasks.py               BaseModel, DetectionModel, SegmentationModel, PoseModel,
                           ClassificationModel, OBBModel, RTDETRDetectionModel,
                           WorldModel, YOLOEModel, YOLOESegModel, Ensemble,
                           guess_model_task(), load_checkpoint(), SafeUnpickler,
                           parse_model(), yaml_model_load(), attempt_load_weights(),
                           attempt_load_one_weight()
    autobackend.py         AutoBackend: loads and runs inference for ALL 18+ formats.
                           Format detection, input normalization, output parsing
    text_model.py          TextModel, CLIP, MobileCLIP, MobileCLIPTS
                           (text encoders for YOLO-World/YOLOE open-vocabulary)
    modules/
      activation.py        AGLU (Adaptive GLU), custom activations
      block.py             DFL, Proto, HGStem, HGBlock, SPP, SPPF, C1, C2, C3,
                           C2f, C3k2, C2fAttn, RepC3, BiFPN, SCDown, A2C2f, ...
      conv.py              Conv, Conv2, LightConv, DWConv, GhostConv, RepConv,
                           ChannelAttention, SpatialAttention, CBAM, Concat, Index
      head.py              Detect, Segment, Pose, Classify, OBB, RTDETRDecoder,
                           WorldDetect, YOLOEDetect, YOLOESegment
      transformer.py       TransformerLayer, TransformerBlock, MLPBlock,
                           LayerNorm2d, AIFI, DeformableTransformerDecoder, ...
      utils.py             multi_scale_deformable_attn_pytorch(),
                           bias_init_with_prob(), linear_init(),
                           inverse_sigmoid(), multi_head_attention_forward()
    backends/              Per-format inference backends (one file per format)
  optim/
    muon.py                MuSGD (Muon-enhanced SGD), Muon
                           Newton-Schulz orthogonalization for gradient updates
  solutions/
    solutions.py           BaseSolution, SolutionAnnotator, SolutionResults
    config.py              SolutionConfig (pydantic-style validation)
    ai_gym.py              AIGym -- workout repetition counting
    analytics.py           Analytics -- chart generation from tracking
    distance_calculation.py DistanceCalculation -- real-world distance
    heatmap.py             Heatmap -- density visualization
    instance_segmentation.py InstanceSegmentation -- per-object masking
    object_blurrer.py      ObjectBlurrer -- privacy blurring
    object_counter.py      ObjectCounter -- line/region counting
    object_cropper.py      ObjectCropper -- extract detected objects
    parking_management.py  ParkingManagement, ParkingPtsSelection
    queue_management.py    QueueManager -- queue length estimation
    region_counter.py      RegionCounter -- zone-based counting
    security_alarm.py      SecurityAlarm -- email alerts on detection
    similarity_search.py   VisualAISearch, SearchApp -- image similarity
    speed_estimator.py     SpeedEstimator -- object speed from tracking
    streamlit_inference.py Inference -- Streamlit web UI
    trackzone.py           TrackZone -- zone-specific tracking
    vision_eye.py          VisionEye -- perspective visualization
  trackers/
    basetrack.py           TrackState (New/Tracked/Lost/Removed), BaseTrack
    byte_tracker.py        STrack (single track), BYTETracker
    bot_sort.py            BOTrack, BOTSORT, ReID (re-identification)
    track.py               on_predict_start(), on_predict_postprocess_end()
                           (hooks into predict pipeline)
    utils/
      gmc.py               GMC: ECC, sparseOptFlow, ORB feature matching
      kalman_filter.py     KalmanFilterXYAH, KalmanFilterXYWH
      matching.py          linear_assignment(), iou_distance(),
                           embedding_distance(), fuse_score()
  utils/
    __init__.py            SimpleClass, IterableSimpleNamespace, YAML (read/write),
                           SettingsManager, JSONDict, TryExcept, Retry,
                           ThreadingLocked, ONLINE, IS_DOCKER, IS_JUPYTER,
                           IS_COLAB, IS_KAGGLE, MACOS, LINUX, WINDOWS,
                           colorstr(), set_logging(), DEFAULT_CFG, ROOT, ...
    autodevice.py          GPUInfo: gpu_count(), memory(), utilization(),
                           select_idle_gpu()
    autobatch.py           check_train_batch_size(), autobatch()
    benchmarks.py          ProfileModels, RF100Benchmark
    callbacks/
      base.py              Default callback dictionaries
      clearml.py           ClearML integration callbacks
      comet.py             Comet ML integration callbacks
      dvc.py               DVC integration callbacks
      hub.py               Ultralytics HUB callbacks (heartbeat, upload)
      mlflow.py            MLflow integration callbacks
      neptune.py           Neptune integration callbacks
      raytune.py           Ray Tune integration callbacks
      tensorboard.py       TensorBoard integration callbacks
      wb.py                Weights & Biases integration callbacks
    checks.py              check_version(), check_requirements(),
                           check_imgsz(), check_file(), check_yaml(),
                           check_pip_update_available(), is_ascii(),
                           check_yolo(), print_args(), cuda_device_count(),
                           cuda_is_available()
    dist.py                DDP utilities: find_free_network_port(),
                           generate_ddp_file(), ddp_cleanup()
    downloads.py           safe_download(), download(), unzip_file(),
                           is_url(), get_google_drive_file_info()
    errors.py              HUBModelError (custom exception)
    export/                Per-format export utilities:
                           axelera.py, coreml.py, engine.py, executorch.py,
                           imx.py, mnn.py, ncnn.py, onnx.py, openvino.py,
                           paddle.py, rknn.py, tensorflow.py, torchscript.py
    files.py               WorkingDirectory context manager,
                           increment_path(), file_age(), file_date(),
                           file_size(), get_latest_run(), spaces_in_path()
    instance.py            Instances: bboxes, segments, keypoints containers
    loss.py                v8DetectionLoss, v8SegmentationLoss, v8PoseLoss,
                           PoseLoss26, v8ClassificationLoss, v8OBBLoss,
                           E2EDetectLoss, FocalLoss, VarifocalLoss,
                           BboxLoss, RotatedBboxLoss, DFLoss, KeypointLoss,
                           RLELoss
    metrics.py             ConfusionMatrix, Metric, DetMetrics, SegmentMetrics,
                           PoseMetrics, ClassifyMetrics, OBBMetrics,
                           ap_per_class(), smooth(), compute_ap(),
                           box_iou(), mask_iou(), kpt_iou()
    nms.py                 TorchNMS: fast_nms(), batched_nms()
    ops.py                 Profile (timing context manager),
                           non_max_suppression(), scale_boxes(),
                           scale_coords(), xyxy2xywh(), xywh2xyxy(),
                           xyxy2xywhn(), xywhn2xyxy(), xyxy2ltwh(),
                           ltwh2xyxy(), xywhr2xyxyxyxy(), xyxyxyxy2xywhr(),
                           crop_mask(), process_mask(),
                           masks2segments(), clip_boxes(), clip_coords(),
                           make_divisible(), clean_str()
    patches.py             Monkey-patches for torch.save, cv2.imread,
                           cv2.imwrite, PIL.Image.open (Unicode path support)
    plotting.py            Annotator: box_label(), masks(), kpts(),
                           text(), rectangle(), circle()
                           Colors: color palette for classes
                           plot_images(), plot_results(), plot_labels(),
                           save_one_box(), feature_visualization()
    tal.py                 TaskAlignedAssigner: forward(), get_targets(),
                           select_highest_overlaps()
                           RotatedTaskAlignedAssigner
                           (dynamic label assignment for training)
    torch_utils.py         select_device(), ModelEMA, EarlyStopping,
                           torch_distributed_zero_first(),
                           smart_inference_mode(), get_flops(),
                           model_info(), de_parallel(), strip_optimizer(),
                           fuse_conv_and_bn(), init_seeds(),
                           one_cycle(), copy_attr()
    triton.py              TritonRemoteModel (Triton Inference Server client)
    tuner.py               run_ray_tune() (wrapper for Ray Tune)

docker/                    14 Dockerfiles (see Section 4.1)
tests/
  test_cli.py              CLI command tests
  test_cuda.py             CUDA-specific tests
  test_engine.py           Engine (model/trainer/predictor/validator) tests
  test_exports.py          Export format tests
  test_integrations.py     Integration (HUB, callbacks) tests
  test_python.py           Python API tests
  test_solutions.py        Solutions tests
```

### 8.2 Architecture Deep Dive

#### Engine Layer

The engine layer orchestrates all operations. The `Model` class is the primary API surface:

```python
class Model:
    """Unified model class that handles all YOLO tasks."""

    def __init__(self, model="yolo11n.pt", task=None, verbose=False):
        # 1. Determine if model is a .pt file, .yaml file, or HUB URL
        # 2. Load weights or architecture
        # 3. Lazy-load task_map on first operation

    @property
    def task_map(self):
        """Returns {mode: class} mapping for this model's task."""
        # Overridden by subclasses (YOLO, SAM, RTDETR, etc.)
        # Example for detection:
        # {"train": DetectionTrainer, "val": DetectionValidator,
        #  "predict": DetectionPredictor, "export": Exporter}

    def train(self, **kwargs):
        # 1. Merge user kwargs with defaults
        # 2. Instantiate task_map["train"] (e.g., DetectionTrainer)
        # 3. Call trainer.train()

    def predict(self, source=None, **kwargs):
        # 1. Instantiate task_map["predict"] (e.g., DetectionPredictor)
        # 2. Call predictor(source=source)
        # 3. Return Results objects

    def export(self, **kwargs):
        # 1. Instantiate Exporter
        # 2. Call exporter(model=self.model)
        # 3. Return exported file path
```

**Task map resolution flow:**

```
Model("yolo11n.pt")
  |
  v
torch.load() -> ckpt["train_args"]["task"] = "detect"
  |
  v
YOLO.task_map["detect"] = {
    "model": DetectionModel,
    "trainer": DetectionTrainer,
    "validator": DetectionValidator,
    "predictor": DetectionPredictor,
}
```

#### Neural Network Layer

`nn/tasks.py` defines the model architectures. All inherit from `BaseModel`:

```
BaseModel (nn.Module)
  +-- DetectionModel (default head: Detect)
  |     +-- SegmentationModel (head: Segment)
  |     +-- PoseModel (head: Pose)
  |     +-- ClassificationModel (head: Classify)
  |     +-- OBBModel (head: OBB)
  +-- RTDETRDetectionModel (transformer-based)
  +-- WorldModel (open-vocabulary, CLIP text encoder)
  +-- YOLOEModel (efficient open-vocabulary)
        +-- YOLOESegModel
  +-- Ensemble (multi-model ensemble)
```

**Model construction from YAML:**

```python
def parse_model(d, ch, verbose=True):
    """Parse a YAML model architecture definition into PyTorch modules."""
    # 1. Read backbone and head layer definitions
    # 2. For each layer:
    #    - Resolve module class (Conv, C3k2, Detect, etc.)
    #    - Apply depth_multiple (repeat scaling)
    #    - Apply width_multiple (channel scaling)
    #    - Build nn.Module instance
    # 3. Wire up skip connections (from indices)
    # 4. Return nn.Sequential model
```

#### AutoBackend (Multi-Format Inference)

`nn/autobackend.py` provides format-agnostic inference:

```python
class AutoBackend(nn.Module):
    def __init__(self, weights, device, data=None, fp16=False, ...):
        # 1. Detect format from file suffix (.pt, .onnx, .engine, ...)
        # 2. Load format-specific runtime:
        #    .pt      -> torch.load + SafeUnpickler
        #    .onnx    -> onnxruntime.InferenceSession
        #    .engine  -> tensorrt.Runtime
        #    .xml     -> openvino.Core().compile_model()
        #    .tflite  -> tensorflow.lite.Interpreter
        #    ... etc for all 18+ formats

    def forward(self, im, ...):
        # 1. Normalize input (0-1 range, CHW format, correct dtype)
        # 2. Run format-specific inference
        # 3. Parse output into standard format
        # 4. Return predictions tensor
```

#### Data Pipeline

```
Image File/Stream
       |
       v
  data/loaders.py         (LoadImagesAndVideos, LoadStreams, ...)
       |
       v
  data/dataset.py          (YOLODataset.get_image_and_label())
       |
       v
  data/augment.py          (Compose pipeline: Mosaic -> CopyPaste ->
       |                    RandomPerspective -> MixUp -> LetterBox ->
       |                    RandomHSV -> RandomFlip)
       v
  data/build.py            (InfiniteDataLoader wraps DataLoader)
       |
       v
  engine/trainer.py        (batch to device, forward pass)
```

### 8.3 Code Patterns

#### Lazy Imports

```python
# ultralytics/__init__.py
def __getattr__(name):
    """Lazy-import model classes on first access."""
    if name == "YOLO":
        from ultralytics.models.yolo.model import YOLO
        return YOLO
    # ... similar for SAM, SAM2, RTDETR, FastSAM, NAS, etc.
    raise AttributeError(f"module 'ultralytics' has no attribute '{name}'")
```

This pattern keeps `import ultralytics` fast (~100ms) while providing convenient top-level access to all model classes.

#### Callback System

Callbacks are registered as dictionaries mapping event names to lists of functions:

```python
# Default callbacks (utils/callbacks/base.py)
default_callbacks = {
    "on_pretrain_routine_start": [],
    "on_pretrain_routine_end": [],
    "on_train_start": [],
    "on_train_epoch_start": [],
    "on_train_batch_start": [],
    "optimizer_step": [],
    "on_before_zero_grad": [],
    "on_train_batch_end": [],
    "on_train_epoch_end": [],
    "on_fit_epoch_end": [],      # After train + val
    "on_model_save": [],
    "on_train_end": [],
    "on_val_start": [],
    "on_val_batch_start": [],
    "on_val_batch_end": [],
    "on_val_end": [],
    "on_predict_start": [],
    "on_predict_batch_start": [],
    "on_predict_postprocess_end": [],
    "on_predict_batch_end": [],
    "on_predict_end": [],
    "on_export_start": [],
    "on_export_end": [],
}
```

Integration callbacks (W&B, MLflow, etc.) register functions into these hooks at import time. The trainer/predictor/validator call each hook at the appropriate point.

#### Task Maps

Each model variant overrides `task_map` to register its own trainer/predictor/validator:

```python
# models/yolo/model.py
class YOLO(Model):
    @property
    def task_map(self):
        return {
            "detect": {
                "model": DetectionModel,
                "trainer": DetectionTrainer,
                "validator": DetectionValidator,
                "predictor": DetectionPredictor,
            },
            "segment": {
                "model": SegmentationModel,
                "trainer": SegmentationTrainer,
                "validator": SegmentationValidator,
                "predictor": SegmentationPredictor,
            },
            # ... classify, pose, obb, world, yoloe
        }
```

#### NaN Recovery Pattern

```python
# engine/trainer.py (simplified)
def _do_train(self, world_size=1):
    nan_attempts = 0
    for epoch in range(self.start_epoch, self.epochs):
        for batch in self.train_loader:
            loss = self.model(batch)
            if torch.isnan(loss):
                nan_attempts += 1
                if nan_attempts >= 3:
                    raise RuntimeError(
                        f"Training failed: NaN persisted for {nan_attempts} epochs"
                    )
                # Restore from last valid checkpoint
                self.model.load_state_dict(self.last_valid_state)
                continue
            nan_attempts = 0  # Reset on valid loss
            loss.backward()
            self.optimizer.step()
```

### 8.4 Class Hierarchy

#### Model Classes

```
ultralytics.engine.model.Model
  +-- ultralytics.models.yolo.model.YOLO
  +-- ultralytics.models.yolo.model.YOLOWorld
  +-- ultralytics.models.yolo.model.YOLOE
  +-- ultralytics.models.sam.model.SAM
  +-- ultralytics.models.sam.model.SAM2
  +-- ultralytics.models.sam.model.SAM3
  +-- ultralytics.models.rtdetr.model.RTDETR
  +-- ultralytics.models.fastsam.model.FastSAM
  +-- ultralytics.models.nas.model.NAS
```

#### Trainer Classes

```
ultralytics.engine.trainer.BaseTrainer
  +-- ultralytics.models.yolo.detect.train.DetectionTrainer
  +-- ultralytics.models.yolo.segment.train.SegmentationTrainer
  +-- ultralytics.models.yolo.classify.train.ClassificationTrainer
  +-- ultralytics.models.yolo.pose.train.PoseTrainer
  +-- ultralytics.models.yolo.obb.train.OBBTrainer
  +-- ultralytics.models.yolo.world.train.WorldTrainer
  +-- ultralytics.models.yolo.yoloe.train.YOLOETrainer
  +-- ultralytics.models.rtdetr.train.RTDETRTrainer
```

#### Predictor Classes

```
ultralytics.engine.predictor.BasePredictor
  +-- ultralytics.models.yolo.detect.predict.DetectionPredictor
  +-- ultralytics.models.yolo.segment.predict.SegmentationPredictor
  +-- ultralytics.models.yolo.classify.predict.ClassificationPredictor
  +-- ultralytics.models.yolo.pose.predict.PosePredictor
  +-- ultralytics.models.yolo.obb.predict.OBBPredictor
  +-- ultralytics.models.yolo.world.predict.WorldPredictor
  +-- ultralytics.models.yolo.yoloe.predict.YOLOEPredictor
  +-- ultralytics.models.sam.predict.SAMPredictor
  +-- ultralytics.models.sam.predict.SAM2Predictor
  +-- ultralytics.models.rtdetr.predict.RTDETRPredictor
  +-- ultralytics.models.fastsam.predict.FastSAMPredictor
  +-- ultralytics.models.nas.predict.NASPredictor
```

#### Validator Classes

```
ultralytics.engine.validator.BaseValidator
  +-- ultralytics.models.yolo.detect.val.DetectionValidator
  +-- ultralytics.models.yolo.segment.val.SegmentationValidator
  +-- ultralytics.models.yolo.classify.val.ClassificationValidator
  +-- ultralytics.models.yolo.pose.val.PoseValidator
  +-- ultralytics.models.yolo.obb.val.OBBValidator
  +-- ultralytics.models.yolo.world.val.WorldValidator
  +-- ultralytics.models.yolo.yoloe.val.YOLOEValidator
  +-- ultralytics.models.rtdetr.val.RTDETRValidator
  +-- ultralytics.models.fastsam.val.FastSAMValidator
  +-- ultralytics.models.nas.val.NASValidator
```

#### Solution Classes

```
ultralytics.solutions.solutions.BaseSolution
  +-- AIGym
  +-- Analytics
  +-- DistanceCalculation
  +-- Heatmap
  +-- InstanceSegmentation
  +-- ObjectBlurrer
  +-- ObjectCounter
  +-- ObjectCropper
  +-- ParkingManagement
  +-- QueueManager
  +-- RegionCounter
  +-- SecurityAlarm
  +-- SimilaritySearch (VisualAISearch, SearchApp)
  +-- SpeedEstimator
  +-- Inference (Streamlit)
  +-- TrackZone
  +-- VisionEye
```

### 8.5 Loss Functions

| Loss Class | Task | Description |
|---|---|---|
| `v8DetectionLoss` | Detection | Box (CIoU) + Classification (BCE) + DFL |
| `v8SegmentationLoss` | Segmentation | Detection loss + mask BCE loss |
| `v8PoseLoss` | Pose estimation | Detection loss + keypoint OKS loss |
| `PoseLoss26` | Pose (YOLO26) | Updated pose loss for v26 architecture |
| `v8ClassificationLoss` | Classification | Cross-entropy loss |
| `v8OBBLoss` | OBB | Rotated box loss (probiou) + cls + DFL |
| `E2EDetectLoss` | End-to-end detection | One-to-one matching loss (no NMS needed) |
| `FocalLoss` | General | Focal modulation of BCE (gamma weighting) |
| `VarifocalLoss` | General | Variable focal loss (IoU-weighted) |
| `DFLoss` | General | Distribution Focal Loss (DFL) |
| `BboxLoss` | Detection | CIoU + DFL combined box regression |
| `RotatedBboxLoss` | OBB | Prob-IoU for rotated boxes + DFL |
| `KeypointLoss` | Pose | OKS-based keypoint regression |
| `RLELoss` | Pose | Residual Log-likelihood Estimation loss |

**Loss composition (detection example):**

```
Total Loss = box_weight * BboxLoss(CIoU + DFL)
           + cls_weight * BCE(class predictions)
```

Where `BboxLoss` uses `TaskAlignedAssigner` for dynamic positive/negative sample assignment during training.

### 8.6 Metrics

| Metric Class | Task | Key Metrics |
|---|---|---|
| `DetMetrics` | Detection | mAP50, mAP50-95, precision, recall per class |
| `SegmentMetrics` | Segmentation | Box mAP + Mask mAP |
| `PoseMetrics` | Pose | Box mAP + Keypoint mAP (OKS-based) |
| `ClassifyMetrics` | Classification | Top-1 accuracy, Top-5 accuracy |
| `OBBMetrics` | OBB | Rotated box mAP50, mAP50-95 |
| `ConfusionMatrix` | All | True/false positive/negative matrix, generates plots |

**Key metric functions:**
- `ap_per_class()` -- per-class AP computation
- `box_iou()` -- standard IoU for axis-aligned boxes
- `mask_iou()` -- IoU for segmentation masks
- `kpt_iou()` -- OKS-based IoU for keypoints
- `smooth()` -- smooth precision-recall curve

### 8.7 Testing Strategy

| Test File | Coverage | Key Tests |
|---|---|---|
| `test_cli.py` | CLI commands | All `yolo` CLI modes, argument validation, settings |
| `test_cuda.py` | GPU operations | CUDA training, mixed precision, multi-GPU |
| `test_engine.py` | Core engine | Model load/save, train/val/predict/export lifecycle |
| `test_exports.py` | Export formats | Export to each format, inference with exported models |
| `test_integrations.py` | Integrations | HUB, callbacks, external services |
| `test_python.py` | Python API | All Python API methods, Results objects |
| `test_solutions.py` | Solutions | Each solution end-to-end |

**Running tests:**

```bash
# Full test suite
pytest tests/

# Specific test file
pytest tests/test_engine.py -v

# CUDA tests (requires GPU)
pytest tests/test_cuda.py -v

# Skip slow tests
pytest tests/ -m "not slow"
```

### 8.8 Contributing

**Development install:**

```bash
git clone https://github.com/ultralytics/ultralytics.git
cd ultralytics
pip install -e ".[dev]"
```

**Code style:**
- Ruff for linting and formatting
- Google-style docstrings
- Type hints encouraged
- All public methods documented

**PR requirements:**
- Tests pass (`pytest tests/`)
- Code formatted (`ruff format .`)
- Linting passes (`ruff check .`)
- AGPL-3.0 license header on new files
- CLA signed

---

## 9. Integration Points

### 9.1 Ultralytics HUB

HUB provides cloud-based model training, dataset management, and deployment:

```python
from ultralytics import HUB

# Login with API key
HUB.login(api_key="YOUR_API_KEY")

# Or browser-based authentication
HUB.login()

# Training session (syncs metrics + checkpoints to HUB)
from ultralytics import YOLO
model = YOLO("https://hub.ultralytics.com/models/MODEL_ID")
model.train()
```

**HUB data flow:**
1. `HUBTrainingSession` created on `model.train()` start
2. Training metrics sent via `session.upload_metrics()` after each epoch
3. Model checkpoints uploaded at `save_period` intervals
4. Heartbeat keeps session alive
5. Final best model uploaded on training completion

**HUB authentication methods:**
- API key: `HUB.login(api_key="...")`
- Browser: `HUB.login()` (opens browser for OAuth)
- Cookie: Automatic if previously authenticated
- Settings: `yolo settings api_key=YOUR_API_KEY`

### 9.2 Callback Integrations

All integrations follow the same pattern: register functions into the callback hooks. They activate automatically when the package is installed and the settings toggle is enabled.

#### Weights & Biases (W&B)

```bash
pip install wandb
wandb login
```

```python
# Automatically logs: metrics, images, model artifacts, hyperparameters
model.train(data="coco.yaml", project="my-yolo-project")
```

**Logged data:** Training/validation metrics per epoch, sample predictions, model checkpoints, confusion matrices, PR curves, hyperparameters.

#### MLflow

```bash
pip install mlflow
```

```python
import os
os.environ["MLFLOW_TRACKING_URI"] = "http://mlflow-server:5000"
model.train(data="coco.yaml")
```

**Logged data:** Parameters, metrics, model artifacts. Uses `mlflow.log_param()`, `mlflow.log_metric()`, `mlflow.log_artifact()`.

#### ClearML

```bash
pip install clearml
clearml-init
```

**Logged data:** Full experiment tracking with metrics, plots, model snapshots, debug images.

#### Comet ML

```bash
pip install comet_ml
export COMET_API_KEY=your_key
```

**Logged data:** Metrics, hyperparameters, model graphs, predictions, confusion matrices.

#### Neptune

```bash
pip install neptune
export NEPTUNE_API_TOKEN=your_token
```

**Logged data:** Metrics, hyperparameters, model artifacts, system monitoring.

#### DVC (Data Version Control)

```bash
pip install dvclive
```

**Logged data:** Metrics in DVC-compatible format for experiment tracking and pipeline management.

#### TensorBoard

```python
# Enabled by default, no extra install needed (ships with PyTorch)
model.train(data="coco.yaml")
```

```bash
tensorboard --logdir runs/detect/train
```

#### Ray Tune (Hyperparameter Optimization)

```python
from ultralytics import YOLO

model = YOLO("yolo11n.pt")
model.tune(
    data="coco8.yaml",
    epochs=30,
    iterations=100,
    optimizer="AdamW",
    space={
        "lr0": (1e-5, 1e-1),
        "momentum": (0.6, 0.98),
        "weight_decay": (0.0, 0.001),
    },
)
```

### 9.3 Streamlit (Built-in Inference UI)

```python
from ultralytics.solutions import Inference

# Launch Streamlit web UI for interactive inference
app = Inference()
app.inference()
```

Or via CLI:

```bash
yolo streamlit-predict
```

Provides a web-based UI for:
- Image upload and inference
- Model selection
- Confidence/IoU threshold adjustment
- Result visualization

### 9.4 SAHI (Sliced Inference)

For detecting small objects in high-resolution images:

```python
from sahi import AutoDetectionModel
from sahi.predict import get_sliced_prediction

detection_model = AutoDetectionModel.from_pretrained(
    model_type="ultralytics",
    model_path="yolo11n.pt",
    confidence_threshold=0.3,
)

result = get_sliced_prediction(
    "high_res_image.jpg",
    detection_model,
    slice_height=640,
    slice_width=640,
    overlap_height_ratio=0.2,
    overlap_width_ratio=0.2,
)
```

### 9.5 Albumentations

When the `albumentations` package is installed, the data pipeline automatically integrates Albumentations transforms:

```bash
pip install albumentations
```

```python
# Automatically used during training when installed
model.train(data="coco.yaml")
# Albumentations transforms are added to the augmentation pipeline
```

Default Albumentations transforms applied:
- `Blur(p=0.01)`
- `MedianBlur(p=0.01)`
- `ToGray(p=0.01)`
- `CLAHE(p=0.01)`

Custom Albumentations transforms can be passed by subclassing the trainer and overriding the augmentation pipeline.

### 9.6 Roboflow

```python
# Download datasets from Roboflow
from roboflow import Roboflow

rf = Roboflow(api_key="YOUR_API_KEY")
project = rf.workspace("workspace").project("project")
dataset = project.version(1).download("yolov8")

# Train with downloaded dataset
model.train(data=f"{dataset.location}/data.yaml")
```

### 9.7 Triton Inference Server

```python
from ultralytics.utils.triton import TritonRemoteModel

# Connect to Triton server
model = TritonRemoteModel(url="http://triton-server:8000/model_name")

# Run inference (same API as local model)
results = model(image)
```

### 9.8 Error Messages Quick Reference

| Error Message | Cause | Fix |
|---|---|---|
| `Invalid 'mode={mode}'. Valid modes are {MODES}` | Unrecognized CLI mode | Use: `train`, `val`, `predict`, `export`, `track`, `benchmark` |
| `Invalid 'task={task}'. Valid tasks are {TASKS}` | Unrecognized task | Use: `detect`, `segment`, `classify`, `pose`, `obb` |
| `'{k}={v}' is of invalid type` | Wrong config value type | Check expected type (int, float, bool) |
| `'{k}={v}' is an invalid value. Valid '{k}' values are between 0.0 and 1.0` | Fraction out of range | Use value in [0.0, 1.0] |
| `model='{model}' should be a *.pt PyTorch model` | Non-PT model for training | Provide `.pt` file for training |
| `'{name}' model does not support '{mode}' mode` | Unsupported mode/task combo | Check model's supported tasks |
| `AutoBatch with batch<1 not supported for Multi-GPU training` | `batch=-1` with DDP | Set explicit batch size |
| `Resume checkpoint not found` | Missing checkpoint for resume | Verify `runs/` directory has `last.pt` |
| `model='{model}' is not a supported model format` | Unknown file suffix | Use a supported export format |
| `Ultralytics TF.js inference is not currently supported` | TF.js inference attempted | Export to a different format for inference |
| `No images or videos found in {p}` | Empty or wrong source path | Verify path contains images/videos |
| `'source=0' webcam not supported in Colab and Kaggle` | Webcam in cloud | Use file/URL source instead |
| `data.yaml flip_idx length must equal kpt_shape[0]` | Keypoint config mismatch | Match `flip_idx` length to keypoint count |

### 9.9 Known Issues (from GitHub)

| Issue | Description | Status |
|---|---|---|
| #24257 | Fails to load media when path contains square brackets | Open |
| #24205 | YOLO26-OBB shows lower accuracy than YOLO11 OBB | Open |
| #24168 | Segmentation mask shrinks when object moves closer to camera | Open |
| #23941 | SAM3 video vs image predictor produce different results | Open |
| #24253 | IoU metric not working intermittently | Open |

---

_Generated May 2026 from ultralytics/ultralytics at f4d0fda. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._