# TensorFlow Models — User Guide

## 1. Quick Start

TensorFlow Models is Google's official collection of production-quality machine learning models built on TensorFlow 2.x and Keras. The repository provides state-of-the-art implementations for computer vision, natural language processing, recommendation systems, and reinforcement learning — all designed for distributed training on GPUs and TPUs.

**Key features:**

- Pre-built vision models: image classification (ResNet, EfficientNet, MobileNet, MaxViT), object detection (RetinaNet, Mask R-CNN, YOLO, CenterNet, DETR), semantic segmentation (DeepLab, Panoptic), and video classification (MoViNet, S3D)
- NLP models: BERT, ALBERT, Longformer, BigBird, Perceiver, and fine-tuning pipelines for GLUE, SQuAD, and SuperGLUE
- Recommendation engine with uplift modeling support
- Orbit training loop library for custom training with `tf.distribute`
- YAML/Gin-based experiment configuration with a registry/factory system
- Built-in support for TPU, multi-GPU mirrored, and multi-worker distributed training strategies

**Prerequisites:** Python 3.7+, TensorFlow 2.20+

**Install:**

```bash
pip install tf-models-official
```

**Minimal example — use a pre-trained BERT encoder:**

```python
import tensorflow_models as tfm

encoder = tfm.nlp.layers.TransformerEncoderBlock(
    num_attention_heads=2, inner_dim=10, inner_activation='relu')
```

```text
<tf_keras.src.layers.TransformerEncoderBlock object at 0x...>
```

**What to try next:** See the Installation section for full setup options, or jump to Core Usage to train a vision or NLP model.

> [!TIP]
> For the fastest path to results, install via pip and use the YAML experiment configs in `official/vision/configs/` or `official/nlp/configs/` — they provide battle-tested hyperparameters for common tasks.

---

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.7 | 3.10+ |
| TensorFlow | 2.20.0 | 2.20.0 (matching `tf-models-official` version) |
| RAM | 8 GB | 16 GB+ |
| GPU | Optional (CPU works) | NVIDIA with CUDA 12.x |
| Disk | 2 GB (package + deps) | 10 GB+ (with datasets) |

### Install Methods

=== "pip (recommended)"
    ```bash
    pip install tf-models-official
    ```

=== "pip (nightly)"
    ```bash
    pip install tf-models-nightly
    ```

=== "Source (development)"
    ```bash
    git clone https://github.com/tensorflow/models.git
    cd models
    pip install -r official/requirements.txt
    export PYTHONPATH="$PYTHONPATH:$(pwd)"
    ```

> [!IMPORTANT]
> The `tf-models-official` package version tracks the TensorFlow version. Version 2.20.0 requires `tensorflow~=2.20.0` and `tensorflow-text~=2.20.0`. Mismatched versions will cause import errors.

### Post-Install Verification

```bash
python -c "import tensorflow_models as tfm; print(tfm.nlp, tfm.vision)"
```

```text
<module 'tensorflow_models.nlp' from '...'> <module 'tensorflow_models.vision' from '...'>
```

### Additional Dependencies for Specific Projects

| Project | Extra Dependencies |
|---|---|
| Object Detection (research) | `pycocotools`, `Cython`, `Pillow`, `opencv-python-headless` |
| MoViNet | `mediapy` |
| Deep Speech | `nltk>=3.3`, `sox>=1.3.3`, `soundfile>=0.10.2` |
| VGGish/YAMNet | `resampy`, `soundfile`, `tf_slim` |
| Unified Detector | `shapely>=1.8.1`, `apache_beam>=2.37.0` |
| DELF | `tensorflow-hub>=0.6.0` |
| Waste Identification (DETR deploy) | `tritonclient[all]`, `supervision`, `cupy-cuda12x` |

```bash
# Example: install research/object_detection dependencies
pip install pycocotools Cython Pillow opencv-python-headless
```

### Install Troubleshooting

### Error: `ModuleNotFoundError: No module named 'official'`

**Cause:** Running from source without setting `PYTHONPATH`.

**Solution:**
1. Set the path to the repository root:
   ```bash
   export PYTHONPATH="$PYTHONPATH:/path/to/models"
   ```
2. Verify: `python -c "from official.core import train_lib"`

**Prevention:** Add the export to your shell profile or use the pip package instead.

### Error: `cannot import name 'runtime_version' from 'google.protobuf'`

**Cause:** Incompatible `protobuf` version (GitHub issue #11192).

**Solution:**
1. Upgrade protobuf:
   ```bash
   pip install --upgrade protobuf
   ```

**Prevention:** Use a virtual environment with pinned dependencies.

### Error: `Installation Issues on Python 3.13`

**Cause:** Some dependencies have not yet released Python 3.13 wheels (GitHub issue #13587).

**Solution:**
1. Use Python 3.10–3.12 until all dependencies support 3.13
2. Or build problematic packages from source

**Prevention:** Check the issue tracker for 3.13 compatibility updates.

### Error: `HTTP Error 403` when downloading ImageNet MobileNet checkpoints

**Cause:** Google Cloud Storage bucket access changes (GitHub issue #13561).

**Solution:**
1. Check the TensorFlow Model Garden documentation for updated checkpoint URLs
2. Use `tensorflow_hub` modules instead of raw checkpoints where available

### Error: Docker image build failure

**Cause:** Dependency resolution issues in containerized builds (GitHub issue #13567).

**Solution:**
1. Pin dependency versions in your `requirements.txt`
2. Use the official TensorFlow Docker base images

**Prevention:** Test Docker builds in CI with locked dependencies.

### Error: `Tag version compatible with Tensorflow 2.20`

**Cause:** No tagged release matching the latest TF version (GitHub issue #13607).

**Solution:**
1. Install from `main` branch:
   ```bash
   pip install git+https://github.com/tensorflow/models.git#subdirectory=official/pip_package
   ```

### Error: `Message type "object_detection.protos.DetectionModel" has no field named "center_net"`

**Cause:** Protobuf compilation with the wrong version or missing proto files (GitHub issue #13518).

**Solution:**
1. Recompile the protos:
   ```bash
   cd models/research
   protoc object_detection/protos/*.proto --python_out=.
   ```

**Prevention:** Always recompile protos after updating the repository.

### Error: `Seq flow lite does not build`

**Cause:** Custom TFLite ops require specific build toolchain (GitHub issue #10737).

**Solution:**
1. Follow the build instructions in `research/seq_flow_lite/demo/colab/`
2. Ensure Bazel and the correct TensorFlow version are installed

### Error: `AccessDenied for tf_model_garden vision models`

**Cause:** Cloud Storage permission changes for hosted model artifacts (GitHub issue #13610).

**Solution:**
1. Use the latest checkpoint URLs from the official documentation
2. Fall back to TensorFlow Hub modules

### Error: `unexpected dimension of score tensor in yamnet`

**Cause:** Input audio preprocessing mismatch (GitHub issue #13605).

**Solution:**
1. Ensure audio is resampled to 16 kHz mono before inference
2. Check input tensor shape matches expected dimensions

> [!WARNING]
> The `research/` directory models are community-maintained and may not track the latest TensorFlow version. The `official/` models receive active support and are the recommended starting point.

---

## 3. Configuration

TensorFlow Models uses a layered configuration system combining YAML experiment configs, Python dataclasses (`ExperimentConfig`), Gin bindings, and command-line flag overrides.

### Quick Configuration Overview

Most training runs are configured through YAML files located in each project's `configs/experiments/` directory. These files define the full experiment: task, model architecture, training schedule, and data pipeline.

```yaml
# Example: official/vision/configs/experiments/image_classification/imagenet_resnet50_gpu.yaml
task:
  model:
    backbone:
      type: 'resnet'
      resnet:
        model_id: 50
    num_classes: 1001
  train_data:
    input_path: '/path/to/imagenet/train*'
    global_batch_size: 256
trainer:
  train_steps: 90000
  validation_interval: 1000
  optimizer_config:
    learning_rate:
      type: 'cosine'
```

### Overriding Configuration at Runtime

Override any config value via the `--params_override` flag:

```bash
python official/vision/train.py \
  --experiment=resnet_imagenet \
  --config_file=path/to/config.yaml \
  --params_override="task.train_data.global_batch_size=128,trainer.train_steps=50000"
```

### Environment Variables

| Variable | Purpose | Default |
|---|---|---|
| `TF_CONFIG` | Multi-worker cluster specification (JSON) | `{}` |
| `PYTHONPATH` | Must include repository root for source installs | — |

> [!NOTE]
> `TF_CONFIG` is read by `official/common/distribute_utils.py` to configure multi-worker training. Set it on each worker with the cluster spec and task index. See the Admin Guide for full configuration reference.

### Distribution Strategy Selection

| Strategy Flag | When to Use |
|---|---|
| `mirrored` | Single host, multiple GPUs (default) |
| `tpu` | Google Cloud TPU |
| `multi_worker_mirrored` | Multiple hosts with GPUs |
| `one_device` | Single GPU or CPU |
| `off` | No distribution strategy |

```bash
# GPU training with MirroredStrategy (default)
python official/vision/train.py --experiment=resnet_imagenet --mode=train \
  --model_dir=/tmp/resnet --config_file=config.yaml

# TPU training
python official/vision/train.py --experiment=resnet_imagenet --mode=train \
  --model_dir=gs://bucket/resnet --config_file=config.yaml \
  --params_override="runtime.distribution_strategy=tpu,runtime.tpu=grpc://TPU_IP:8470"
```

> [!TIP]
> For TPU training, set `steps_per_loop` to 1000 or higher. Running training steps inside a `tf.function` loop significantly increases TPU utilization.

---

## 4. Core Usage

### Training Vision Models

The unified training driver at `official/vision/train.py` handles image classification, object detection, segmentation, and video classification through the experiment registry.

```bash
python official/vision/train.py \
  --experiment=resnet_imagenet \
  --mode=train_and_eval \
  --model_dir=/tmp/resnet50 \
  --config_file=official/vision/configs/experiments/image_classification/imagenet_resnet50_gpu.yaml
```

```text
I0507 12:00:00.000000 Train num_replicas_in_sync 2
Step 100/90000 - loss: 6.2341 - accuracy: 0.0156
Step 200/90000 - loss: 5.8921 - accuracy: 0.0312
```

**Supported modes:** `train`, `eval`, `train_and_eval`, `continuous_eval`

#### Object Detection (Legacy Detection)

The legacy detection entry point supports RetinaNet, Mask R-CNN, OLN-Mask, and ShapeMask:

```bash
python official/legacy/detection/main.py \
  --mode=train \
  --model=retinanet \
  --config_file=official/legacy/detection/configs/retinanet_config.yaml \
  --training_file_pattern="/data/coco/train-*.tfrecord" \
  --model_dir=/tmp/retinanet
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--mode` | `str` | `train` | `train`, `eval`, or `eval_once` |
| `--model` | `str` | `retinanet` | `retinanet`, `mask_rcnn`, `shapemask` |
| `--enable_xla` | `bool` | `False` | Enable XLA compilation for GPU |
| `--training_file_pattern` | `str` | `None` | Glob pattern for training TFRecords |
| `--eval_file_pattern` | `str` | `None` | Glob pattern for eval TFRecords |
| `--checkpoint_path` | `str` | `None` | Checkpoint path for `eval_once` mode |

> [!WARNING]
> Models in the `official/legacy/` directory are maintained primarily for benchmarking. Use `official/vision/` for new projects — it has the same architectures with a modern configuration system and active support.

### Training NLP Models

#### BERT Fine-Tuning for Text Classification (GLUE)

```bash
export BERT_DIR=gs://cloud-tpu-checkpoints/bert/keras_bert/uncased_L-24_H-1024_A-16
export GLUE_DIR=gs://some_bucket/datasets
export TASK=MRPC

python run_classifier.py \
  --mode='train_and_eval' \
  --input_meta_data_path=${GLUE_DIR}/${TASK}_meta_data \
  --train_data_path=${GLUE_DIR}/${TASK}_train.tf_record \
  --eval_data_path=${GLUE_DIR}/${TASK}_eval.tf_record \
  --bert_config_file=${BERT_DIR}/bert_config.json \
  --init_checkpoint=${BERT_DIR}/bert_model.ckpt \
  --train_batch_size=4 \
  --eval_batch_size=4 \
  --learning_rate=2e-5 \
  --num_train_epochs=3 \
  --model_dir=/tmp/bert_mrpc \
  --distribution_strategy=mirrored
```

> [!NOTE]
> For GPU memory of 16 GB or smaller, use `BERT-Base` (`uncased_L-12_H-768_A-12`) instead of `BERT-Large`. Reduce `train_batch_size` to 4 or lower if out-of-memory errors persist.

#### BERT Fine-Tuning for Question Answering (SQuAD)

```bash
export BERT_DIR=gs://cloud-tpu-checkpoints/bert/keras_bert/uncased_L-24_H-1024_A-16
export SQUAD_DIR=gs://some_bucket/datasets

python run_squad.py \
  --input_meta_data_path=${SQUAD_DIR}/squad_v1.1_meta_data \
  --train_data_path=${SQUAD_DIR}/squad_v1.1_train.tf_record \
  --predict_file=${SQUAD_DIR}/dev-v1.1.json \
  --vocab_file=${BERT_DIR}/vocab.txt \
  --bert_config_file=${BERT_DIR}/bert_config.json \
  --init_checkpoint=${BERT_DIR}/bert_model.ckpt \
  --train_batch_size=4 \
  --predict_batch_size=4 \
  --learning_rate=8e-5 \
  --num_train_epochs=2 \
  --model_dir=/tmp/bert_squad \
  --distribution_strategy=mirrored
```

**Getting predictions:**

```bash
python run_classifier.py \
  --mode='predict' \
  --input_meta_data_path=${GLUE_DIR}/${TASK}_meta_data \
  --eval_data_path=${GLUE_DIR}/${TASK}_eval.tf_record \
  --bert_config_file=${BERT_DIR}/bert_config.json \
  --eval_batch_size=4 \
  --model_dir=/tmp/bert_mrpc \
  --distribution_strategy=mirrored
```

```text
Output written to test_results.tsv in the model directory.
Each line contains class probabilities for one sample.
```

#### Pre-training Data Preparation

```bash
python models/official/nlp/data/create_pretraining_data.py \
  --input_file=$WORKING_DIR/input/input.txt \
  --output_file=$WORKING_DIR/output/tf_examples.tfrecord \
  --vocab_file=$BERT_DIR/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=5
```

### Using the `tensorflow_models` Python API

After installing via pip, you can import and use models programmatically:

```python
import tensorflow_models as tfm

# Vision: SqueezeExcitation layer
se_layer = tfm.vision.layers.SqueezeExcitation(
    in_filters=8, out_filters=4, se_ratio=1)

# NLP: Transformer encoder block
encoder_block = tfm.nlp.layers.TransformerEncoderBlock(
    num_attention_heads=2, inner_dim=10, inner_activation='relu')

# Common: learning rate schedules
warmup = tfm.optimization.LinearWarmup(
    after_warmup_lr_sched=0.0, warmup_steps=10, warmup_learning_rate=0.1)

# Hyperparameters
config = tfm.hyperparams.Config()
```

### Using Orbit for Custom Training Loops

Orbit provides the `Controller` class that manages the outer training loop:

```python
import orbit
import tensorflow as tf

# Define your trainer
class MyTrainer(orbit.StandardTrainer):
    def train_step(self, iterator):
        # Your training logic here
        pass

# Create the controller
controller = orbit.Controller(
    global_step=tf.Variable(0),
    trainer=my_trainer,
    evaluator=my_evaluator,
    steps_per_loop=100,
    checkpoint_manager=tf.train.CheckpointManager(
        checkpoint, directory='/tmp/ckpts', max_to_keep=5),
    summary_interval=100,
    summary_dir='/tmp/summaries')

# Run training
controller.train(steps=10000)

# Or interleaved training and evaluation
controller.train_and_evaluate(
    train_steps=10000,
    eval_steps=100,
    eval_interval=1000)
```

> [!TIP]
> Orbit's `Controller` supports four loop modes: `train`, `evaluate`, `train_and_evaluate`, and `evaluate_continuously` (for monitoring a checkpoint directory). Use `train_and_evaluate` for most workflows.

### Research Models

The `research/` directory contains implementations of published papers. Each subdirectory is a self-contained project with its own README and dependencies.

| Project | Task | Key Scripts |
|---|---|---|
| `research/object_detection` | Object detection & segmentation | `model_main_tf2.py` |
| `research/deeplab` | Semantic segmentation | `train.py`, `eval.py` |
| `research/slim` | Image classification (TF-Slim) | `train_image_classifier.py` |
| `research/delf` | Local feature extraction | `extract_features.py`, `match_images.py` |
| `research/autoaugment` | Data augmentation policies | training scripts |
| `research/deep_speech` | Speech recognition | `deep_speech.py` |
| `research/audioset` | Audio classification (VGGish/YAMNet) | inference scripts |
| `research/adversarial_text` | Adversarial text classification | `train.py` |

### DELF Feature Extraction (CLI)

The DELF (DEep Local Features) project provides CLI tools for image feature extraction and matching:

```bash
# Extract bounding boxes
python research/delf/delf/python/examples/extract_boxes.py \
  --detector_path=/tmp/d2r_frcnn_20190411/ \
  --list_images_path=list_images.txt \
  --output_dir=test_boxes \
  --detector_thresh=0.0

# Extract DELF features
python research/delf/delf/python/examples/extract_features.py \
  --config_path=delf_config_example.pbtxt \
  --list_images_path=list_images.txt \
  --output_dir=test_features

# Match two images
python research/delf/delf/python/examples/match_images.py \
  --image_1_path=test_images/image_1.jpg \
  --image_2_path=test_images/image_2.jpg \
  --features_1_path=test_features/image_1.delf \
  --features_2_path=test_features/image_2.delf \
  --output_image=test_match.png
```

| Flag | Default | Description |
|---|---|---|
| `--detector_path` | `/tmp/d2r_frcnn_20190411/` | Path to exported detector model |
| `--detector_thresh` | `0.0` | Confidence threshold for box detection |
| `--config_path` | `delf_config_example.pbtxt` | Path to DelfConfig proto text file |
| `--list_images_path` | `list_images.txt` | File listing images to process |
| `--output_dir` | varies | Directory for output files |
| `--output_viz_dir` | (empty) | If set, saves detection visualizations with `_viz.jpg` suffix |

### MARCO Automated Analysis (CLI)

```bash
python research/marco/Automated_Marco.py \
  --PATH=/path/to/image/directory \
  --MODEL_PATH=./savedmodel
```

| Flag | Default | Description |
|---|---|---|
| `--PATH` | (required) | Path to image directory (recursively finds images) |
| `--MODEL_PATH` | `./savedmodel` | Path to the TensorFlow saved model |

---

## 5. Common Tasks

### Task: Train an Image Classifier on Custom Data

**Goal:** Train a ResNet-50 on your own image dataset.

**Prerequisites:** TFRecord dataset, `tf-models-official` installed.

**Steps:**
1. Prepare your data as TFRecords (or use TFDS)
2. Copy and modify a config file:
   ```bash
   cp official/vision/configs/experiments/image_classification/imagenet_resnet50_gpu.yaml my_config.yaml
   ```
3. Update the config with your data paths, number of classes, and batch size
4. Train:
   ```bash
   python official/vision/train.py \
     --experiment=resnet_imagenet \
     --mode=train_and_eval \
     --model_dir=/tmp/my_model \
     --config_file=my_config.yaml
   ```
5. Monitor with TensorBoard:
   ```bash
   tensorboard --logdir=/tmp/my_model
   ```

**Verify:** Check that loss decreases and accuracy increases in TensorBoard.

**If it fails:** Reduce batch size if you get OOM errors. Verify TFRecord paths with `tf.data.TFRecordDataset('/path/to/data').take(1)`.

### Task: Fine-Tune BERT on a Custom Classification Task

**Goal:** Adapt BERT to your domain-specific text classification.

**Prerequisites:** GLUE-format data converted to TFRecords, BERT checkpoint.

**Steps:**
1. Prepare fine-tuning data:
   ```bash
   python official/nlp/data/create_finetuning_data.py \
     --input_data_dir=${DATA_DIR} \
     --vocab_file=${BERT_DIR}/vocab.txt \
     --train_data_output_path=${OUTPUT_DIR}/train.tf_record \
     --eval_data_output_path=${OUTPUT_DIR}/eval.tf_record \
     --meta_data_file_path=${OUTPUT_DIR}/meta_data \
     --fine_tuning_task_type=classification \
     --max_seq_length=128 \
     --classification_task_name=CUSTOM
   ```
2. Run training:
   ```bash
   python run_classifier.py \
     --mode=train_and_eval \
     --input_meta_data_path=${OUTPUT_DIR}/meta_data \
     --train_data_path=${OUTPUT_DIR}/train.tf_record \
     --eval_data_path=${OUTPUT_DIR}/eval.tf_record \
     --bert_config_file=${BERT_DIR}/bert_config.json \
     --init_checkpoint=${BERT_DIR}/bert_model.ckpt \
     --train_batch_size=32 \
     --learning_rate=2e-5 \
     --num_train_epochs=3 \
     --model_dir=/tmp/custom_bert
   ```

**Verify:** Run with `--mode=predict` and check `test_results.tsv`.

### Task: Switch from GPU to TPU Training

**Goal:** Run the same training job on Cloud TPU for faster iteration.

**Prerequisites:** GCP project, TPU VM, data on GCS.

**Steps:**
1. Upload data and checkpoints to Google Cloud Storage
2. Add TPU flags to your command:
   ```bash
   --distribution_strategy=tpu \
   --tpu=grpc://${TPU_IP_ADDRESS}:8470
   ```
3. Increase `batch_size` (32 or higher for TPU efficiency)
4. Set `steps_per_loop=1000` for better TPU utilization
5. Ensure `model_dir` is a GCS path (`gs://...`)

> [!IMPORTANT]
> When switching to TPU, `use_bfloat16` is automatically enabled and `use_sync_bn` is turned on. If you see numerical differences from GPU training, this is expected behavior.

### Task: Run Object Detection with the Research API

**Goal:** Train an object detection model using the research API.

**Prerequisites:** `pycocotools`, compiled protobuf files.

**Steps:**
1. Compile protobuf files:
   ```bash
   cd models/research
   protoc object_detection/protos/*.proto --python_out=.
   ```
2. Set Python path:
   ```bash
   export PYTHONPATH="$PYTHONPATH:$(pwd):$(pwd)/slim"
   ```
3. Run training:
   ```bash
   python object_detection/model_main_tf2.py \
     --pipeline_config_path=configs/tf2/ssd_mobilenet_v2.config \
     --model_dir=/tmp/od_model
   ```

### Task: Use Pre-Trained TF Hub Modules Instead of Checkpoints

**Goal:** Simplify model loading by using TF Hub.

**Steps:**
1. Replace `--init_checkpoint` with `--hub_module_url`:
   ```bash
   python run_classifier.py \
     --mode=train_and_eval \
     --hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/1 \
     --train_data_path=${GLUE_DIR}/${TASK}_train.tf_record \
     --eval_data_path=${GLUE_DIR}/${TASK}_eval.tf_record \
     --input_meta_data_path=${GLUE_DIR}/${TASK}_meta_data \
     --model_dir=/tmp/hub_bert
   ```

### Task: Restore from a Checkpoint

**Goal:** Load pre-trained weights into a model.

```python
import tensorflow as tf

model = tf.keras.Model()  # Your model definition
checkpoint = tf.train.Checkpoint(model=model)
checkpoint.restore('path/to/checkpoint')
```

### Task: Set Up Multi-Worker Training

**Goal:** Distribute training across multiple GPU hosts.

**Steps:**
1. Set `TF_CONFIG` on each worker:
   ```bash
   export TF_CONFIG='{"cluster":{"worker":["host1:port","host2:port"]},"task":{"type":"worker","index":0}}'
   ```
2. Run with multi-worker mirrored strategy:
   ```bash
   python official/vision/train.py \
     --experiment=resnet_imagenet \
     --config_file=config.yaml \
     --params_override="runtime.distribution_strategy=multi_worker_mirrored"
   ```

> [!WARNING]
> When using `multi_worker_mirrored`, only `ring` and `nccl` are valid values for `all_reduce_alg`. NCCL requires compatible GPU hardware across all workers.

### Task: Serve the Waste Identification Prediction API

**Goal:** Deploy the CircularNet waste detection model as a REST endpoint.

**Steps:**
1. Build and run the Docker container:
   ```bash
   cd official/projects/waste_identification_ml/docker_solution/prediction_api
   docker build -t waste-detection .
   docker run -p 5000:5000 waste-detection
   ```
2. Send a prediction request:
   ```bash
   curl -X POST http://localhost:5000/predict \
     -F "image=@test_image.jpg"
   ```

```json
{"predictions": [{"detection_classes": [...], "detection_scores": [...]}]}
```

---

## 6. Troubleshooting

**Diagnostic decision tree:**

1. **Import errors?** → Check TensorFlow and `tf-models-official` version compatibility
2. **OOM during training?** → Reduce batch size, enable mixed precision, or use gradient accumulation
3. **NaN loss?** → Check learning rate, data pipeline, and input preprocessing
4. **Slow training?** → Increase `steps_per_loop`, enable XLA, use TPU
5. **Checkpoint loading failures?** → Verify checkpoint format matches model architecture

### Error: `ValueError: Unrecognized Distribution Strategy`

**Cause:** Invalid value passed to `--distribution_strategy`.

**Solution:**
1. Use one of the supported values: `off`, `one_device`, `mirrored`, `parameter_server`, `multi_worker_mirrored`, `tpu`
2. The flag is case-insensitive

**Prevention:** Check `official/common/distribute_utils.py` for the current list of supported strategies.

### Error: `ValueError: num_gpus can not be negative`

**Cause:** Negative value passed to `--num_gpus`.

**Solution:**
1. Set `--num_gpus` to 0 (CPU) or a positive integer

### Error: `ValueError: OneDeviceStrategy can not be used for more than one device`

**Cause:** `--distribution_strategy=one_device` with `--num_gpus > 1`.

**Solution:**
1. Use `mirrored` strategy for multi-GPU training
2. Or set `--num_gpus=1`

### Error: `ValueError: Must specify task_index when number of workers > 1`

**Cause:** Multi-worker setup without proper `TF_CONFIG`.

**Solution:**
1. Set `TF_CONFIG` with cluster specification and task index on each worker
2. Example:
   ```bash
   export TF_CONFIG='{"cluster":{"worker":["host1:12345","host2:12345"]},"task":{"type":"worker","index":0}}'
   ```

### Error: `ValueError: ExperimentConfig.task.init_checkpoint must be a directory`

**Cause:** In `continuous_train_and_eval` mode, the checkpoint path must be a directory, not a file.

**Solution:**
1. Point `init_checkpoint` to the directory containing checkpoint files

### Error: `OverflowError: Loss is nan`

**Cause:** Training divergence due to learning rate, data issues, or numerical instability.

**Solution:**
1. Lower the learning rate
2. Add gradient clipping (`gradient_clip_norm` in config)
3. Verify input data has no corrupt samples
4. Enable mixed precision carefully (avoid all-bfloat16 on sensitive operations)

> [!CAUTION]
> NaN loss during adversarial text training (`research/adversarial_text`) terminates training immediately with an `OverflowError`. Save checkpoints frequently to avoid losing progress.

### Error: `ValueError: Model not supported` (legacy detection)

**Cause:** Invalid model name passed to `--model`.

**Solution:**
1. Use one of: `retinanet`, `mask_rcnn`, `olnmask`, `shapemask`

### Error: `ValueError: Unrecognized file_type`

**Cause:** Dataset configuration specifies unsupported file type.

**Solution:**
1. Use `tfrecord` or `tfrecord_compressed` as the dataset type

### Error: `maskconver is lacking a decoder code`

**Cause:** Missing implementation in the MaskConver project (GitHub issue #13585).

**Solution:**
1. Check the issue tracker for updates on MaskConver completion

### Error: `MoViNets: Permission 'storage.objects.get' denied`

**Cause:** Access denied to Google Cloud Storage model artifacts (GitHub issue #13611).

**Solution:**
1. Use updated model URLs from the project README
2. Or download models via TF Hub

> [!WARNING]
> Several pre-trained model checkpoint URLs have been affected by Google Cloud Storage access changes. If you encounter 403 errors or access denied messages, check the GitHub issues for updated URLs before debugging your authentication.

---

## 7. FAQ

**Q: What is the difference between `official/` and `research/` models?**

The `official/` directory contains Google-maintained, production-quality models that track the latest TensorFlow API and receive regular updates. The `research/` directory contains implementations of published papers, maintained by their original authors or community contributors. For new projects, start with `official/`.

**Q: Can I use these models with TensorFlow Lite?**

Yes. The official vision models support TFLite export. The `ai-edge-litert` package (required dependency) provides the conversion tools. Research models like `seq_flow_lite` are specifically designed for on-device inference.

**Q: What is Orbit and when should I use it?**

Orbit is a training loop library that manages checkpointing, evaluation scheduling, and summary writing. Use it when you need a custom training loop with `tf.distribute` support but don't want to build the outer loop boilerplate yourself. The official models use Orbit internally.

**Q: Which distribution strategy should I use?**

Use `mirrored` (default) for single-machine multi-GPU training. Use `tpu` for Cloud TPUs. Use `multi_worker_mirrored` for multi-machine GPU clusters. Use `one_device` for single-GPU debugging. Use `off` to disable distribution entirely.

**Q: How do I use a TF Hub module instead of a checkpoint?**

Replace `--init_checkpoint` with `--hub_module_url` in your training command. For example: `--hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/1`.

**Q: What BERT checkpoints are available?**

BERT-Base and BERT-Large in uncased, cased, whole-word-masking, multilingual, and Chinese variants. All are available from `gs://cloud-tpu-checkpoints/bert/keras_bert/` and via TF Hub.

**Q: How do I create SQuAD 2.0 fine-tuning data?**

Add `--version_2_with_negative=True` to the `create_finetuning_data.py` command.

**Q: Is there a pretrained model for waste detection?**

Yes. The `official/projects/waste_identification_ml` project (CircularNet) provides Mask R-CNN models for waste identification with a Docker-based prediction API.

**Q: How do I enable XLA compilation?**

Pass `--enable_xla=True` to the training script. XLA can significantly speed up GPU training but may increase compilation time on the first step.

**Q: Where can I find YAML configs for specific experiments?**

Each project directory under `official/projects/` and `official/vision/configs/experiments/` contains task-specific YAML files organized by architecture and dataset.

---

_Generated May 2026 from tensorflow/models at 971ded9. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._