# Twitter's "The Algorithm" — User Guide

Twitter's recommendation algorithm is the system that decides what content appears in your Home timeline, who shows up in "Who to Follow," what tweets surface in Search, and which notifications you receive. This repository is the open-sourced codebase behind those systems — a massive Scala/Java/Python/Rust monorepo containing the services, ML models, and data pipelines that power Twitter's core product experience.

## 1. Quick Start

### What This Repository Contains

Twitter's "The Algorithm" is not a single program you run — it is the collection of interconnected backend services that together form Twitter's recommendation engine. The major subsystems are:

- **Home Timeline Ranking** — decides which tweets appear in "For You" and "Following" tabs
- **Follow Recommendations** — powers "Who to Follow" suggestions using graph algorithms and ML
- **Tweet Search (Earlybird)** — real-time search indexing and retrieval built on Apache Lucene
- **Content Recommendations (CR-Mixer)** — content-based tweet recommendations
- **Push Notifications** — the heavy/light ranking models that decide which notifications to send
- **Trust & Safety Models** — NSFW detection, toxicity classification, abusive content filtering
- **ML Infrastructure** — twml (Twitter ML library) and Navi (Rust model serving)

### Key Features

- Real-time tweet indexing and retrieval across billions of tweets
- Multi-stage candidate generation → ranking → filtering pipeline
- Graph-based recommendation algorithms (RealGraph, SimClusters, SALSA)
- Approximate Nearest Neighbor (ANN) service supporting HNSW, Annoy, FAISS, and brute-force algorithms
- Content visibility rules engine with hundreds of safety and policy rules
- Distributed ML training framework with TensorFlow integration
- Multi-backend model serving (TensorFlow, PyTorch, ONNX) via Navi
- Unified user action tracking across all product surfaces

### Prerequisites

- Java 11+ and Scala 2.12
- [Bazel](https://bazel.build/) build system
- Python 3.7+ (for ML components)
- Rust toolchain (for Navi model serving)
- Access to internal Twitter infrastructure services (Aurora, HDFS, Manhattan, Kafka) for full operation

> [!IMPORTANT]
> This codebase was designed to run within Twitter's internal infrastructure. Many services depend on internal systems (Aurora for scheduling, Manhattan for storage, Kafka for streaming, Strato for data access) that are not publicly available. You can study the code and build individual components, but running the full system requires these backend dependencies.

### Build a Component

```bash
./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder:indexbuilder-deploy --bundle-jvm-archive=zip
```

```text
INFO: Build completed successfully, 1 total action
```

> [!TIP]
> Start with a self-contained subsystem like `navi/` (Rust, builds with Cargo) or `twml/` (Python, installs with pip) if you want to run code rather than just study it.

### What to Explore Next

- **Home timeline ranking** — start at `home-mixer/` and `timelineranker/`
- **Follow recommendations** — start at `follow-recommendations-service/`
- **Search** — start at `src/java/com/twitter/search/`
- **ML training** — start at `twml/`
- **Model serving** — start at `navi/`

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Java | JDK 11 | JDK 11 |
| Scala | 2.12 | 2.12 |
| Python | 3.7 | 3.7 |
| Rust | 1.65+ | Latest stable |
| Bazel | 5.x | Per `.bazelversion` |
| RAM | 8 GB | 32 GB+ |
| Disk | 10 GB | 50 GB+ |

### Clone the Repository

```bash
git clone https://github.com/twitter/the-algorithm.git
cd the-algorithm
```

```text
Cloning into 'the-algorithm'...
```

### Building Scala/Java Services

All Scala and Java services use Bazel. Each service has its own build target:

```bash
# Build the Follow Recommendations Service
./bazel build follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations:follow_recommendations

# Build the Graph Feature Service server
./bazel build graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server:server
```

> [!WARNING]
> This is a very large repository (7,651 files). Building the full repository with `./bazel build //...` will require substantial memory and time. Build specific targets instead.

### Building Navi (Rust Model Serving)

Navi supports three ML backends via Cargo feature flags:

```bash
cd navi/navi

# Build with TensorFlow backend
cargo build --release --features tf

# Build with PyTorch backend
cargo build --release --features torch

# Build with ONNX backend
cargo build --release --features onnx
```

> [!NOTE]
> On Linux, the ONNX and TensorFlow builds automatically enable CUDA and TensorRT support. On other platforms, GPU acceleration features are excluded.

### Building twml (Twitter ML Library)

```bash
cd twml
pip install -e .
```

The native C++ components in `twml/libtwml/` require a C++ compiler:

```bash
cd twml/libtwml
make
```

### Post-Install Verification

Verify Bazel can resolve the dependency graph:

```bash
./bazel query 'deps(//home-mixer/server/...)' --output label 2>/dev/null | head -5
```

### Install Troubleshooting

| Problem | Cause | Solution |
|---|---|---|
| Bazel version mismatch | Wrong Bazel version installed | Check `.bazelversion` file and install the matching version |
| `cargo build` fails on `ort` | ONNX Runtime native library not found | Ensure `ort` crate can download or locate the ONNX Runtime; check proxy settings |
| `pip install twml` fails | Missing C++ compiler for native extensions | Install `gcc`/`g++` and `python3-dev` |
| Scala compilation OOM | Insufficient JVM heap for Bazel | Add `startup --host_jvm_args=-Xmx8g` to `.bazelrc` |
| TensorFlow import fails | Python version mismatch | twml targets Python 3.7 specifically; check compatibility |
| Protobuf build errors in Navi | `tonic-build` cannot find proto files | Ensure you build from the `navi/navi/` directory, not the repo root |
| Missing `thrift` dependency | Thrift compiler not installed | Install Apache Thrift for Rust bindings: `cargo install thrift` |
| CUDA not detected for Navi | CUDA toolkit not installed or not in PATH | Install CUDA 11.2+ and set `LD_LIBRARY_PATH` |
| Bazel sandbox errors on macOS | macOS sandbox restrictions | Try `--spawn_strategy=local` flag |
| `gh` CLI auth failures | GitHub CLI not authenticated | Run `gh auth login` before cloning private forks |

## 3. Configuration

Configuration in this repository is service-specific. Each service maintains its own configuration files, typically in YAML format within `server/src/main/resources/config/` directories.

### Service Configuration Locations

| Service | Config Path | Format |
|---|---|---|
| Follow Recommendations | `follow-recommendations-service/server/src/main/resources/config/` | YAML |
| Timeline Ranker | `timelineranker/server/config/` | YAML |
| CR-Mixer | `cr-mixer/server/src/main/resources/config/` | YAML |
| TweetyPie | `tweetypie/server/config/` | YAML |
| SimClusters ANN | `simclusters-ann/server/src/main/resources/config/` | YAML |
| Topic Social Proof | `topic-social-proof/server/src/main/resources/config/` | YAML |
| Representation Manager | `representation-manager/server/src/main/resources/config/` | YAML |
| Representation Scorer | `representation-scorer/server/src/main/resources/config/` | YAML |
| User Signal Service | `user-signal-service/server/src/main/resources/config/` | YAML |
| Tweet Mixer | `tweet-mixer/server/src/main/scala/com/twitter/tweet_mixer/config/` | Scala |
| Earlybird | `src/java/com/twitter/search/earlybird/config/` | Java |
| Recos Injector | `recos-injector/server/config/` | YAML |

### twml Environment Variables

The ML training framework reads these environment variables:

| Variable | Purpose |
|---|---|
| `TWML_JOB_CLUSTER` | Cluster identifier for distributed training jobs |
| `TWML_JOB_ROLE` | Role identifier for the training job |
| `TWML_JOB_ENV` | Environment name (staging, production) |
| `TWML_DISTRIBUTED_BASE_JOBNAME` | Base job name for distributed training |
| `HADOOP_CONF_DIR` | HDFS configuration directory for data access |

> [!TIP]
> For full configuration reference with every setting and default value, see the Admin Guide's Configuration Reference section.

## 4. Core Usage

### Understanding the Recommendation Pipeline

Twitter's recommendation system follows a multi-stage pipeline pattern across all product surfaces:

```text
User Request
    │
    ▼
┌─────────────────────┐
│ Candidate Generation │  ← Multiple sources produce thousands of candidates
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Light Ranking     │  ← Fast model filters to hundreds
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Heavy Ranking     │  ← Full model scores remaining candidates
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Filtering & Rules  │  ← Visibility rules, dedup, policy enforcement
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Product Mixing    │  ← Final assembly for the product surface
└─────────────────────┘
```

### Exploring the Home Timeline

The Home timeline ("For You" tab) pipeline is orchestrated by **Home Mixer** and uses **Timeline Ranker** for scoring:

1. **Candidate generation** — Earlybird retrieves in-network tweets; CR-Mixer provides out-of-network recommendations; SimClusters ANN finds topic-relevant content
2. **Feature hydration** — Graph Feature Service computes user-user graph features; User Signal Service provides behavioral signals
3. **Ranking** — Timeline Ranker applies the scoring model using features from multiple sources
4. **Filtering** — VisibilityLib applies safety rules and content policies
5. **Assembly** — Home Mixer assembles the final timeline using Product Mixer framework

> [!NOTE]
> The Product Mixer framework (`product-mixer/`) is Twitter's generic pipeline framework. Home Mixer, Tweet Mixer, and other product surfaces are all built on top of it. The framework provides pipeline orchestration, feature hydration, scoring, and response marshalling.

### Exploring Follow Recommendations ("Who to Follow")

The Follow Recommendations Service generates account suggestions through multiple candidate sources:

| Candidate Source | Algorithm | What It Finds |
|---|---|---|
| **RealGraph** | User-user engagement graph | Accounts you interact with most |
| **Sims (CosineFollow)** | Cosine similarity on follow graph | Accounts similar to ones you follow |
| **Sims (CosineList)** | Cosine similarity on list membership | Accounts on similar lists |
| **Sims (Follow2Vec)** | Word2Vec on follow sequences | Accounts with similar follower patterns |
| **SALSA** | Personalized random walk on bipartite graph | People You May Know via mutual follows |
| **Strong Tie Prediction** | Mutual follow prediction (online + offline) | High-probability mutual follow candidates |
| **Two-Hop Random Walk** | Engagement-weighted 2nd-degree expansion | Friends of friends ranked by engagement |
| **User-User Graph** | Real-time follow tracking (GraphJet) | Accounts your recent follows also followed |
| **Address Book** | Phone/email contact matching | People from your synced contacts |
| **Crowd Search** | Search click aggregation by country | Most-searched accounts in your country |
| **Pop Geo** | Geographic popularity (±20 km precision) | Most-followed accounts near you |
| **PPMI Locale Follow** | Pointwise Mutual Information by locale | Most-followed accounts for your language+country |
| **Triangular Loops** | Graph cycle detection (u→v→w→u) | Closing triangles in the follow graph |
| **Top Organic Follows** | Organic follow aggregation by country | Most organically followed accounts |
| **Sims Expansion** | 2nd-hop Sims on engaged/followed accounts | Similar accounts to your recent interactions |
| **Social Graph** | Recent follow expansion | Accounts your followings recently followed |
| **Recent Engagement** | Explicit + implicit engagement tracking | Accounts you recently liked, retweeted, or visited |
| **Promoted Accounts** | Ads server | Sponsored account suggestions |

After candidate generation, candidates flow through ranking (multi-tower MLP model trained on Profile Sidebar impression data) and filtering before display.

> [!TIP]
> Each candidate source has its own README in `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/`. These contain detailed algorithm explanations with academic references.

### Exploring Tweet Search (Earlybird)

Earlybird is Twitter's real-time search system, built on Apache Lucene:

- **Realtime cluster** — indexes all public tweets from the past ~7 days
- **Protected cluster** — indexes protected tweets for the same timeframe
- **Archive cluster** — indexes all tweets ever posted, up to ~2 days ago

Each cluster is partitioned and replicated. Earlybird roots fan out requests to partitions and merge responses.

Key data flow:

```text
Tweets → Kafka → Ingesters → Extract fields/features → Kafka → Earlybird (index + serve)
                                                                     ↑
Feature Update Service → engagement counts (likes, retweets, replies) ┘
```

> [!NOTE]
> The Earlybird system uses a distributed inverted index with incremental (real-time) indexing. Its single-writer, multiple-reader architecture enables concurrent updates and queries while maintaining low latency. See the [original Earlybird paper](http://notes.stephenholiday.com/Earlybird.pdf) for the foundational design.

### Using the twml Training Framework

twml is Twitter's ML training library built on TensorFlow. It provides a `Trainer` class with standardized training, evaluation, and export workflows.

```bash
python my_model.py \
  --save_dir hdfs:///user/$USER/my_model \
  --train.batch_size 32 \
  --train.learning_rate 0.002 \
  --train.steps 10000 \
  --eval.steps 1000 \
  --optimizer Adam
```

> [!WARNING]
> The `--distributed` flag enables distributed training but is incompatible with early stopping (`--train.early_stop_patience`). Do not use both simultaneously.

### Using Navi for Model Serving

Navi is a high-performance Rust-based model inference server that supports TensorFlow, PyTorch, and ONNX models. It serves predictions via gRPC using the TensorFlow Serving API protocol.

Run Navi with a specific backend:

```bash
# TensorFlow backend
./navi --model-dir /path/to/models --config /path/to/config.json

# PyTorch backend
./navi_torch --model-dir /path/to/models --config /path/to/config.json

# ONNX backend
./navi_onnx --model-dir /path/to/models --config /path/to/config.json
```

Navi supports:
- Automatic model version detection and hot-reloading
- Batched prediction with configurable batch sizes and timeouts
- Prometheus metrics export via HTTP
- mTLS with x509 certificate verification
- Multiple model serving from a single instance

## 5. Common Tasks

### Navigate the Codebase by Subsystem

**Goal:** Find the code responsible for a specific product feature.

1. Identify the subsystem from the top-level directories:

| Directory | Subsystem |
|---|---|
| `home-mixer/` | Home timeline assembly |
| `timelineranker/` | Timeline ranking and scoring |
| `follow-recommendations-service/` | Who to Follow |
| `cr-mixer/` | Content recommendations |
| `product-mixer/` | Pipeline framework (shared) |
| `tweetypie/` | Tweet storage and hydration |
| `pushservice/` | Push notifications |
| `visibilitylib/` | Content moderation rules engine |
| `src/java/com/twitter/search/` | Tweet search (Earlybird) |
| `src/scala/com/twitter/simclusters_v2/` | Topic clustering and embeddings |
| `twml/` | ML training framework |
| `navi/` | ML model serving (Rust) |
| `ann/` | Approximate Nearest Neighbor service |
| `graph-feature-service/` | Graph-based features |
| `user-signal-service/` | User behavior signals |
| `unified_user_actions/` | Event tracking and analytics |
| `representation-manager/` | Embedding storage and retrieval |
| `representation-scorer/` | Embedding similarity scoring |
| `recos-injector/` | Recommendation event injection |
| `trust_and_safety_models/` | Safety ML models (NSFW, toxicity) |
| `topic-social-proof/` | Topic relevance scoring |
| `simclusters-ann/` | SimCluster-based nearest neighbors |
| `tweet-mixer/` | Tweet mixing and blending |
| `timelines/` | Timeline data processing pipelines |

2. Within each subsystem, follow the standard directory structure:
   - `server/src/main/scala/` — service implementation
   - `server/src/main/resources/config/` — configuration
   - `common/` — shared models and utilities
   - `thrift/` — Thrift IDL definitions
   - `client/` — client libraries

### Build and Bundle a Service for Deployment

**Goal:** Create a deployable artifact for a service.

```bash
./bazel bundle follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations:follow_recommendations-deploy --bundle-jvm-archive=zip
```

```text
INFO: Build completed successfully
```

The output bundle appears in `dist/`.

### Build an ANN Index

**Goal:** Build an Approximate Nearest Neighbor index from embeddings on HDFS.

```bash
export JOB_NAME=ann_index_builder
export OUTPUT_PATH=hdfs:///user/$USER/${JOB_NAME}_test

aurora job create smf1/$USER/devel/$JOB_NAME \
  ann/src/main/aurora/index_builder/aurora_builder.aurora \
  --bind=profile.name=$JOB_NAME \
  --bind=profile.role=$USER \
  --bind=profile.output_dir=$OUTPUT_PATH \
  --bind=profile.entity_kind=user \
  --bind=profile.embedding_args='--input.embedding_format tab --input.embedding_path /path/to/embeddings' \
  --bind=profile.num_dimensions=300 \
  --bind=profile.algo=hnsw \
  --bind=profile.ef_construction=200 \
  --bind=profile.max_m=16 \
  --bind=profile.expected_elements=10000000 \
  --bind=profile.metric=InnerProduct \
  --bind=profile.concurrency_level=32 \
  --bind=profile.hadoop_cluster=dw2-smf1
```

> [!NOTE]
> ANN supports three index algorithms: **HNSW** (Cosine, L2, InnerProduct), **Annoy** (Cosine, L2), and **brute_force** (Cosine, L2, InnerProduct). FAISS is also available for load testing.

### Train an ML Model with twml

**Goal:** Train a ranking model using the twml framework.

1. Define your model class extending twml's `Trainer`
2. Configure training parameters:

```bash
python train.py \
  --save_dir hdfs:///user/$USER/model_output \
  --train.batch_size 64 \
  --train.learning_rate 0.001 \
  --train.max_steps 50000 \
  --eval.batch_size 64 \
  --eval.period 600 \
  --eval.delay 120 \
  --optimizer Adam \
  --config.save_checkpoints_secs 600 \
  --config.keep_checkpoint_max 20
```

3. Monitor training via TensorBoard (enabled by default on an auto-assigned port)
4. Export the trained model for serving

**If training stalls:** Check `--train.early_stop_patience` and `--train.early_stop_tolerance` settings. Use `--train.log_metrics` for debugging (note: this slows training by 10-20%).

### Run a Load Test on an ANN Query Service

**Goal:** Benchmark ANN query service performance.

```bash
aurora job create smf1/$USER/staging/ann-loadtest \
  ann/src/main/aurora/loadtest/loadtest.aurora \
  --bind=profile.name=ann-loadtest \
  --bind=profile.role=$USER \
  --bind=profile.duration_sec=10 \
  --bind=profile.number_of_neighbors=10 \
  --bind=profile.qps=200 \
  --bind=profile.algo=hnsw \
  --bind=profile.metric=Cosine \
  --bind=profile.index_id_type=int \
  --bind=profile.hnsw_ef=400,600,800 \
  --bind=profile.embedding_dimension=3 \
  --bind=profile.concurrency_level=8 \
  --bind=profile.loadtest_type=remote \
  --bind=profile.service_destination=/srv#/staging/local/$USER/ann-server-test \
  --bind=profile.with_random_queries=True \
  --bind=profile.random_queries_count=50000
```

Load test results are printed to the Aurora job's stdout.

### Compute Feature Importance

**Goal:** Determine which features matter most in a trained model.

```bash
python my_model.py \
  --feature_importance.algorithm tree \
  --feature_importance.sensitivity 0.03 \
  --feature_importance.example_count 10000 \
  --feature_importance.metric roc_auc \
  --feature_importance.is_metric_larger_the_better \
  --feature_importance.data_dir hdfs:///user/$USER/eval_data
```

> [!TIP]
> The `tree` algorithm is faster for models with many features — it groups features by name prefix and only expands groups whose removal impacts the metric above the sensitivity threshold. Use `serial` only when you need exact per-feature importance values.

### Generate Sample Query Sets for ANN Benchmarking

**Goal:** Create query, index, and truth sets for recall benchmarking.

```bash
./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline:ann-offline-deploy

export QUERY_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/embeddings
export INDEX_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/embeddings
export METRIC=InnerProduct
export NEIGHBOURS=10

oscar hdfs \
  --screen --tee log.txt \
  --bundle ann-offline-deploy \
  --host hadoopnest1.smf1.twitter.com \
  --tool com.twitter.ann.scalding.offline.KnnTruthSetGenerator -- \
  --neighbors $NEIGHBOURS \
  --metric $METRIC \
  --query_entity_kind user \
  --query.embedding_path $QUERY_EMBEDDINGS_PATH \
  --query.embedding_format tab \
  --query_sample_percent 50.0 \
  --index_entity_kind user \
  --index.embedding_path $INDEX_EMBEDDINGS_PATH \
  --index.embedding_format tab \
  --index_sample_percent 90.0 \
  --query_set_output.embedding_path /user/$USER/query_set \
  --query_set_output.embedding_format tab \
  --index_set_output.embedding_path /user/$USER/index_set \
  --index_set_output.embedding_format tab \
  --truth_set_output_path /user/$USER/truth_set \
  --reducers 100
```

## 6. Troubleshooting

### Error: `ValueError: Path for zookeeper dataset pointer is None`

**Cause:** twml's `BlockFormatDataset` requires a ZooKeeper path for the dataset pointer, but none was specified.

**Solution:**
1. Ensure you pass a valid ZooKeeper dataset path in your data pipeline configuration
2. If running locally without ZooKeeper, provide data via a local file path instead

**Prevention:** Always specify `--data_dir` or the appropriate dataset path argument when running training.

### Error: `CheckpointNotFoundError: ... not found`

**Cause:** The training framework cannot locate a checkpoint at the specified `--save_dir` path.

**Solution:**
1. Verify the `--save_dir` path exists and is accessible (check HDFS permissions if using HDFS)
2. If starting fresh, use `--overwrite_save_dir` to skip checkpoint restoration

**Prevention:** Ensure HDFS paths use the correct prefix format (`hdfs:///user/...`).

### Error: `ModuleNotFoundError: Tensorflow has not found the GPU`

**Cause:** The trust and safety model training scripts require GPU access via TensorFlow, but CUDA is not properly configured.

**Solution:**
1. Verify CUDA toolkit installation: `nvidia-smi`
2. Ensure TensorFlow GPU package is installed: `pip install tensorflow-gpu`
3. Check `LD_LIBRARY_PATH` includes CUDA libraries

**Prevention:** Use the provided Dockerfile at `ann/src/main/python/dataflow/worker_harness/Dockerfile` as a reference for a working CUDA environment (CUDA 11.2, cuDNN 8, Ubuntu 20.04).

### Error: `PEM parsing failed` (Navi)

**Cause:** Navi's mTLS bootstrap cannot parse the provided PEM certificate file.

**Solution:**
1. Verify the certificate file is valid PEM format: `openssl x509 -in cert.pem -text -noout`
2. Ensure the file path in the Navi configuration is correct
3. Check file permissions allow the Navi process to read the certificate

**Prevention:** Always validate certificates before deploying model-serving instances.

### Error: `model sync cli empty` (Navi)

**Cause:** Navi's predict service could not find a model sync command in the configuration.

**Solution:**
1. Verify the model configuration JSON includes the `model_dir` field with valid paths
2. Ensure model directories contain versioned subdirectories with the expected model artifacts

**Prevention:** Use the config validation in `navi/navi/src/lib.rs` (`read_config`) to check configuration before starting the service.

### Error: `unsupport input tensor type` (Navi)

**Cause:** The input tensor type does not match what the model expects.

**Solution:**
1. Check the model's input signature to verify expected tensor types
2. Ensure the Thrift `BatchPredictionRequest` contains features in the correct format (see `DataRecord` structure in `navi/thrift_bpr_adapter/`)

> [!WARNING]
> Navi's error messages use Rust's `{:?}` debug formatting. The actual error will include the mismatched type information — read the full message carefully.

### Error: `Unsupported params.learning_rate_decay`

**Cause:** An invalid learning rate decay schedule name was passed to twml.

**Solution:** Use one of the supported schedules:
- `exponential` (requires `--decay_steps`, `--exponential_decay_rate`)
- `polynomial` (requires `--decay_steps`, `--end_learning_rate`)
- `piecewise_constant` (requires boundary and value lists)
- `inverse` (requires `--decay_steps`, `--decay_rate`, `--min_learning_rate`)
- `cosine_decay` (requires `--decay_steps`, `--alpha`)
- `cosine_restarts_decay` (requires `--first_decay_steps`, `--t_mul`, `--m_mul`, `--alpha`)

**Prevention:** Verify the schedule name and all required companion arguments before launching training.

### Bazel Build Fails with Memory Errors

**Cause:** The repository is very large (7,651 files) and Bazel's analysis phase can exceed available memory.

**Solution:**
1. Increase Bazel's JVM heap: add `startup --host_jvm_args=-Xmx8g` to `.bazelrc`
2. Build individual targets rather than entire directory trees
3. Use `--local_ram_resources` to limit parallel actions

**Prevention:** Build specific service targets rather than using broad patterns like `//...`.

### Service Fails at Startup: Missing Internal Dependencies

**Cause:** Services depend on internal Twitter infrastructure (Aurora, Manhattan, Strato, Kafka, ZooKeeper) that is not available in the open-source distribution.

**Solution:**
1. This is expected — the repository was open-sourced for transparency, not for standalone deployment
2. Study the service code and configuration to understand the architecture
3. For individual components (Navi, twml, trust_and_safety_models), local execution is possible with appropriate data substitution

> [!CAUTION]
> Do not attempt to run the full service mesh without the internal infrastructure dependencies. The services will fail at startup when they cannot connect to Aurora, Manhattan, or internal Kafka clusters.

### Error: `This loss name is not valid` (Trust & Safety)

**Cause:** An invalid loss function name was passed to the toxicity model trainer.

**Solution:** Use one of: `BCE`, `masked BCE`, `CCE`, `sCCE`, `Focal_BCE`, `inv_KL_loss`.

**Prevention:** Check `trust_and_safety_models/toxicity/load_model.py` for the current list of supported loss functions.

### Error: `This is not an experiment id as defined in the settings file`

**Cause:** The toxicity trainer received an experiment ID that does not exist in the settings configuration.

**Solution:**
1. Check `trust_and_safety_models/toxicity/settings/` for available experiment definitions
2. Ensure the experiment ID string matches exactly

**Prevention:** List available experiment IDs from the settings file before launching training.

## 7. FAQ

**Q: Can I run these services locally?**

Most services cannot run standalone because they depend on Twitter's internal infrastructure (Aurora for deployment, Manhattan for storage, Kafka for streaming, Strato for data access). However, individual components like Navi (model serving), twml (ML training), and the trust_and_safety_models can be run independently with appropriate data inputs.

**Q: What build system does this use?**

Bazel for all Scala/Java services. Rust components (Navi, segdense, dr_transform) use Cargo. Python components (twml, trust_and_safety_models, pushservice models) use standard pip/setuptools.

**Q: What is the Product Mixer framework?**

Product Mixer (`product-mixer/`) is Twitter's internal framework for building product recommendation pipelines. It provides standardized abstractions for pipeline orchestration, candidate generation, feature hydration, scoring, filtering, and response marshalling. Home Mixer, Tweet Mixer, and other product surfaces are built on top of it.

**Q: How does the "For You" tab work at a high level?**

The For You tab combines in-network tweets (from accounts you follow, retrieved by Earlybird) with out-of-network recommendations (from CR-Mixer and SimClusters ANN). These candidates pass through feature hydration (Graph Feature Service, User Signal Service), heavy ranking (Timeline Ranker), visibility filtering (VisibilityLib), and final assembly (Home Mixer).

**Q: What are SimClusters?**

SimClusters is Twitter's topic clustering system. It maps users and tweets into a shared embedding space based on engagement patterns, enabling topic-based recommendations. The `simclusters_v2` directory contains embedding generation pipelines (Scalding/Scio batch jobs) and the `simclusters-ann` service provides nearest-neighbor lookups on these embeddings.

**Q: What distance metrics does the ANN service support?**

**InnerProduct**, **Cosine**, and **L2**. Not all algorithms support all metrics: HNSW and brute_force support all three; Annoy supports only Cosine and L2.

**Q: What ML frameworks are used?**

TensorFlow is the primary ML framework, accessed through Twitter's `twml` library. PyTorch and ONNX are supported via Navi's runtime backends. The trust and safety models use Keras (TensorFlow). The push notification heavy ranker uses a custom ClemNet architecture built with `tf.keras.layers`.

**Q: What is the VisibilityLib?**

VisibilityLib (`visibilitylib/`) is Twitter's content moderation rules engine. It evaluates hundreds of visibility rules against content and users to determine what should be shown, filtered, or hidden. Rules cover safety policies, legal requirements, and product-specific display logic. The `visibilitylib/src/main/scala/com/twitter/visibility/models/` directory contains 24 model files defining content labels, actions, and rule outcomes.

**Q: What is RealGraph?**

RealGraph is a user-user graph dataset that measures relationship strength between users based on real-time and historical engagement signals (favorites, replies, retweets, clicks, follows, mutes, blocks). It powers follow recommendations, timeline ranking, and various feature hydration pipelines. Based on the paper "Realgraph: User interaction prediction at Twitter" (KDD'14 UEO workshop).

**Q: What entity types does the ANN service support?**

The ANN service supports embeddings indexed by: `user`, `tweet`, `word`, `url`, and `tfwId` entity types. The entity kind determines how IDs are serialized and stored in the index.

**Q: How do push notifications get ranked?**

Push notifications use a four-stage pipeline: candidate generation → light ranking (fast MLP to pre-filter) → heavy ranking (ClemNet multi-task model predicting open + engage probabilities) → quality control. The models are in `pushservice/src/main/python/models/`.

---

_Generated May 2026 from twitter/the-algorithm at c54bec0. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._