DisTrO Administrator & Developer Guide

This guide is written for operators, architects, researchers, and developers who need to evaluate, plan for, or integrate DisTrO (Distributed Training Over-the-Internet) into a real system. It treats the NousResearch/DisTrO repository honestly: as a research announcement artifact, not a deployable system. The operational substance lives downstream in the Psyche project.

Important

Do not plan a deployment against this repository. It contains only a PDF report (A_Preliminary_Report_on_DisTrO.pdf) and a README.md research timeline. There are no binaries, no libraries, no configs, no deployment manifests, and no runtime dependencies here. Deployment concerns for the DisTrO optimizer family are properly owned by Psyche. This guide translates what the preliminary report says into operational language so you can plan intelligently before moving to Psyche.

1. Executive Summary

What this repository is

A two-file release artifact announcing the DisTrO family of distributed optimizers for neural network training. It consists of:

• A 7-page preliminary technical report (PDF, ~1.2 MB).
• A README.md with a chronological timeline of the research and its production descendants.

What this repository is not

• A runnable service.
• A Python or Rust library.
• A training framework.
• An active support channel for production DisTrO deployments.

What you operate against instead

• PsycheFoundation/psyche — the production training network and canonical DisTrO implementation.
• bloc97/DeMo — the original seed research code for DeMo (the optimizer family DisTrO descends from).
• huggingface/nanotron — the pre-training framework used in the 1.2B benchmark reported here.

Why DisTrO matters operationally

The DisTrO family of optimizers was demonstrated in the preliminary report to reduce per-step inter-GPU communication by 857x (measured) on a 1.2B parameter pre-training run — from 74.4 GB per step under standard AdamW + All-Reduce to 86.8 MB per step under DisTrO-AdamW — with no degradation in convergence rate or final model quality on six GPT4All zero-shot benchmarks.

Operational consequence: training large models is no longer gated on specialized high-bandwidth data-center interconnects. Consumer-grade internet connections — including asymmetric home bandwidth profiles (e.g., 100 Mbps down / 10 Mbps up) — become viable training participants, per the analysis in Section 3.1 of the preliminary report.

Who should read this document

2. Architecture

Because this repository contains no code, this section describes the conceptual operational architecture of DisTrO as the preliminary report presents it. Engineering specifics (wire format, checkpoint format, consensus mechanism, peer discovery, and so on) are Psyche concerns and are not claimed here.

2.1 What DisTrO replaces

Standard Distributed Data Parallel (DDP) training synchronizes full gradients every step:

  +---------+      All-Reduce      +---------+
  | GPU 0   | <------------------> | GPU 1   |
  | full M  |    ~2.3 GB/step per  | full M  |
  +---------+    node (1.2B ex.)   +---------+
         \                          /
          \                        /
           \      +---------+     /
            <-----| GPU 2   | <---
                  | full M  |
                  +---------+

Every node maintains a full copy of the model M. Every step, full gradient tensors are shared via All-Reduce, requiring extremely high bandwidth between all nodes. In the 1.2B Llama 2 benchmark with 32 H100s, that bandwidth worked out to 74.4 GB received per node per step.

2.2 What DisTrO looks like operationally

DisTrO operates as a drop-in optimizer replacement that transmits a far smaller tensor per step:

  +---------+    86.8 MB / step    +---------+
  | GPU 0   | <------------------> | GPU 1   |
  | full M  |   (857x reduction    | full M  |
  +---------+    vs. All-Reduce)   +---------+
         \                          /
          \                        /
           \      +---------+     /
            <-----| GPU 2   | <---
                  | full M  |
                  +---------+

Key differences from All-Reduce-based DDP:

• No optimizer state synchronization. DisTrO does not sync optimizer states across nodes. This is called out explicitly in Section 2 of the preliminary report.
• Drop-in AdamW substitution. In the benchmark, the team "swapped out AdamW with DisTrO-AdamW without changing hyper-parameters and disabled every instance of the All-Reduce operation in Nanotron."
• Clock-synchronous. Every node executes the same arithmetic and takes the same wall time per step, like SGD or Adam.
• Stateful vs. stateless variants. The report tested the stateful variant. A stateless variant exists (conceptually similar to Federated Averaging) but was not benchmarked in the preliminary report.

Note

The preliminary report uses DDP (every node holds the full model in VRAM). DisTrO's interaction with FSDP or tensor-parallel strategies — where the model is sharded across nodes — is explicitly listed in Section 3.3 as future work, not a current capability.

2.3 Deployment topologies

Section 3.1 of the preliminary report discusses three viable topology patterns for the data exchange:

Peer-to-peer

Every node sends to and receives from every other node. With 32 nodes and an 857x reduction, each node uploads and downloads ~86.8 MB per step. Symmetric bandwidth required.

Aggregation servers

Each training node uploads a single copy of its contribution (~2.8 MB per step in the 32-node case) to one or more aggregation servers, then receives the aggregated result (~86.8 MB per step). Asymmetric bandwidth profile fits consumer ISPs, where download is typically 10x upload.

Hybrid

A mix of peer-to-peer and aggregation, trading off coordinator dependence against bandwidth symmetry.

Tip

The asymmetric (aggregation server) topology is the operational sweet spot for consumer internet. With 100 Mbps down / 10 Mbps up, the worst-case per-step latency was reported as 6.94 s for downloading and 2.24 s for uploading — total 6.94 s per step wall-clock contribution from network I/O.

2.4 Component inventory (this repository)

2.5 Component inventory (downstream, not in this repository)

3. System Requirements

This section translates the benchmark and the design claims into operational requirements. All figures below are from the preliminary report. Nothing about install paths, OS compatibility, package versions, or runtime dependencies is claimed — because nothing in this repository establishes those facts.

3.1 Hardware requirements (as-benchmarked)

The preliminary report's 1.2B Llama 2 training run used:

3.2 Minimum per-node compute requirements

Because DDP keeps the full model in VRAM on every node, per-node VRAM requirements are driven by model size, not by DisTrO itself. The report says DisTrO was shown to converge "consistently" on the 1.2B scale and that smaller models (50M and 300M tested at 10B tokens) did not converge as well under DisTrO. This implies a practical lower bound on model size for DisTrO's bandwidth benefits to apply.

Warning

Do not plan a DisTrO deployment for models smaller than roughly 1B parameters without independent validation. Section 3.2 of the preliminary report is explicit that the 1.2B model is "the smallest size that we found that DisTrO works with consistently."

3.3 Network requirements

The report characterizes DisTrO as network-agnostic but quantifies the per-step byte budgets under DDP:

Against a 100 Mbps down / 10 Mbps up consumer connection:

3.4 Software requirements

This repository does not publish software requirements. For the production network, see the Psyche repository. The benchmark used:

• Nanotron (huggingface/nanotron) — minimalistic LLM 3D-parallelism training framework.
• AdamW baseline with β1 = 0.9, β2 = 0.95, peak learning rate 4 × 10⁻⁴, cosine decay schedule, weight decay 0.1, 400 warmup steps.
• Dataset: the first 105B tokens of a random 10% sample of Dolma v1.7 (huggingface.co/datasets/emozilla/dolma-v1_7-305B).

Note

The preliminary report does not publish the exact Nanotron commit, Dolma sampling script, or the DisTrO-AdamW source used in the benchmark. Reproducing the exact numbers requires reconstructing the benchmark against the DeMo v2 paper and the Psyche codebase.

3.5 Scale boundaries

4. Configuration Reference

4.1 Configuration surface in this repository

None. There are no configuration files, environment variables, CLI flags, or manifest schemas in this repository. Every setting discussed below is inferred from the preliminary report's benchmark setup, not extracted from executable code.

4.2 Hyper-parameter reference (as-tested)

The preliminary report used the following hyper-parameters for the DisTrO + AdamW run. These are the only configuration facts that can be documented from this repository.

Model hyper-parameters (Table 1 of the preliminary report):

AdamW hyper-parameters:

DisTrO-specific configuration: the preliminary report notes that the team "swapped out AdamW with DisTrO-AdamW without changing hyper-parameters and disabled every instance of the All-Reduce operation in Nanotron." The specific DisTrO-AdamW parameters (compression ratios, decoupled-momentum settings, etc.) are not documented in this preliminary report and must be looked up in the DeMo v2 paper or in Psyche's distro.rs.

Caution

If you are rebuilding the benchmark from scratch, the DisTrO-internal parameters are non-trivial and are intentionally out of scope for the preliminary report. The report repeatedly notes that a more detailed paper with full methodology was planned — this is now the DeMo v2 paper (Oct 2025). Use DeMo v2 as the authoritative configuration reference.

5. Deployment

5.1 Deploying this repository

There is nothing to deploy from this repository. Cloning it gives you a PDF and a README.

5.2 Deploying DisTrO-based training

Deployment of a DisTrO-based training system is the responsibility of downstream projects. The canonical production deployment is the Psyche Network — a permissionless, decentralized training network.

For a permissionless (Psyche) deployment:

• [ ] Review the Psyche Network announcement for the high-level operational model.
• [ ] Read the DeMo v2 paper for the current optimizer semantics.
• [ ] Clone and build PsycheFoundation/psyche. Follow its deployment documentation.
• [ ] Join the Nous Research Discord for coordination on active training runs.

For an internal (non-Psyche) DisTrO deployment:

• [ ] Choose a deployment topology (peer-to-peer, aggregation server, or hybrid — see Section 2.3).
• [ ] Size per-node bandwidth for the chosen topology and target model size. Budget ~86.8 MB per step per node in the download direction for a 32-node 1.2B run, and scale from there.
• [ ] Choose a training framework. The benchmark used Nanotron; any DDP-capable framework should work in principle.
• [ ] Integrate a DisTrO-family optimizer. Today, the only publicly available production-grade implementation is in the Psyche Rust codebase.
• [ ] Validate convergence on a small run before scaling up.

Warning

Do not treat the 857x figure as a guaranteed outcome. The report measured 857x on a specific benchmark (32 H100s, 1.2B Llama 2, Dolma v1.7). Section 3.2 explicitly says bandwidth scaling behavior as model size changes is an open research question. Section 3 also notes that achievable reduction depends on hyper-parameter choices. Run your own calibration.

5.3 Roll-forward / roll-back

This repository is an immutable announcement. The only relevant roll-forward is to follow the research timeline to its current state:

Aug 2024: Preliminary Report (this repo)
    |
    v
Dec 2024: DeMo paper + code (bloc97/DeMo)
    |
    v
Dec 2024: 15B DisTrO training run (distro.nousresearch.com)
    |
    v
May 2025: Psyche Network goes live
    |
    v
May 2025: Consilience 40B trained on Psyche
    |
    v
Oct 2025: DeMo v2 paper + Psyche production code (distro.rs)

"Rolling back" to an earlier state is not an operational concern — each artifact is published independently. If you are planning to use DisTrO, plan against DeMo v2 and the current Psyche codebase, not against this preliminary report.

6. Security

6.1 Attack surface of this repository

The attack surface is negligible. This repository contains:

• A static PDF file (A_Preliminary_Report_on_DisTrO.pdf).
• A static Markdown README.

There is no executable code, no build system, no published dependency, no supply-chain surface, and no network service. Cloning this repository does not install or execute anything.

6.2 Security considerations for DisTrO training networks

The preliminary report (Section 3.3.2) discusses — but does not implement — security concerns that arise once DisTrO is deployed in a permissionless decentralized setting:

• Malicious / Byzantine nodes. The report says "DisTrO is remarkably resilient to a small amount of degraded or dropped nodes during training" and that "cryptographic primitives and incentive structures could be layered on to recruit participants and mitigate risks of non-trusted nodes damaging a run using adversarial attacks." This is a design-intent statement, not a demonstrated threat model.
• Adversarial gradient attacks. Not analyzed in the preliminary report. This is a live research area for any federated / decentralized training system.
• Data-privacy properties. The report explicitly declines to focus on privacy ("we do not focus on the data privacy aspect of training"), while noting that DisTrO's stateless variant is structurally suitable for federated learning patterns.

Caution

The preliminary report does not claim any specific cryptographic or Byzantine-fault-tolerance guarantee. If you are evaluating DisTrO for a threat-sensitive deployment, the security properties you care about are properties of the full system (Psyche or your internal integration), not of the optimizer alone.

6.3 Secrets and credentials

There are no secrets, API keys, or credentials in this repository, and none are required to read the PDF or README.

6.4 Transport security

Tensor transfer between nodes is a downstream concern handled by whatever training network you use (e.g., Psyche). The preliminary report does not prescribe TLS, authentication schemes, or peer authentication — these are implementation details of the training system, not of the optimizer.

7. Operations

7.1 Monitoring

There is nothing to monitor in this repository. It is a static release artifact.

For a DisTrO-based training run, the operationally interesting signals to monitor are:

Tip

Use GPT4All zero-shot benchmarks (ARC-c, ARC-easy, Hellaswag, OpenBookQA, PIQA, WinoGrande) as your intermediate-checkpoint quality signal. The preliminary report used exactly these six benchmarks and found DisTrO matches or exceeds the All-Reduce baseline across all six — they are therefore the natural validation target for DisTrO convergence claims.

7.2 Start / stop / restart

Not applicable to this repository. For a Psyche deployment, refer to the Psyche repository's operations documentation.

7.3 Backup and recovery

Not applicable to this repository. Standard checkpointing practices apply to any DisTrO-based training run.

7.4 Performance tuning

The preliminary report gives three operationally relevant tuning signals:

1. Careful hyper-parameter selection can push bandwidth reduction from 857x to 1000x–3000x during pre-training. The report does not list which hyper-parameters — defer to DeMo v2 for specifics.
2. Fine-tuning can reach up to 10000x reduction with no noticeable loss in quality.
3. Smaller models do not converge as well — the practical operating range begins around 1.2B parameters.

Tip

If your operating regime is fine-tuning, not pre-training, expect roughly an order of magnitude better bandwidth reduction than the 857x headline number.

8. Developer Internals

8.1 Repository structure

NousResearch/DisTrO/
├── A_Preliminary_Report_on_DisTrO.pdf   The 7-page technical announcement
└── README.md                            Chronological research timeline

This is the complete repository. There are no source directories, build systems, test suites, CI configurations, or dependency manifests.

8.2 Research lineage and code provenance

Understanding where the code comes from is the key developer task for anyone trying to work with DisTrO:

8.3 The DisTrO optimizer mechanism (from the preliminary report)

The preliminary report gives a high-level description of the stateful variant's mechanism without publishing the full algorithm (deferred to DeMo v2):

• Decoupled momentum: DisTrO adapts the Decoupled Momentum Optimization (DeMo) technique to the distributed setting.
• No All-Reduce on gradients: instead of synchronizing full gradient tensors, each node computes a compact representation that can be aggregated with far fewer bytes.
• No optimizer state sync: each node maintains its own optimizer state independently. Only the compact gradient representation is transmitted.
• Clock-synchronous updates: all nodes advance at the same wall-clock rate, preserving the synchronous convergence properties of SGD/Adam.

For the complete algorithm, including compression details, momentum decoupling formulation, and convergence analysis, read the DeMo v2 paper at openreview.net/pdf?id=U9oewpa7cn and the production implementation at distro.rs.

8.4 Open research questions (from Section 3.2 of the preliminary report)

The preliminary report explicitly flags these as unresolved:

1. Bandwidth scaling with model size: it is an open question whether the reduction factor grows or shrinks as model parameters scale beyond 1.2B. The intuition is that larger models may benefit more, but this is not quantified.
2. Sub-1B convergence: DisTrO did not converge consistently on 50M and 300M parameter models in 10B-token runs. The reason is not analyzed in the report.
3. Compression: current implementation transmits tensors without compression. Adding compression is listed as a future improvement that could push the 857x reduction higher.
4. MoE and sparse architectures: DisTrO is demonstrated only on dense networks. Applicability to sparse models is not claimed.
5. FSDP and model-parallel integration: the benchmark uses DDP (full model per node). Compatibility with FSDP or tensor parallelism (which shard the model) is future work.

8.5 Contributing

This repository is a static release artifact and is not actively maintained as a development target. Contributions to DisTrO technology go to:

• PsycheFoundation/psyche — production code (Rust).
• bloc97/DeMo — research-grade code.
• Nous Research Discord (discord.com/invite/jqVphNsB4H) — community discussion and coordination.

Issues filed against this repository are unlikely to receive timely responses. GitHub issue #2 ("why not publish the code same time as paper") and #3 ("Request for Update and Method Documentation") are both closed — the answer to both is now the DeMo v2 paper and Psyche codebase.

9. Integrations

9.1 Integration surface of this repository

None. This repository has no API, SDK, plugin system, or integration points.

9.2 Integration surface of DisTrO as a concept

DisTrO integrates into the training stack at the optimizer layer. In a DDP-based framework, this typically means:

1. Replacing the optimizer class (e.g., AdamW) with a DisTrO-family equivalent (e.g., DisTrO-AdamW).
2. Disabling framework-level All-Reduce collectives — DisTrO handles inter-node communication internally.
3. Leaving per-node model initialization, dataloading, and loss computation unchanged.

The preliminary report demonstrates exactly this pattern with Nanotron:

"For the DisTrO training run, we swapped out AdamW with DisTrO-AdamW without changing hyper-parameters and disabled every instance of the All-Reduce operation in Nanotron."

Note

No "DisTrO adapter" interface is defined in this repository. The integration shape is defined by whichever implementation you use — in practice, Psyche's Rust implementation.

9.3 Future integration opportunities the report calls out

Section 3.3 of the preliminary report identifies three directions where DisTrO may compose with other systems, all framed as forward-looking:

• Federated Learning. DisTrO's stateless variant is structurally compatible with Federated Averaging-style patterns.
• Decentralized Training. DisTrO + cryptographic incentive mechanisms → a permissionless training network. This is what Psyche is.
• Memory-parallelism methods. DisTrO + FSDP or SWARM Parallelism could extend training to smaller-VRAM consumer hardware.

10. Troubleshooting

Operational troubleshooting for this repository maps to what to do when users land here expecting code. Deeper training-run troubleshooting belongs to Psyche.

Error: "There's nothing to run in this repository"

Cause: This repository is a research announcement. It has no runnable code.

Solution: Redirect the user or the deployment pipeline to PsycheFoundation/psyche, which is the canonical production implementation.

Prevention: Treat any repository whose README is a chronological link list as an announcement artifact, not a software distribution.

Error: "The 857x bandwidth reduction doesn't match what I measured"

Cause: The 857x figure is specific to the benchmark in Table 3 of the preliminary report — 32 H100s, 1.2B Llama 2, stateful DisTrO-AdamW, no compression. Different model sizes, batch sizes, node counts, or topologies will produce different ratios.

Solution: Benchmark your own configuration. Use the 857x number as a lower bound of demonstrated performance, not as an expected outcome. Consult DeMo v2 for the current parameter space.

Prevention: Quote the reduction factor with its operating regime attached. Pre-training ≈ 857x–3000x; fine-tuning up to 10000x. These are not interchangeable.

Error: "DisTrO-AdamW doesn't converge on my 200M model"

Cause: Expected. Section 3.2 of the preliminary report says 50M and 300M models tested for 10B tokens did not converge well under DisTrO. The practical lower bound for DisTrO convergence is around 1.2B parameters.

Solution: Validate DisTrO on 1B+ models. For smaller models, stick with standard AdamW + All-Reduce.

Prevention: Plan DisTrO adoption for model scales above the demonstrated floor.

Error: "The DisTrO wall time was longer than All-Reduce wall time"

Cause: Expected on a high-bandwidth cluster. The preliminary report benchmark showed 19.8 h for DisTrO + AdamW vs. 17.1 h for All-Reduce + AdamW on the same 32x H100 setup. DisTrO's value is not wall-time reduction inside a cluster — it is making training feasible when inter-node bandwidth is the bottleneck (e.g., internet-scale deployment).

Solution: Evaluate DisTrO against the scenario it targets: heterogeneous, low-bandwidth, geographically distributed nodes. In that scenario, All-Reduce-based training is infeasible, and DisTrO's wall-clock contribution is bounded by per-step network latency (reported as ~7s at 100 Mbps down).

Prevention: Do not benchmark DisTrO on a single-datacenter high-bandwidth cluster and conclude it is "slower." You are measuring the wrong thing.

Error: "I need the exact DisTrO-AdamW source from the benchmark"

Cause: The preliminary report did not release the benchmark code. It was an announcement, with code release deferred to a follow-up paper.

Solution: Read the DeMo v2 paper (Oct 2025) and refer to Psyche's distro.rs.

Prevention: Treat "preliminary report" as shorthand for "method announcement, code follows later."

Warning

The Nanotron commit, Dolma sampling script, and DisTrO-AdamW source from the original 1.2B benchmark are not publicly archived anywhere as of the source commit date. Exact reproduction of Table 3 may not be achievable without direct access to the original experiment environment.

11. Quick Reference

11.1 File inventory

11.2 Key operational numbers (from the preliminary report)

11.3 External resources

11.4 Deployment decision matrix

11.5 Upstream / downstream map

 UPSTREAM (context)
  - Llama 2 architecture (baseline)
  - AdamW optimizer (drop-in replaced)
  - DDP parallelism strategy
  - Nanotron training framework

 THIS REPOSITORY
  - A_Preliminary_Report_on_DisTrO.pdf
  - README.md

 DOWNSTREAM (production / live code)
  - bloc97/DeMo (seed research code)
  - PsycheFoundation/psyche (production network + distro.rs)
  - DeMo v2 paper (current method reference)
  - Live training runs on the Psyche Network