DisTrO User Guide

DisTrO (Distributed Training Over-the-Internet) is a family of architecture-agnostic and network-agnostic distributed optimizers that reduce inter-GPU communication during neural network training by four to five orders of magnitude. This guide helps you understand what DisTrO is, what this repository contains, and where to find the code you can actually run.

Important

This repository is a research announcement, not a software package. It contains the preliminary report (PDF) and a timeline of follow-up work. There is no installable library, CLI, or API in this repo. The production implementation lives in the PsycheFoundation/psyche project — see Where The Code Lives below.

1. Quick Start

If you are new to DisTrO, do these three things in order:

1. Read the README timeline in the repository root — it shows the chronological progression of the research from the first report to the current production network.
2. Read the preliminary report — A_Preliminary_Report_on_DisTrO.pdf in the repository root.
3. Follow the trail to the current state of the work:
   • The updated optimizer paper: DeMo Optimization v2 (Oct 2025)
   • The production training network: Psyche Network
   • The production optimizer implementation: distro.rs in the Psyche repository

Tip

If you want to use DisTrO for a training run today, skip directly to the Psyche repository. This NousResearch/DisTrO repository is historical — it marks the original announcement.

2. What Is DisTrO

DisTrO is a family of optimizers designed to make large-model training possible over ordinary internet connections instead of specialized data-center interconnects.

Modern large language model (LLM) training traditionally requires thousands of GPUs wired together with specialized high-speed interconnects (NVLink, InfiniBand) inside purpose-built data centers. At every training step, the GPUs exchange full gradient tensors using an All-Reduce collective, and that exchange is so bandwidth-heavy that geographical distribution is infeasible.

DisTrO changes that assumption. Its optimizers act as drop-in replacements for standard optimizers like AdamW. They do the same job — update model weights based on gradients — but exchange a radically smaller amount of data between nodes each step. The original report demonstrated a 857x reduction in bandwidth during pre-training of a 1.2B Llama 2-style model, with no loss in convergence rate or final model quality.

Why This Matters

• Training decentralizes. Groups without hyperscale data centers can pool GPUs over the public internet.
• Consumer hardware becomes viable. Asymmetric home internet (fast download, slow upload) suits DisTrO's communication profile.
• Federated and permissionless training become practical for frontier-scale models.

Key Properties (From the Preliminary Report)

Note

DisTrO was demonstrated only on dense neural network architectures (Llama 2, 1.2B parameters). Extending it to sparse architectures such as Mixture-of-Experts is not claimed in the preliminary report.

3. What This Repository Contains

This repository is intentionally minimal — it is the announcement artifact for the DisTrO research line, not a software distribution.

NousResearch/DisTrO/
├── A_Preliminary_Report_on_DisTrO.pdf   (~1.2 MB, 7 pages)
└── README.md                            (research timeline with links)

A_Preliminary_Report_on_DisTrO.pdf

The 7-page preliminary technical report that introduced DisTrO. It contains:

• The abstract and introduction framing the bandwidth problem in distributed LLM training.
• Empirical results from a 1.2B-parameter LLM pre-training run on 32x NVIDIA H100 GPUs.
• A discussion section covering data-streaming patterns, bandwidth scaling, and future applications (federated learning, decentralized training, environmental impacts).
• References to the prior literature this work builds on (DiLoCo, SWARM Parallelism, FSDP, Deep Gradient Compression, Federated Averaging, CocktailSGD, GaLore, and others).

Authors: Bowen Peng, Jeffrey Quesnelle, Dillon Rolnick, Ari Lotter, Umer H. Adil, and Esteban La Rocca — all from Nous Research.

Note

The preliminary report is self-described as "preliminary." A more rigorous academic paper was promised alongside code release. The follow-up work (DeMo v2 and the Psyche production code) now carries that torch — see The Research Timeline.

README.md

A short chronological timeline of the DisTrO research line and its descendants. Each checkpoint links to the corresponding paper, model, or codebase. The README is the single best starting point if you arrive at this repository with no prior context — it tells you where the frontier of the work currently lives.

4. The Research Timeline

The README in this repository documents a clear chronology. This section expands on each milestone so you know what to read next.

August 26, 2024 — DisTrO Preliminary Report

• What was released: The PDF in this repository.
• What it claimed: First empirical proof that DisTrO-AdamW matches standard AdamW + All-Reduce in convergence rate while cutting per-step inter-GPU bandwidth by 857x.
• Scale demonstrated: 1.2B parameters, 105B training tokens, 25000 steps, 32x H100 GPUs.

December 2, 2024 — DeMo Optimization (Original Paper) and Code

• Paper: arxiv.org/abs/2411.19870
• Code: github.com/bloc97/DeMo
• What it is: The "original seed research/idea for DisTrO." DeMo (short for Decoupled Momentum Optimization) is the foundational technique that the DisTrO family extends.

December 2, 2024 — Nous Trains a 15B Model Using DisTrO

• Announcement: distro.nousresearch.com
• Significance: First public training run of a 15B-parameter model using DisTrO in production, scaling well beyond the 1.2B demonstration in the preliminary report.

May 14, 2025 — Psyche Network

• Announcement: nousresearch.com/nous-psyche
• What it is: A permissionless, decentralized training network built on DisTrO. Psyche is the productionization of the decentralized-training vision sketched in Section 3.3 of the preliminary report.

May 14, 2025 — Consilience 40B LLM

• Training run: psyche.network/runs/consilience-40b-1/0
• Model weights: huggingface.co/PsycheFoundation/consilience-40b-7Y9v38s5
• Significance: A 40B-parameter model trained on the Psyche network using DisTrO-family optimizers.

October 14, 2025 — DeMo Optimization v2 (Academic Paper) and Production Code

• Paper (v2): openreview.net/pdf?id=U9oewpa7cn
• Production implementation: distro.rs in the Psyche repository
• Significance: The fully detailed academic paper promised in the preliminary report, plus the canonical production implementation in Rust.

Tip

If you are trying to replicate results or cite the method, cite the DeMo v2 paper rather than the preliminary report. The preliminary report established the empirical claim; DeMo v2 is the peer-reviewable detailed write-up.

5. Where The Code Lives

Because this repository contains no code, this section is deliberately prominent.

Warning

None of these external repositories are maintained by the NousResearch/DisTrO repository. Their licenses, issue trackers, and contribution processes are their own. Follow links to the upstream projects for installation, usage, and support.

6. Reading The Preliminary Report

The PDF is short (7 pages) and dense. Read it in this order to extract the most value with the least effort.

Step 1: Abstract and Introduction (Pages 1–2)

These frame the problem: inter-GPU bandwidth is the bottleneck that forces LLM training into expensive monolithic data centers. Skim this to confirm you understand why DisTrO matters before you dive into numbers.

Step 2: Results Tables (Pages 3–4)

Three tables contain every quantitative claim in the paper:

Step 3: Discussion (Pages 4–5)

Section 3 of the report covers the parts most useful for planning a DisTrO-based training run:

• 3.1 Data Streaming — explains the asymmetric bandwidth pattern (2.8 MB upload, 86.8 MB download per step in the 32-node case) and why it suits consumer internet.
• 3.2 Bandwidth Scaling — discusses whether the reduction factor grows or shrinks with model size (open question in the preliminary report).
• 3.3 Future Applications — federated learning, decentralized training, environmental impacts. Section 3.3.2 specifically describes the shape of what became Psyche.

Step 4: References (Pages 5–7)

Twenty-six references. The most relevant for context:

• [3] DiLoCo (Douillard et al., 2023) — a prior low-communication distributed training method DisTrO distinguishes itself from.
• [6] Nanotron — the Hugging Face training framework used for the 1.2B benchmark.
• [10] PyTorch Distributed — the All-Reduce baseline DisTrO replaces.
• [12] Federated Averaging (McMahan et al.) — the closest conceptual cousin to DisTrO's stateless variant.
• [14] SWARM Parallelism — discussed as a candidate to layer on top of DisTrO for memory scaling.
• [26] PyTorch FSDP — the other standard distributed-training baseline.

Tip

The preliminary report is only 7 pages. Reading it front-to-back takes under an hour for anyone already familiar with distributed training. Going directly to Section 3 (Discussion) first gives the fastest path to the practical implications.

7. Common Use Cases

Even without code in this repository, you may have arrived here to decide how DisTrO applies to your work. This section maps common intents to next actions.

"I want to train my own model with DisTrO"

1. Go to the Psyche repository.
2. Follow its installation and setup instructions — that repository is the software distribution.
3. Read the DeMo v2 paper to understand the current optimizer semantics (which may have evolved since the preliminary report).

"I want to join the Psyche network as a compute contributor"

1. Visit nousresearch.com/nous-psyche.
2. Join the community Discord (link at the bottom of the README in this repository).
3. Follow setup instructions from the Psyche project itself.

"I want to cite DisTrO in a paper"

1. For the original empirical claim: cite A_Preliminary_Report_on_DisTrO.pdf (Peng et al., 2024). The report is hosted in this repository.
2. For the peer-reviewable method paper: cite the DeMo v2 paper (Peng et al., 2025) at openreview.net/pdf?id=U9oewpa7cn.

"I want to reproduce the 857x bandwidth reduction"

1. Read the preliminary report for the exact setup (Nanotron, 1.2B Llama 2, 32x H100, 105B tokens of Dolma v1.7, AdamW hyper-parameters β1=0.9, β2=0.95, peak LR 4×10⁻⁴).
2. Cross-reference the DeMo v2 paper for the current optimizer definition.
3. Use the Psyche production code as the reference implementation.
4. Expect to invest significant compute — the benchmark was a 25000-step run on 32 H100s.

"I want to understand the research conceptually"

Read the preliminary report (7 pages), skim the DeMo v2 paper abstract and results, then read Section 3 of the preliminary report for implications. That is a 1–2 hour reading block for someone already familiar with distributed training.

8. Getting Involved

The README ends with an invitation to the community Discord:

Join us on Discord if you're interested in helping research and build the future of distributed training.

Discord invite: discord.com/invite/jqVphNsB4H

Active development happens in two places:

• Psyche Foundation — production code, training runs, network operations. See github.com/PsycheFoundation.
• Nous Research — the organization that published DisTrO. See nousresearch.com.

9. Troubleshooting

The issues below are the real frictions users hit when they land on this repository.

"I cloned the repo but there's nothing to install."

Cause: This repository is a research announcement, not a software distribution. It is intentionally minimal — a PDF and a README.

Solution: Go to PsycheFoundation/psyche for the production code. Follow that repository's installation instructions.

Prevention: Check the README.md first on any repository labeled "preliminary report" or similar — it will point to the current state of the work.

"The preliminary report describes hyper-parameters but not the optimizer algorithm."

Cause: The preliminary report was explicit that a "more rigorous and fully detailed academic paper is in progress." Algorithmic details were deferred to the follow-up.

Solution: Read the DeMo v2 paper at openreview.net/pdf?id=U9oewpa7cn. That paper contains the detailed method.

Prevention: Treat the preliminary report as an empirical announcement. Treat DeMo v2 as the method reference.

"I want to see the exact optimizer code used in the 1.2B benchmark."

Cause: The preliminary report did not release the benchmark code. The production implementation in Psyche is the closest public reference.

Solution: Read distro.rs in Psyche. Cross-reference with DeMo v2 for the algorithm definition.

Prevention: When a paper is marked "preliminary," assume code follows later. Subscribe to updates from the author organization.

"I opened an issue months ago and there is no response."

Cause: This repository is not actively maintained as a support channel — it is a release artifact. Active discussion happens in the Nous Research Discord and in the Psyche project.

Solution: Join the Discord (link above) or open an issue in the Psyche repository if your question is about the production code.

Prevention: For research announcement repositories, treat issues as archival rather than as a support queue.

"I see a bandwidth reduction number that's different in different places."

Cause: The preliminary report reports 857x from the actual 1.2B benchmark, 1000x–3000x as achievable during pre-training with careful hyper-parameter tuning, and up to 10000x for post-training/fine-tuning. These numbers apply to different regimes.

Solution: When quoting the number, specify the regime (benchmark vs. tuned pre-training vs. fine-tuning).

Prevention: Read Section 3 of the preliminary report — it lists all three numbers explicitly.

Caution

The 857x reduction was measured on a specific benchmark: 32x H100, 1.2B Llama 2, Dolma v1.7, stateful DisTrO-AdamW, no compression. Do not report this number out of context — different model sizes, node counts, and topologies will produce different ratios.

10. Frequently Asked Questions

Q: Is there a Python package I can pip install?

No package is published from this repository. The production implementation is written in Rust inside the Psyche repository. Check that repository's README for current integration options.

Q: Is DisTrO a single optimizer, or a family?

A family. The preliminary report introduces the "DisTrO family" and specifically benchmarks the variant DisTrO-AdamW (the AdamW analogue). Other variants — including a stateless version — are mentioned but not benchmarked in the report.

Q: Can I use DisTrO as a drop-in replacement for AdamW in my existing training code?

Conceptually yes — the preliminary report demonstrates this by swapping AdamW for DisTrO-AdamW in the Nanotron framework without changing hyper-parameters. In practice, you need a DisTrO implementation. Today, that is the Rust implementation in Psyche. There is no PyTorch reference implementation published from this repository.

Q: Does DisTrO work for Mixture-of-Experts (MoE) or sparse models?

The preliminary report says DisTrO is demonstrated on dense neural networks (the 1.2B Llama 2 architecture). The report explicitly contrasts its coverage with sparse networks like mixture-of-experts. Extending DisTrO to sparse architectures is not claimed in the report.

Q: What GPUs were used for the 857x benchmark?

32 NVIDIA H100 GPUs, per Table 3 of the preliminary report. Wall-clock time was 17.1 hours for All-Reduce + AdamW versus 19.8 hours for DisTrO + AdamW on the same hardware.

Q: Why did DisTrO take slightly longer in wall-clock time (19.8h vs 17.1h) if it uses less bandwidth?

The preliminary report does not provide a direct explanation. Possible factors include per-step computational overhead of the DisTrO optimizer itself, or the particular networking setup in the benchmark. What the report emphasizes is that DisTrO's bandwidth profile makes internet-distributed training feasible — not that it beats All-Reduce on a high-bandwidth cluster.

Q: Does DisTrO reduce GPU memory use?

The preliminary report does not claim GPU memory reductions. Each GPU still holds the full model in VRAM under the Distributed Data Parallelism strategy tested. Memory-saving methods like FSDP and SWARM Parallelism are mentioned as potential future integrations, not as current features of DisTrO.

Q: Is DisTrO compatible with Federated Learning?

Potentially — the preliminary report says DisTrO "imposes no requirements on how the data is processed or assigned to individual GPU nodes, and has a stateless variant (similar to Federated Averaging), thus can be adapted for federated learning in the future." That is a forward-looking statement in the report; it is not yet benchmarked.

Q: Is compression applied to the transmitted tensors?

No — the preliminary report states "we currently do not use any compression and transmit the tensors directly." The 857x reduction is already without compression. Compression is listed as a potential future improvement.

Q: How did the 1.2B DisTrO-trained model compare to TinyLlama on quality?

Table 2 of the preliminary report shows DisTrO + AdamW matches or exceeds both the All-Reduce baseline and a same-token TinyLlama checkpoint across all six GPT4All benchmarks tested (ARC-c, ARC-easy, Hellaswag, OpenBookQA, PIQA, WinoGrande).

Q: Is DisTrO related to Bitcoin mining or any cryptocurrency?

No. The preliminary report mentions Bitcoin mining infrastructure only as an analogy for tapping underutilized power grids (Section 3.3.3). DisTrO itself is a neural-network optimizer and has no on-chain component. Decentralized incentive mechanisms layered on top (e.g., in Psyche) are a separate engineering concern from the optimizer.

Q: Why are some community issues on this repository closed without code being shipped here?

This repository was intentionally kept minimal as an announcement artifact. Active development moved to DeMo and then to Psyche. If you are looking for code, those are the right destinations. Issue #3 specifically requested method documentation and an update — the DeMo v2 paper (Oct 2025) is the answer to that request.

Q: Can I run DisTrO training on my home GPU?

In principle, yes — that is one of the core motivations. In practice, you need: a working DisTrO implementation (Psyche), enough VRAM to hold the target model (DDP keeps the full model on every node, so a 1.2B model needs roughly 24GB+ per GPU), and a functioning connection to a training coordinator. Start with the Psyche project documentation for current onboarding.

Q: Is there a conference or journal citation I should use?

For the empirical claim and the DisTrO name, cite the preliminary report hosted in this repository. For the peer-review-ready method description, cite DeMo v2 (Peng et al., 2025) at openreview.net/pdf?id=U9oewpa7cn.

Q: Where can I follow ongoing research from this group?

Follow nousresearch.com and the Nous Research Discord. For production training infrastructure, follow PsycheFoundation.

Q: Are there pre-trained model weights I can download from this repository?

Not from this repository. The 1.2B benchmark model is not published here. For pre-trained models from the Psyche network, see huggingface.co/PsycheFoundation — for example, the Consilience 40B checkpoint.

Q: Does DisTrO need a specific network topology?

No. The preliminary report explicitly calls DisTrO "agnostic to the telecommunication network's topology." Peer-to-peer and dedicated-aggregator topologies are both discussed (Section 3.1).

Q: What does "clock-synchronous" mean?

From a footnote in the preliminary report: "each training step uses the same arithmetic operations and takes the same amount of time." This is in contrast to asynchronous distributed optimizers where different nodes may execute different operations or proceed at different rates. DisTrO, like SGD or Adam, advances all nodes in lockstep per step.

Q: Is DisTrO resilient to node failures?

The preliminary report says "evidence suggests that when implemented, DisTrO is remarkably resilient to a small amount of degraded or dropped nodes during training, and it can be easily made to accommodate new nodes joining" (Section 3.3.2). That is a qualitative claim in the report — for quantitative fault-tolerance characteristics, see the Psyche production system.

11. Reference

Repository Contents At A Glance

External Resources Referenced From This Repository

Key Numbers From The Preliminary Report

Authors

• Bowen Peng — [email removed]
• Jeffrey Quesnelle — [email removed]
• Dillon Rolnick — [email removed]
• Ari Lotter — [email removed]
• Umer H. Adil — [email removed]
• Esteban La Rocca — [email removed]

All authors listed as affiliated with Nous Research in the preliminary report.