Table of Contents
The Chutes team has now publicly outlined Parallax, an experimental decentralized AI training architecture designed to distribute model training across geographically dispersed hardware while dramatically reducing the infrastructure requirements typically associated with large-scale AI development.
The project was introduced through a new video and accompanied by a detailed technical report authored by Chutes core contributor and backend developer Jon Durbin. Together, they provide the clearest picture yet of how Chutes believes large AI models could eventually be trained without relying exclusively on massive data centers and tightly coupled GPU clusters.
"We believe open models shouldn't require $200 million and the permission of three cloud providers just to train models. [Parallax] is what the Chutes team has been working on."
What Is Chutes Parallax?
At a high level, Parallax is a decentralized training framework built around sparse Mixture-of-Experts (MoE) models.
Modern AI training generally assumes that GPUs are located inside the same data center and connected through extremely high-bandwidth networking. While sparse MoE architectures reduce the amount of compute required per token, most existing implementations still require constant communication between GPUs during training.
Parallax attempts to remove that requirement.
Instead of treating a model as a single monolithic system that must remain synchronized across a cluster, Parallax breaks responsibility across multiple independent participants, which the paper refers to as "composers." Each composer owns only a portion of the model's experts while maintaining lightweight approximations of the remaining experts. This allows training to proceed locally without requiring constant remote communication between every participant.
The result is a training architecture designed for fragmented compute environments, where individual GPUs, gaming PCs, workstations, and small clusters can contribute to model training. By breaking training workloads into smaller components, Parallax seeks to lower the hardware requirements for each participant while expanding the pool of machines capable of contributing compute.
According to Chutes, the team has already trained a 20-billion-parameter model using decentralized GPUs distributed around the world for less than $10 per hour.
Moreover, the technical report describes completed 20B model experiments conducted across non-colocated H100 GPUs, RTX 6000 Ada systems, and RTX 4090 worker nodes, demonstrating the feasibility of the approach across heterogeneous hardware environments.
The introductory Abstract section is as follows:
"Parallax is a training decomposition for sparse Mixture-of-Experts models on heterogeneous GPUs that are not colocated. Each composer owns a disjoint shard of routed experts and keeps detached low-rank surrogates for the rest. The router still selects across the global expert namespace, but the MoE layer runs locally: owned selections call real experts, non-owned selections call surrogates, and no token-level expert all-to-all sits on the step path. Owners refresh surrogate state in the background. Router, latent-interface, and backbone parameters synchronize through tiered RDA-DiLoCo-style cadences. A worker-offload variant moves routed-expert optimizer state and expert update work to remote GPUs using compressed activation sketches and Taylor-proxy expert objectives. The report gives point estimates from completed 20B runs. It does not report replicated equivalence tests or measured wall-clock speedups. A 4-composer run on non-colocated H100 GPUs reaches median validation loss 3.1896 at the 30k baseline-equivalent comparison; the interpolated 4×B300 end-to-end baseline is 3.1990 at the same token count. An 8-composer L40S/NVIDIA RTX 6000 Ada run with remote NVIDIA RTX 4090 expert workers reaches 3.2521 at 50k baseline-equivalent steps and a best exported median of 3.2104 after further training. For the same 20B architecture, a rank-64 surrogate uses 48× fewer expert hidden dimensions than a real routed expert. At C = 8, the analytic model gives 7.6× lower routed-expert training compute and 2.16× lower total active compute for direct ownership; with expert offload, it gives 20.9× lower composer-side routed-expert compute and 2.43× lower total active compute."
Why MoE Models Matter
Much of Parallax's efficiency comes from how it handles Mixture-of-Experts architectures.
Durbin's core observation is that most parameters in modern MoE models exist inside routed experts that are only activated when needed. Instead of requiring every participant to store and train every expert, Parallax distributes ownership of those experts across the network.
Participants maintain compact surrogate versions of experts they do not own, dramatically reducing memory and compute requirements while preserving the model's ability to route across the full expert namespace.
In his post on X, Durbin summarized the approach:
"MoE models' params are mostly routed experts, and you can massively reduce VRAM and FLOPS per participant by splitting up those experts."
The report estimates that a rank-64 surrogate can require 48 times fewer hidden dimensions than a full routed expert, significantly lowering the amount of hardware needed for training.
Privacy Without Sharing Raw Data
Another major focus of the project is privacy.
Traditional distributed training often requires participants to exchange large amounts of model state, gradients, or raw training data. Parallax attempts to avoid that through what it calls "activation sketches."
The system doesn't send datasets to workers. Instead, it sends compact compressed representations containing only the information necessary to improve a specific expert. Those workers never receive raw text data and do not need access to the full model.
Durbin notes that some protections would still be needed around the earliest layers of a model, where gradient inversion attacks could theoretically reconstruct portions of training data. Beyond those initial layers, however, he argues that workers would lack sufficient context to reconstruct the original dataset.
The result is a training framework where distributed participants contribute compute without directly accessing the underlying training corpus.
Beyond Decentralized Training
While decentralized training is the headline, Durbin argues that it is not the ultimate objective. It's to create systems that are economical to operate long after training is complete.
"The thing is, decentralized training is nice and becoming a bit necessary given compute and power constraints, but really it's just a means to an end," he wrote.
That end, according to Durbin, is making advanced AI more accessible to build, train, and serve. Throughout both the technical report and his accompanying commentary, he emphasizes reducing the amount of compute, memory, networking, and infrastructure required to develop increasingly capable models, saying the industry should be optimizing for "intelligence per dollar" rather than benchmark scores alone.
Future iterations of Parallax will explore combinations of LatentMoE architectures, ternary weights, and hybrid Mamba-Transformer designs, all aimed at improving efficiency without sacrificing model quality.
In his post accompanying the report, Durbin described a future where powerful AI models can be trained and served without relying on a small number of centralized providers.
"The entire world's knowledge trains these models. They should in turn be trained by and accessible to the entire world."
He envisions privacy-preserving AI systems running inside trusted execution environments that can approach frontier-model capabilities while requiring significantly less hardware and infrastructure. In that vision, decentralized training helps ensure that access to advanced AI remains broadly available, instead of being concentrated among a handful of organizations.
Disclaimer: This article is for informational purposes only and does not constitute financial, investment, or trading advice. The information provided should not be interpreted as an endorsement of any digital asset, security, or investment strategy. Readers should conduct their own research and consult with a licensed financial professional before making any investment decisions. The publisher and its contributors are not responsible for any losses arising from reliance on the information presented.
