Table of Contents
Macrocosmos has taken its IOTA training system live with a 16-billion-parameter model running across geographically distributed compute.
Brian McCrindle, a founding engineer at Macrocosmos, said the run is training over the internet using Nvidia RTX 4090 and 5090 GPUs, with support from Green Compute and Targon. McCrindle said the system has reached model FLOP utilization, or MFU, "upwards of 40%," with compute stretching from the United States to Hong Kong.
The launch follows Macrocosmos' earlier Orion-100B distributed training run, which showed the team's ability to coordinate a 100-billion-parameter model across distributed A100 infrastructure. The new IOTA run is smaller by parameter count, but it moves closer to the system's longer-term goal of training large models across heterogeneous, permissionless, globally distributed hardware rather than a more controlled fleet.
Bart Hillerich
The IOTA dashboard gives the test a public surface while Macrocosmos works through the operational challenges of running AI training across less predictable infrastructure.
IOTA Moves From Demonstration Toward Stress Test
IOTA, short for Incentivized Orchestrated Training Architecture, is Macrocosmos' attempt to turn Bittensor Subnet 9 into a cooperative training network. Instead of asking every miner to train a full model independently, IOTA splits work across participants so different machines can contribute to a shared model.
Conventional data-parallel training requires each participating machine to hold the full model, meaning only operators with enough memory and high-end hardware can participate meaningfully. IOTA uses a data- and pipeline-parallel approach, where model layers and training work can be distributed across a broader pool of miners.
The system was designed for heterogeneous and unreliable devices in adversarial environments. An orchestrator coordinates training, miners process assigned portions of the model, and validators evaluate whether miners are performing honest and useful work. The architecture is meant to reward contribution to the shared model rather than turning subnet competition into a winner-takes-all contest.
The latest 16B run tests that premise in a more difficult environment. RTX 4090s and 5090s are not the same as a tightly controlled datacenter cluster. They can sit behind different network conditions, operate with different levels of reliability, and create more coordination overhead.
The reported 40% MFU figure is relevant here. MFU measures how much of the theoretical compute capacity is being converted into useful model training work, so higher utilization suggests the distributed system is spending less time waiting on communication, synchronization, or stalled participants.
Faults Provide Next Steps
Macrocosmos co-founder Will Squires said the system faced a series of faults during launch, including provider network bottlenecks during weight merging, a caching-service overload after running roughly 18 runs at once, and planned shutdowns that included a 48-hour full checkpoint restart.
Those faults show where decentralized training has to prove itself. Training a model across distributed hardware is a networking, orchestration, storage, monitoring, and incentives problem as much as a machine learning one.
"[The faults] are important steps for us making this thing bombproof for production. we have some tweaks to improve on this, and can occasionally be perfectionists, but thought it was best to get this into the wild," Squires said.
In a centralized training cluster, engineers can assume relatively consistent hardware, high-bandwidth interconnects, and controlled failure domains. A permissionless or semi-permissionless training network has to handle machines joining, leaving, slowing down, failing, or behaving unpredictably. IOTA's architecture is built around that reality, but the live run is where the assumptions are tested under practical load.
Squires said the team added automated monitoring after one provider's network bottlenecked during weight merging and improved autoscaling logic after the caching layer was overloaded. He also described the run as a "gym" for Macrocosmos, with the team planning to use it through August to test different parts of the system and intentionally push it toward failure.
"We will also be onboarding more and more bittensor compute! first tests went with this, so we are paying out based on contribution now - exact details on our first pass approach for this later this week, but we do want lots of community feedback!"
