> ## Content Index
> Fetch the complete content index at: https://www.tao.media/llms.txt
> Use this file to discover other available public pages before exploring further.

# Chutes Reports Nearly 18,000 Tokens per Second for Parallax on One RTX 5090
- URL: https://www.tao.media/chutes-reports-nearly-18-000-tokens-per-second-for-parallax-on-one-rtx-5090/
- Published: 2026-11-11T14:00:00.000Z
- Updated: 2026-11-12T15:19:00.000Z
- Description: The consumer-GPU preview combines 256 concurrent requests with a compact model and low-precision expert processing.
- Author: Tristan Hillerich
- Tags: Chutes, Subnets, AI, News

Chutes reports that Parallax can generate nearly 18,000 tokens per second on a single NVIDIA RTX 5090 while handling 256 requests simultaneously, using a modified version of the vLLM inference engine.

The [serving preview](https://x.com/chutes%5Fai/status/2120532426862571607?ref=tao.media) tests the other half of Chutes' consumer-hardware strategy. Chutes trains models across distributed gaming GPUs, and it also aims to make the resulting models inexpensive to run. The figure measures aggregate throughput across the concurrent requests, rather than the generation speed of any individual response.

The [Opentensor Foundation identified the model as Parallax 8B](https://x.com/opentensor/status/2120557109137821760?ref=tao.media). The demonstration draws on Chutes' work training models across hundreds of consumer GPUs worldwide. Chutes says the serving performance to a small model, 4-bit processing in its expert layers, and a fixed-size cache for much of its sequence processing.

## Serving More Requests With Sparse Experts

Parallax uses a mixture-of-experts architecture, in which a router selects specialized neural-network components for each token. That allows a model to retain a larger bank of parameters without executing all of them for every generated token.

The [technical report for Parallax's lambda training run](https://parallax.chutes.ai/tech-report.pdf?v=18f1de4333&ref=tao.media) gives that model approximately 7.77 billion total parameters, with about 1.20 billion active per token. Each of its 32 routed layers selects 12 of 128 experts. Its expert weights are ternary (minus one, zero, or plus one, with scaling factors) and use structured sparsity to restrict which weights can be nonzero.

The serving implementation combines that compact representation with hardware-friendly arithmetic. At a batch of 256 scheduled tokens, the report's vLLM-based server uses sparse ternary experts with 4-bit floating-point activations. At smaller batch sizes, it switches to packed bfloat16 experts. Ternary weight storage reduces model size, while the 4-bit activation path supports low-precision expert computation at larger batches.

The report also contains a separate set of RTX 5090 measurements on an intermediate checkpoint, exported after 242.4 billion training tokens. Those tests produced higher rates than the latest social-post preview, under the specific configurations below.

| One-GPU Workload                        | FP32 Recurrent State | FP16 Recurrent State |
| --------------------------------------- | -------------------- | -------------------- |
| Decode, 256 requests, 32-token prompts  | 23,294 tokens/s      | 28,634 tokens/s      |
| Decode, 256 requests, 128-token prompts | 22,551 tokens/s      | 27,520 tokens/s      |
| Decode, one request, 32-token prompt    | 289 tokens/s         | 295 tokens/s         |

For these report measurements, each request generates 128 tokens. Throughput counts the 127 cached predictions after the first output, excludes prompt processing, and is the median of five trials following four warm-ups. The server reserves 8,192 context tokens per request and uses synchronized arrivals and CUDA graphs, which reduce repeated GPU execution overhead.

These are distinct measurements, not interchangeable versions of the nearly 18,000-token preview. They show how batch size, prompt length, and recurrent-state precision change the amount of output one card can produce. Serving more requests together raises total output by sharing GPU execution across the batch; it does not give each request the full aggregate token rate.

## Keeping Sequence Memory Compact

A second part of Parallax's design addresses the memory needed to retain context while generating text.

Conventional attention stores keys and values for earlier tokens so the model can refer back to them. That key-value cache grows with context length and is maintained separately for each active request. At high concurrency, context storage becomes a substantial part of the GPU memory budget.

Lambda combines three forms of sequence processing: 18 Erase-then-Delta Attention recurrent layers, eight sliding-window attention layers, and six global-attention layers. The recurrent layers update a fixed-size state rather than retaining an expanding collection of token-level keys and values. Sliding-window attention limits its cache to a bounded recent history, while global attention preserves access across the wider context.

The architecture reduces the number of layers that need expanding caches while preserving global attention in selected parts of the network. Its global-attention layers still retain context-dependent information, so the fixed-size memory benefit applies to the recurrent portion, not the entire model.

## From Distributed Training to Local Inference

The inference work follows Chutes' plan to build an open-source AI lab on Bittensor Subnet 64, extending a network built around serverless inference into model development.

In its [lab announcement](https://chutes.ai/news/democratizing-intelligence-building-an-open-source-ai-lab-on-a-decentralized-network?ref=tao.media), Chutes reported processing roughly 901 billion tokens in less than six days for an 8B model, using less than $7,000 in compute. The lambda report records a launch fleet of 208 RTX 5090 GPUs across 26 eight-GPU hosts, connected over the public internet and run by one operator. After automatic replacements for failed expert owners, the assigned fleet ended at 152 GPUs across 19 hosts.

Training distributes responsibility differently from serving. Every host trains small adaptations to the experts it uses. Designated expert owners merge that evidence into full-precision master weights and publish updated ternary versions. Shared model updates pass through CPU relays on a separate synchronization schedule, reducing the need for every training machine to communicate after every local step.

Parallax's [earlier technical report](https://chutes.ai/parallax.pdf?ref=tao.media), dated June, describes a different training decomposition tested on 20B-parameter models. Training machines own subsets of experts and use lightweight local surrogates for the others, avoiding a remote expert round trip inside each training step. Its four-composer H100 run reached median validation loss of 3.1896 against 3.1990 for an interpolated four-B300 baseline at the same token count.

Chutes' planned lab will publish checkpoints, datasets with lineage, training and synchronization code, evaluation tools, and benchmark results. Its existing inference business will continue alongside that work and provide a distribution channel for the models the network develops.