Table of Contents
Harvard researchers and collaborators from Chutes have released a yearlong dataset of production LLM inference metadata, one of the larger public traces available for studying how real users interact with large language models in production.
The dataset, called A Year in LLM Serving, covers 6.12 billion requests across 9,174 models over one year of traffic from Chutes. It was announced by Juncheng Yang, an assistant professor at Harvard University, who said the release is intended to support research into “real-world LLM serving workload understanding, system design and infrastructure optimization.”
The release includes the dataset, analysis code, and an accompanying arXiv paper titled “A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing.” The paper lists William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, and Juncheng Yang as authors.
The release also gives outside researchers a look at usage patterns from a production decentralized AI serving network built on Bittensor’s infrastructure stack.
What the Chutes Dataset Includes
The released trace focuses on metadata rather than raw prompt text. According to the dataset hub, the schema includes fields such as request IDs, function names, chute and instance identifiers, rotating user IDs, request start and completion times, input tokens, output tokens, cache tokens, and time to first token.
That structure makes the dataset useful for systems research without requiring publication of full user conversations. Researchers can study when requests arrive, how long they run, how many tokens they consume, how cache behavior changes across workloads, and how traffic varies across many models.
The dataset hub says user IDs rotate every three months, meaning the same person may appear under different identifiers across the full year. That limits some longitudinal user-level analysis, but it also exposes operational patterns without making the trace a direct identity record.
The release is part of a broader Harvard MadSys open-data hub for LLM and agent-serving research. In addition to the Chutes trace, the hub lists released GPU and serving telemetry from FreeInference, plus planned datasets covering Chutes conversational traffic, agent-serving traces, and sanitized raw prompt data pending further review.
Why Real LLM Serving Traces Matter
LLM serving infrastructure is difficult to optimize from synthetic benchmarks alone. Production systems have uneven request arrivals, long-tail model demand, repeated user-model interactions, variable prompt and completion lengths, and changing cache behavior over time.
The Harvard paper argues that existing LLM serving workload studies have often been limited by short observation windows or restricted visibility into real production use. By releasing a one-year trace, the researchers aim to give system designers a more realistic basis for testing batching, scheduling, cache eviction, routing, and capacity-planning strategies.
The dataset includes several findings for inference infrastructure. The project’s data hub says repeat requests from the same user to the same model show strong temporal locality, with 99% of repeats arriving within 15 minutes. It also reports that least-recently-used cache eviction often matches or outperforms more complex alternatives, and that cache-aware routing can raise token hit ratios, though with simulated load imbalance of 5% to 7%.
Those are practical findings for LLM serving operators. Prefix-cache reuse can reduce redundant computation when users repeatedly interact with the same model or maintain similar context across requests. Routing, eviction, and batching decisions therefore affect latency, GPU efficiency, and serving costs.
Chutes’ Role in Bittensor Infrastructure
Chutes is a decentralized serverless compute platform for open-source AI models, built for deploying, scaling, and running those models in production.
On Bittensor, Chutes operates as Subnet 64, a GPU application deployment platform. The subnet lets developers define GPU requirements, deploy containerized FastAPI applications, expose functions as API endpoints, and rely on the platform for node selection, scaling, GPU validation, routing, and billing.
Bart Hillerich
That makes the Harvard dataset more than a conventional cloud-systems trace. It comes from a production environment tied to Bittensor’s decentralized AI economy, where inference workloads, miner hardware, model deployment, and user demand all interact through an open compute network.
Furthermore, by publishing the trace and accompanying paper, Harvard MadSys and Chutes are giving the AI infrastructure community a larger empirical base for understanding how LLM serving behaves at scale. For Chutes, the collaboration places its Bittensor-based network inside a more formal research conversation about decentralized AI serving.
Sami Kassab of Unsupervised Capital framed the release as part of Chutes’ broader shift from infrastructure provider to research-driven AI lab. In a post on X, Kassab wrote:
“Chutes is publishing research, running an open inference network averaging 27B tokens/day, and preparing to release a model trained with a novel decentralized MoE architecture. Their positioning as an open source AI lab is about to be very obvious”
