Table of Contents
OpenAI has rebuilt the backend serving its main shared code repository to handle the parallel development generated by Codex. The new system, Pando, now powers all development in OpenAI’s monorepo and has reduced median push latency by nearly 80%, while retaining the Git protocol and workflows used by its engineers and coding agents.
In its technical account of Pando, OpenAI reports that the system delivers nearly a petabyte of Git data per day. The company built it with Codex, taking 10 weeks from kickoff to production, and began serving production traffic in April.
The pressure came from multiplying activity around each engineer’s work. Codex allows several changes to reach continuous integration, or CI, at once, and each push can trigger multiple jobs downloading the same code. OpenAI’s previous monorepo infrastructure struggled with slow transfers and failed merges, with reliability sometimes falling below 90%.
Pando addresses that workload with two architectural choices: coordinate writes independently for each branch, and serve reusable bundles of repository data rather than generate a customized response for every reader.
Letting Independent Branches Move Independently
Git separates repository contents from the references that point to them. Files, directories, and commits are stored as immutable objects identified by hashes. A branch is a movable reference, or ref, pointing to a commit. A push sends new objects and asks the server to advance a reference.
Publishing that new branch tip in a distributed Git service also means ensuring readers can retrieve the objects behind it. Coordinating those steps across replicas can put otherwise unrelated changes in each other’s way.
OpenAI evaluated designs based on replicated repositories coordinated through three-phase commit and a shared write-ahead log in blob storage. The first required repeated communication among replicas to publish an update. In the second, writers uploaded data in parallel but competed to update a common manifest recording repository changes.
Pando instead stores an incoming push as an immutable packfile, a compressed bundle of Git objects, in blob storage. A separate metadata store records a ref version containing the new branch tip, its packfile, and the earlier versions it depends on. Updating that record and the branch pointer makes the new commit available to readers.
Competing pushes need coordination when they target the same ref, rather than simply because they belong to the same repository. OpenAI can expand capacity by adding Pando servers and scaling the metadata and blob stores alongside them.
The consistency model preserves the history needed to reconstruct a particular revision, but it does not provide a single point-in-time snapshot of every branch. Two clients reading several branches during concurrent updates can observe different combinations of branch versions. That tradeoff removes repository-wide coordination from work that only needs an individual branch’s ordered history.
Reusing Packs Instead of Rebuilding Every Fetch
Conventional Git servers determine which objects a client needs by comparing its existing history with the commits it requests. They then assemble a packfile tailored to that client. Repeating that work for many readers consumes CPU and disk resources, even when those readers want much of the same code.
Pando selects existing packs instead. Through Git’s packfile-uri feature, it returns links that let clients download those packs separately. A delivery service called Express caches the files regionally, allowing many clients to receive identical bytes without repeated server-side packing.
OpenAI reports cache hit rates above 99% and says clone and fetch operations saturated 10 Gbps network links on standard cloud virtual machines. In one hour, Pando served more than 57 terabytes of Git packs, with over 99% of the bytes coming from local disk cache. The company says it can deliver a petabyte per day with only a handful of CPU cores on the serving side.
Making packs reusable also changes how pushes are negotiated. By default, when a client pushes a new feature branch, Pando advertises only a designated trunk branch as the shared base. The client builds a pack that another reader with that trunk history can also use.
This keeps negotiation small as the number of branches grows. It can increase upload size, though, since repeated pushes to a feature branch, or work stacked on another feature branch, resend objects outside trunk. In OpenAI’s monorepo, the median pack produced this way was below 20 KiB, and the 99th percentile was below 3 MiB.
Keeping Fast-Moving History Practical to Download
Reusing every original push indefinitely would leave a different problem. A client catching up with a busy repository could need tens of thousands of small packfiles for a single day’s changes.
Pando uses background compaction to group small packs into progressively larger units called covers. Its smallest packs are labeled L0, and OpenAI’s monorepo currently uses covers through L4. Workers combine objects into larger packfiles, improving compression and reducing the number of downloads needed to catch up.
The hierarchy also speeds up fetch planning. Rather than inspect every intervening version individually, Pando can skip across larger covered ranges to select the packs a reader needs. OpenAI says the required metadata reads grow logarithmically with the distance between the client’s existing history and its requested revision.
Pando retains packs at every level rather than discarding the smaller ones after compaction. That lets it choose different combinations for different readers; fetching a single object through an API can also require less data from a small pack than from a more heavily compressed large one.
The result separates the work of negotiating Git history from delivering repository bytes. Servers handling pushes validate packs as they stream into blob storage, while fetch servers select existing packs and leave bulk transfer to the caching layer.
Building With Agents and Feeding Changes Back to Git
Pando is written in Rust. OpenAI says one engineer designed its core, teams of agents implemented it, and a handful of engineers deployed it. Validation included a conformance suite against standard Git behavior and deterministic simulations to uncover race conditions and errors in negotiation and pack selection.
The company also used GPT-6 Astra agents to write Lean and TLA+ specifications, formalizing system behavior to find bugs and prove properties. Continuous agent sessions ran for hundreds of hours checking edge cases, and OpenAI runs ongoing red-teaming of Git and Pando with Daybreak-capable models.
Deployment began with Pando acting as a read- and write-through proxy to the previous source-control system. That kept the two synchronized while traffic moved over without downtime. An internal Git command-line wrapper supported phased client migration, alongside the transfer of hundreds of integrations relying on older APIs, webhooks, and repository downloads. Codex skills helped developers migrate their environments and services.
The work also produced changes to Git itself. OpenAI fixed problems involving concurrent pack downloads, interrupted transfers, and completed-download resumption, and added tracing for packfile-URI downloads. Its public Git distribution supports downloading and indexing independent packs in parallel.
Another addition, “explicit haves,” separates a branch’s current tip from the history a pushing client may omit. On Pando, that lets --force-with-lease check that a feature branch has not moved while still building a reusable pack against trunk alone.
Several improvements have reached upstream Git, with others available through OpenAI’s repository and public patch tracker. OpenAI is exploring additional internal workloads where many agents collaborate in one repository, as well as ways to share Pando’s underlying technology with users.
