Table of Contents
Actual Computer, the team behind Bittensor Subnet 95, has released toks, a source-available tokenizer that returns the same token IDs as Hugging Face tokenizers while running substantially faster on CPU workloads.
The company announced the release in a blog post from founder and CEO Thomas Lynch, who described toks as Actual Computer's first public software release and "the first tokenizer designed for the quadrillion-token era."
Tokenizers convert text into the numeric tokens that language models process. That makes them part of nearly every LLM workflow, including inference prompts and training data pipelines. Actual's argument is that as AI systems move toward ever-larger token volumes, tokenization becomes an infrastructure bottleneck that should be optimized closer to the hardware.
Actual Computer Says toks Prioritizes Speed And Exact Parity
toks is built as a small library of assembly hot paths, with C glue and a Python interface. Actual says the library is 13x to 151x faster than Hugging Face tokenizers across every cell of its speed table when running on one CPU core with fresh text.
toks is faster than OpenAI's tiktoken in all 195 benchmark cells tiktoken can run, and faster than gigatoken in 252 of 255 cells. In one example cited by Actual, Llama 3 tokenization on English prose reached 269.8 MB/s on one NVIDIA GB10 CPU core with toks, compared with 30.3 MB/s for tiktoken and 5.36 MB/s for Hugging Face. In another, Kimi K3 tokenization on code reached 474.3 MB/s with toks, compared with 26.2 MB/s for tiktoken and 3.84 MB/s for Hugging Face.
Actual also introduced a parallel mode, toks_par, which it says encodes a single 16 MiB input at 1.6 GB/s on eight GB10 cores and 1.8 GB/s on eight Zen 5 cores while returning the same IDs as a serial call.
Speed only helps if the tokenizer keeps the same outputs developers expect from existing tooling. Actual says toks returns the same IDs as Hugging Face tokenizers 0.23.2, checked across millions of cases per model on every CPU tier with zero differences. A tokenizer toks cannot reproduce exactly is refused at load time, with the missing feature named, instead of being loaded in a compatibility mode that could silently return different IDs.
A Source-Available Release With A Quadrillion-Token Threshold
toks is free for any organization processing fewer than one quadrillion tokens a year, including production use, modification, vendoring, and shipping toks inside another product.
Actual equates that threshold to roughly 32 million tokens per second running continuously. Organizations at or above the limit need a commercial license from Actual Computer. Each version is set to convert to Apache 2.0 four years after its public release, with toks 0.3.0 scheduled to convert on October 5, 2030.
Python wheels for CPython 3.10 through 3.14 and C bundles for Linux arm64, Linux x86-64, and macOS arm64 are attached to the v0.3.0 release on GitHub.
