Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
And screw all the 0.1% haters on here, this is great stuff.
[dead]
Valuable not only for inference, but for training too (think proprietary datasets).
I would add, a single individual did this.
One person can make a difference :-)
How can I join this? Sounds interesting
Could I join this?
[dead]
There is a serious research community on tokenization, and we are quite interested in this work.
As I said, It just might be the low information density approach to talking recently that seems off to me.
To explain a bit more, I kept reading and waiting for the penny drop but nothing.
“I cloned your repo” — okay then what happened? Nothing? I put my shoes on this morning.
“I know the tokrniziation community wants to absord tje lessons” okay? Tell me the point please! I know the AI community wants to understand the universe.
“These are useful ideas”. Yes.
Of course, you could argue that my comment falls in the same category in the sense that I am not actually contributing to the topic at hand but I am peeved with all low the information noise.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8
sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4
input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%)
input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%)
input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%)
Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.
If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
Tips & Tricks on parameters/settings?
What happens at peak? Do people have to wait now? Increase of latency?
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
[dead]
Have you considered publishing a rust crate as well? (If not, I volunteer.)
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.
It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.
Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQ
Another Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.
In my case, I recently optimized for https://uxwizz.com the session playback: before, it was saving the entire recording in one row, with updates, and loading it client-side in one chunk. I asked thr AI to, instead, store the recorded chunks in separate rows in db, and when replaying, to stream those chunks as needed. It implemented it perfectly, and everything is much snappier and lighter now.
So, the idea was "store in chunks and stream the chunks", and the LLM does it. Yes, there were a few bugs, but those systems genetally either work or don't. So testing for a 30 minutes after was enough, and now the system is live, and a client actually told me in person yesterday he loved the playback improvements.
I said in another comment somewhere, now knowing how to code isn't important, what's important is that you know what to ask, and what to look for when testing.
Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)
Good models now already follow best practices when implementing, better than junior devs.
I wrote a bit about this, I call it "AI slap", lol:
We're clearly thinking of very different "optimizations" here I think :) Do you have any concrete examples of this sort of optimizations you'd get automatically? In my experience, you get what you prompt for, if I don't include to think about performance, they won't think about performance, not sure what model would automatically consider things like that. Most of the time I use whatever SOTA OpenAI has on maximum reasoning level, fwiw.
If you ask to implement a custom dropdown that does something, it often comes with good spacing, aria-accessible tags, keyboard accessibility, etc. A junior dev wouldn't think of all of those.
Also, it will probably choose the right HTML elements to use for it (i.e. maybe the modern native popover functionality instead of implementing it with custom JS, which would indeed be more efficient and less code).
I am not saying it would add caching by default (even though, it might suggest that), but it's more likely to choose whatever the best options are and to use them as they should, including going around knowing limitations and gotchas.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
In practice every training project I've worked on does tokenization in a separate data processing phase.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
Ah, Python.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?
Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:
• 4,700 American homes
• a week of British tea
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
If this increases efficiency, it has real impact. 0.1% is A LOT at scale.
But look at e.g. Biome; optimizing the formatter didn't seem worth it for a long time because it only took <1s to format most files. But it's <1s times millions of files, billions of times a day when you add up every developer that used Prettier for their code formatting - it adds up.
And I'm convinced Biome triggered or was part of a bigger effort to convert JS based tools to native code. This saved time and energy, which in turn allows for faster and / or more feedback loops, which in turn allows faster turnaround cycles for software development (bet it human or LLM assisted), etc. It's a compound effect.
I don't know enough about tokenization or whatever to judge this one, but if it's 1000x as fast as it used to be, there will be less need to try and avoid or minimize tokenization which may lead to new applications.
So much for "human programming is obsolete".
In the final stages of the project, AI was used to assist:
Implementing the user-facing API Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization Porting SIMD strategies between AVX512/AVX2/NEON Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy Refactoring and code reuse
BTW throughput is measured for a 12 GiB file. Would be interesting to see the throughput for something more like 32 KiB, with cold start (token cache not yet populated).
marcelroed is modest, this speedup is not. Good work.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.
> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
[deleted]