9 October 2026
ttok 1.0 released, fixing a default tokenizer problem
First reported
Simon Willison ran this on .
- A new version, ttok 1.0, has been released as a tool for counting text in units the AI model reads.
- The update was prompted by a bug where the tool defaulted to a tokenizer (the piece that splits text into chunks for the model) from GPT-4, an OpenAI model.
