The premium
AI models read, bill and limit text in tokens, the word pieces a tokenizer splits text into. For the same meaning, African languages need more tokens than English. That token premium is paid on every query: in cost, in how much text a model can read at once, and in speed.
A tokenizer is fixed into every model trained with it, so its premium is paid for the life of that model. Cutting it at the tokenizer is among the least expensive ways to make AI in African languages cheaper, faster and able to read more.
The approach
African text is first split with Gemma 3's own tokenizer. New merges are learned on top, with each language weighted equally. Any merge that would change English or French text is discarded, and the 32,000 survivors are appended after the Gemma 3 vocabulary. Training text is filtered against both test sets.
# Using the tokenizer from tokenizers import Tokenizer tok = Tokenizer.from_file("tokenizer/gemma3-afx32k.json") enc = tok.encode("Huku shirika la Save the Children likitela pamoja timu", add_special_tokens=False) print(len(enc.ids), enc.tokens)
Your language. Your voice. Your terms.
The result
Nzinga needs fewer tokens than all 12 other tokenizers tested on all 12 Tier 1 languages, including Gemma 4, Qwen 3.8 and gpt-oss. Across the 12 it needs 22% fewer tokens than the best of them, and 6 of the 12 already meet the 1.3 to 1.5 target. English and French are within one to three tokens of Gemma 3.
In Swahili, a model with a 128,000-token context window can read about 79,100 words at once with Nzinga, against 71,900 with the best global tokenizer.
These are token counts only. Whether a model trained with Nzinga performs better has not been measured yet. The experiment is built and waiting for GPU time, and its results will be published whether positive or not.