The premium
A tokenizer is fixed into every model trained with it, so its premium is paid for the life of that model. African languages need more tokens than English for the same meaning, in cost, context and speed.
The approach
African text is first split with Gemma 3's own tokenizer. New merges are learned on top, with each language weighted equally. Any merge that would change English or French text is discarded.
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer/gemma3-afx32k.json")
enc = tok.encode("Huku shirika la Save the Children likitela pamoja timu",
add_special_tokens=False)
print(len(enc.ids), enc.tokens)
Your language. Your voice. Your terms.
Brand line
,
Nzinga
The result
Nzinga needs 22% fewer tokens than the best other tokenizer across the 12 Tier 1 languages, and leaves English and French unchanged. Model accuracy is not measured yet.
