A 30.9M-parameter encoder built from the ground up for Hausa, Yoruba, Igbo, and Nigerian Pidgin — with a tokenizer that beats GPT-4o’s on all four.
Most “African language support” in NLP is an afterthought: a massive multilingual model where Yoruba got a sliver of the training budget and the tokenizer shreds every word into fragments. Today we’re releasing something different — mist-encoder-base-ng, a compact encoder pretrained entirely from scratch on Nigerian text, open-sourced on Hugging Face under Apache 2.0.
It covers the four major languages of Nigeria — Hausa, Yoruba, Igbo, and Nigerian Pidgin — and it’s built to be a workhorse base: load the encoder body, attach a head, and fine-tune for classification, sentiment, language ID, or sentence embeddings.
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("olaverse/mist-encoder-base-ng")
enc = AutoModel.from_pretrained("olaverse/mist-encoder-base-ng")
Why from scratch?
Fine-tuning an existing multilingual model inherits that model’s tokenizer — and for Nigerian languages, general-purpose tokenizers are quietly terrible. Every extra subword a tokenizer needs per word means less real signal fits in a fixed context window, slower inference, and weaker per-token representations.
So before training the model, we built the tokenizer: otk-bpe-50k, a unified byte-level BPE for Naija text with a ~50k vocabulary, NFC diacritic preservation, code-mixed English support, and 0% unknown tokens. The fertility numbers (tokens per word — lower is better) speak for themselves:
| Tokenizer | Hausa | Yoruba | Igbo | Pidgin |
|---|---|---|---|---|
| otk-bpe-50k (ours) | 1.231 | 1.296 | 1.416 | 1.249 |
| GPT-4o (o200k) | 1.589 | 1.687 | 1.807 | 1.304 |
| AfroXLMR | 1.604 | 2.277 | 2.570 | 1.401 |
Our tokenizer is more efficient on Nigerian text than the tokenizer behind GPT-4o — on all four languages. On Igbo, AfroXLMR needs over 2.5 tokens per word; ours needs 1.4.
The model
On top of that tokenizer sits a ModernBERT architecture — 6 layers, hidden size 384, 1024 max positions — pretrained with masked language modeling on ~480k carefully prepared chunks. The corpus combines FineWeb-2, the audited WURA corpus, and a fresh Nigerian Pidgin dataset, all cross-deduplicated (FineWeb-2 and WURA both descend from Common Crawl, so overlap was real), language-balanced so abundant Hausa didn’t drown out scarce Igbo and Pidgin, and chunked into windows so no text was wasted on truncation. Every source is attribution-only licensed, which is what makes the clean Apache 2.0 release possible.
The result is 30.9M parameters — and since the embedding table is about two-thirds of that, the transformer itself is only ~11M. For comparison: AfriBERTa is 126M, mBERT is 178M, XLM-R is 270M.
What a 30.9M specialist can do
On MasakhaNEWS topic classification (macro-F1), the model is genuinely competitive at a quarter of the baselines’ size:
| Model | Params | Hausa | Yoruba | Igbo | Pidgin |
|---|---|---|---|---|---|
| mist-encoder-base-ng | 30.9M | 0.878 | 0.859 | 0.803 | 0.898 |
| AfriBERTa | 126M | 0.924 | 0.921 | 0.914 | 0.991 |
| mBERT | 178M | 0.806 | 0.886 | 0.805 | 0.967 |
It beats mBERT on Hausa and ties it on Igbo — with under a fifth of the parameters. That efficiency is the point: this is a model you can fine-tune on a laptop and deploy on-device or in low-resource environments where 126M+ models are impractical.
What it can’t do (yet) — and exactly why
We publish our weaknesses with the same rigor as our wins. Named-entity recognition is this model’s honest weak spot: it trails AfriBERTa and mBERT by roughly 10–20 entity-F1 on MasakhaNER 2.0, with Hausa NER lowest at 0.656.
And we didn’t just report the gap — we diagnosed it. The deficit is structural, not a tuning artifact: it persists across random seeds and labeling schemes. Two causes: the unified 50k tokenizer fragments entity words (person and place names) more than language-specific tokenizers — on Hausa NER text, ~61% of entity words split into multiple subwords versus ~21% for AfriBERTa — and at 30.9M parameters the model has less capacity to reassemble meaning from those fragments. If NER accuracy is critical to your application, use a larger model. If sentence-level understanding at minimal cost is what you need, this is your base.
The model card also lays out an evidence-backed v2 roadmap: more capacity (~50–80M), a less-fragmenting tokenizer for token-level tasks, and more Pidgin and Hausa data. Notably, longer pretraining isn’t on the list — eval loss plateaued by epoch 11, so we know that lever is spent.
Part of a bigger picture
mist-encoder-base-ng joins the Mist Embedder collection and sits alongside our LID models (which are fine-tuned classifiers) as complementary infrastructure: a from-scratch, Nigerian-first foundation others can build on. Two fine-tunes of it already exist on the Hub, and there’s a demo Space to try it.
- Model: https://huggingface.co/olaverse/mist-encoder-base-ng
- Tokenizer: https://huggingface.co/olaverse/otk-bpe-50k
- Collection: https://huggingface.co/collections/olaverse/mist-embedder
Apache 2.0, free for commercial use. If you fine-tune it for a Nigerian-language task — sentiment, topic tagging, embeddings, anything — open a discussion on the model page and show us. That’s the whole point of releasing the base.
Olaverse Lab builds open-source AI with first-class support for underserved languages. Explore all our models at huggingface.co/olaverse.
Leave a Reply