Tag: small language models

A diacritic model that cannot corrupt your text

Diacritic restoration is easy to state. Given se eranko naa si gbo o, produce ṣé ẹranko náà sì gbọ́ ọ. Every character keeps its identity; some acquire marks.

The standard approach treats it as translation. Bare text in, marked text out, generated token by token. That is what diacnet did — ByT5 fine-tuned on 7 million sentences.

It works, mostly. But “mostly” hides something. On Hausa, only 94.7% of diacnet-1.1 outputs still stripped back to their input. The other 5.3% weren’t mis-accented — they were different text. Words dropped. Clauses rewritten. Punctuation invented. A model asked to add accents had quietly decided to edit.

That is not a tuning problem. A generative decoder can emit any token at any position, so nothing in the architecture prevents it. You can penalise it in training and it gets rarer, but it never becomes impossible — and “rare” is not a property you can put in a contract.

We wanted one where it is impossible.

Reframing: classify, don’t generate

The task has a hard structural constraint that generation throws away. The output has exactly as many characters as the input, in the same order, with the same base letters. Only the marks change.

So don’t generate. Classify. For each character, predict which diacritics it carries, and copy the base character through.

 input      s      e      r      a      n      k      o
            │      │      │      │      │      │      │
            ▼      ▼      ▼      ▼      ▼      ▼      ▼
     ╔═══════════════════════════════════════════════════╗
     ║          character transformer encoder            ║
     ╚═══════════════════════════════════════════════════╝
            │      │      │      │      │      │      │
   SHAPE   DOT     ·      ·      ·      ·      ·      ·
   TONE     ·      ·      ·    ACUTE    ·      ·      ·
            │      │      │      │      │      │      │
            ▼      ▼      ▼      ▼      ▼      ▼      ▼
 output     ṣ      e      r      á      n      k      o
            └──────┴──────┴──────┴──────┴──────┴──────┘
              base letters COPIED, never predicted

The base character is not an output of the network. There is no mechanism by which the model can change it, so strip(output) == strip(input) holds by construction — for a trained model, an untrained one, or the int8 export.

We test exactly that: the invariant suite runs against a randomly initialised model. If the guarantee only held for a good model, it would not be a guarantee.

The model

Twelve layers, 512 hidden, 37.6M parameters — against 580M for the seq2seq model it replaces.

      graphemes  [B,T]              language id  [B]
           │                              │
           ▼                              ▼
    ┌─────────────┐                ┌─────────────┐
    │  char emb   │                │  lang emb   │   11th slot =
    │   403×512   │                │    11×512   │   "unknown language"
    └──────┬──────┘                └──────┬──────┘
           │                              │
           └──────────────► + ◄───────────┘   additive: no <yor> token,
                            │                 no position spent
                            ▼
        ╔═══════════════════════════════════════╗
        ║          × 12  encoder block          ║
        ║  ┌─────────────────────────────────┐  ║   bidirectional —
        ║  │ RMSNorm → MHA(8 heads, RoPE)    │  ║   a diacritic depends on
        ║  │ + residual                      │  ║   what follows as much
        ║  │ RMSNorm → SwiGLU 512→1344→512   │  ║   as what precedes
        ║  │ + residual                      │  ║
        ║  └─────────────────────────────────┘  ║
        ╚═══════════════════╤═══════════════════╝
                            │  H  [B, T, 512]
        ┌──────────┬────────┴────────┬───────────┐
        ▼          ▼                 ▼           ▼
   ┌─────────┐┌─────────┐      ┌──────────┐┌──────────┐
   │  SHAPE  ││  TONE   │      │   MLM    ││   LID    │
   │ 512→15  ││ 512→8   │      │ 512→403  ││ 512→10   │
   │per char ││per char │      │ tied emb ││mean-pool │
   └────┬────┘└────┬────┘      └────┬─────┘└────┬─────┘
        │          │                │           │
        └────┬─────┘           training only  language
             ▼
      legality mask  →  argmax  →  compose(base, shape, tone)

Where the parameters go:

component	shape	params
character embedding	403 × 512	206k
language embedding	11 × 512	6k
12 × encoder block	attn 4·512² + FFN 3·512·1344	37.4M
SHAPE head	512 × 15	7.7k
TONE head	512 × 8	4.1k
LID head	512 × 10	5.1k
MLM head	tied to embedding	0

The task heads together are 12k parameters — 0.03% of the model. Nearly everything is shared encoder, which is why ten languages cost roughly the same as one.

loss = 1.0·CE(shape) + 1.0·CE(tone) + 0.3·CE(masked_char) + 0.05·CE(language)

SHAPE and TONE are the task. MLM is auxiliary and dropped at inference — it forces lexical knowledge into the encoder, which is what tone disambiguation needs. Its target is always the fully marked character; masking ú and accepting u would teach the encoder that bare vowels are valid Yorùbá, the exact failure being fixed. LID is nearly free and makes the API restore(text) instead of restore(text, lang).

Three choices that carry weight:

RoPE, not learned positions. Documents are processed with a sliding window, so the model sees offsets at inference it never saw in training. Learned absolute positions generalise badly to that; rotary ones don’t.

Language as an additive embedding. No <yor> prefix token — nothing to confuse with content, no position spent. During training the language embedding is dropped 12% of the time, giving a genuine “unknown language” mode, so being wrong about the language degrades gracefully instead of catastrophically.

Trained from scratch. We considered starting from an existing encoder, but its tokenizer was trained on diacritized text while our inputs are stripped — every subword boundary would be wrong in the direction that matters most.

Splitting shape from tone

The naive version assigns each character one label from a flat set: acute, grave, underdot, circumflex-plus-acute. That works for Spanish and falls apart on Vietnamese, where stacked marks produce ~60 classes, most of them rare.

Instead, factorize:

   ẹ́   →   e  +  DOT_BELOW   +  acute        (Yorùbá)
   ộ   →   o  +  CIRCUMFLEX  +  dot-below    (Vietnamese)
   ữ   →   u  +  HORN        +  tilde        (Vietnamese)
   ń   →   n  +     —        +  acute        (Polish)

SHAPE is marks that change letter identity. TONE is the mark that changes pitch or stress. Two heads, two label sets, two loss terms.

The subtlety — and the reason it works — is that which marks belong to which slot is language-dependent:

   U+0323  dot below      →  SHAPE in Yorùbá     (ẹ and e are different letters)
                          →  TONE  in Vietnamese (nặng, one of six tones)

   U+0303  tilde          →  SHAPE in Spanish    (ñ is a letter)
                          →  TONE  in Vietnamese (ngã)

A flat label set cannot express that. It sees ọ and assigns one class, blind to the fact that the same codepoint plays different grammatical roles in the two languages.

Factorized, Vietnamese collapses to 5 shapes × 6 tones, every component seen thousands of times. Across all ten languages the final space is 15 shapes and 8 tones.

The bigger payoff is diagnostic. Yorùbá’s dominant failure is tone direction — the model knows a mark belongs there and picks the wrong one, writing pátápátá for pàtàpàtà. In a flat label space that is indistinguishable from any other error. Split, it gets its own head, its own confidence and its own metric:

checkpoint	Yorùbá DER	tone portion
10k steps	0.1306	87%
30k steps	0.1138	81%
120k steps	0.0836	83%

Four fifths of the remaining error, at every stage of training, is one failure mode. Aggregate DER would have shown a number going down and told us nothing about what to do next.

Graphemes, not codepoints

Yorùbá stacks marks: ọ̀ is o with an underdot and a grave. Unicode has no precomposed codepoint for that combination, so even after NFC normalisation it stays two codepoints. len("ọ̀rọ̀") is 5, not 3.

Iterating over the string misaligns every label after the first stacked character. Everything here operates on graphemes — a base codepoint plus any combining marks that follow it. A small detail that silently determines whether the hardest language in the set trains at all.

What the corpus was teaching us

This is the finding that changed the result more than any architectural decision.

Before training, we plotted diacritic density — the fraction of eligible characters actually carrying a mark — per language. Nine of ten gave a single hump. Yorùbá gave two:

  count
    │        ▁▄██▇▅▃▂▁
    │       ▂████████▃                       ▁▂▃▃▂▁
    │      ▃██████████▅▂                   ▁▃██████▃▁
    └──────┴───────────┴──────────────────┴──────────┴────► density
          0.10                                0.43
        89% of sentences                 11% of sentences
        UNDER-MARKED                        correct

Correctly marked Yorùbá lands around 0.30–0.45 — nearly half its characters are vowels, most carry a high or low tone, plus underdots on ẹ, ọ and ṣ.

Which means the large mode is the broken one. Eighty-nine per cent of our Yorùbá training data was under-marked: real words with their tone marks simply omitted. And we were using it as ground truth. Every one of those sentences was teaching the model that bare vowels are correct.

That is diacnet-1.1‘s documented failure, sitting in a histogram. It also explains something we had noticed and shrugged at: diacnet-1.1 is worse than 1.0 on all three African languages. More data, trained longer, learned to omit tone more confidently.

The fix is not to delete the data

The obvious move is to drop the low-density mode. We didn’t, because under-marked Yorùbá is still Yorùbá — the words are real, the morphology is real. It is excellent data for learning the language. It is poisonous data for learning tone marks.

Since we already had two objectives on one encoder, we could separate them. Sentences below the detected threshold keep their masked-character targets and contribute no diacritic supervision at all:

   density 0.458   ABOVE floor  →  23 diacritic labels,  8 MLM labels
   density 0.000   BELOW floor  →   0 diacritic labels,  8 MLM labels
                                    └── still teaches vocabulary

The MLM head trains on all 1.9M Yorùbá sentences. The tagger heads train on the clean ~205k. Same corpus, two different notions of what counts as a label.

Detecting it without eyeballing a threshold

We wanted this automatic. The first attempt used Otsu’s method, the standard image-binarisation algorithm — and it reported every language as bimodal.

Otsu returns a split for any distribution. On a single right-skewed hump — which is what density distributions look like — that split lands near the left edge, where “the peak below the cut” is two nearly-empty bins and every statistic derived from it is noise. Our test fixtures used symmetric Gaussians, which hid the failure completely.

The fix was to find peaks directly and measure how deep the trough between them falls, requiring both sides to hold at least 5% of the mass so a tail cannot masquerade as a mode. Measured that way the answer is not close:

   yor   ████████████████████████  0.55   BIMODAL
   pol   ████                      0.10
   tur   ███                       0.07
   ita   ▊                         0.02
   ibo                             0.00

Igbo and Hausa are correctly left alone, and that matters: standard Igbo orthography omits tone by convention, and Hausa rarely writes it at all. Their low density is a property of the writing system, not a defect. No threshold should touch them.

What falls out of the architecture

Per-character classification gives you several things that are awkward or impossible in a generative setup.

Automatic language detection. A fourth head classifies the language from the mean-pooled encoder state. It costs almost nothing — DER 0.0132 with the language supplied, 0.0133 with it detected — and most integrations don’t reliably know the input language.

Per-character confidence. The joint shape × tone distribution gives a calibrated probability per character (temperature-fitted on validation, T = 1.14). Threshold it, and characters below are left exactly as the user typed them:

threshold	coverage	DER on committed
0.00	100.0%	0.0132
0.90	97.1%	0.0039
0.99	91.9%	0.0008

At 0.90: 97% of characters restored at 99.6% accuracy, the rest flagged. This is the number to put in front of a customer, because it answers the question they have. A wrong tone mark changes meaning; a missing one is merely incomplete. The threshold is per-request, so a CMS pre-fill and a legal pipeline sit at different points on the same curve.

Legality masking. For each (language, base character) we record which (shape, tone) pairs were ever observed and mask the rest to −∞ at decode time. No acute on Turkish vowels, no Vietnamese horn on Polish. An interesting result: turning it off changes DER by zero to four decimal places. The model learned the orthographies well enough that the constraint never binds. We keep it as a guarantee for out-of-distribution input — it costs nothing.

Protected spans. URLs, emails, @handles, inline code and CONSTANT_NAMES pass through untouched, and were excluded from the training loss. An accent inside a URL is never correct, and it is the error a reader notices first.

Documents, not sentences. No sentence splitting. Overlapping windows are planned across the input and only the centre of each is trusted, so every character is predicted with context on both sides. The window plan is proved to partition the input exactly — 500 randomised cases in CI.

Results

diacbench, 1000 sentences per language.

lang	DER	shape	tone	exact	compliance
ita	0.0002	0.0000	0.0002	0.991	1.0000
fra	0.0012	0.0003	0.0009	0.952	1.0000
tur	0.0016	0.0016	0.0000	0.961	1.0000
por	0.0019	0.0006	0.0014	0.925	1.0000
pol	0.0022	0.0018	0.0003	0.935	1.0000
spa	0.0022	0.0001	0.0021	0.917	1.0000
hau	0.0041	0.0040	0.0001	0.741	1.0000
ibo	0.0122	0.0110	0.0013	0.483	1.0000
vie	0.0164	0.0073	0.0124	0.650	1.0000
yor	0.0836	0.0203	0.0695	0.084	1.0000
all	0.0132	0.0055	0.0086	0.764	1.0000

Yorùbá 0.2006 → 0.0836, a 58% reduction. Hausa 0.0593 → 0.0041, 93% — though much of the Hausa gain is the compliance guarantee rather than better modelling, since a large share of the old error was corruption rather than wrong accents.

37.6M parameters against 580M.

Against frontier LLMs

300 sentences per language, matching how the baselines were run.

lang	`diactag-1.0`	Claude Sonnet 4.5	GPT-4o-mini
yor	0.0933	0.1913	0.2811
ibo	0.0110	0.0427	0.1277
hau	0.0041	0.0178	0.1432
vie	0.0166	0.0107	0.0399
fra	0.0014	0.0052	0.0023

Best on 7 of 10. Vietnamese and Portuguese genuinely lose.

Those LLM numbers deserve a footnote, because they are the charitable ones. They come from a harness that checks whether the output still strips back to the input and throws it away when it doesn’t. Raw, against the floor of copying the input unchanged:

lang	copy-input floor	Claude, raw
hau	0.0236	0.3509
ibo	0.1171	0.2261
fra	0.0471	0.1171

On three languages, a frontier model makes the text worse than doing nothing at all. Hausa is already 97.6% correct if you leave it alone; Claude returns it 35% wrong.

That fallback harness is our architecture, reimplemented externally and imperfectly. The difference is that here there is nothing to discard.

Deployment

backend	chars/s	p50	size	DER
PyTorch CPU	105	591 ms	150 MB	0.0105
ONNX fp32	202	286 ms	150.9 MB	0.0105
ONNX int8	244	200 ms	38.3 MB	0.0108

Three times faster and four times smaller for +0.03pp DER. Compliance stays 1.0000 under quantisation — the guarantee is architectural, not a property of a particular numeric precision.

A 38MB artifact at 244 characters per second on one CPU core takes the GPU off the serving bill entirely.

One honest caveat: p95 is 788–1099 ms across all backends while p50 varies 3×. The tail is length-driven — long sentences need more windows — not backend-driven. Quantisation doesn’t fix it.

Reproducing it

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

Data. 7.0M sentences from a web + Wikipedia corpus, plus 131k from a Q&A passage set — the only non-encyclopedic register in the mix — gated on diacritic density and deduplicated against the eval splits by stripped-form hash.

Two filters worth copying if you build something similar. Script filtering: all ten languages are Latin, but Hausa Wikipedia carries Ajami and Yorùbá articles carry Korean and Devanagari names; those contribute nothing but identity labels. Cross-language filtering: a Vietnamese name inside a Turkish sentence factorizes into composite classes that exist in no language, because Turkish declares no tone slot for those marks to occupy. Without both, the label space inflates from 15 shape classes to 99, most of them junk competing in the same softmax.

Training. AdamW, lr 3e-4 with 2000 warmup steps and cosine decay to 5%, weight_decay 0.05, batch 128 × 2 accumulation = 256, seq len 256, bf16, 120k steps. About 13 hours on a single A100.

Language balancing. Temperature sampling at α=0.5. At α=1 Yorùbá and Igbo would be 54% of the mix; at α=0 Hausa’s 400k sentences get seen 5× per Yorùbá epoch and overfit.

Augmentation. partial_mark_prob=0.30 with keep range [0.05, 0.9], case perturbation (2% upper, 3% lower, 1% title), 3% protected-span injection, mlm_prob=0.15, lang_dropout=0.12.

Evaluation. Report shape_DER and tone_DER separately, and compliance as a hard assertion rather than a metric. If compliance is not exactly 1.0, the architecture is broken and nothing else in the table means anything.

What it still gets wrong

Yorùbá. DER 0.0836, 83% of it tone direction. Sentence-level exact match is 0.084 — 92 of every 100 Yorùbá sentences contain at least one wrong mark. A harsher framing than DER, and a truer one.

Igbo and Hausa tone numbers are not achievements. tone_DER of 0.0013 and 0.0001 looks superb and means little, because those orthographies barely write tone. Nothing to learn, nothing to get wrong.

No typo correction. The architecture cannot insert or delete characters, so it cannot fix Ile → Ilé and teh → the in one pass. That is the price of the guarantee, and we think it is the right trade — a model that reliably does one thing beats one that sometimes does two. The per-character confidence is a natural trigger for a separate corrector.

Dense input degrades. The Polish pangram Zażółć gęślą jaźń has density 0.565 against a Polish median of 0.068 — nine times denser than real Polish — and the model misses six characters, despite a Polish DER of 0.0022.

Some errors are irreducible. Viaggio and Viaggiò are both valid Italian and the stripped form contains no information distinguishing them. Every language has a nonzero floor.

Three things worth taking away

Plot your data before you train. The bimodal Yorùbá distribution took ten minutes to find and changed the outcome more than any modelling decision. We nearly skipped it.

Make your metric decompose along your failure modes. Aggregate DER would have gone down and taught us nothing. Splitting shape from tone told us, at every checkpoint, that four fifths of the problem was one thing — and that the other nine languages had already converged.

Put your guarantees in the architecture, not the loss function. A constraint the model can violate is one you have to monitor, apologise for, and wrap in a fallback. A constraint it cannot violate is one you can sell.

Weights, ONNX exports and label space: olaverse/diactag-1.0. Documentation: olaverse-labs.github.io/olaverse/models/diactag. Apache 2.0.

August 4, 2026

Introducing DiacNet-1.1: Multilingual Diacritic Restoration for 10 Languages

Overview

Diacritics—such as accents, tone marks, and special characters—are essential for clarity, pronunciation, and meaning in many languages. Yet, they are often stripped away in digital text due to technical limitations, user input habits, or data processing errors. This loss can lead to ambiguity, mispronunciation, and even miscommunication.

Today, we’re excited to introduce DiacNet-1.1, a state-of-the-art model designed to restore diacritics to text across 10 languages in a single, unified system. Developed by olaverse, DiacNet-1.1 is fine-tuned from Google’s ByT5-base (580M parameters) and trained on 7 million sentences of commercially licensed text. Whether you’re working with Yorùbá, Vietnamese, Polish, or Spanish, DiacNet-1.1 can help you recover the nuances of written language with remarkable accuracy.

Supported Languages

DiacNet-1.1 supports the following 10 languages:

Yorùbá (<yor>)
Igbo (<ibo>)
Hausa (<hau>)
Vietnamese (<vie>)
Polish (<pol>)
Turkish (<tur>)
Portuguese (<por>)
Spanish (<spa>)
French (<fra>)
Italian (<ita>)

You can also use the <auto> tag to let the model infer the language automatically, with minimal impact on accuracy.

How It Works

DiacNet-1.1 is a sequence-to-sequence model that takes stripped text (text without diacritics) and predicts the fully diacritized version. The model is trained on a diverse corpus of web and encyclopedic text, ensuring robustness across different domains.

Example Usage

Here’s how you can use DiacNet-1.1 in just a few lines of Python:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load the model and tokenizer
tok = AutoTokenizer.from_pretrained("olaverse/diacnet-1.1")
model = AutoModelForSeq2SeqLM.from_pretrained("olaverse/diacnet-1.1")

# Restore diacritics to Yorùbá text
text = "<yor> se eranko naa si gbo o?"
inputs = tok(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))
# Output: ṣé ẹranko náà sì gbọ́ ọ?

Language Tagging

Prefix your input text with the appropriate language tag to guide the model:

# Vietnamese
text = "<vie> Toi khong biet tieng Viet"
# Output: Tôi không biết tiếng Việt

# Polish
text = "<pol> Lodz jest piekna"
# Output: Łódź jest piękna

# Auto-detect language
text = "<auto> El nino esta en la casa"
# Output: El niño está en la casa

Performance

DiacNet-1.1 was evaluated on diacbench, a benchmark of 1,000 held-out sentences per language, sourced independently from the training corpus. The model was tested using the following metrics:

DER (Diacritic Error Rate): Errors restricted to diacritic-eligible characters.
WER (Word Error Rate): Measures errors at the word level.
CER (Character Error Rate): Measures errors at the character level.
Exact Match: Percentage of sentences restored perfectly.

Benchmark Results

Language	DER	WER	CER	Exact Match
Italian	0.0002	0.10%	0.04%	99.3%
Portuguese	0.0031	1.39%	0.23%	88.0%
French	0.0053	0.77%	0.18%	94.7%
Polish	0.0058	2.65%	0.33%	85.3%
Turkish	0.0068	1.50%	0.35%	91.7%
Spanish	0.0081	2.06%	0.35%	83.0%
Vietnamese	0.0460	11.98%	3.06%	36.0%
Igbo	0.0508	4.78%	1.29%	41.0%
Hausa	0.0593	2.53%	0.51%	59.7%
Yorùbá	0.2006	42.18%	14.50%	0.3%

Note: The lower exact-match scores for Vietnamese and Yorùbá reflect the high density of diacritic marks in these languages. Even a single error can cause the entire sentence to fail the exact-match metric. DER is the most meaningful metric for these languages.

Improvements Over DiacNet-1.0

DiacNet-1.1 represents a significant leap forward from its predecessor, DiacNet-1.0, with improvements across multiple languages:

Language	DER (1.0 → 1.1)	WER (1.0 → 1.1)	Verdict
Vietnamese	0.1264 → 0.0460	21.98% → 11.98%	1.1 — 2.7× better
Turkish	0.0447 → 0.0068	19.20% → 1.50%	1.1 — 6.6× better
Polish	0.0357 → 0.0058	15.12% → 2.65%	1.1 — 6.2× better
Italian	0.0015 → 0.0002	0.66% → 0.10%	1.1 — 7.5× better
Portuguese	0.0072 → 0.0031	1.80% → 1.39%	1.1 better
Spanish	0.0084 → 0.0081	2.19% → 2.06%	~equal
French	0.0038 → 0.0053	1.46% → 0.77%	Mixed
Igbo	0.0359 → 0.0508	8.34% → 4.78%	Mixed
Hausa	0.0383 → 0.0593	8.58% → 2.53%	Mixed
Yorùbá	0.1554 → 0.2006	27.69% → 42.18%	1.0 better

Key Observations

Major Gains: DiacNet-1.1 achieves 2.7× to 7.5× better performance on Vietnamese, Turkish, Polish, and Italian.
Mixed Results: For French, Igbo, and Hausa, the model shows trade-offs between DER and WER. For example, while DER increased slightly for Hausa, WER improved dramatically (8.58% → 2.53%), and exact-match scores jumped from 0% to 59.7%.
Yorùbá Challenge: The model’s performance on Yorùbá regressed due to the lower quality of diacritic annotations in the larger training corpus. DiacNet-1.0 was trained on ~2,000 high-quality Yorùbá passages, while DiacNet-1.1’s corpus, though much larger, had a lower diacritic density. This highlights a critical trade-off: more data isn’t always better if annotation quality suffers.

Priority for Next Release: Improving Yorùbá, Igbo, and Hausa performance by incorporating higher-quality annotated data.

Comparison with General-Purpose LLMs

We benchmarked DiacNet-1.1 against Claude Sonnet 4.5 and GPT-4o-mini on the same diacbench dataset. The results reveal interesting trade-offs between accuracy and reliability:

Diacritic Error Rate (DER)

System	Yorùbá	Igbo	Hausa	Vietnamese	Polish	Turkish	Portuguese	Spanish	French	Italian
Copy-Input (Floor)	0.4358	0.1171	0.0236	0.3062	0.0915	0.1113	0.0476	0.0426	0.0471	0.0333
DiacNet-1.0	0.1554	0.0359	0.0383	0.1264	0.0357	0.0447	0.0072	0.0084	0.0038	0.0015
DiacNet-1.1	0.2006	0.0508	0.0593	0.0460	0.0058	0.0068	0.0031	0.0081	0.0053	0.0002
Claude 4.5 (Strict)	0.3353	0.2261	0.3509	0.0278	0.0343	0.0241	0.0172	0.0215	0.1171	0.0005
Claude 4.5 (Fallback)	0.1913	0.0427	0.0178	0.0107	0.0035	0.0019	0.0011	0.0040	0.0052	0.0005
GPT-4o-mini (Strict)	0.4607	0.5035	0.6368	0.0847	0.0430	0.0261	0.0280	0.0555	0.0341	0.0085
GPT-4o-mini (Fallback)	0.2811	0.1277	0.1432	0.0399	0.0070	0.0038	0.0024	0.0089	0.0023	0.0052

Key Takeaways

Strict vs. Fallback Scoring:

Strict scoring counts all errors, including cases where the LLM rewrites or translates the input.
Fallback scoring assumes a pipeline that detects and skips rewritten outputs, focusing only on diacritic restoration.

DiacNet-1.1 vs. Claude (Strict): DiacNet-1.1 outperforms Claude on 8 out of 10 languages when scored strictly.
DiacNet-1.1 vs. Claude (Fallback): With fallback scoring, Claude outperforms DiacNet-1.1 on 9 out of 10 languages. This highlights that while LLMs can achieve higher accuracy on compliant outputs, they fail to return usable text on 20-35% of inputs (e.g., 34% of Hausa, 26% of Yorùbá, 21% of Igbo).
Reliability: DiacNet-1.1 never rewrites or alters base characters in 94.7–100% of cases, depending on the language. In contrast, LLMs alter 20–50% of inputs, making them unreliable for large-scale processing.

Output Compliance

System	Yorùbá	Igbo	Hausa	Vietnamese	Polish	Turkish	Portuguese	Spanish	French	Italian
DiacNet-1.1	100%	96.3%	94.7%	99.3%	100%	99.7%	100%	99.7%	99.7%	100%
DiacNet-1.0	98.0%	99.0%	98.3%	96.3%	99.3%	100%	99.7%	99.7%	100%	100%
Claude 4.5	74.3%	79.3%	65.7%	97.7%	96.7%	98.0%	97.7%	98.3%	88.3%	100%
GPT-4o-mini	68.0%	57.3%	49.3%	93.7%	96.0%	94.7%	97.7%	97.3%	96.7%	99.7%

Conclusion: If you need maximum accuracy on a few sentences and can tolerate retries, a frontier LLM like Claude may be the best choice. However, if you need to process millions of sentences reliably, deterministically, and offline, DiacNet-1.1 is the tool for the job.

Training Details

Aspect	Description
Base Model	google/byt5-base (580M parameters, byte-level)
Data	7.02M sentences from FineWeb-2 (ODC-By) and Wikipedia (CC BY-SA), filtered for diacritic density, NFC-normalized, and deduplicated.
Objective	Sequence-to-sequence: `<lang> stripped text` → fully diacritized text.
Augmentation	Partial stripping (30%), character noise (7%), `<auto>` tag (12%), optional gloss conditioning (~6%).
Precision	BF16, max 512 bytes.
Correction Pass	Continued training on high-density Yorùbá with 35% replay of other languages, learning rate 2e-5.

Note: All training data is commercially licensed. No non-commercial sources (e.g., MENYO-20k, JW300) were used.

Limitations

While DiacNet-1.1 is a powerful tool, it has some limitations:

Yorùbá Tone Marks: The model under-marks tones in Yorùbá due to under-annotated training data. Output should not be treated as publication-ready without review.
Irreducible Errors: Stripping diacritics is lossy. For example, in Italian, both Viaggiò (3rd-person past) and Viaggio (1st-person present) reduce to Viaggio—no model (or human) can recover the original distinction.
Proper Nouns: Names and places (e.g., Àlàmú vs. Àlámù) are unreliable due to insufficient context. Foreign names may also be over-nativized (e.g., Vietnamese Tom → Tôm).
Gloss Conditioning: The model supports optional [g: word=meaning] hints, but their effect on diacbench was neutral to slightly negative. This feature is documented but not recommended for general use.
Structural Guarantees: As a seq2seq model, DiacNet-1.1 can alter base characters. However, measured compliance on diacbench is 94.7–100%, so constrained decoding is advisable for high-stakes pipelines.
Register Bias: The model was trained on web and encyclopedic text. Performance on conversational, poetic, or code-switched text is untested.
Benchmark Limitations: The diacbench dataset contains a small number of defective reference sentences (~1%), which slightly caps measured scores.

Intended Use Cases

DiacNet-1.1 is ideal for:

✅ Restoring diacritics in scraped or user-typed text.
✅ Cleaning corpora for downstream NLP tasks.
✅ Assisting typing in languages where diacritics are commonly omitted.

DiacNet-1.1 is not designed for:

❌ Text generation.
❌ Translation.
❌ Spelling correction.
❌ General-purpose language modeling.

Best Practice: Use DiacNet-1.1 with human review for Yorùbá and any application where a wrong tone mark could change meaning.

Citation

If you use DiacNet-1.1 in your research or applications, please cite it as follows:

@misc{diacnet11,
  author = {olaverse},
  title  = {diacnet-1.1: multilingual diacritic restoration},
  year   = {2026},
  url    = {https://huggingface.co/olaverse/diacnet-1.1}
}

DiacNet-1.1 is built on ByT5 (Xue et al.), FineWeb-2, and Wikipedia. Please attribute these sources per their respective licenses.

Try It Yourself

Ready to restore diacritics to your text? Head over to the DiacNet-1.1 Hugging Face page to explore the model, download it, and start using it in your projects!

Questions or feedback? Open an issue on the Hugging Face repository or reach out to the olaverse team.

July 31, 2026

Pretrained From Scratch for Naija: Introducing mist-encoder-base-ng
A 30.9M-parameter encoder built from the ground up for Hausa, Yoruba, Igbo, and Nigerian Pidgin — with a tokenizer that beats GPT-4o’s on all four.

Most “African language support” in NLP is an afterthought: a massive multilingual model where Yoruba got a sliver of the training budget and the tokenizer shreds every word into fragments. Today we’re releasing something different — mist-encoder-base-ng, a compact encoder pretrained entirely from scratch on Nigerian text, open-sourced on Hugging Face under Apache 2.0.

It covers the four major languages of Nigeria — Hausa, Yoruba, Igbo, and Nigerian Pidgin — and it’s built to be a workhorse base: load the encoder body, attach a head, and fine-tune for classification, sentiment, language ID, or sentence embeddings.
```
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("olaverse/mist-encoder-base-ng")
enc = AutoModel.from_pretrained("olaverse/mist-encoder-base-ng")
```
Why from scratch?

Fine-tuning an existing multilingual model inherits that model’s tokenizer — and for Nigerian languages, general-purpose tokenizers are quietly terrible. Every extra subword a tokenizer needs per word means less real signal fits in a fixed context window, slower inference, and weaker per-token representations.

So before training the model, we built the tokenizer: otk-bpe-50k, a unified byte-level BPE for Naija text with a ~50k vocabulary, NFC diacritic preservation, code-mixed English support, and 0% unknown tokens. The fertility numbers (tokens per word — lower is better) speak for themselves:

Tokenizer Hausa Yoruba Igbo Pidgin
otk-bpe-50k (ours) 1.231 1.296 1.416 1.249
GPT-4o (o200k) 1.589 1.687 1.807 1.304
AfroXLMR 1.604 2.277 2.570 1.401

Our tokenizer is more efficient on Nigerian text than the tokenizer behind GPT-4o — on all four languages. On Igbo, AfroXLMR needs over 2.5 tokens per word; ours needs 1.4.

The model

On top of that tokenizer sits a ModernBERT architecture — 6 layers, hidden size 384, 1024 max positions — pretrained with masked language modeling on ~480k carefully prepared chunks. The corpus combines FineWeb-2, the audited WURA corpus, and a fresh Nigerian Pidgin dataset, all cross-deduplicated (FineWeb-2 and WURA both descend from Common Crawl, so overlap was real), language-balanced so abundant Hausa didn’t drown out scarce Igbo and Pidgin, and chunked into windows so no text was wasted on truncation. Every source is attribution-only licensed, which is what makes the clean Apache 2.0 release possible.

The result is 30.9M parameters — and since the embedding table is about two-thirds of that, the transformer itself is only ~11M. For comparison: AfriBERTa is 126M, mBERT is 178M, XLM-R is 270M.

What a 30.9M specialist can do

On MasakhaNEWS topic classification (macro-F1), the model is genuinely competitive at a quarter of the baselines’ size:

Model Params Hausa Yoruba Igbo Pidgin
mist-encoder-base-ng 30.9M 0.878 0.859 0.803 0.898
AfriBERTa 126M 0.924 0.921 0.914 0.991
mBERT 178M 0.806 0.886 0.805 0.967

It beats mBERT on Hausa and ties it on Igbo — with under a fifth of the parameters. That efficiency is the point: this is a model you can fine-tune on a laptop and deploy on-device or in low-resource environments where 126M+ models are impractical.

What it can’t do (yet) — and exactly why

We publish our weaknesses with the same rigor as our wins. Named-entity recognition is this model’s honest weak spot: it trails AfriBERTa and mBERT by roughly 10–20 entity-F1 on MasakhaNER 2.0, with Hausa NER lowest at 0.656.

And we didn’t just report the gap — we diagnosed it. The deficit is structural, not a tuning artifact: it persists across random seeds and labeling schemes. Two causes: the unified 50k tokenizer fragments entity words (person and place names) more than language-specific tokenizers — on Hausa NER text, ~61% of entity words split into multiple subwords versus ~21% for AfriBERTa — and at 30.9M parameters the model has less capacity to reassemble meaning from those fragments. If NER accuracy is critical to your application, use a larger model. If sentence-level understanding at minimal cost is what you need, this is your base.

The model card also lays out an evidence-backed v2 roadmap: more capacity (~50–80M), a less-fragmenting tokenizer for token-level tasks, and more Pidgin and Hausa data. Notably, longer pretraining isn’t on the list — eval loss plateaued by epoch 11, so we know that lever is spent.

Part of a bigger picture

mist-encoder-base-ng joins the Mist Embedder collection and sits alongside our LID models (which are fine-tuned classifiers) as complementary infrastructure: a from-scratch, Nigerian-first foundation others can build on. Two fine-tunes of it already exist on the Hub, and there’s a demo Space to try it.
- Model: https://huggingface.co/olaverse/mist-encoder-base-ng
- Tokenizer: https://huggingface.co/olaverse/otk-bpe-50k
- Collection: https://huggingface.co/collections/olaverse/mist-embedder
Apache 2.0, free for commercial use. If you fine-tune it for a Nigerian-language task — sentiment, topic tagging, embeddings, anything — open a discussion on the model page and show us. That’s the whole point of releasing the base.

Olaverse Lab builds open-source AI with first-class support for underserved languages. Explore all our models at huggingface.co/olaverse.
July 19, 2026
The Small Model Behind a Better Chat List: Introducing mist-tg-0.3b
A 300M-parameter open-source model that turns a user’s first message into a clean chat title — instantly, on almost any hardware.

Open any chat app you’ve built and look at the sidebar. If it’s full of entries like “New chat”, “Untitled”, or the raw first 40 characters of whatever the user typed, you already know the problem mist-tg-0.3b solves.

Today we’re releasing mist-tg-0.3b, a tiny title-generation model, open-sourced on Hugging Face under Apache 2.0. Give it a user’s first message and it returns a short, readable title — the kind you’d actually want in a conversation list.
```
"My laptop keeps freezing every time I open more than
five browser tabs, any idea why?"
        ↓
"Laptop Freezing Impact"
```
Why a dedicated model for something so small?

Because the obvious alternatives are all worse in practice. Truncating the first message gives ugly, uninformative titles. Calling a large LLM for every new conversation adds cost and latency to the single most frequent event in a chat product. A purpose-built small model hits the sweet spot: at ~300M parameters, mist-tg-0.3b runs comfortably on CPU, adds negligible cost per conversation, and does exactly one job well.

This is the same philosophy behind the rest of the Mist family: not every problem needs a frontier model. Some need a fast, cheap, reliable specialist.

How it works

mist-tg-0.3b is fine-tuned from google/byt5-small, a byte-level seq2seq T5. It was trained for one epoch on SupraLabs/chat-titles-filtered-115K, a filtered dataset of 115K real chat messages paired with titles.

The byte-level base turns out to matter. Because ByT5 operates on raw bytes rather than a fixed subword vocabulary, there’s no tokenizer to fight and no prompt template to remember — you pass the raw message string in, and a title comes out:
```
from transformers import AutoTokenizer, T5ForConditionalGeneration

tok = AutoTokenizer.from_pretrained("olaverse/mist-tg-0.3b")
model = T5ForConditionalGeneration.from_pretrained("olaverse/mist-tg-0.3b")

message = "How do I get my sourdough starter to rise faster in a cold kitchen?"
inputs = tok(message, return_tensors="pt", truncation=True, max_length=256)
output_ids = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(output_ids[0], skip_special_tokens=True))
```
Inputs are capped at 256 bytes and outputs at 64 — which fits the task: a first message and a sidebar-length title.

An unexpected bonus: Latin-script transfer

The model was trained end-to-end on English, and English is where it’s production-ready. But byte-level modeling produced a genuinely interesting side effect: because Latin-script languages share bytes with English, same-language title extraction often just works for them — despite the model never being trained on non-English targets.

In an informal cross-language check across 25 languages (hand-written chat-style messages, qualitative judgment rather than a scored benchmark), roughly 17–19 of 25 languages produced usable same-language titles — almost all of them Latin-script: French, German, Dutch, Vietnamese, Indonesian, Yoruba, Igbo, Hausa, Swahili, Zulu, Xhosa, Shona, Somali, Afrikaans, and more.

Where it doesn’t work — and why we’re telling you

We believe model cards should say what a model can’t do as plainly as what it can. mist-tg-0.3b is not reliable for non-Latin scripts. All four non-Latin-script languages we tested — Japanese, Korean, Hindi, and Amharic — failed to produce same-language output, generating unrelated English text instead. The byte-level transfer trick depends on shared script; where the bytes diverge, so does the behavior. Fixing this properly would require training on non-English targets, which is a natural direction for a future version.

If your user base writes in CJK, Hangul, Devanagari, or Ethiopic scripts, this model isn’t the right tool yet. If your users write in English or Latin-script languages — including many African languages that rarely get this kind of support — it very likely is.

Getting started
- Model: https://huggingface.co/olaverse/mist-tg-0.3b
- Training data: https://huggingface.co/datasets/SupraLabs/chat-titles-filtered-115K
- The full Mist Generators collection: https://huggingface.co/collections/olaverse/mist-generators
It’s Apache 2.0 — free for commercial use, no strings. If you drop it into a chat product, we’d love to hear how it holds up on your real traffic. Open a discussion on the model page and tell us.

Olaverse Lab builds open-source AI with first-class support for underserved languages. Explore all our models at huggingface.co/olaverse, or read about mist-qg-1.5b, our multilingual question generator.
July 18, 2026
One Passage In, Real Questions Out: Introducing mist-qg-1.5b
A compact, open-source question generator for 25 languages — and a data factory for building better retrieval systems.

Every retrieval system, quiz app, and search engine shares a hidden hunger: it needs questions. Questions to train retrievers, questions to test rerankers, questions to turn static content into interactive learning. For English, that data exists in abundance. For most of the world’s languages, it barely exists at all.

Today we’re releasing mist-qg-1.5b, a compact multilingual question generation model, open-sourced on Hugging Face under Apache 2.0. Give it a passage in any of 25 languages and it returns natural, search-style questions that the passage directly answers — the kind of thing a real user would actually type into a search box.

25 languages, one small model

Coverage spans the languages your users actually speak:
- European: English, French, German, Spanish, Portuguese, Italian, Dutch, Russian, Polish
- Asian & Middle Eastern: Hindi, Japanese, Korean, Vietnamese, Indonesian, Turkish
- African: Yoruba, Igbo, Hausa, Swahili, Amharic, Zulu, Xhosa, Shona, Somali, Afrikaans
At ~1.5 billion parameters (fine-tuned from Qwen2.5-1.5B-Instruct), it runs comfortably on a single modest GPU — no cluster required.

Two jobs, one model

mist-qg-1.5b is deliberately dual-use:

1. A question-generation service. Wrap it behind an endpoint and you have instant question generation for education platforms, quiz builders, FAQ tooling, and content pipelines. The model returns strict JSON ({"questions": [...]}), so outputs slot straight into production systems — and for guaranteed structural validity, serve it with vLLM guided JSON decoding.

2. A data factory. This is the quieter superpower. Every question it generates, paired with its source passage, is a (query, positive) training pair — exactly what you need to fine-tune retrievers and rerankers for RAG. For languages where such data has never existed at scale, this model can mint it.

The interesting part: round-trip verification

Question generation models often produce questions that look plausible but aren’t actually answerable from the source. We attacked this at the data level.

Training data was distilled from the Aya collection using Qwen2.5-32B-Instruct — but every candidate question had to pass a test before entering the training set: embedded alongside its source passage and a pool of distractors, the question had to retrieve its own passage at rank one. Fail the round trip, and the question was discarded. Roughly 150k questions across ~50k passages survived this filter.

We evaluate the same way. On 625 held-out passages never seen during training, the model achieves an overall round-trip keep-rate of 0.955 — meaning 95.5% of generated questions successfully retrieve their source passage from a distractor pool (scored with the larger Qwen3-Embedding-4B, a stricter judge than the one used in training).

Honest numbers, including the weak spots

Six languages — English, Spanish, Portuguese, Turkish, Indonesian, and Afrikaans — scored a perfect 1.000 on the held-out eval, and most high-resource languages cluster between 0.95 and 1.00.

We want to be equally upfront about the other end: Amharic, Somali, and Shona sit at 0.58–0.70, and outputs in those three languages should be treated with lower confidence. Our analysis suggests this gap tracks fine-tuning data volume (~2,000 source passages per language) rather than a hard model-capacity ceiling, which makes it a natural target for the next data-scaling pass. We’re publishing these numbers because per-language transparency is the only way users can make informed decisions — and because we intend to close the gap.

Try it in one minute
```
pip install -U transformers
```
```
import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "olaverse/mist-qg-1.5b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

passage = "Tides are caused by the gravitational pull of the moon and, to a lesser extent, the sun, acting on Earth's oceans."

messages = [
    {"role": "system", "content": "You write search-style questions that a passage directly answers."},
    {"role": "user", "content": f'''You are given a passage. Write 3 questions that the passage directly answers.

Rules:
- Each question must be answerable using ONLY this passage.
- Vary the type: factual, yes/no, and a comparison or "why/how".
- Natural, like a real user search query. Do NOT write "according to the passage".
- Write the questions in English.

Return ONLY JSON: {{"questions": ["...", "...", "..."]}}

Passage: {passage}'''},
]

input_ids = tokenizer.apply_chat_template(messages, tokenize=True,
    add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(input_ids, max_new_tokens=200, do_sample=False,
    pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id)
text = tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True)
print(json.loads(text[text.index("{"): text.rindex("}") + 1])["questions"])
# ["What causes ocean tides?", "Does the sun affect tides?",
#  "Which has a bigger effect on tides, the moon or the sun?"]
```
Or skip the setup entirely and try the live demo Space.

Links
- Model: https://huggingface.co/olaverse/mist-qg-1.5b
- Training dataset: https://huggingface.co/datasets/olaverse/qg-passages-multi
- Demo: https://huggingface.co/spaces/olaverse/Mist_Question_Gen
- GitHub: https://github.com/Olaverse-Labs
If you use mist-qg-1.5b to build something — a quiz app, a retriever for a language nobody else supports, anything — we’d genuinely love to hear about it. Open a discussion on the model page or reach us on Twitter/X.

Olaverse Lab builds open-source AI with first-class support for underserved languages. Follow us on Hugging Face and LinkedIn.
July 16, 2026

Tokenizer	Hausa	Yoruba	Igbo	Pidgin
otk-bpe-50k (ours)	1.231	1.296	1.416	1.249
GPT-4o (o200k)	1.589	1.687	1.807	1.304
AfroXLMR	1.604	2.277	2.570	1.401

Model	Params	Hausa	Yoruba	Igbo	Pidgin
mist-encoder-base-ng	30.9M	0.878	0.859	0.803	0.898
AfriBERTa	126M	0.924	0.921	0.914	0.991
mBERT	178M	0.806	0.886	0.805	0.967

Tag: small language models

A diacritic model that cannot corrupt your text

Reframing: classify, don’t generate

The model

Splitting shape from tone

Graphemes, not codepoints

What the corpus was teaching us

The fix is not to delete the data

Detecting it without eyeballing a threshold

What falls out of the architecture

Results

Against frontier LLMs

Deployment

Reproducing it

What it still gets wrong

Three things worth taking away

Introducing DiacNet-1.1: Multilingual Diacritic Restoration for 10 Languages

Overview

Supported Languages

How It Works

Example Usage

Language Tagging

Performance

Benchmark Results

Improvements Over DiacNet-1.0

Key Observations

Comparison with General-Purpose LLMs

Diacritic Error Rate (DER)

Key Takeaways

Output Compliance

Training Details

Limitations

Intended Use Cases

Citation

Try It Yourself

Pretrained From Scratch for Naija: Introducing mist-encoder-base-ng

Why from scratch?

The model

What a 30.9M specialist can do

What it can’t do (yet) — and exactly why

Part of a bigger picture

The Small Model Behind a Better Chat List: Introducing mist-tg-0.3b

Why a dedicated model for something so small?

How it works

An unexpected bonus: Latin-script transfer

Where it doesn’t work — and why we’re telling you

Getting started

One Passage In, Real Questions Out: Introducing mist-qg-1.5b

25 languages, one small model

Two jobs, one model

The interesting part: round-trip verification

Honest numbers, including the weak spots

Try it in one minute

Links