Tag: Open Source

---

Diacritic restoration is easy to state. Given se eranko naa si gbo o, produce ṣé ẹranko náà sì gbọ́ ọ. Every character keeps its identity; some acquire marks.

The standard approach treats it as translation. Bare text in, marked text out, generated token by token. That is what diacnet did — ByT5 fine-tuned on 7 million sentences.

It works, mostly. But “mostly” hides something. On Hausa, only 94.7% of diacnet-1.1 outputs still stripped back to their input. The other 5.3% weren’t mis-accented — they were different text. Words dropped. Clauses rewritten. Punctuation invented. A model asked to add accents had quietly decided to edit.

That is not a tuning problem. A generative decoder can emit any token at any position, so nothing in the architecture prevents it. You can penalise it in training and it gets rarer, but it never becomes impossible — and “rare” is not a property you can put in a contract.

We wanted one where it is impossible.

---

Reframing: classify, don’t generate

The task has a hard structural constraint that generation throws away. The output has exactly as many characters as the input, in the same order, with the same base letters. Only the marks change.

So don’t generate. Classify. For each character, predict which diacritics it carries, and copy the base character through.

 input      s      e      r      a      n      k      o
            │      │      │      │      │      │      │
            ▼      ▼      ▼      ▼      ▼      ▼      ▼
     ╔═══════════════════════════════════════════════════╗
     ║          character transformer encoder            ║
     ╚═══════════════════════════════════════════════════╝
            │      │      │      │      │      │      │
   SHAPE   DOT     ·      ·      ·      ·      ·      ·
   TONE     ·      ·      ·    ACUTE    ·      ·      ·
            │      │      │      │      │      │      │
            ▼      ▼      ▼      ▼      ▼      ▼      ▼
 output     ṣ      e      r      á      n      k      o
            └──────┴──────┴──────┴──────┴──────┴──────┘
              base letters COPIED, never predicted

The base character is not an output of the network. There is no mechanism by which the model can change it, so strip(output) == strip(input) holds by construction — for a trained model, an untrained one, or the int8 export.

We test exactly that: the invariant suite runs against a randomly initialised model. If the guarantee only held for a good model, it would not be a guarantee.

---

The model

Twelve layers, 512 hidden, 37.6M parameters — against 580M for the seq2seq model it replaces.

      graphemes  [B,T]              language id  [B]
           │                              │
           ▼                              ▼
    ┌─────────────┐                ┌─────────────┐
    │  char emb   │                │  lang emb   │   11th slot =
    │   403×512   │                │    11×512   │   "unknown language"
    └──────┬──────┘                └──────┬──────┘
           │                              │
           └──────────────► + ◄───────────┘   additive: no <yor> token,
                            │                 no position spent
                            ▼
        ╔═══════════════════════════════════════╗
        ║          × 12  encoder block          ║
        ║  ┌─────────────────────────────────┐  ║   bidirectional —
        ║  │ RMSNorm → MHA(8 heads, RoPE)    │  ║   a diacritic depends on
        ║  │ + residual                      │  ║   what follows as much
        ║  │ RMSNorm → SwiGLU 512→1344→512   │  ║   as what precedes
        ║  │ + residual                      │  ║
        ║  └─────────────────────────────────┘  ║
        ╚═══════════════════╤═══════════════════╝
                            │  H  [B, T, 512]
        ┌──────────┬────────┴────────┬───────────┐
        ▼          ▼                 ▼           ▼
   ┌─────────┐┌─────────┐      ┌──────────┐┌──────────┐
   │  SHAPE  ││  TONE   │      │   MLM    ││   LID    │
   │ 512→15  ││ 512→8   │      │ 512→403  ││ 512→10   │
   │per char ││per char │      │ tied emb ││mean-pool │
   └────┬────┘└────┬────┘      └────┬─────┘└────┬─────┘
        │          │                │           │
        └────┬─────┘           training only  language
             ▼
      legality mask  →  argmax  →  compose(base, shape, tone)

Where the parameters go:

component	shape	params
character embedding	403 × 512	206k
language embedding	11 × 512	6k
12 × encoder block	attn 4·512² + FFN 3·512·1344	37.4M
SHAPE head	512 × 15	7.7k
TONE head	512 × 8	4.1k
LID head	512 × 10	5.1k
MLM head	tied to embedding	0

The task heads together are 12k parameters — 0.03% of the model. Nearly everything is shared encoder, which is why ten languages cost roughly the same as one.

loss = 1.0·CE(shape) + 1.0·CE(tone) + 0.3·CE(masked_char) + 0.05·CE(language)

SHAPE and TONE are the task. MLM is auxiliary and dropped at inference — it forces lexical knowledge into the encoder, which is what tone disambiguation needs. Its target is always the fully marked character; masking ú and accepting u would teach the encoder that bare vowels are valid Yorùbá, the exact failure being fixed. LID is nearly free and makes the API restore(text) instead of restore(text, lang).

Three choices that carry weight:

RoPE, not learned positions. Documents are processed with a sliding window, so the model sees offsets at inference it never saw in training. Learned absolute positions generalise badly to that; rotary ones don’t.

Language as an additive embedding. No <yor> prefix token — nothing to confuse with content, no position spent. During training the language embedding is dropped 12% of the time, giving a genuine “unknown language” mode, so being wrong about the language degrades gracefully instead of catastrophically.

Trained from scratch. We considered starting from an existing encoder, but its tokenizer was trained on diacritized text while our inputs are stripped — every subword boundary would be wrong in the direction that matters most.

---

Splitting shape from tone

The naive version assigns each character one label from a flat set: acute, grave, underdot, circumflex-plus-acute. That works for Spanish and falls apart on Vietnamese, where stacked marks produce ~60 classes, most of them rare.

Instead, factorize:

   ẹ́   →   e  +  DOT_BELOW   +  acute        (Yorùbá)
   ộ   →   o  +  CIRCUMFLEX  +  dot-below    (Vietnamese)
   ữ   →   u  +  HORN        +  tilde        (Vietnamese)
   ń   →   n  +     —        +  acute        (Polish)

SHAPE is marks that change letter identity. TONE is the mark that changes pitch or stress. Two heads, two label sets, two loss terms.

The subtlety — and the reason it works — is that which marks belong to which slot is language-dependent:

   U+0323  dot below      →  SHAPE in Yorùbá     (ẹ and e are different letters)
                          →  TONE  in Vietnamese (nặng, one of six tones)

   U+0303  tilde          →  SHAPE in Spanish    (ñ is a letter)
                          →  TONE  in Vietnamese (ngã)

A flat label set cannot express that. It sees ọ and assigns one class, blind to the fact that the same codepoint plays different grammatical roles in the two languages.

Factorized, Vietnamese collapses to 5 shapes × 6 tones, every component seen thousands of times. Across all ten languages the final space is 15 shapes and 8 tones.

The bigger payoff is diagnostic. Yorùbá’s dominant failure is tone direction — the model knows a mark belongs there and picks the wrong one, writing pátápátá for pàtàpàtà. In a flat label space that is indistinguishable from any other error. Split, it gets its own head, its own confidence and its own metric:

checkpoint	Yorùbá DER	tone portion
10k steps	0.1306	87%
30k steps	0.1138	81%
120k steps	0.0836	83%

Four fifths of the remaining error, at every stage of training, is one failure mode. Aggregate DER would have shown a number going down and told us nothing about what to do next.

---

Graphemes, not codepoints

Yorùbá stacks marks: ọ̀ is o with an underdot and a grave. Unicode has no precomposed codepoint for that combination, so even after NFC normalisation it stays two codepoints. len("ọ̀rọ̀") is 5, not 3.

Iterating over the string misaligns every label after the first stacked character. Everything here operates on graphemes — a base codepoint plus any combining marks that follow it. A small detail that silently determines whether the hardest language in the set trains at all.

---

What the corpus was teaching us

This is the finding that changed the result more than any architectural decision.

Before training, we plotted diacritic density — the fraction of eligible characters actually carrying a mark — per language. Nine of ten gave a single hump. Yorùbá gave two:

  count
    │        ▁▄██▇▅▃▂▁
    │       ▂████████▃                       ▁▂▃▃▂▁
    │      ▃██████████▅▂                   ▁▃██████▃▁
    └──────┴───────────┴──────────────────┴──────────┴────► density
          0.10                                0.43
        89% of sentences                 11% of sentences
        UNDER-MARKED                        correct

Correctly marked Yorùbá lands around 0.30–0.45 — nearly half its characters are vowels, most carry a high or low tone, plus underdots on ẹ, ọ and ṣ.

Which means the large mode is the broken one. Eighty-nine per cent of our Yorùbá training data was under-marked: real words with their tone marks simply omitted. And we were using it as ground truth. Every one of those sentences was teaching the model that bare vowels are correct.

That is diacnet-1.1‘s documented failure, sitting in a histogram. It also explains something we had noticed and shrugged at: diacnet-1.1 is worse than 1.0 on all three African languages. More data, trained longer, learned to omit tone more confidently.

The fix is not to delete the data

The obvious move is to drop the low-density mode. We didn’t, because under-marked Yorùbá is still Yorùbá — the words are real, the morphology is real. It is excellent data for learning the language. It is poisonous data for learning tone marks.

Since we already had two objectives on one encoder, we could separate them. Sentences below the detected threshold keep their masked-character targets and contribute no diacritic supervision at all:

   density 0.458   ABOVE floor  →  23 diacritic labels,  8 MLM labels
   density 0.000   BELOW floor  →   0 diacritic labels,  8 MLM labels
                                    └── still teaches vocabulary

The MLM head trains on all 1.9M Yorùbá sentences. The tagger heads train on the clean ~205k. Same corpus, two different notions of what counts as a label.

Detecting it without eyeballing a threshold

We wanted this automatic. The first attempt used Otsu’s method, the standard image-binarisation algorithm — and it reported every language as bimodal.

Otsu returns a split for any distribution. On a single right-skewed hump — which is what density distributions look like — that split lands near the left edge, where “the peak below the cut” is two nearly-empty bins and every statistic derived from it is noise. Our test fixtures used symmetric Gaussians, which hid the failure completely.

The fix was to find peaks directly and measure how deep the trough between them falls, requiring both sides to hold at least 5% of the mass so a tail cannot masquerade as a mode. Measured that way the answer is not close:

   yor   ████████████████████████  0.55   BIMODAL
   pol   ████                      0.10
   tur   ███                       0.07
   ita   ▊                         0.02
   ibo                             0.00

Igbo and Hausa are correctly left alone, and that matters: standard Igbo orthography omits tone by convention, and Hausa rarely writes it at all. Their low density is a property of the writing system, not a defect. No threshold should touch them.

---

What falls out of the architecture

Per-character classification gives you several things that are awkward or impossible in a generative setup.

Automatic language detection. A fourth head classifies the language from the mean-pooled encoder state. It costs almost nothing — DER 0.0132 with the language supplied, 0.0133 with it detected — and most integrations don’t reliably know the input language.

Per-character confidence. The joint shape × tone distribution gives a calibrated probability per character (temperature-fitted on validation, T = 1.14). Threshold it, and characters below are left exactly as the user typed them:

threshold	coverage	DER on committed
0.00	100.0%	0.0132
0.90	97.1%	0.0039
0.99	91.9%	0.0008

At 0.90: 97% of characters restored at 99.6% accuracy, the rest flagged. This is the number to put in front of a customer, because it answers the question they have. A wrong tone mark changes meaning; a missing one is merely incomplete. The threshold is per-request, so a CMS pre-fill and a legal pipeline sit at different points on the same curve.

Legality masking. For each (language, base character) we record which (shape, tone) pairs were ever observed and mask the rest to −∞ at decode time. No acute on Turkish vowels, no Vietnamese horn on Polish. An interesting result: turning it off changes DER by zero to four decimal places. The model learned the orthographies well enough that the constraint never binds. We keep it as a guarantee for out-of-distribution input — it costs nothing.

Protected spans. URLs, emails, @handles, inline code and CONSTANT_NAMES pass through untouched, and were excluded from the training loss. An accent inside a URL is never correct, and it is the error a reader notices first.

Documents, not sentences. No sentence splitting. Overlapping windows are planned across the input and only the centre of each is trusted, so every character is predicted with context on both sides. The window plan is proved to partition the input exactly — 500 randomised cases in CI.

---

Results

diacbench, 1000 sentences per language.

lang	DER	shape	tone	exact	compliance
ita	0.0002	0.0000	0.0002	0.991	1.0000
fra	0.0012	0.0003	0.0009	0.952	1.0000
tur	0.0016	0.0016	0.0000	0.961	1.0000
por	0.0019	0.0006	0.0014	0.925	1.0000
pol	0.0022	0.0018	0.0003	0.935	1.0000
spa	0.0022	0.0001	0.0021	0.917	1.0000
hau	0.0041	0.0040	0.0001	0.741	1.0000
ibo	0.0122	0.0110	0.0013	0.483	1.0000
vie	0.0164	0.0073	0.0124	0.650	1.0000
yor	0.0836	0.0203	0.0695	0.084	1.0000
all	0.0132	0.0055	0.0086	0.764	1.0000

Yorùbá 0.2006 → 0.0836, a 58% reduction. Hausa 0.0593 → 0.0041, 93% — though much of the Hausa gain is the compliance guarantee rather than better modelling, since a large share of the old error was corruption rather than wrong accents.

37.6M parameters against 580M.

Against frontier LLMs

300 sentences per language, matching how the baselines were run.

lang	diactag-1.0	Claude Sonnet 4.5	GPT-4o-mini
yor	0.0933	0.1913	0.2811
ibo	0.0110	0.0427	0.1277
hau	0.0041	0.0178	0.1432
vie	0.0166	0.0107	0.0399
fra	0.0014	0.0052	0.0023

Best on 7 of 10. Vietnamese and Portuguese genuinely lose.

Those LLM numbers deserve a footnote, because they are the charitable ones. They come from a harness that checks whether the output still strips back to the input and throws it away when it doesn’t. Raw, against the floor of copying the input unchanged:

lang	copy-input floor	Claude, raw
hau	0.0236	0.3509
ibo	0.1171	0.2261
fra	0.0471	0.1171

On three languages, a frontier model makes the text worse than doing nothing at all. Hausa is already 97.6% correct if you leave it alone; Claude returns it 35% wrong.

That fallback harness is our architecture, reimplemented externally and imperfectly. The difference is that here there is nothing to discard.

---

Deployment

backend	chars/s	p50	size	DER
PyTorch CPU	105	591 ms	150 MB	0.0105
ONNX fp32	202	286 ms	150.9 MB	0.0105
ONNX int8	244	200 ms	38.3 MB	0.0108

Three times faster and four times smaller for +0.03pp DER. Compliance stays 1.0000 under quantisation — the guarantee is architectural, not a property of a particular numeric precision.

A 38MB artifact at 244 characters per second on one CPU core takes the GPU off the serving bill entirely.

One honest caveat: p95 is 788–1099 ms across all backends while p50 varies 3×. The tail is length-driven — long sentences need more windows — not backend-driven. Quantisation doesn’t fix it.

---

Reproducing it

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

Data. 7.0M sentences from a web + Wikipedia corpus, plus 131k from a Q&A passage set — the only non-encyclopedic register in the mix — gated on diacritic density and deduplicated against the eval splits by stripped-form hash.

Two filters worth copying if you build something similar. Script filtering: all ten languages are Latin, but Hausa Wikipedia carries Ajami and Yorùbá articles carry Korean and Devanagari names; those contribute nothing but identity labels. Cross-language filtering: a Vietnamese name inside a Turkish sentence factorizes into composite classes that exist in no language, because Turkish declares no tone slot for those marks to occupy. Without both, the label space inflates from 15 shape classes to 99, most of them junk competing in the same softmax.

Training. AdamW, lr 3e-4 with 2000 warmup steps and cosine decay to 5%, weight_decay 0.05, batch 128 × 2 accumulation = 256, seq len 256, bf16, 120k steps. About 13 hours on a single A100.

Language balancing. Temperature sampling at α=0.5. At α=1 Yorùbá and Igbo would be 54% of the mix; at α=0 Hausa’s 400k sentences get seen 5× per Yorùbá epoch and overfit.

Augmentation. partial_mark_prob=0.30 with keep range [0.05, 0.9], case perturbation (2% upper, 3% lower, 1% title), 3% protected-span injection, mlm_prob=0.15, lang_dropout=0.12.

Evaluation. Report shape_DER and tone_DER separately, and compliance as a hard assertion rather than a metric. If compliance is not exactly 1.0, the architecture is broken and nothing else in the table means anything.

---

What it still gets wrong

Yorùbá. DER 0.0836, 83% of it tone direction. Sentence-level exact match is 0.084 — 92 of every 100 Yorùbá sentences contain at least one wrong mark. A harsher framing than DER, and a truer one.

Igbo and Hausa tone numbers are not achievements. tone_DER of 0.0013 and 0.0001 looks superb and means little, because those orthographies barely write tone. Nothing to learn, nothing to get wrong.

No typo correction. The architecture cannot insert or delete characters, so it cannot fix Ile → Ilé and teh → the in one pass. That is the price of the guarantee, and we think it is the right trade — a model that reliably does one thing beats one that sometimes does two. The per-character confidence is a natural trigger for a separate corrector.

Dense input degrades. The Polish pangram Zażółć gęślą jaźń has density 0.565 against a Polish median of 0.068 — nine times denser than real Polish — and the model misses six characters, despite a Polish DER of 0.0022.

Some errors are irreducible. Viaggio and Viaggiò are both valid Italian and the stripped form contains no information distinguishing them. Every language has a nonzero floor.

---

Three things worth taking away

Plot your data before you train. The bimodal Yorùbá distribution took ten minutes to find and changed the outcome more than any modelling decision. We nearly skipped it.

Make your metric decompose along your failure modes. Aggregate DER would have gone down and taught us nothing. Splitting shape from tone told us, at every checkpoint, that four fifths of the problem was one thing — and that the other nine languages had already converged.

Put your guarantees in the architecture, not the loss function. A constraint the model can violate is one you have to monitor, apologise for, and wrap in a fallback. A constraint it cannot violate is one you can sell.

---

Weights, ONNX exports and label space: olaverse/diactag-1.0. Documentation: olaverse-labs.github.io/olaverse/models/diactag. Apache 2.0.