Six ways to spell Njinga

The victory lap

Last week I wrote about discovering that kimbundu.org's "missing" vocabulary was never missing. The search was burying it. I fixed the ranking, verified the fixes in production, and replied to a two-year-old review that had gone unanswered.

I recorded the result in the audit file, in a comment at the top, so future-me would know where things stood:

Validated against every EXISTS row in this file: 72% top-1, 89% top-3, 100% top-10, 0 misses.

Every number in that sentence is real. I measured them. They are also close to meaningless, and it took me until this week to notice why.

What the number was measuring

The audit file has 116 concepts confirmed to exist in the corpus, each with a Portuguese label, an English label, and the entry IDs that should come back. That is a test set: 232 queries, if you use both languages.

I had not used both languages. I had hand-picked about twenty Portuguese queries, scored them at top-3, and written the result down as though it covered the file. "100% top-10, 0 misses" described a set I chose, containing the cases I had just fixed.

So I wrote a script that runs the search code, bundled from the module, not a reimplementation that can drift, against all 232 queries, and reports top-1, top-5, top-10 and MRR (Mean Reciprocal Rank; 1.0 is perfect).

Not a regression. The old check still passed. It had simply never been asked the harder question.

The split underneath was the useful part. Portuguese scored 57.8% top-1; English scored 47.4%. A ten-point gap in a dictionary whose entire English layer I had generated and shipped with equal confidence.

Two bugs behind the gap

The first was that scoring cannot express "this entry is the answer." Muxima's definition literally reads Heart; innermost being; conscience. The query heart returned it at position twelve. Muxima has five senses, its aggregate text field is long, and BM25 length normalisation handed the top spots to single-sense entries that mention hearts in passing. Portuguese coração worked only because fewer entries competed for it.

So exact whole-segment equality now bypasses scoring entirely and pins to the top. If a gloss segment equals your query, no amount of term statistics should outrank it. That one change moved top-1 from 52.6% to 66.4%.

The second was duller. The curated alias file (the hand-written list that anchors vento to Kitembu, whose 1940s definition says Ventania and never vento) was entirely Portuguese. Sixteen entries, twenty-one terms, not one in English. vento found Kitembu at #1; wind missed completely. I mirrored every alias into English and the language gap inverted: English top-1 went from 47.4% to 69.8%, now ahead of Portuguese.

Overall: 52.6% → 67.7% top-1, 84.5% → 91.8% top-5.

Then I stopped tuning and went to look at what people had typed.

The miss log is 12% of the story

The site logs a search_no_results event when a search returns nothing. I had been treating that as the failure log. Over ninety days it holds 327 events.

In the same window, search_performed (searches that returned something) holds 2,472.

A search that returns the wrong answer is still a failure. It just returns the wrong answer confidently, and my failure log could not see it by construction. I had been reading twelve percent of the problem and calling it the problem.

My first attempt at the other 88% was to measure click-through: did the user open an entry after searching? The answer came back 10.1%, an "abandonment rate" of nearly ninety percent, which is the kind of number that should make you check your instrument before you write it down.

The search results page renders the full dictionary entry inline. The definition is already on screen. Clicking into the word page is unnecessary, so not clicking is what success looks like. Ranked by that metric, the worst-performing queries were agua, mar and coração, all of which return the correct entry at position one.

What users do when search fails

They search again.

If someone searches, gets results, and immediately searches something nearly identical, the first search failed them, and the term they settle on tells you what they meant. That signal needs no interpretation. It is the user labelling the failure themselves.

Twenty-three percent of all searches sit inside one of these retry chains. 649 of 2,799.

"jinga" → "jninga" → "njinga" → "Jinga" → "njinga" → "jninga" → "jinga"

Somebody spent ninety seconds trying to find Queen Njinga, cycling through six spellings of her name, every one of which returned results, none of which were her. Not one of those searches appeared in my failure log.

"quitanda" → "kitanda" → "quitanda" → "Quitanda"
"Ucamba"  → "Kukamba" → "UJAMBA"   → "UKAMBA"
"Elevo"   → "Elebo"   → "Elevo"    → "Elebo"

Most of these are not typos. They are the same word spelled the way different people write Kimbundu. The language was standardised in 1987, recorded in this corpus in the 1940s, and typed today by people using Portuguese orthographic habits.

Two bugs, both documented correctly and implemented wrong

The search already folded spelling variants. It handled qu by rewriting it to ku. The comment above the rule gave an example: "oquivu" → "okivu".

Read those two carefully. The comment drops the u. The code does not. In Portuguese the u after q is silent before i and e (quilo, queijo), so quitanda was being rewritten to "kuitanda", which matches nothing. Searching kitanda returned Kitánda at #1. Searching quitanda did not return it at all. The comment had been right for however long the code had been wrong.

Fixing that wasn't enough, because of a second one. The merge step that combines results across spelling variants was documented as "keeping the best score from any variant match." It kept the first. A weak fuzzy match from your original spelling permanently outranked the strong exact match from the corrected one.

I have written both of those comments-that-disagree-with-their-code before. They are what happens when you fix the description and not the thing, or write the intention and then implement something adjacent to it.

Measuring the fix without marking my own homework

The obvious move was to test the spelling fixes against the audit file. But I had tuned against the audit file. A number produced that way tells you how well you fitted, not how well you work.

So the test set came from the retry chains: pairs of spellings that users typed back-to-back, mined from ninety days of sessions. Nobody tuned against it. It was collected before the fixes existed.

The metric needs no answer key, which is the point: I have no idea what kitembulangi means, but I know it must not return something different from kitembulangy. Two spellings of one word should produce one result.

My first extraction gave 145 pairs and a convergence rate of 25.2%. It was also contaminated. Keying on edit distance had swept in amar/amor, esta/estar, obrigada/obrigado, Portuguese inflections — different words, which should return different things. Restricting inclusion to differences explainable by a known orthographic alternation left 38 pairs.

The nasal-prefix step added coverage for jinga/njinga while pushing the denominator up by one, which nudged the rate down slightly. The pair it fixed was the one that started this investigation.

Each rule came from reading what was still broken, not from what I imagined Kimbundu spelling did. The final-vowel rule accounted for five of eleven remaining failures on its own.

Through every one of those changes, the reverse-lookup score stayed at 67.7% / 91.8% / 0.787, unchanged to three decimal places. That is the result I wanted. These fixes target how people spell Kimbundu words; the gold set measures finding Kimbundu words from Portuguese concepts. Movement would have meant collateral damage.

jinga and njinga now return identical results.

The half I can't fix

Forty-seven percent of failed searches are multi-word, and many are whole sentences. Vai procurar a tua mãe. Salve a força das ervas, salve a força dos ancestrais.

People are trying to translate. The site cannot translate. It is a word dictionary built from a 1940s source, and no amount of ranking work changes that.

What it was doing was worse than failing: a fallback pass matched individual words from the sentence and presented them as an ordinary result list. You asked for a sentence and got a page of unrelated words with no indication anything had gone wrong.

Now, when nothing matches the phrase as a whole, it says so (that this is a word dictionary and not a translator) and offers to look up each word separately. The partial matches still appear, correctly labelled. That is not a fix. It is an honest failure. That is the most a dictionary can offer someone who wants a translator.

Two things I broke while looking

Every local development session had been writing to the production analytics project. The events proxy passes straight through, and initialisation was gated only on the key existing. Ninety days of data contained six of my own events, negligible, and pure luck that I test search so rarely.

And every search was logging itself two to eight times, because the capture effect depended on the analytics client reference, which changes as the SDK loads. Every count I quoted from that data was inflated. The factor varied from two to eight times depending on session and SDK state, so the raw totals (327, 2,472, 649) are approximate. The proportions should be close.

What I'd take from this

A metric you tuned against cannot estimate how well you generalize. It tells you how well you fitted. It can catch regressions, but it cannot tell you how users experience the system.

Check the instrument when the number is extreme. Ninety percent abandonment was not a finding about my users. It was a finding about my metric.

Your failure log is the failures you thought to log. Zero results was a failure I had imagined. Six spellings of a queen's name was not, and it was twice as common.

Users will tell you what's broken, in a channel you're not reading. Last time it was a Reddit thread that sat unanswered for two years. This time it was somebody typing njinga six ways in ninety seconds. Both were there the whole time.

Previously: The Missing Words That Weren't, on discovering the words were never missing.