Canonical: https://thehedgeling.com/benchmarks/RESULTS.html
Language: en-US

# Hedgeling benchmark results

`@hedgeling/i18n` 1.3.1

Eight real open-source applications, each extracted with its own tuned
configuration: a zero-config start, then `hedgeling tune` (its deterministic
tier and its LLM tier), then a hand-tuned pass driven by the diagnostics
report. Every figure below comes from that final configuration. Precision is
measured on a 100-string sample per project; recall is measured by a strip
test on a fully internationalized app.

## Headline

| Measure | Result |
|---|---|
| Recall, strip test (Keycloak, 2,181 inlined `t()` messages) | **99.13%** (2,162 recovered) |
| Precision, apps with hardcoded English UI (6 apps) | **82% to 100%** |
| Strings found for migration (6 apps) | **19,952** |
| Developer time saved finding those strings by hand | **about 1,000 hours** (3 minutes per string) |
| Real i18n gaps found in already-localized apps (Keycloak, Directus) | **about 180** of 241 strings surfaced |
| Code covered | 8,403 files, 1,124,670 lines, extracted in 11.2 s |

## Apps that start with hardcoded English (migration)

| Project | Framework | Files | LOC | Strings | Precision |
|---|---|---:|---:|---:|---:|
| Bulletproof React | React | 128 | 6,960 | 103 | 100% |
| Appwrite Console | Svelte | 1,445 | 154,639 | 4,129 | 98% |
| HF Chat UI | Svelte | 360 | 39,934 | 528 | 100% |
| PocketBase Admin | vanilla JS | 185 | 34,650 | 992 | 96% |
| Analog | Angular | 219 | 10,080 | 116 | 82% |
| Supabase Studio | React | 4,142 | 578,458 | 14,084 | 97% |

Supabase Studio is the largest app here and was also used to test the
extractor before this release, so treat its row as a scale result, not an
independent one.

Precision is the share of surfaced strings that are real user-facing copy.
Everything else was a sampled miss: a selector, a code sample, a mock name,
a template.

## Apps that already ship i18n (audit)

| Project | Framework | Files | LOC | Strings surfaced | Real gaps |
|---|---|---:|---:|---:|---:|
| Keycloak Admin Console | React | 618 | 95,840 | 122 | 70% |
| Directus | Vue | 1,306 | 204,109 | 119 | 83% |

These apps already externalize their copy, so what Hedgeling surfaces is the
residue their own i18n pass missed. "Real gaps" is the share of surfaced
strings that turned out to be genuinely untranslated UI copy rather than
noise.

## Recall

2,181 plain-text react-i18next messages in the
Keycloak Admin Console were inlined back into the source as hardcoded English
and the tuned configuration was run against the result. It recovered
2,162 of them as catalog strings or diagnostics
(**99.13%**).

## Against i18next-cli

Same eight repositories, same pinned commits, i18next-cli 1.87.0. Its side was
tuned as far as its documented options allow: per-project file ignores, a
custom scorer that skips noise and raises the confidence of everything else,
every lint tag accepted, and the community Vue and Svelte lint plugins where
its own docs point to them (there is no Angular plugin). Its results were
labeled on 100-string samples with the same rubric as ours.

| Measure | Hedgeling, tuned config | i18next-cli, tuned config |
|---|---:|---:|
| Recall, same Keycloak strip test (2,181 inlined messages) | **99.13%** (19 missed) | 91.61% (183 missed) |
| Precision, migration apps (6) | **96%** | 84% |
| Precision, audit apps (2) | **77%** | 53% |

Recall is measured the same way for both: a message counts as found when it
appears in what the tool shows the developer (our catalog and diagnostics; its
instrument candidates and lint findings).

Out of the box, i18next-cli finds 72.81% of the same messages, missing about
one in four; Hedgeling with no configuration at all finds 98.90%
(24 missed). Tuned as far as its options allow it still misses
183, against Hedgeling's 19. About seven in ten
of those remaining misses are the same pattern: a success or error message
passed straight to a notification helper, which i18next-cli does not treat as
user-facing copy, so toast and alert text is invisible to it.

The [per-project comparison](i18next-cli/RESULTS.html) has every row.

## Questions

Want the per-project samples and configurations, or a run against your own
codebase? Email [support@thehedgeling.com](mailto:support@thehedgeling.com).