Lingo.dev Compiler vs Hedgeling — comparison
Hedgeling 1.7.2, each project with its tuned configuration, against the Lingo.dev Compiler 0.4.14, which has no extraction settings to tune. Both tools ran on the same pinned project commits.
Recall on the same strip test
2,181 messages in the Keycloak Admin Console were unwrapped from their t() calls back into hardcoded English, then both tools were run on the result. A message counts as found when it appears in what the tool shows the developer.
| tool and configuration | messages found | messages missed | recall |
|---|---|---|---|
| Lingo.dev Compiler | 1,086 | 1,095 | 49.79% |
| Hedgeling, out of the box | 2,157 | 24 | 98.90% |
| Hedgeling, tuned | 2,162 | 19 | 99.13% |
The Lingo.dev Compiler finds about half of these messages and misses 1,095; Hedgeling with no
configuration at all misses 24. The compiler has a single row because it has no setting that
changes what it extracts. Before it ran, the unwrapped text was rewritten into the form a developer would type by hand
(label="Save" and <b>Save</b> instead of label={"Save"} and
<b>{"Save"}</b>), because the compiler does not read the second form. Without that step it finds
1 of the 2,181.
Where the missed messages sit
| the message is | for example | messages missed |
|---|---|---|
| an attribute outside its list of nine | helpText, labelIcon, tooltip and other component props | 669 |
| a message passed to a notification helper | success and error alerts | 152 |
| a property of an object | select options, table columns, form definitions | 135 |
| another position in the code | variables, return values, default parameters | 77 |
| text chosen by an expression | one label or another depending on state | 51 |
| an argument of another function call | 9 | |
| a plain .ts file | 2 |
The compiler translates text written between JSX tags and the values of nine attributes: title, aria-label, aria-description, alt, label, description, placeholder, content and subtitle. About six in ten of its misses are text in some other attribute, and most of the rest are strings that are not markup at all: messages handed to a notification helper, option lists, validation text. Hedgeling reads all of these.
Real copy one tool finds that the other also finds
String counts are not comparable one to one: the compiler joins an element's text and its inline markup into a single message, Hedgeling keeps the pieces. So this table starts from copy instead: the strings in each tool's sample that were labeled real user-facing copy, looked up in the other tool's full output.
| project | fw | what the Lingo.dev Compiler reads | Hedgeling copy the compiler also finds | compiler copy Hedgeling also finds |
|---|---|---|---|---|
| keycloak-admin-ui | react | 542 .jsx and .tsx files | 24/70 | 13/14 |
| directus | vue | nothing: no .jsx or .tsx files | 0/83 | — |
| bulletproof-react | react | 56 .jsx and .tsx files | 79/100 | 64/67 |
| analog | angular | 4 .jsx and .tsx files | 9/82 | 9/9 |
| pocketbase-admin | svelte | nothing: no .jsx or .tsx files | 0/96 | — |
| hf-chat-ui | svelte | nothing: no .jsx or .tsx files | 0/100 | — |
| appwrite-console | svelte | nothing: no .jsx or .tsx files | 0/98 | — |
| supabase-studio | react | 2,074 .jsx and .tsx files | 52/96 | 89/97 |
| all projects | 23% | 94% | ||
| React projects only | 58% | 93% | ||
On the three React applications the compiler finds 155 of the 266 strings in Hedgeling's samples that are real copy. Hedgeling finds 166 of the 178 in the compiler's samples. The compiler is built for React, so it reads nothing in the Vue and Svelte applications and only the four React files of the Angular one.
Brownfield
Already internationalized. Precision is a residue ratio; the TP count is the headline metric (real i18n gaps found).
| project | fw | Hedgeling strings | Compiler strings | Hedgeling precision | Hedgeling TP | Compiler precision | Compiler TP |
|---|---|---|---|---|---|---|---|
| keycloak-admin-ui | react | 122 | 131 | 70% | 70/100 | 14% | 14/100 |
| directus | vue | 119 | 0 not read | 83% | 83/100 | — | — |
| all brownfield projects | 77% | 153/200 | 14% | 14/100 | |||
Greenfield
No existing i18n pass. Precision is the headline accuracy number.
| project | fw | Hedgeling strings | Compiler strings | Hedgeling precision | Hedgeling TP | Compiler precision | Compiler TP |
|---|---|---|---|---|---|---|---|
| bulletproof-react | react | 103 | 67 | 100% | 100/100 | 100% | 67/67 |
| analog | angular | 116 | 19 | 82% | 82/100 | 47% | 9/19 |
| pocketbase-admin | svelte | 992 | 0 not read | 96% | 96/100 | — | — |
| hf-chat-ui | svelte | 528 | 0 not read | 100% | 100/100 | — | — |
| appwrite-console | svelte | 4,129 | 0 not read | 98% | 98/100 | — | — |
| supabase-studio | react | 14,083 | 5,834 | 96% | 96/100 | 97% | 97/100 |
| all greenfield projects | 95% | 572/600 | 93% | 173/186 | |||
label attributes, and the compiler takes those keys for text, which is why its precision there is low.
The React files of Analog are a documentation site that already wraps its text in a translation component, with the
same effect.
Hedgeling counts come from each project's tuned configuration. Lingo.dev Compiler counts are the texts it would
translate during a build, read from the compiler itself without running a translation. Tests, stories and mocks were
left out for both tools. Precision for both tools comes from samples of up to 100 strings labeled with the same
rubric; a project where the compiler reads no file has no sample.