Repository navigation
fix(i18n): guard messages.pot against source drift and regenerate it - #44467
Merged
Merged
Conversation
superset/translations/messages.pot is the extraction template every language catalog is generated from, but nothing in CI ever compared the committed template against a fresh extraction — the babel-extract job regenerates messages.pot from scratch in throwaway worktrees before comparing translated/fuzzy counts (.github/workflows/superset-translations.yml), a check scoped to catching a reworded source string invalidating an existing translation, not to template/source parity. The template had drifted 432 strings behind source and carried 109 stale entries, so those 432 strings could not reach any of the 30 .po catalogs and were untranslatable in every language. Add scripts/translations/check_pot_drift.py, which re-extracts from source with the same pybabel invocation as babel_update.sh and fails when the committed template's msgid set differs from a fresh extraction (ignoring whitespace-only wrapping differences). Wire it into the babel-extract CI job as a hard gate. Regenerate messages.pot and propagate to all 30 .po catalogs via babel_update.sh's extract/apply-do-not-translate/update sequence so the new guard starts green; adding the guard alone, without regenerating, would immediately fail CI against the already-stale template. Fixes #44403 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e drift guard
Two follow-ups to the messages.pot drift guard.
1. `useExploreAdditionalActionsMenu/index.tsx` was almost entirely invisible
to `pybabel extract`: its `return /[",\r\n]/.test(s) ? ... : s;` puts a
regex literal directly after `return`, and babel's JavaScript lexer reads
that `/` as division, so the `"` inside the character class opens a phantom
string literal that swallows the next ~290 lines of the file. Only 3 of the
module's 34 translatable strings survived extraction. The committed
template still carried the missing 31 only because it predated the file's
current shape — so regenerating it (as the guard requires) deleted 10
still-live, already-translated strings from all 30 catalogs, among them
"Export All Data", "Export Current View" and "New chart". Hoist the regex
into a module constant so the lexer never sees it in `return` position, and
regenerate from the base catalogs so those translations are preserved
rather than dropped. No behavior change: the hoisted literal is identical.
2. `check_pot_drift.py` collapsed whitespace in every msgid before comparing.
That was unnecessary — gettext continuation lines concatenate back to the
exact original, so `read_po` already yields identical msgids for a wrapped
and an unwrapped copy of the same string — and harmful: it silently
accepted a whitespace-only source reword ("Save chart" -> "Save chart"),
which gettext's exact-match lookup treats as a completely different message
and which therefore leaves the template and every catalog stale. Drop the
normalization and cover the case with a test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sadpandajoe
requested review from
betodealmeida,
dpgaspar,
eschutho,
geido,
hainenber,
kgabryje,
mistercrunch,
nytai,
rusackas,
sha174n and
villebro
as code owners
September 21, 2026 01:50
Contributor
|
Bito Automatic Review Skipped - Large PR |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #44467 +/- ##
==========================================
- Coverage 80.65% 80.65% -0.01%
==========================================
Files 2942 2942
Lines 175691 175692 +1
Branches 40788 40788
==========================================
Hits 141703 141703
- Misses 31326 31327 +1
Partials 2662 2662
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The prior regeneration (549dc5b) ran pybabel extract under Python 3.12 and Babel 2.18.0 instead of this project's pinned Python 3.11 / Babel 2.17.0. Python 3.12's PEP 701 f-string tokenization changes how Babel's extractor reads the nested-quote f-string at superset/common/query_context_processor.py:888, producing a spurious msgid "valuename" that does not correspond to any real translatable string. CI runs on Python 3.11 and correctly does not extract it, so the new drift guard (check_pot_drift.py) flagged the committed template as stale relative to a genuine extraction. Regenerate messages.pot and all 30 .po catalogs under Python 3.11 with Babel 2.17.0 pinned, matching CI and the committed Generated-By header. The diff is exactly the removal of the one spurious entry from every catalog; no other msgid changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rusackas
approved these changes
Sep 21, 2026
rusackas
left a comment
Member
There was a problem hiding this comment.
LGTM! Nicely researched fix, and I appreciate the real extraction test on top of the mocked edge cases.
9 tasks
2 of 9 tasks
This was referenced Sep 22, 2026
9 tasks
9 tasks
3 tasks done
9 tasks
6 tasks
9 tasks
3 tasks done
rusackas
added a commit
that referenced
this pull request
Oct 1, 2026
babel-extract runs check_pot_drift.py, added in #44467 to fail a PR that adds or renames a translatable string without regenerating messages.pot. It is not currently in .asf.yaml's required contexts, so it can be red (or, on a PR whose branch predates the script's own existence, error outright) without blocking merge. That gap let #44465 land 18 minutes after #44467 regenerated the template, reintroducing drift immediately: #44465's branch was cut before #44467 merged, so check_pot_drift.py did not exist on it yet, and nothing required babel-extract to be green (or even present) on the up-to-date branch before allowing the merge. #44509 cleaned up the resulting drift. babel-extract is also the only translation-drift check gated on both Python and frontend changes (unit-tests-required, which independently covers check_pot_drift.py via check_pot_drift_test.py, only runs when a PR touches Python), so it is the one that has to be required for a frontend-only string addition to be caught at all. This does not by itself close the race that let #44465 through -- that needs required_status_checks.strict: true, a separate, repo-wide decision left for a follow-up PR stacked on this one. This PR only makes the existing check_pot_drift.py gate binding on the PR's own branch. Co-Authored-By: Evan Rusackas <evan@preset.io> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
superset/translations/messages.potis the extraction template every languagecatalog is generated from, but nothing in CI ever compared the committed
template against a fresh extraction. The
babel-extractjob regeneratesmessages.potfrom scratch in throwaway worktrees before comparingtranslated/fuzzy counts — a check scoped to catching a reworded source string
invalidating an existing translation, not template/source parity. The template
had drifted 432 strings behind source and carried 109 stale entries, so those
432 strings could not reach any of the 30
.pocatalogs and were untranslatablein every language.
This PR does three things.
1. Adds a drift guard.
scripts/translations/check_pot_drift.pyre-extractsfrom source with the same
pybabel extractinvocation asbabel_update.shandfails when the committed template's msgid set differs from a fresh extraction.
It is wired into the
babel-extractjob as a hard gate, placed beforebabel_update.shoverwrites the workspace template. Comparison is on msgid setsonly, so it is insensitive to
POT-Creation-Datechurn and to thedo-not-translate /
python-formatpost-processing steps. Line wrapping isdeliberately not normalized away: gettext continuation lines concatenate back
to the exact original, so any whitespace difference that survives parsing is a
real source reword that gettext's exact-match lookup would miss.
2. Unbreaks extraction of the explore additional-actions menu.
useExploreAdditionalActionsMenu/index.tsxwas almost entirely invisible topybabel extract:return /[",\r\n]/.test(s) ? ... : s;puts a regex literaldirectly after
return, and babel's JavaScript lexer reads that/asdivision, so the
"inside the character class opens a phantom string literalthat swallows the next ~290 lines. Only 3 of the module's 34 translatable
strings survived extraction. The stale template still carried the other 31 only
because it predated the file's current shape — so regenerating it would have
deleted 10 still-live, already-translated strings ("Export All Data", "Export
Current View", "New chart", "Superset Chart", …) from all 30 catalogs. Hoisting
the regex into a module constant fixes extraction with no behavior change.
3. Regenerates the template and all 30 catalogs via
babel_update.sh's ownsequence, so the new guard starts green. Adding the guard without regenerating
would fail CI immediately against the already-stale template.
Regeneration changes zero existing translation values (verified by parsing
both revisions with
babel.messages.pofile.read_pofor all 30 catalogs, andindependently re-verified in review): the diff is additions plus the removal of
100 msgids that no longer exist in source, none of which still appear inside a
t()/_()/__()/tct()call. Theit/messages.pocatalog's diff looks largebecause
babel_update.shalso normalizes it from CRLF to LF line endings(master's copy was the only CRLF-encoded catalog) — that's a whitespace-only
normalization, not a retranslation; the zero-changed-values check above covers
it.
Note:
babel_update.sh'smsgcat --sort-by-msgid --no-wrapstep is a no-op inCI today —
--sort-by-msgidis not an msgcat option, the script has noset -e, so msgcat exits without writing. This PR reproduces current CI outputrather than introducing a new convention; the flag itself is addressed
separately in #44395. Its other two effects are redundant, since
--sort-outputand
--no-locationare already on thepybabel extractcall.Out of scope, left untouched (separate, pre-existing issues):
flask_babel.lazy_gettextcall sites (e.g.superset/views/filters.py:330) are not reachable bypybabel extractat all, since neither babel's default keywords nor this project's-klist includelazy_gettext— worth its own follow-up issue.Fixes #44403
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
Guard passes on this branch:
Guard catches the bug it exists for — restore the pre-fix template and re-run:
Unit tests (mocked cases plus a real end-to-end extraction against this
checkout):
Frontend behavior of the hoisted regex is unchanged:
Regeneration is idempotent — re-running
./scripts/translations/babel_update.shproduces no diff beyond
POT-Creation-Date.ADDITIONAL INFORMATION