Ongoing Translation-Quality QA for a Multilingual AI Agent, Per Language

Quick answer

Run a recurring, per-language QA process, separate from and after whatever build-time work made the agent multilingual in the first place, that samples real conversations in each supported language on a fixed schedule, has them reviewed by a fluent speaker (not just a back-translation check) against the same accuracy and tone rubric used for your primary language, and tracks quality per language as its own metric rather than assuming build-time testing in one or two languages generalizes to all supported languages indefinitely. Translation and response quality in non-primary languages tends to drift or lag silently, since most internal review, by default, happens in whatever language the team building the product speaks natively.

Why this needs a distinct, ongoing process from the build-time work

Supporting multiple languages without separate per-language builds is a real, valuable architectural decision, covered in supporting multiple languages in an AI agent. That post is about the build architecture: how one underlying system serves multiple languages, whether through a multilingual model, a translation layer, or some combination. It does not answer a separate, ongoing question: once that architecture is live, how do you know response quality is actually holding up, on a continuing basis, in each of the languages it claims to support, and not just at the moment it first launched.

This distinction matters because architecture and quality can diverge over time even without any architecture change. A model update, a prompt change made and tested primarily in English, a new product feature whose instructions or copy were written and reviewed only in the primary language, or a knowledge-base update authored only in the primary language and machine-translated on the fly, can each degrade quality in secondary languages without anyone noticing, because the people making the change may not be fluent enough in the affected languages to notice the degradation themselves.

The core problem: most internal review happens in the team's native language

The default failure mode here is entirely structural, not a matter of anyone being careless: most product, engineering, and QA teams building an AI agent are fluent in one or two primary languages, and most day-to-day review, casual spot-checks, demo conversations, informal "does this feel right" testing, naturally happens in those languages. Quality in every other supported language depends entirely on whether a deliberate, scheduled process exists to check it, because it will not get checked incidentally the way primary-language quality does.

Build the ongoing QA loop per language, not as one combined check

A meaningful per-language QA program needs a few specific elements:

• A fluent human reviewer for each supported language, reviewing actual production conversations in that language, not machine-translating the conversation back into the team's primary language and reviewing the translation. Back-translation review misses exactly the kind of subtle tone, register, and idiom problems that matter most, since a back-translation can look perfectly reasonable even when the original response was awkward, overly formal, overly casual, or used an idiom that does not carry the intended meaning in that language.
• The same accuracy and tone rubric used for the primary language, translated and adapted (not just literally translated) for cultural and linguistic appropriateness, so quality is being measured on a comparable basis across languages rather than each language inventing its own ad hoc bar.
• A fixed, recurring schedule, not a one-time check after launch. This can reasonably piggyback on the same kind of ongoing production monitoring described in production quality metrics you should track for a live AI agent, with an explicit requirement that the sample is stratified by language so lower-volume languages still get meaningful, regular coverage rather than being crowded out by sampling that is proportional to raw conversation volume, since a low-volume language's issues are just as real to the customers experiencing them even if they generate fewer total conversations.

Specific failure modes to watch for beyond raw mistranslation

Per-language QA needs to look for more than "is this technically the wrong words." Some of the most common and highest-impact failure modes are structural rather than lexical:

• Register and formality mismatches: many languages carry formal/informal distinctions (a wrong choice here can feel rude or oddly familiar) that do not exist in English and are easy to get wrong if the underlying prompt or model behavior was primarily tuned against English conversation norms.
• Idioms and quick-answer phrasing that do not translate meaningfully, where a phrase that reads naturally in the primary language becomes stilted, confusing, or literally nonsensical when translated, particularly in FAQ and quick-answer blocks written to be punchy in the primary language first.
• Locale-specific facts silently staying in the primary locale: currency, units, regional terminology, or region-specific policy details that were correct for the primary market but wrong or irrelevant when the same underlying content serves a customer in a different region and language.
• Feature or product-update lag: a new capability or a changed policy documented and tested only in the primary language, with the non-primary-language knowledge base or prompt content simply not yet updated to match, producing confidently wrong or outdated answers in that language specifically.

Track quality per language as its own metric, and flag drift explicitly

Aggregate quality metrics that blend all languages together can hide a real, sustained quality gap in a specific language behind an acceptable-looking overall number, exactly as handling an AI agent that performs well overall but poorly for one segment or geography describes for other kinds of segment-specific underperformance. Tracking per-language quality as a separate, visible line, rather than folding it into one global score, is what makes a language-specific quality gap or a quality regression after a recent change visible enough to act on, instead of averaging away.

Escalate genuinely low-volume languages differently

For a language with low conversation volume, a fixed percentage sample may not generate enough conversations in any given period to review meaningfully. In that case, set a minimum absolute number of reviewed conversations per period regardless of what percentage that represents, so low-volume languages still get real, regular attention rather than effectively falling out of the QA process by virtue of their volume being too small for a percentage-based sample to catch anything.

FAQ

Is this the same as the work needed to build multilingual support in the first place? No. Building multilingual support is an architecture decision made largely once (with ongoing maintenance). This is the recurring, ongoing quality-assurance process that checks whether response quality is actually holding up in each supported language over time, independent of the underlying architecture.

Is back-translation review (translating the response back to the primary language to check it) good enough? No, it misses tone, register, and idiom problems that are only visible to a fluent reviewer reading the response in its original language. Back-translation review should be treated as a weak substitute at best, not a real replacement for native-fluent review.

How should sampling be split across languages with very different conversation volumes? Use a minimum absolute review count per language per period, not a pure percentage of volume, so low-volume languages still receive meaningful, regular review instead of being effectively skipped.

What is the most commonly missed failure mode in non-primary-language QA? Feature or policy update lag: a recent product change tested and documented only in the primary language, with non-primary-language content quietly left out of date, producing confidently wrong answers that a primary-language reviewer would never think to check for.