Notebook / Human assessment

Turn flags into decisions

Agree what the review can decide

AI-assisted review works best when the team agrees on its authority before the first flag appears. A model may propose that an instruction lost a condition. A language reviewer assesses that claim. A writer clarifies ambiguous intent. A localization lead resolves policy conflicts. None of those responsibilities should disappear inside an automated “quality” label.

Write a short review charter covering languages, content types, build versions and exclusions. State whether the task checks terminology, meaning, technical integrity or style. Separate language review from certification, accessibility evaluation and cultural consultation. Those activities can inform one another without being interchangeable.

Choose the evidence needed for each outcome. A spelling correction may need little context; a change to a character’s relationship or a progression instruction needs more. A clear threshold makes it easier for reviewers to say “not enough information” without appearing to block the process.

A localization lead planning review responsibilities with editors around a naturally lit meeting table
Define who can interpret, approve and verify a change before triage begins.

Build a queue that respects attention

Deduplicate repeated flags without losing their affected string IDs. A single terminology policy issue may appear in many resources; showing every occurrence as a separate emergency creates noise. Group by likely cause, then keep the individual examples available for verification.

Prioritize player impact and release relevance rather than model confidence. A suspected reversal in a purchase confirmation deserves prompt attention even when the model is uncertain. A high-confidence punctuation suggestion may wait. Mark whether the observation comes from a rule, a model, a tester or a reviewer so its origin remains visible.

Each queue item should contain the original source and translation, a narrowly stated concern, relevant context and a proposed next check. Keep the model’s suggestion separate from the translation field. An automated rewrite must not quietly become the new baseline before anyone has assessed it.

Use outcomes that preserve uncertainty

Reviewers need more than “accept” and “reject.” Confirmed error means the evidence supports a problem. Rejected flag means the concern does not hold in this context. Needs context means a decision is not yet possible. Approved exception means a deliberate departure has an owner and a defined scope.

Example decision record

Observation: the target instruction omits “after the shield breaks.”
Assessment: confirmed meaning omission; the action is unavailable before that event.
Next step: revise the instruction, then verify it in the tutorial state where the shield breaks.

Record a short rationale for rejected flags too. Those records reveal repeated weaknesses in a checker and protect editors from revisiting the same harmless construction every week. Avoid broad suppressions: an exception for one speaker or scene should not disable a term check across the entire game.

Calibrate impact together

Have reviewers assess a small shared sample before dividing the queue. Discuss disagreements about category, severity and acceptable adaptation. Multidimensional Quality Metrics, commonly called MQM, provides a useful vocabulary for error types, but teams still need project-specific definitions and examples.

Severity should describe the consequence in context. A wrong faction name could be a minor distraction in optional lore or a major obstacle in a quest objective. A typo in a legal notice may require specialist escalation even if it does not block play. Do not let category alone assign severity.

The matrix below is a starting point, not a universal scoring standard. Agree escalation paths and release criteria with the people responsible for the product.

Language specialists comparing review notes and discussing a finding in a bright, informal meeting area
Calibration makes the reasoning more consistent without demanding identical phrasing.
Illustrative player-impact severity matrix
LabelImpact to investigateReview response
CriticalA serious safety, legal or irreversible-action riskEscalate immediately to the responsible specialist and release owner
MajorMeaning changes an objective, choice or required actionConfirm context, assign a fix and retest the affected state
MinorA localized defect with limited impact on understandingSchedule a correction and check related occurrences
Preference / no defectAn acceptable alternative without demonstrated harmDo not count as a confirmed translation error

Protect the material and verify closure

Before sending game content to an AI service, check permission, contractual confidentiality, retention settings and access controls. Remove player identifiers and unnecessary personal information. Proprietary dialogue is not made safe merely by removing the project title. Use only processing arrangements approved for the material involved.

After a correction is accepted, verify the actual resource and the rendered result in the relevant build. Check placeholders, line breaks, timing and any linked terminology. “Edited” and “verified” are different states; keep them separate so a ticket cannot close just because someone proposed better wording.

Finally, inspect a sample of unflagged content. Otherwise the team can measure how useful the queue is but cannot learn much about what it misses. Feed reviewer decisions back into rules and examples through a controlled update, then rerun the calibration set before expanding the check.

Next: make findings actionable