MultiLangBench: Judge Legacy Code by Its Behavior

Evaluate observable behavior in the language and environment you actually maintain. OpenAI’s newly released MultiLangBench repository describes an evaluation in which models implement SQLite behavior in multiple programming languages. Its scoring includes test pass rate and API cost. Read the repository before treating a headline result as evidence for your own codebase.

The useful idea is a shared behavioral interface. A model can produce plausible syntax while changing rounding, ordering, file formats or error handling. A benchmark can demonstrate progress on its chosen task without proving that an unrelated legacy migration is safe.

The repository describes initial database setup, a candidate query and a validation query checked with a reference implementation. It also specifies environment restrictions and a standard test subset. Those details are part of the result, not optional footnotes.

For your own project, ask the analogous questions: what input is provided, what state can change, what output is compared and what counts as failure? If a migration passes only examples shown inside the prompt, you do not yet know whether it generalizes to unseen cases.

Choose a small, nonproduction component with a known reference implementation. A fictional fixed-width invoice parser is a useful exercise. Specify character encoding, field positions, sign handling, decimal rules, blank fields and invalid-record behavior.

Keep original input files unchanged and create a separate expected-results set. Include awkward but valid examples and deliberately invalid records. Do not use real customer invoices or personal data. A behavior contract should expose the details that a generic translation prompt might smooth over.

Case	Why it belongs in the contract
Leading zeros	Detects unintended numeric normalization
Negative value	Checks sign and field-width behavior
Blank optional field	Distinguishes empty from missing
Boundary amount	Checks overflow and rounding assumptions
Invalid record length	Tests the failure path

Provide the source, contract and target environment. Keep a private set of additional tests outside the prompt if you want a meaningful check of unseen behavior.

Translate only this component: {{source_component}}.
Target language and runtime: {{target_environment}}.
Behavior contract and visible examples: {{contract_and_examples}}.
Preserve input/output formats, errors and numeric semantics.
List assumptions before code. Do not replace fixed-width fields
or decimal rules with convenient defaults. Return the bounded
implementation and tests; flag any behavior you cannot preserve.

An illustrative failure is converting a fixed-width identifier into an integer, which removes meaningful leading zeros. The generated code may look cleaner while violating the actual file contract. That is a semantic defect, not a stylistic disagreement.

Run both implementations on synthetic fixtures in an isolated environment. Compare outputs and state changes, including error paths. Record runtime and dependency versions. Investigate mismatches before expanding the migration scope.

Do not describe this local exercise as a MultiLangBench score. To reproduce that evaluation, follow its exact current instructions and disclose deviations. In particular, do not run an environment-cleanup script against your normal workstation; inspect and prepare a disposable environment as required by the evaluation.

A passing bounded exercise supports confidence in the tested contract, not an entire system rewrite. Add a regression test for each discovered mismatch and get a maintainer to review the translated behavior. Keep unsupported language features or external dependencies visible.

For the broader project workflow, see the AI-written code checks. Your next step is choosing one small legacy behavior and writing the test contract before asking for a translation. That habit is useful regardless of which model currently leads a benchmark.

Reviewed on 22 October 2026. Primary-source reporting and editorial analysis; the exercises and example results are illustrative, not hands-on performance measurements. Product access, regional rollout and commercial terms can change.

• OpenAI: MultiLangBench evaluation repository

01 Choose a tool: compare the fit. 02 Use a prompt: start with context. 03 Check the result: keep your judgment.