What OpenAI published on October 19
OpenAI described two coding evaluations aimed at languages with less training code and documentation, including COBOL, Fortran, and Visual Basic. MultiLangBench asks a model to build a SQLite-compatible database engine in one of 20 languages; reference SQLite checks the resulting database. A separate COBOL migration evaluation checks whether a rewrite preserves the original program’s behavior. OpenAI also released the MultiLangBench test setup and suite on GitHub.
A passing rewrite must preserve more than syntax
A program can compile, finish without an error, and still change an important result. OpenAI points to fixed-point arithmetic in COBOL: excess decimal digits can be discarded unless rounding is requested. A rewrite that rounds them instead may produce different financial results. The same risk applies to output formatting, missing records, default values, and side effects that have become part of a system’s behavior.
What the reported score tells you
OpenAI reports that GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort in its COBOL migration evaluation, compared with 5.4% for GPT-5.6 Sol at the same setting. Those are OpenAI’s results on its evaluation, not an independent assessment or a result from your codebase. The announcement does not establish that a particular legacy system can be migrated safely.
Try a small behavior-preservation check
- Illustrative example: imagine rewriting a fictional invoice routine. Before changing it, run the old routine on fake cases with extra decimal places, a negative adjustment, and a missing record. Save its exact outputs as the comparison baseline; do not guess what rounding or missing-data rule the old program uses.
- Ask for a narrow rewrite of one routine and keep the fixtures, inputs, and output format fixed. Run the old and candidate versions on the same fake cases in an isolated test environment.
- Compare outputs and any recorded side effects exactly. Treat every mismatch as a failure to investigate, even if the new program compiles and its result looks reasonable.
- Have a developer who understands the business rules review each difference. Expand the test set before a wider migration, and keep checking dependencies and system-level behavior that a small fixture cannot cover.
Benchmark results are a starting point
The figures above are vendor-reported evaluation results. A benchmark score does not replace tests against your own rules, dependencies, and real execution environment. We did not run MultiLangBench or test a model on legacy code.