OpenAI models achieve 97.9% accuracy in coding and migrating legacy programming languages

OpenAI has significantly enhanced its models' ability to write software in legacy programming languages and migrate existing code. The latest results indicate an accuracy of 97.9% in a test designed to evaluate the migration of legacy programs. These improvements could assist businesses in modernizing software that is critical to their daily operations. Banks and financial institutions still rely on code from decades ago, making replacement efforts highly risky as developers must retain existing calculations and business rules. Handling legacy COBOL code remains deeply embedded in financial infrastructure.

IBM estimates that COBOL supports over 40% of online banking systems, 80% of face-to-face credit card transactions, and 95% of automated teller machine transactions. Fortran also maintains significance in scientific computing, including weather forecasting and nuclear reactor safety analysis. The training materials for these two languages are far less than those for popular options like Python and JavaScript.

Challenges of Legacy Programming Languages

OpenAI states that developers and enterprise customers report difficulties when coding in these legacy languages. Documentation may be incomplete, and critical business rules may only exist in the original code, leading to issues during the migration process. A rewritten program may run without errors, but the results could differ. For instance, COBOL may discard extra decimal places unless the program explicitly requests rounding. Changing this behavior could impact financial calculations. OpenAI's COBOL migration assessment tests whether the model can convert existing programs to C++, Java, Kotlin, Python, or Rust without altering behavior.

GPT-6.1 Sol achieved an accuracy of 86.6% at low reasoning difficulty, while GPT-5.6 Sol only reached 5.4%. OpenAI also reported a 39% reduction in API costs under this setup. At higher reasoning difficulty, GPT-6.1 Sol achieved an accuracy of 97.9%. The assessment required the converted programs to pass every hidden test, including comparisons of their observable outputs with the original programs. This figure reflects the performance of OpenAI's test suite rather than a guarantee of error-free migration.

Developers still need to validate based on actual workloads and undocumented dependencies.

Multi-Language Test Results

OpenAI also tested its models on MultiLangBench, which measures their ability to build SQLite-compatible database engines in 20 programming languages. GPT-6 Astra achieved an average pass rate of 87.4%, compared to just 26.8% for GPT-5.6 Sol. Astra showed improvements across all tested languages. Its Fortran score rose from 21.4% to 83.5%, while Visual Basic increased from 27.5% to 89.4%.

Although COBOL remains the most challenging language, Astra's score improved from 10.9% to 72.1%. GPT-6 Luna had a lower average pass rate of 40.3%, but the cost per sample was approximately lower than GPT-5.6 Sol's 78%. OpenAI estimated the API costs at US$0.77 (approximately HK$6) and US$3.48 (approximately HK$27). These findings indicate stronger support for maintaining aging software; however, developers still need to ensure that modernized systems retain the behaviors upon which businesses rely.

Benchmark tests also indicate areas needing further improvement. Astra's results varied significantly across different languages, with COBOL still presenting the greatest challenge. Meanwhile, Luna's lower cost may allow engineering teams to conduct more experiments without significantly increasing budgets. These results provide encouraging evidence of progress, but teams must carefully validate production deployments before trusting migrated software for critical operations.

Impact of OpenAI Models on Legacy Systems

OpenAI has made significant strides in coding and migrating legacy programming languages, particularly in the financial sector, where many systems still rely on decades-old code. As businesses seek to modernize their operations, these technological advancements can effectively reduce replacement risks and ensure the retention of business rules. However, developers still face challenges with incomplete documentation and business logic, which could affect the accuracy of migrations. OpenAI's test results indicate that despite progress, further validation is needed to ensure the reliability of new systems, which is critical for essential business operations.