Recently, OpenAI released 719 model-generated solutions to mathematical problems, covering multiple open research questions. Although the company stated it consulted a mathematics and AI advisory panel prior to release, its process does not yet fully meet the panel's standards, particularly in adequate safeguards for human understanding, formal verification, and peer review. OpenAI's advisory panel is led by the Institute for Advanced Study at Princeton, with members including nine mathematics researchers from institutions worldwide. The panel previously recommended halting the use of proprietary models to test advanced mathematics problems, but OpenAI this time explicitly used a proprietary model to evaluate open mathematics problems.
OpenAI's math problem solutions have multiple shortcomings
Among the 719 manuscripts, only 10 disclosed the reasoning process by which the model reached its conclusions, while another 42% of proofs were not formally processed. A new paper by mathematics researchers at the University of Cambridge and King's College London notes that in OpenAI's solution to a problem derived from the Navier-Stokes equations, there are at least two discrepancies between the natural language proof and the formalized code. These discrepancies do not necessarily mean the proof is wrong, but they show that models may mistranslate when converting textual reasoning into verifiable code.
The advisory panel previously requested that labs provide machine-readable linking information to correspond natural language proofs with formalized results, but OpenAI did not provide this. Mathematicians argue that machine-generated proofs still require the same peer review as traditional proofs. Terence Tao also noted that if the person prompting the model to solve a problem cannot explain the results, participate in the report, or respond to peer questions, the findings cannot truly integrate into mathematical research. Harvard mathematics professor Melanie Wood stated that when models output solutions, human understanding often has not yet formed, and subsequent verification work is just beginning.