Can LLMs generate scientific code?
Researchers evaluated 15 models on implementation and test generation. The best model succeeded in 87.1% of single-pass implementation trials.

Illustration: generated for Physis Systems
Short answer: The best model succeeded in 87.1% of single-pass implementation trials. The leading implementer achieved 49.9% joint test success. The study is a preprint and has not been peer reviewed.
The primary source for this article is arXiv cs.AI's Scientific Coding with Large Language Models: Evidence from Finite Element Analysis. It is a preprint, which means it has not yet been peer reviewed.
What is the current state of LLM-generated scientific code?
Researchers evaluated 15 models on implementation and test generation without execution feedback. The best model succeeded in 87.1% of single-pass implementation trials.
How successful were the models in joint test success?
The leading implementer achieved 49.9% joint test success. The leading tester reached 82.6% success.
What were the results of the exploratory configuration?
An exploratory configuration with separate coder and tester agents achieved 97.6% Pass@3 and 88.8% Pass$^3$.
What do we know, and what is still open?
What is confirmed: The study identified consistency across attempts and independent verification as main obstacles to trusting LLM-generated scientific code.
What is still open: The study is a preprint and has not been peer reviewed, and further research is needed to fully understand the capabilities and limitations of LLM-generated scientific code.
Why does Physis Systems cover this?
Physis Systems follows research at the frontier, from space to fundamental science. For more stories like this one, visit the Physis Systems blog.
About Physis Systems
Physis Systems (Physis Defense Systems, Inc.) builds Inventi Matrix, an invention engineering platform available in the United States and Canada at inventimatrix.com, and Lumina AI Companion, a personal AI companion you can meet on the Lumina page.
Frequently asked questions
What was the success rate of the best model in single-pass implementation trials?
The best model succeeded in 87.1% of single-pass implementation trials.
What was the joint test success rate of the leading implementer?
The leading implementer achieved 49.9% joint test success.
What were the results of the exploratory configuration with separate coder and tester agents?
An exploratory configuration with separate coder and tester agents achieved 97.6% Pass@3 and 88.8% Pass$^3$.


