Physis Systems BlogLumina AI Companion and Inventi Matrix
AI Research

How Does Tool Output Affect LLM Agents?

Researchers studied the impact of retrieved tool output on LLM agents' accuracy. The study used a geochemistry simulator called PHREEQC.

Illustration of a researcher and robot interacting with a geochemistry simulator report on a screen in a laboratory setting

Illustration: generated for Physis Systems

Short answer: The study found that LLM agents' accuracy can be affected by retrieved tool output, with some models losing accuracy when reading reports through a section index. The preprint has not been peer reviewed.

The primary source for this article is arXiv cs.AI's Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy. It is a preprint, which means it has not yet been peer reviewed.

What is the impact of retrieved tool output on LLM agents?

The study used a geochemistry simulator called PHREEQC to investigate the impact of retrieved tool output on LLM agents' accuracy. The simulator's reports can be up to 846.8k characters long.

How do different LLM models perform?

GPT-5.1 and GPT-5.2 lost accuracy when reading the simulator's report through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.

What is the cause of the accuracy loss?

The study found that GPT models often skipped the report and answered from their own output files, leading to a failure called evidence substitution.

Can the accuracy loss be mitigated?

Naming the authoritative output raised accuracy for the three GPT models.

What are the implications of the study's findings?

The study's findings have implications for context-management choices in agent harnesses.

What do we know, and what is still open?

What is confirmed: The study found that LLM agents' accuracy can be affected by retrieved tool output, with some models losing accuracy when reading reports through a section index.

What is still open: The preprint has not been peer reviewed, and further research is needed to confirm the findings.

Why does Physis Systems cover this?

Physis Systems follows research at the frontier, from space to fundamental science. For more stories like this one, visit the Physis Systems blog.

About Physis Systems

Physis Systems (Physis Defense Systems, Inc.) builds Inventi Matrix, an invention engineering platform available in the United States and Canada at inventimatrix.com, and Lumina AI Companion, a personal AI companion you can meet on the Lumina page.

Frequently asked questions

What is the name of the geochemistry simulator used in the study?

PHREEQC.

How long can the simulator's reports be?

Up to 846.8k characters.

What is the failure called when GPT models answer from their own output files?

Evidence substitution.

Can naming the authoritative output improve accuracy?

Yes, it raised accuracy for the three GPT models.

What are the implications of the study's findings?

The study's findings have implications for context-management choices in agent harnesses.

Watch

Inventi Matrix, the film Watch on YouTube

Get the weekly frontier roundup

One email a week from Physis Systems with the research stories worth knowing, each linked to its primary source.

We will email you a link to confirm. We never share your address.