All 30 Simulated Chatbot Nutrition Plans Had Safety Issues
Higher completeness on a guideline-based scoring rubric did not establish safe nutrition planning: every analyzed chatbot plan had a major or critical safety issue.
Evidence
All 30 analyzed chatbot-generated plans for total parenteral nutrition—delivering all nutrition through a vein—contained at least one major or critical safety issue in simulated cases. Seven plans (23.3%) contained a critical concern, and 19 (63.3%) were rejected in the study assessment.
The chatbot parenteral-nutrition benchmark, published October 6, 2026, evaluated ChatGPT, Gemini, Copilot, and Perplexity using 10 standardized clinical cases representing critically ill adults. It was a cross-sectional simulation study: a snapshot assessment of generated plans, not a trial in patients. Researchers assessed 30 plans using a predefined, guideline-based rubric—a structured scoring checklist—covering accuracy, completeness, safety, acceptability, error types, and numerical precision.
The most commonly reported error categories involved electrolytes, minerals that carry an electric charge (90.0%); micronutrients, nutrients needed in small amounts (70.0%); and monitoring (66.7%).
Median completeness—the middle score for coverage of the rubric—was 64.6% for ChatGPT, 41.9% for Gemini, and 32.3% for Perplexity. The overall comparison gave a p-value of 0.003, indicating that differences this large or larger would be unusual under a no-difference model. Kendall’s W, an effect-size measure of consistency in the chatbots’ rankings across cases, was 0.57.
ChatGPT also scored higher than Perplexity after correction for multiple comparisons, with an adjusted p-value of 0.006. The reported rank-biserial correlation, an effect-size measure summarizing the direction and strength of paired ranked-score differences, was 1.00.
Analysis
Interpretation: Completeness and safety measured different things. Higher checklist coverage coexisted with major or critical safety problems; it did not demonstrate safe planning. The reported completeness ranking should therefore not be read as a safety ranking.
The error pattern also extended beyond a single nutrient calculation to nutrient components and monitoring. This supports examining the content and seriousness of errors alongside completeness, rather than treating a fuller plan as sufficient evidence of quality.
Limitations
This is a single-study report based on only 10 simulated cases, not corroboration from separate studies. The narrow case set limits generalizability and may have reduced the ability to detect meaningful differences between systems. Findings depend on the selected cases, prompts, rubric, reference plans, and review process.
Statistical differences in completeness do not establish clinical significance or patient benefit. The simulation design cannot establish performance in routine patient care, and safety issues identified in generated plans are not measurements of actual patient harm.
What to watch
A larger, more varied simulation benchmark could report each chatbot’s major-or-critical safety-issue rate alongside completeness scores. Consistently fewer safety issues in higher-scoring plans would strengthen confidence that completeness is informative about safety within simulations, without establishing safety in patients.