Chatbots in Parenteral Nutrition Planning

This single-study explainer examines whether publicly available artificial-intelligence chatbots could formulate total parenteral nutrition plans for critically ill adults in a standardized simulation benchmark. The investigators compared ChatGPT, Gemini, Copilot, and Perplexity against expert pharmacist reference plans using a predefined, guideline-based assessment that covered accuracy, completeness, safety, acceptability, error types, numerical precision, and agreement. The work tested chatbot outputs in constructed cases, not decisions made for actual patients. pmid:42903103

Evidence

The reported design was cross-sectional and simulation-based. Investigators used 10 standardized clinical cases involving critically ill adults and prompted each chatbot to create a total parenteral nutrition plan. The study reported 30 plans for analysis: ChatGPT generated a plan for every case, whereas Gemini generated six and Perplexity generated four. Copilot declined to create a patient-specific plan under the standardized prompt; that refusal was retained as a performance outcome, while Copilot was excluded when an analyzable prescription was necessary. pmid:42903103

The evaluation framework went beyond a simple correctness check. Plans were assessed by a predefined, guideline-based rubric for accuracy, completeness, safety, acceptability, error types, and numerical precision. Comparative analyses used Friedman tests, Kendall’s W, and Holm-adjusted Wilcoxon signed-rank tests. The researchers also assessed numerical agreement and concordance between chatbot recommendations and pharmacist-derived reference values, including Bland-Altman analysis. These methods framed chatbot responses as outputs to be audited against a pharmacist reference rather than as independently validated clinical prescriptions. pmid:42903103

Completeness differed across the three chatbots with analyzable plans. Median completeness was 64.6% for ChatGPT, 41.9% for Gemini, and 32.3% for Perplexity; the overall comparison had p = 0.003 and Kendall’s W = 0.57. In a pairwise comparison, ChatGPT was more complete than Perplexity, with Holm p = 0.006 and rank-biserial r = 1.00. The abstract also reported median accuracy values for the three chatbots and no statistically significant overall accuracy difference, while reporting a ChatGPT–Perplexity pairwise accuracy difference. pmid:42903103

Safety and acceptability findings were unfavorable across the analyzed plans. Every plan had at least one major or critical safety issue. Seven plans, 23.3% of the 30 plans, contained a critical concern, and 19 plans, 63.3%, were rejected. The source characterized the evaluated chatbots as not having sufficient accuracy, safety, or concordance for independent use in this benchmark. Statistical results on a rubric score and safety classifications describe the tested outputs; they do not measure patient outcomes. pmid:42903103

The reported error pattern concentrated in nutrition-support details that require internally consistent prescribing and follow-up instructions. The most common errors involved electrolytes, micronutrients, and monitoring. The discussion further described recurrent omissions involving trace elements, multivitamin administration, monitoring protocols, and refeeding precautions across the chatbots. The source contrasted relatively better chatbot performance for energy, protein, dextrose, and lipid recommendations with poorer performance for electrolytes, micronutrients, and monitoring. pmid:42903103

Numerical concordance with pharmacist-derived values was another separate assessment domain. The abstract reported that chatbot recommendations often exceeded pharmacist-derived values and that Bland-Altman analysis showed considerable variability in agreement. Thus, a response that included plan components was not necessarily numerically aligned with the reference plan. This distinction matters within the study’s rubric: completeness, accuracy, safety, acceptability, error taxonomy, and numerical agreement were related but separately assessed characteristics of simulated prescriptions. pmid:42903103

Analysis — Benchmark interpretation

The observed pattern is that ChatGPT produced more complete simulated total parenteral nutrition plans than the other analyzable chatbots, while the safety review identified major or critical problems in every analyzed output. Completeness differences and pairwise statistical results establish variation within this particular set of standardized cases and this rubric; they do not establish that a more complete output was clinically sufficient, safer, or beneficial for a critically ill person. The comparator was an expert pharmacist-derived reference, so the benchmark addresses concordance with that reference and rubric-based review rather than comparative outcomes under clinical care. pmid:42903103

The study also separates two questions that can otherwise be conflated: whether a chatbot responds with a plan and whether that plan withstands detailed safety and numerical review. Copilot’s refusal was recorded as a performance outcome, whereas its lack of an analyzable prescription limited its inclusion in prescription-level comparisons. The repeated error categories show that the tested systems’ weaknesses were not restricted to one macronutrient calculation; they extended to electrolytes, micronutrients, monitoring, and refeeding-related content. Because cases were simulated and outputs were evaluated cross-sectionally, this is an observed benchmark pattern consistent with the selected cases, prompts, reference plans, and review process. pmid:42903103

Limitations

The retrospective design cannot establish causation. More specifically, this was a cross-sectional simulation benchmark, not a study of care delivered to patients, so it cannot establish clinical safety, efficacy, or patient benefit. The 10-case selection process may not represent the diversity of routine critical-care populations, and the limited case set may have reduced statistical power to detect clinically meaningful differences, increased the possibility of type II error, and restricted generalizability. pmid:42903103

Selection and comparator boundaries are central to interpretation. The cases were standardized, chatbot performance depended on the applied prompt and whether a chatbot returned a patient-specific plan, and prescription-level analyses excluded Copilot because no analyzable prescription was produced. Outcomes measured were rubric-based accuracy and completeness, safety and acceptability classifications, error types, numerical precision, and agreement with pharmacist-derived references. The supplied material describes considerable variability in agreement but does not provide a basis here for broader claims about performance in real-world clinical workflows. pmid:42903103