LLM-Generated ICU Summaries Improve Mortality Prediction, but Mostly Reorganize Clinical Notes

A preprint updated on arXiv on Aug. 14, 2026 reports that fusing LLM-generated summaries of ICU notes with physiological data improves in-hospital mortality prediction on the MIMIC-III database. The study also asks a harder question: how much of that improvement is genuinely new signal, and how much is just a reorganized version of what the raw notes already contain? The answer, according to the authors, is that the summaries help, but most of the gain is redundant with the notes themselves.

The finding is relevant to hospital AI teams deciding whether to add note summarization to clinical risk models. Summaries can improve a mortality model without necessarily contributing new clinical knowledge.

What the study tested

The analysis used MIMIC-III, a widely used critical care dataset, and covered 19,211 first ICU stays with a recorded mortality rate of 12.83%. The researchers encoded three representations of each patient: 48 hours of physiological data, the raw clinical notes, and LLM-generated summaries of those notes. The summaries were created under instructions that explicitly forbade prognostication, meaning the model was not allowed to add its own predictions or outcome judgments.

The evaluation was performed on 3,843 held-out ICU stays. A fused model combining physiology, notes and summaries reached an area under the precision-recall curve of 0.4977 and an AUROC of 0.8429. Physiology alone scored 0.3625 AUPRC and 0.7770 AUROC. The gap is large, especially on AUPRC, which is more informative in a dataset where mortality occurs in only about one in eight stays.

How redundancy was measured

The main scientific contribution of the preprint is in the redundancy analysis. Rather than stopping at the fused-model improvement, the authors tried to determine whether the summaries introduced information that could not be recovered from the original notes.

First, they used ridge regression to test whether note embeddings could predict the summary embeddings. The notes explained 40.8% of the variance in the summary representations, confirming substantial overlap.

Second, they created a note-orthogonal residual by removing from the summary embeddings everything that could be predicted from the notes. When this residual was used in place of the full summaries, the model retained only a minority of the original improvement: +0.0258 AUPRC, with a 95% confidence interval from 0.005 to 0.047, or about 28% of the total gain.

Third, they tested a patient-shuffled version of the summary embeddings. When summary embeddings were assigned to different patients and the model was retrained, performance fell below the physiology reference. That result indicates the summaries’ value is patient-specific rather than a generic text artifact.

What the results showed

The authors conclude that summaries improve prediction patient-specifically, but predominantly by reorganizing information the notes already contain. That is a useful distinction for model development. A summary may make existing note content easier for a model to use, but it should not be treated as a new diagnostic signal.

Key numbers from the preprint:

| Metric or design | Result |
| Dataset | MIMIC-III: 19,211 first ICU stays; 12.83% mortality |
| Held-out evaluation set | 3,843 ICU stays |
| Fused model, physiology + notes + summaries | AUPRC 0.4977; AUROC 0.8429 |
| Physiology alone | AUPRC 0.3625; AUROC 0.7770 |
| Redundancy check | Note embeddings explained 40.8% of summary-embedding variance |

Why it matters

For teams building ICU risk tools, the practical takeaway is that LLM summaries can produce a measurable improvement in in-hospital mortality prediction, but the improvement is mostly about representation, not discovery. The summaries may help a transformer or a fused tabular-text model attend to relevant details more efficiently, while the underlying clinical facts were already present in the notes.

This also matters for benchmark design. A simple comparison of “physiology only” versus “physiology plus LLM summaries” can overstate the value of the summary step. Without an orthogonality check or an ablation that removes note-redundant signal, a model’s gain could be credited to the LLM when it should be credited to the raw notes.

What remains unclear

The paper is a preprint and has not completed peer review. The public abstract reports the core results but does not name the specific LLM used for summarization, the note encoder architecture, or all the intermediate ablation numbers. Those details are important for reproducing the result in a hospital setting.

The study is also based on a single public dataset. MIMIC-III is well known and widely used, but it does not guarantee the same behavior on newer ICU notes, different note styles, or patients outside the original institution’s population. External validation and real-time clinical testing would be needed before drawing operational conclusions.

Source: https://arxiv.org/abs/2411.16818

Datos clave