Summarize this article. Four words, dead simple, feels like the most basic thing you could ask an AI to do. And yet it’s one of the places where AI fails in the most expensive way: confidently, readably, and wrong in a way you won’t catch unless you go back to the original.
The output looks like a summary. It’s shorter than the source. It’s grammatically cleaner. It covers the general topic. But there’s a consistent failure mode that almost nobody talks about, and once you see it you can’t unsee it.
What actually gets dropped
Models don’t summarize the way a careful human does. A human re-reads a document, decides what the author was actually trying to argue, and writes that. A language model predicts what a summary of this kind of text tends to look like — which sounds similar but produces a different result in practice.
What gets predicted as “summary-shaped” is usually: the most frequently mentioned concepts, the opening and closing claims (which anchor most strongly in the context window), and whatever was stated most directly. What gets dropped is usually: the actual numbers, the specific caveat the author buried in paragraph four, the comparison that changes the meaning of the conclusion, and any claim that required reading two sections together to understand.
The caveat problem is the one that stings most. A lot of well-written technical content has the structure: here’s a general claim, here are three supporting points, and here’s the specific condition under which this all breaks down. The summary often reproduces the general claim and the three points and silently removes the condition — because that’s what makes the summary shorter without obviously changing the topic.
A concrete example of how this plays out
Take a research post that says: “Model X outperforms Model Y on coding benchmarks — but only on Python, and the gap disappears on tasks involving external library calls.” A standard “summarize this” prompt will very likely produce something like: “Model X outperforms Model Y on coding benchmarks.” The qualifier is gone. The summary is technically about the same document. It is also meaningfully wrong.
If you then use that summary to make a decision — say, choosing which model to use for a project that involves a lot of third-party library code — you’ve been misled by a tool you trusted to save you reading time. The failure was silent. Nothing crashed. You got a clean paragraph back.
Why asking for bullet points doesn’t fix it
The reflex solution is to say “summarize as bullet points” instead. This helps a little, because bullets force the model to treat each point as a separate unit rather than blending everything into prose. But it doesn’t fix the selection problem. The model still has to choose which points to include, and it still tends to choose the high-frequency, obvious-sounding claims over the specific, load-bearing ones.
Bullet-point summaries of complex documents often end up as a list of the document’s least surprising sentences. Everything true, nothing useful.
What actually improves the output
Three things help, and they work for different reasons.
Ask for the argument, not the content. “What is this document actually arguing?” produces a different kind of summary than “summarize this document.” The first forces the model to reconstruct the claim being made; the second just asks for shorter text on the same topic. For anything that has a thesis — an opinion piece, a research post, a case study — this change alone noticeably shifts what you get back.
Ask specifically for what was qualified or limited. “What claims in this document come with conditions or caveats?” is a separate follow-up worth running on anything that matters. You’re explicitly routing the model toward the part of the document it’s most likely to drop. It doesn’t catch everything, but it catches a lot.
Give it a target reader with a specific question. “Summarize this for someone deciding whether to switch from Model Y to Model X for a project with heavy library dependencies” will surface the relevant qualifier that a generic summary drops — because you’ve described the exact context where that qualifier matters. This is just being specific about what “done” looks like, which is the same fix that works everywhere in prompting.
When summarization is actually fine
Not every summary needs to be precise. If you’re skimming a news roundup to decide which three articles to actually read, a rough AI summary that gets the general topic right is exactly what you need. The cost of dropping a caveat is low when you’re triaging.
The problem is applying that same default behavior to something where precision matters — a research paper, a contract, a competitor’s technical documentation, a policy document you’re making decisions from. The model is equally confident either way. It doesn’t know whether you’re triaging or deciding.
That distinction is yours to make, not the model’s.
One habit worth building
Before you use an AI summary for anything that actually affects a decision, run one extra prompt: “What did you leave out?” or “What would someone miss about this document if they only read your summary?” Models are surprisingly good at this — probably because generating the critique doesn’t require retrieving the document’s subtleties, just knowing what summaries typically miss, which is a pattern with plenty of training signal behind it.
It won’t catch everything. But it catches the obvious gaps, which are usually the ones that matter most.
“Summarize this” is a fine starting point. Treat it as a finished output for anything you’re about to act on, and you’re trusting the wrong part of the process.
