Stochastic Parrots: Why LLMs Don't Think

LLMs predict the next token. That's it. They've mastered linguistic form (syntax, grammar, the shape of arguments) without possessing communicative intent. The danger is that fluent output tricks humans into seeing a mind where there's only statistics.

The Stochastic Parrot Hypothesis

Bender, Gebru, McMillan-Major & Shmitchell coined the term "Stochastic Parrot" to describe what these models actually are: systems for probabilistic string prediction trained to minimize perplexity. They produce coherent text because coherent text was in the training data, not because they understand what coherence means.

The risk is "Ersatz Fluency," output so polished that humans reflexively assume understanding. This triggers automation bias: we overlook logical failures because the form is correct. When an LLM hallucinates, it isn't lying (lying requires intent). It's generating a statistically probable continuation that happens to be false. No reference to truth, only to probability.

GSM-Symbolic: The Reasoning Collapse

The GSM-Symbolic benchmark destroyed the "LLMs can reason" narrative. Mirzadeh, Alizadeh, Shahrokhi et al. at Apple generated thousands of variations of standard math word problems with the same logic but different names and numbers.

Two findings stand out. First, changing surface features (swapping "Alice" for "Bob" or adjusting numeric values) caused significant performance swings. A system that understands addition knows A + B = C regardless of labels. The LLM's sensitivity to irrelevant features proves it's pattern matching, not reasoning.

Second, and more damning: the "NoOp" collapse. Researchers inserted irrelevant but plausible statements ("Oliver picks 44 kiwis on Friday. He also sees a red car."). Performance dropped up to 65%. A reasoning entity ignores the red car. The LLM, seeing a new object in context, statistically integrates it into the calculation.

What looks like reasoning is template retrieval. Critical thinking remains a human job, and with AI integration accelerating, that job is getting more important, not less.

Sources