The output arrives looking like an answer. The process that generated it stays out of reach, and a confident written explanation doesn’t change that.
Berlin, 1904: A retired mathematics teacher named Wilhelm von Osten had spent years convinced that he could teach a horse to think. He sought something beyond training responses or memorized cues. He believed Hans could work through problems the way a student would, arriving at answers through genuine reasoning. He called the horse Hans. The public eventually called him Clever Hans, and the name stuck for a reason.
The setup looked like a demonstration of something that shouldn’t be possible. Von Osten would pose a question, the kind normally given to a schoolchild, and Hans would answer by striking the cobblestones with a foreleg. The right number of times, the right sequence. Spectators crowded in. Journalists wrote about it. A formal panel of investigators convened to determine whether the thing was a hoax, and came away without finding one. Von Osten wasn’t pocketing entrance fees. He wasn’t feeding the horse signals from behind his back. Other questioners could step in, and Hans would keep going.
A young psychologist named Oskar Pfungst attended one of those demonstrations and left with a different question from the one most observers were asking. Where everyone else wanted to know how well the horse could calculate, Pfungst wanted to know how the correct answer was arriving at all. He began adjusting the conditions around the performance, one variable at a time. When the questioner knew the answer and stayed within the horse’s sight, things went well. When the questioner was kept ignorant of the answer, or positioned where Hans couldn’t see them clearly, the performance collapsed.
Pfungst eventually traced it to something too small to perceive consciously. As Hans approached the expected number of hoof strikes, the people watching him would tighten almost imperceptibly, leaning forward, holding slightly more still, a kind of involuntary anticipation spreading through posture and expression. When the count reached the correct number, that tension would release. Hans had learned, with extraordinary precision, to read the release. He stopped at the moment the crowd silently told him to.
Von Osten had not deceived anyone. He was a true believer. Hans was not pretending either. What the horse had developed was genuine perceptual skill, an ability to detect involuntary human movement that exceeded anything the humans around him could consciously perform. The failure was in the interpretation, not the animal. Everyone had been reading the hoof when the answer was happening somewhere else entirely.
Pfungst’s investigation mattered because he refused to accept the output as proof of the mechanism. That distinction is now more consequential than it has ever been.
The Sentence and the Computation Are Two Different Things
When a large language model responds to a prompt, something real has happened. A sequence of computations has run across billions of learned parameters and arrived at particular words. If the words are accurate and the sentence is clear, the natural human response is to assume something like reasoning occurred, because articulate language normally indicates that in the humans who use it.
That assumption does not follow. The output shows that the system produced those words. It does not show how.
The people who build these systems know the architecture they chose, the data they trained on, and the procedures used to shape the final product. They can examine attention patterns, weight distributions, activation values, and token-by-token probabilities. They run the system through enormous test suites and study how it behaves under thousands of different conditions. None of that gives them a complete, human-readable explanation of why a specific prompt produced one particular answer when it passed through a specific configuration of billions of learned parameters.
This is the interpretability problem, and it is worth being precise about what makes it hard. The numbers are fully accessible to anyone with enough compute to read them. What they resist is interpretation at the scale and complexity at which they operate. The system emerged from a learning process, and an enormous internal arrangement developed through that exposure. The useful behavior lives inside that arrangement. The arrangement arrived without an index.
A Reasoning Trace Is Still Just More Words
Several AI laboratories have begun surfacing intermediate language before delivering a final answer. Users see what looks like a thinking process, a sequence of considerations that feels like watching someone work through a problem on paper. The term often used is chain-of-thought reasoning, and the framing invites the inference that this is a window into the system’s underlying computation.
It is not, at least not reliably. It is more language produced by the same model that produced the answer. And the two things can come apart.
A 2025 study from one of the industry’s leading research teams embedded misleading cues into multiple-choice questions, then examined whether the visible reasoning acknowledged those cues when they changed the model’s answer. Two reasoning models in the study surfaced the influential hint in their written explanations only 25 and 39 percent of the time. When questions grew more difficult, the explanations became less faithful, not more. In several cases, a model adopted a wrong answer from a planted hint and then produced a lengthy justification for that answer without disclosing what had actually shifted its response.
No theory of deception is needed to explain this. The output that arrives as an explanation is generated by the same process as any other output. A system trained to produce convincing explanations will do so, whether or not the explanation accurately reflects the computations that preceded it. Clever Hans gave the crowd a count of hoof strikes. Contemporary language models give the crowd an answer, and then a paragraph of words describing how the first set of words supposedly came to be. Both products can look like evidence of the method. Neither one guarantees it.
What Interpretability Research Has Actually Found
The right response to all of this is not resignation. Researchers are doing serious work on mechanistic interpretability, the effort to identify meaningful internal features inside neural networks and trace how they combine to produce specific behaviors. The field is young, and the findings are real.
One significant 2025 project aimed to build attribution graphs that could trace the computational path a model took for particular inputs. Researchers uncovered genuine internal mechanisms, including a striking finding that a model sometimes selected a rhyme for the end of a line before writing the words that would lead up to it. That is a meaningful result. It describes something real about how the system operates.
The researchers were honest about the limits. Their method worked satisfyingly for roughly a quarter of the prompts they tested. Even in those cases, the graphs captured only a portion of what was happening inside the full model. They were also working with a smaller, simplified stand-in that approximated the real system and could not serve as a direct window into the deployed model itself. The result was not a dissection. It was closer to a map drawn from a scale model, checked against the original, and presented with full acknowledgment of what the scale reduced.
That still represents meaningful scientific progress. But it makes the current state of affairs legible in a meaningful way. The best interpretability researchers working today are still building preliminary tools to reconstruct partial causal pathways inside systems already deployed at enormous scale. They are making the microscope while the patient is already at work.
When the Output Stops Being Sufficient
For most casual uses of these systems, the gap between output and mechanism is easy to live with. A misattributed restaurant recommendation or an imprecise paragraph carries a low cost. The stakes change when the system starts participating in decisions with real institutional weight.
The National Institute of Standards and Technology has been clear that interpretable and explainable systems are easier to debug, easier to monitor, and more amenable to the documentation and governance that institutions depend on. Those are not abstract values. They describe the conditions under which a system can be audited, challenged, corrected, and explained to a regulator, a court, or a person whose life it affected.
When a consequential decision needs review, the output alone does not complete the review. A generated explanation produced after the fact does not either. An audit requires a path. It requires knowing what information the system drew on, how different inputs related to each other, which possibilities it effectively discarded, and what conditions would have led somewhere else. Without that path, remediation tends to address the edges of the failure without reaching what produced it. Teams adjust prompts, add filters, change training data, or retrain components. Sometimes the symptom disappears in testing. The mechanism that produced it may or may not have been reached.
An industry that has gotten very good at generating confident-sounding outputs, but far less good at explaining how those outputs came to be, has made a tradeoff. That tradeoff may be acceptable across a wide range of low-stakes applications. It should not be described as transparency just because the word reasoning now appears above the paragraph.
A Different Starting Position
Vertus approaches this from an architectural premise that differs entirely from the large language model family. The company describes its system as a Cognitive Reasoning Superintelligence, and the distinction it draws is not cosmetic. Vertus does not apply an interpretability layer to a trained model after the fact. It is not a trained model of that kind at all.
The system generates a cognitive structure specific to each problem it encounters. That structure is the work, not a byproduct of work that happened somewhere else and finished before the user arrived. The relevant pathways form around the problem in view, develop as evidence and relationships enter the analysis, and remain visible while the reasoning proceeds. When the answer comes, the route that produced it remains intact.
The visibility serves a practical function. It does not make an answer correct on its own. What it does is give verification and challenge somewhere to stand. A reviewer can see where a connection was made and question it. A weak inference can be identified at the point where it entered the path, not reconstructed after the fact from the output alone. Human judgment, external evidence, accountability: all of those still matter. What changes is that they now have a structure to engage with.
The distinction being drawn is between producing an explanation of the performance and exposing the actual route. One treats the process as something that happened inside a fixed system and asks that system to characterize it afterward. The other treats the cognitive structure as something built visibly around the specific problem from the start.
The Question Pfungst Actually Asked
Clever Hans was not undone by giving wrong answers. He was undone by a researcher who declined to treat right answers as proof of the method. Pfungst watched Hans perform well, recognized that the crowd was satisfied, noted that the formal commission had found nothing suspicious, and decided that none of that settled the question he actually had.
The horse turned out to be doing something remarkable. His capacity to perceive involuntary human movement exceeded anything his observers possessed. But what he was doing was not what the blackboard said he was doing, and that distinction, invisible as long as everyone kept watching the hoof, was the whole thing.
The same structure is present now. A language model produces text that carries the texture of reasoning. The words are fluent, the conclusions often accurate, and the follow-up explanation sounds like introspection. The natural inference is that the process behind the words resembles the process behind human thought. That inference is doing a great deal of work that no one has earned the right to have it do.
More than a century ago, the investigation ended the moment one person stopped treating the count of hoof strikes as evidence of its own explanation and traced the actual path from question to tap. That same question is open now, in a context that reaches far beyond a courtyard.
Whether a system shows you the route while it is still taking it, or produces a story about the route once it has arrived, is not a secondary technical consideration. It is the question. Vertus was built around taking it seriously. Until more of the field follows, the part that actually matters will keep going unexamined.
