AI ‘thinking’ words it never says: What this tells us about consciousness

AI pioneer Geoffrey Hinton, meanwhile, has consistently said that chatbots have “subjective experience”, challenging the human belief in unique consciousness.

Milestones marking the advancement of AI.

Does the kind of “biological computer” that Australian startup Cortical Labs seems to be developing — a computer with human neurons mounted on silicon — potentially hold greater promise than Claude’s most recent experiment for achieving some sort of sentience? 

“The kind of bio-hybrid computers that Cortical Labs and others are building may well move the needle on the potential for consciousness. The reason is that, all else equal, the more similar our technologies are to real brains, the more plausible consciousness becomes.  The challenge is that we don’t yet know what difference makes the difference,” Seth told The Indian Express.

Inside Claude’s internal processing

So, what exactly does this step-by-step thinking inside the model look like? According to Dave Bergmann, when a reasoning LLM shows you its reasoning steps, that output is the equivalent of someone thinking out loud. (In technical terms, it’s reasoning in the token space.) “Imagine yourself cooking a simple meal and narrating every step out loud, like you’re hosting a cooking show: how you know when the pan is hot enough, why you’re adding salt after a quick taste test, how you’re cutting this vegetable and why. You’re still not saying every thought aloud. There’s a distinct inner monologue in your head that doesn’t get fully or faithfully broadcast in words,” according to Bergmann.

For instance, in the experiment on Claude Sonnet 4.5, Anthropic researchers asked the LLM to count to five and, at the same time, to “introspect deeply”. The AI model complied, dutifully coming up with 1, 2, 3, 4, 5. During this task, the researchers were observing what was going on inside the LLM’s many layers of artificial neural networks.

For a long time, these layers have remained largely impenetrable. But what surprised the researchers when they parsed the model’s internal activations while it counted was that different words kept popping up in the underlying layers of the model. “Countdown” was one such word. Then midway through the output, the word “halfway” appeared. After the model belted out the number five, it said nothing else to the researchers, but the word “done” appeared in its neural layers. 

Anthropic interprets this as evidence of an internal thought process, with representations related to Claude’s eventual outputs but not visible to the user.

Seth is clear in his view that biological traits may be necessary to produce consciousness. And we’re nowhere close to that. Anthropic said its J-space experiments do not show that Claude can have experiences or feel things in the way humans do, meaning there is still no evidence of consciousness. But the J-space appears to support some functions associated with conscious access. It holds the thoughts Claude can report on, deliberately bring to mind, and reason with, while the rest of its processing runs automatically beneath.

“This work is just a first step in what we expect to be an extensive line of research. The J-space looks like a good candidate for the divide between consciously accessible and unconscious processing in a language model, but we’d be surprised if it’s the whole story… And there remain many mysteries about how the J-space works. We don’t know what mechanism decides what enters the J-space in the first place. We’ve seen hints that it’s tied to Claude’s sense of self, something like emotional reactions, and traces of metacognition, without exactly having worked out how. But we now have methods for tackling questions like these. As that work progresses, our understanding of LLM minds — and their relationship to our own — will grow clearer,” Anthropic said in its blog post.