Watching Language Models Think
For years, understanding the internal workings of large language models (LLMs) has been akin to peering into a black box. We provide input, and an output emerges, but the precise cognitive steps, the intermediate reasoning, remained largely opaque. Now, a new development is peeling back that layer of mystery. Inspired by Anthropic's recent 'J-space' paper, a developer has created an open-source tool that visualizes the 'silent words'—emergent internal concepts—that LLMs utilize before generating any visible output.
The J-space paper introduced the concept of these 'silent words,' a small, emergent set of internal concepts that LLMs can report on, control, and use for reasoning. These concepts represent a fraction of the model's total activations, typically less than 10%, yet they appear to be crucial for complex thought processes. The key innovation enabling this insight is the 'Jacobian lens,' a measurement tool that Anthropic developed. This lens, now open-sourced, provides a window into the model's internal state.
Neuronpedia has already provided pre-fitted lenses for the Qwen model, making it easier for researchers and developers to apply this technology. The natural progression was to integrate this powerful new tool into a user-friendly interface, allowing for real-time observation of an AI's internal 'thought' process. This is precisely what the new project achieves: it wires the Jacobian lens into a chat UI, enabling users to literally watch the model think as it processes input and formulates a response.
Real-time Insight into AI Reasoning
The tool, built using Python and integrated with a chat interface, runs the Qwen3.5-4B model on a single 12GB GPU using bf16 precision. It reads the Jacobian lens at nine layers for every token generated, both while the model is parsing the user's message and while it is constructing its reply. Crucially, this process happens at full generation speed, as the lens operation—essentially a matrix multiplication and unembedding per layer—is computationally inexpensive.
This real-time capability offers unprecedented visibility. A particularly striking example highlighted by the project's creator involved a simple arithmetic query: 'is this correct? 12 + 5 = 1'. As the model processed this input, the internal concept corresponding to 'incorrect' lit up within the network *before* any reply tokens were generated. At that exact moment, the model had internally determined the answer was wrong, even though it hadn't yet begun to output any text. This demonstrates that the model's verdict exists internally, as a discernible concept, long before it's communicated externally.
This capability moves beyond simply observing output probabilities or attention weights. It provides a glimpse into the abstract conceptual space the model navigates. Instead of just seeing which words are likely to come next, we can now see which internal ideas or concepts are being activated and manipulated to arrive at that decision. This is analogous to watching a human solve a problem, not just hearing their spoken answer, but observing the flash of insight or the moment of realization in their eyes.
Implications for AI Development and Understanding
The ability to visualize these 'silent words' has profound implications. For developers building AI applications, it offers a powerful debugging tool. Understanding why an LLM might produce an unexpected or incorrect response becomes easier when you can see the internal reasoning path it took. This could lead to more robust and reliable AI systems, as developers can identify and correct flawed internal logic more effectively.
For researchers, this opens new avenues for studying emergent properties of LLMs. The J-space paper suggested these silent words are 'emergent,' implying they are not explicitly programmed but arise spontaneously as models scale. Visualizing them allows for empirical study of this phenomenon. Researchers can investigate how these concepts form, how they are used in different reasoning tasks, and how they vary across different model architectures and sizes. This could be key to developing more interpretable and controllable AI.
The project also raises fascinating philosophical questions. If an AI can internally 'know' an answer before it's articulated, what does that imply about consciousness or internal states? While this is far from sentience, it does provide a concrete, observable manifestation of internal 'understanding' within a non-biological system. It forces us to reconsider what it means for a machine to 'think' or 'reason,' moving beyond black-box outputs to observable internal processes.
The open-sourcing of the Jacobian lens and the creation of this visualization tool are significant steps. They democratize access to a powerful new method for probing LLM internals. As this technology matures, we can expect a new era of AI development and research, one where the internal 'thoughts' of artificial intelligence are no longer entirely hidden from view.
The Future of AI Transparency
This development addresses a critical need in the AI community: transparency. As LLMs become more integrated into critical applications, from healthcare to finance, understanding their decision-making processes is paramount. The 'Jacobian lens' and its visualization in tools like this chat UI are early steps towards demystifying AI. They allow us to move from simply trusting the output to beginning to understand the process that generated it.
The implications extend beyond mere debugging. They touch upon the very nature of intelligence, both artificial and natural. By observing the 'silent words' that underpin AI reasoning, we gain a novel perspective on how complex problem-solving can emerge from vast neural networks. It's a powerful reminder that the most significant advancements often come from finding ways to see what was previously invisible.