How LLMs Think About Space and Time

Do language models actually "think" and reason about the world, or do they merely mimic such abilities by picking up on patterns in data? This question lies at the heart of much of the research aimed at unraveling the inner workings of large language models (LLMs). A new paper titled "Language Models Represent Space and Time" probes how concepts fundamental to human cognition—space and time—emerge in the learned representations of LLMs. 

Wes Gurnee & Max Tegmark of Massachusetts Institute of Technology discover that despite not being directly trained to reason about space and time, LLM models develop specialized neural pathways responsive to spatial and temporal relationships. Even weirder, when they probed the interior of these networks they uncovered spontaneous emergence of individual neurons specifically attuned to spatial relationships like "left" and "right" or temporal orderings like "before" and "after”. This suggests certain core elements of a reasoning mind can arise spontaneously with increasing scale, bringing us closer to understanding how higher-level intelligence may potentially emerge in AI.

How do humans model the world? Does AI need to be embodied in order to understand our physical environment? It was Barbara Tversky who alerted us to the importance of spatial reasoning in human cognition. She demonstrated that human spatial and temporal cognition relies on active mental modeling rather than passive perception. We construct abstract mental frameworks that structure our understanding, imagination, and reasoning about space and time. 

Tversky revealed the subjectivity inherent in human comprehension—our spatial and temporal models incorporate distortions, conceptual framing, and systematic biases that diverge from objective reality. Her work unpacked "common sense" spatial and temporal reasoning humans depend on. She showed that abstraction enables the flexible mental transformations essential for imagination and planning, despite leading to biased modes of thought that violate logic. By rigorously characterizing the flaws and flexibility of human spatial and temporal reasoning, Tversky provided profound insight into the nature of human common sense—how we build useful but imperfect models of space and time through active cognitive processes grounded in perception but distorted by subjectivity. Tversky also demonstrated how spatial reasoning is influenced by conceptual framing and description, rather than purely abstract spatial relationships. 

The specialized spatial and temporal neurons uncovered in this research parallel findings by Tversky on the critical role of mental models in human spatial reasoning. The emergence of innate spatially-sensitive neurons in language models suggests AI systems may be developing their own implicit spaces and frameworks statistically derived from language data. 

While still very primitive compared to human cognition, these innate spatial and temporal encodings hint at neural networks developing their own latent knowledge structures. As models grow in scale and capability, more sophisticated internal representations may form in the network that take on abstract qualities reminiscent of human concepts and categories. 

This raises profound questions—could structures mimicking our own evolved conceptual understanding start to take shape autonomously in AI systems as they expand? Or perhaps there are hard constraints on emergent cognition within the inherent framework of model architectures today. The parallel discoveries of time and space sensitive neurons in both biological and artificial systems suggest deep commonalities in how neural representations can self-organize. As models continue to scale, internal representations drifting farther from human-intended design become a possibility. And this may impact AI safety—we know how to look for concepts such as “time” and “space” but we can’t look for alien concepts that only an AI can “know”.

The methodology in this research is also important. Rather than just testing overall model accuracy, the researchers analyzed the LLM's internal representations using probes. Probes are a way to peer inside the black box of large language models to empirically examine what knowledge has been embedded in the learned neural network weights.

Specifically, the researchers trained probes to predict spatial and temporal relationships based on patterns of activation within the model. The success of these probes in inferring relationships from the model’s hidden states provides significant evidence that concepts of space and time are encoded in the representations learned by LLMs.

This probing methodology enables researchers to move beyond treating advanced AI systems as impenetrable black boxes. By opening up the model and inspecting its inner workings, we gain critical insights into emerging properties of the representations learned based solely on exposure to linguistic data.

If spatial reasoning forms a cornerstone of human cognition, yet contains inherent biases as shown by Tversky, what should we make of AI gaining these skills? The paradox arises from evolution optimizing for efficiency over accuracy. While distorting objective reality, biases like compressing unimportant distances improved real-world navigation and decision-making with limited brainpower. Our spatial heuristics are "good enough" for survival, not mathematical perfection. 

So while absorbing the impressive flexibility of human spatial thinking could benefit AI, models statistically learning these abilities may implicitly encode the useful shortcuts that violate geometry. As AI grows more capable of imagination and abstraction, and as spatial computing becomes our UX, we will want grounded spatial representations without our quirks. 

As always, AI should build on human skills while overcoming our gaps. With the right foundations, AI could combine imaginative mental modeling with rigorous spatial logic—comprehending the world through both flexible human-like reasoning and unbiased geometric relations. If designed well, AI spatial intelligence could build on the strengths of human cognition while correcting innate weaknesses resulting from our evolutionary need for efficient approximation over accuracy.

This research marks an important moment in AI's evolution towards sophisticated intelligence. Envisioning a future where AI doesn't just process but 'imagines' spatial worlds akin to human mental models, the potential for a transformative experience emerges. This hinges on designing interfaces that guide us through AI's conceptualized landscapes, inviting us to explore and interact with machine-learned spatial realms.