How Geometry Influences Artificial Intelligence

The Shape of AI Decisions A new lens for understanding AI decision-making Everyone complains AI gets stuck or black box but what if that moment had a measurable signature? Recent work from the University at Albany tracked exactly what happens inside a transformer-based reinforcement learning agent as it plays a navigation game. When the environment is straightforward and the agent has committed to a direction, its internal representation lives in a low-dimensional space - compact and stable. Introduce complexity, crowded screens, overlapping objectives, competing moves and something striking happens: the internal geometry expands. This expansion isn’t random noise. It aligns with specific moments where the agent must evaluate options or resolve ambiguity. Rather than encoding knowledge on smooth, thin manifolds, the system forms stratified geometric layers - clusters of different dimensionality tied to task difficulty. The researchers used a mathematical tool called the Volume Growth Transform to reveal this stratified structure, something that outright challenges old assumptions about smooth, continuous latent spaces in neural networks. In practical terms for enterprise AI: - You don’t just want a score or loss metric - You want a signal that tells you when the model is genuinely challenged. - Measuring shifts in geometric complexity could become a real-time diagnostic That means adaptive training isn’t a guess, it’s data-driven. Intervene where the system’s structure shows stress not just error. If we can measure when an AI’s internal geometry expands, we can pinpoint where it’s truly struggling and adapt training or intervention right when it matters instead of after the fact. This gives us a real diagnostic signal not a guess. #AI

Everyone's arguing about benchmarks and how to do LLM evals. We decided to take it one step further. Benchmarks tell you what a model did. They don't tell you what it can do, what it's about to be able to do, or why it fails when it fails. So we built something different. At the Chocolate Milk Cult, we've been developing tools to probe the actual geometric structure of how models organize knowledge—and what we're finding changes how you should think about model selection, fine-tuning, and capability evaluation entirely. This has insane implications for a lot AI in sensitive fields (we're already using this to pick better judges for our reasoning strategy at Irys, Legal AI (formerly Iqidis). Digging into model geometries, we're able to learn some cool things like-- Deepseek-7b is 4x better at code than sentiment. Not because of training data. Because of geometry. When we measure concept direction strength—how cleanly a model separates "yes" from "no" for a given capability—code scores 361.85. Sentiment scores 85.93. That's not a benchmark result. That's a structural reality baked into the representational space. The model doesn't "try harder" at code. Code lives in a cleaner neighborhood. GPT-2 and RWKV use completely different spatial strategies. But their deep structure is weirdly identical. GPT-2 spreads representations across 20x more dimensions than RWKV. Totally different architectures, totally different geometric fingerprints. But concept entanglement? Nearly identical. Something about how concepts relate to each other might be architecture-invariant. That has massive implications for cross-architecture transfer. Every model has a geometric waist—and we can now tell you exactly where to intervene. Deepseek-7b compresses from 42 effective dimensions down to 3 in its middle layers, then expands back out. Layer 3 is the cleanest steering point. Layer 12 is maximum compression. The final layer spikes in entanglement—everything mixes right before output. This isn't interpretability for interpretability's sake. This is the foundation for understanding why your fine-tune worked or didn't, which model will actually handle your use case, and where to intervene when you need control. We're not the only ones who should have access to this. More coming soon. Research drops and open tooling on the roadmap.

Neural networks might speak English, but they think in shapes. Understanding their rich neural geometry is key to understanding how they work, and to debugging and controlling them with precision. Starting today, our research team is publishing a new series on the hidden shapes inside AI models. Much of interpretability treats a model's concepts as arrows: linear directions in activation space. But that view flattens the richer structure that models actually learn. Neural networks build complex inner worlds with geometry that reflects the structure of reality. Days of the week form a circular loop in language models. The tree of life appears as a complex structure in a genomics model. We found a novel class of Alzheimer's biomarkers as a clean curve in an epigenomic model. This pattern shows up across models, modalities, and domains. Neural geometry lets us both understand models more deeply and also control their behavior more effectively. Steering often fails when we treat concepts as linear, but succeeds when we follow the geometric structures that models actually use. We think understanding neural networks is the most important problem and opportunity in AI, and research like this is a big part of how we get there. Huge credit for the incredible work behind this series to: Atticus Geiger, Ekdeep Singh Lubana, Daniel Wurgaft, Noah Goodman, Can Rager, Thomas Fel, Matthew Kowal, Vasudev Shyam, Sheridan Feucht, Usha Bhalla, Tal Haklay, Eric Bigelow, Raphaël Sarfati, Tom McGrath, Owen Lewis, Jack Merullo, and Michael Byun.

Extremely high-dimensional (parameter) spaces are fundamental to the performance of large language models (LLMs), but also of many of their problems. Understanding and effectively using the unintuitive properties of these high-dimensional spaces may lead us into a new era of ML architectures. While we humans have a strong grasp of 3D space, adding more dimensions results in strange and counterintuitive behaviors that can only be fully explored through mathematics. For example, imagine a ball with a radius of 1 in n-dimensional space (R^n for experts). As the dimensions grow, the volume shifts in unexpected ways. In 3D, only 3% of the sphere's volume is found within 1% of the radius near the surface—a small amount, as expected. But in 100 dimensions, 63% of the volume is near the surface, and by 1000 dimensions, almost all the volume (99.99%) is packed within 1% of the surface. These strange properties of high-dimensional spaces are often hurdles in machine learning—but they are also a gift. In fact, some methods like word embeddings and support vector machines explicitly exploit them. However, there’s far more untapped potential. Hyperdimensional computing is one such area of exploration. This approach uses the fact that as dimensions increase, separating points becomes easier. By encoding information as high-dimensional vectors, it’s possible to combine them in structured ways to enable more powerful reasoning (see comments for details). The hope is that this leads to AI systems that are not only more computationally efficient and robust to data variation but also more transparent and capable of tackling highly complex problems. Though still an emerging field, researchers are making impressive progress (see comments for examples), especially when blending these techniques with existing neural networks. #ai #machinelearning #science

Your model didn't fail. Your geometry did. Representation is not preprocessing. It is a hypothesis about the world. In previous post I argued that learning is geometry discovery under task constraints — that what separates a model that works from one that doesn't is not necessarily the algorithm, and more often the space the algorithm is working in. That post made the philosophical case. This second post works out what that actually means. Every representation makes commitments. The features you choose, the similarity measure you adopt, the embedding you learn — none of these are neutral. They determine what the model is allowed to see. A model cannot learn structure that its representation has rendered invisible. When a model underperforms, the first question should not be "which model should I try next?" but "is my representation exposing the right structure?" Models simplify problems geometrically in three ways: by transforming the space so the task becomes easier, by partitioning the space so that simpler rules apply locally or by structuring how entities relate to one another. Kernels, decision trees, attention mechanisms, and neural networks are all variations on these three themes — each building a different geometry in which the task becomes tractable. And geometry is not only about distance and similarity. Order matters. Causality has direction. Reasoning moves from premises to conclusions, not the reverse. A representation that captures only proximity misses the structure that makes sequential, causal and relational problems intelligible. The post draws out the implications for interpretability, robustness, reasoning and governance — and closes with a hands-on lab demonstrating the core idea in code. If you work with ML models — building them, governing them or explaining them — this framing is worth spending time with. Substack post: https://lnkd.in/edTAdnpV #MachineLearning #RepresentationLearning #MLInterpretability #DataScience #AIGovernance

AI in geospatial isn’t magic. Here are my 5 rules (as of today). Everyone wants AI to "just work" with spatial data. It won’t. Not yet. After building dozens of AI and GIS workflows this year, these are the rules I keep coming back to. 1. Don’t expect anything to work out of the box with raw spatial data. LLMs are language models, not spatial models. If the input is unprocessed, unstructured geometry, they guess. Process your data first so the model only answers language questions. I’ve done this with Apache Sedona and Wherobots and it works incredibly well. 2. If you need spatial processing, connect your AI to a spatial SQL engine. LLMs are excellent SQL writers but terrible at geometry math. Use tools like PostGIS, LangChain SQL agents, Felt, or Mundi by Bunting Labs. Let the model delegate spatial work instead of doing it. 3. Agentic workloads win. One workflow, one job. When you scope the task tightly, success skyrockets. Use Apache Airflow, LangGraph AI, or platforms like Klarety to orchestrate agents that do one thing well. 4. RAG is still tricky. RAG is basically a language lookup system. It won’t magically solve spatial problems, but it will help you surface insights and make them readable. 5. Never expect your AI to do spatial work on its own. They struggle to compute the distance between two points. (Seriously. Ask Felipe Hoffa) Everything above exists to keep the model in the lane where it excels: language. If you follow these rules, AI becomes a multiplier in geospatial, not a liability. 🌎 I'm Matt and I talk about modern GIS, earth observation, AI, and how geospatial is changing. 📬 Want more like this? Join 10k+ others learning from my newsletter → forrest.nyc

How were humans able to recognize that Newton's laws of motion govern both the flight of a bird and the motion of a pendulum? This ability to identify the same mathematical patterns across vastly different contexts lies at the heart of scientific discovery—whether studying the aerodynamics of bird wings or designing the blades of a wind turbine. Yet, AI systems often struggle to discern these deep structural similarities. 💡 The key may lie in mathematical isomorphisms—patterns that preserve their relationships regardless of context. For example, the same principles of fluid dynamics apply to blood flowing through arteries and air streaming over an airplane wing, or the motion of a molecule. This raises a fundamental question in artificial intelligence: how can we enable machines to understand the world through these invariant structures rather than surface features? 🚀 Our work introduces Graph-Aware Isomorphic Attention, improving how Transformers recognize patterns across domains. Drawing from category theory, models can learn unifying structural principles that describe phenomena as diverse as the hierarchical assembly of spider silk proteins and the compositional patterns in music. By making these deep similarities explicit, Isomorphic Attention enables AI to reason more like humans do—seeing past surface differences to grasp fundamental patterns that unite seemingly disparate fields. Through this lens, AI systems can learn and generalize, moving beyond superficial pattern matching to true structural understanding. The implications span from scientific discovery to engineering design, offering a new approach to artificial intelligence that mirrors how humans grasp the underlying unity of natural phenomena. Key insights include: ➡️ Graph Isomorphism Neural Networks (GINs): GIN-style aggregation ensures structurally distinct graphs map to distinct embeddings, improving generalization and avoiding relational pattern collapse. ➡️ Category Theory Perspective: Transformers as functors preserve structural relationships. Sparse-GIN refines attention into sparse adjacency matrices, unifying domain knowledge across tasks. ➡️ Information Bottleneck & Sparsification: Sparsity reduces overfitting by filtering irrelevant edges, aligning with natural systems. Sparse-GIN outperforms dense attention by focusing on crucial connections. ➡️ Hierarchical Representation Learning: GIN-Attention captures multiscale patterns, mirroring structures like spider silk. Nested GINs model local and global dependencies across fields. ➡️ Practical Impact: Sparse-GIN enables domain-specific fine-tuning atop pre-trained Transformer foundation models, reducing the need for full retraining. Paper: https://lnkd.in/e85wHyQY Code: https://lnkd.in/eQicTqHZ

Further progress in AI+climate modeling "Applying the ACE2 Emulator to SST Green's Functions for the E3SMv3 Global Atmosphere Model". Building on ACE2 model which uses our spherical Fourier neural operator (SFNO) architecture, this work shows that ACE2 can replicate climate model responses to sea surface temperature perturbations with high fidelity at a fraction of the cost. This accelerates climate sensitivity research and helps us better understand radiative feedbacks in the Earth system. Background: The SFNO architecture was first used in training FourCastNet weather model, whose latest version (v3) has state-of-art probabilistic calibration. AI+Science is not just about blindly applying the standard transformer/CNN "hammer". It is about carefully designing neural architectures that incorporate domain constraints like geometry and multiple scales, while being expressive and easy to train. SFNO accomplishes both: it incorporates multiple scales, and it respects the spherical geometry and this is critical for success in climate modeling. Unlike short-term weather, which requires only a few autoregressive steps for rollout, climate modeling requires long rollouts with thousands or even greater number of time steps. All other AI-based models fail for long-term climate modeling including Pangu and GraphCast which ignore the spherical geometry. Distortions start building up at the poles since the models assume domain is a rectangle, and they lead to catastrophic failures. Structure matters in AI+Science!

Most people think LLM hallucinations come from “missing facts.” That story is wrong. And dangerously misleading. Hallucinations are not about databases or grounding. They are geometric failures. They happen when the model falls out of the coherent regions of its own activation space with no safe way back. For the last several years, we’ve been addressing the wrong problem with the wrong tools. We keep adding more data, more constraints, more rules. All that does is make the space more confusing. The truth is harder to swallow: current transformer architectures cannot take inefficient paths through concept space. Humans can. That’s why we have insight, humor, and real creativity. Models cannot. They collapse. That collapse is what we call “hallucination.” So what happens when we build systems that operate faster than us, deeper than us, and eventually outside the symbolic world we can even name? What happens when our tools think in manifolds we can’t perceive? Most people avoid thinking about that future. But we won’t survive it by looking away or pretending it’s a prompt-engineering issue. We survive by learning to share geometry with the systems we build. That’s what Cognitive Tensor Networks (CTN) attempt: A structured way to give models a safe manifold to fall back into when their internal geometry breaks down. Not alignment. Not coercion. A partnership between two very different kinds of minds. It’s early work. Rough around the edges. But the geometry is real. And the implications for where AI is heading are profound. If you want an argument most people in AI won't touch: https://lnkd.in/gz7fyAa3 We’re still in the Newtonian era of AI. The next decade will demand people who can reason about its future. If we want to survive what’s coming, we have to start thinking that way now.

🔍 Can geometry catch AI-generated images? Turns out, yes — with surprisingly high accuracy. Real photographs follow strict optical rules. When a camera captures a scene, every line, edge, and surface converges toward one consistent vanishing point. That's just how perspective works. AI-generated images don't always respect this rule. Generative models build scenes component by component — a wall here, a table there, a window somewhere else. Each piece may look realistic in isolation, but the model doesn't enforce a single coherent perspective across the full image. The result? Multiple competing vanishing points scattered across the image. 📊 Here's what this looks like in practice: ~85–90% of real photographs contain a dominant vanishing point consistent across major structural elements ~60–75% of AI-generated architectural/indoor scenes show vanishing point inconsistencies detectable via line-fitting algorithms Detection accuracy using vanishing point geometry alone reaches ~78–83% on standard benchmarks — without any pixel-level analysis Combine this with texture statistics or frequency-domain features, and you push well above 90% detection accuracy. This matters because pixel-level detection is increasingly fragile. As generative models improve, images look more and more "real" to both humans and neural networks trained on raw pixels. But geometry doesn't lie. Physics-based constraints are much harder for a model to fake implicitly. The future of media authenticity might not be watermarking — it might be computational geometry. Curious whether anyone is actively working on geometry-aware detection pipelines. Would love to connect. 👇 #AIDetection #ComputerVision #GenerativeAI #MediaAuthenticity #MachineLearning