The AIGS Pod

The AIGS
  • 5.0 (1)
  • Natural Sciences
  • Biweekly

The AIGS podcast tackling AI in earth system modeling and more.

  1. 2d ago

    What does attention do with a dynamical system?

    What does a transformer's attention actually do with measurements from a physical system? This paper takes the smallest transformer, one layer and mostly one attention head, and reads it with classical dynamical-systems theory. For a linear system, attention is an autoregressive model whose lag weights come from softmax, so they are positive and add up to one: a single attention-only head can only average its past, and a damped spring that needs weights of opposite sign comes out too smooth. A second head supplies the minus sign. For nonlinear systems seen through a single sensor, attention works like an adaptive delay embedding, in the spirit of Takens' theorem. We walk through the sign problem, how a short history stands in for a missing sensor (on the Van der Pol oscillator, the best transformer's error is more than a thousand times lower than that of an MLP that sees only the present value), why the model needs enough internal dimensions to unfold what it rebuilds (Chafee-Infante), and why flow past a cylinder goes better when the model is told the Reynolds number (without it, rebuilding the velocity field from the model's internal state is about 70 times worse). Then we ask whether good next-step forecasts mean a sound internal state, and what that suggests for machine-learned climate emulators (our extrapolation). Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's interpretability series (Nov 11). Paper: https://arxiv.org/abs/2512.21113

    What does attention do with a dynamical system?

  2. Nov 1

    Do circuits explain model errors?

    A circuit is a small piece of a neural network that is supposed to explain how the whole network does a task. The usual test asks whether the circuit still does the task, on average. This paper asks a sharper question: when the model gets an answer wrong, does the circuit get it wrong in the same way? Mostly, it doesn't. On indirect object identification in GPT-2 small, under mean ablation, the circuits tested agree with the model on 97 to 99.5 percent of the prompts it gets right, but on only 11 to 42 percent of the ones it gets wrong. We walk through the measurement, three attempts to explain the gap away (errors are rare, errors are close calls, the circuit is just too sure), a case study that adds 63 head positions to the classic circuit and reproduces about three quarters of the held-out errors, and the two omitted computations that make the wrong name win. Then we ask what a circuit is an explanation of, and what the same test would mean for climate emulators and their biases (our extrapolation). Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's circuits series (Nov 4). Paper: https://arxiv.org/abs/2609.35686

    Do circuits explain model errors?

  3. Oct 26

    Proving AI truly understands physics

    A fast, enthusiastic two-host pass through LAWFUL, the framework that checks a neural network against a known physical law: a transformer that turns motion capture into radar, the Doppler law it was never shown, counterfactual changes in speed, activation patching, and a circuit of temporal attention heads that seems to compare early and late moments. A companion to episode 25, which takes the same paper slowly and more skeptically. Listen critically: this episode swaps the two attention-patching numbers (patching only the attention pattern gives the smaller drop, 0.35; patching only the values gives 1.31), and it presents as proven several things the paper offers as suggestions, including that the model reinvented adaptive differentiation. Good material to argue about at the meeting (Oct 28). Paper: https://arxiv.org/abs/2607.28672

    Proving AI truly understands physics

  4. Oct 26

    When a network follows the law, has it learned the physics?

    A small transformer predicts radar from motion capture, and when the motion is changed in physically controlled ways, it responds the way the Doppler law says it should. LAWFUL (Law-Aligned Witness for Faithful Use of Latents) turns that into a test: it scores how a model's response to counterfactual inputs tracks a known law, then uses activation patching to find the circuit that carries the response. We walk through what its 0.988 consistency score does and doesn't constrain (a network off by a constant factor would still score perfectly), how far the tested range reaches, the circuit and its weak necessity control, the probes, and the idea that attention heads pick which moments to compare. Then we ask what evidence would justify saying a model has learned the physics, including for climate emulators. Narrated by a synthetic voice (Kokoro, "Bella"). This continues the journal club's circuits series (Oct 28). Paper: https://arxiv.org/abs/2607.28672

    When a network follows the law, has it learned the physics?

  5. Oct 17

    Mapping visual circuits to fix AI

    Finding that a network encodes something is not the same as knowing how it uses it. Edge-based circuits have been defined for large language models, but work on vision models has so far only used neuron-based circuits, which show what is encoded but not how it is routed. The authors introduce Automatic Visual Circuit Discovery (Vi-CD), which finds sparse, edge-based computational subgraphs in vision transformers. They apply it to class-specific classification circuits, to the circuits behind typographic attacks on CLIP, and to circuits that can be steered to correct harmful behavior: on the RoCOCO benchmark, steering the discovered circuits reduces safety violations more than random ablations while keeping retrieval performance close to baseline. This opens the journal club's circuits series (Oct 21). Paper: https://arxiv.org/abs/2604.14477

    Mapping visual circuits to fix AI

  6. Oct 13

    How AI weather models reinvented physics

    Could it be that AI weather models are solving physical equations, although they may not be the equations used by conventional NWP models? The authors compute correlations of forecast skill and Centered Kernel Alignment, providing evidence that different AI weather models represent the atmosphere in similar ways, despite differences in architecture and capacity. They argue that the architecture and training of the AI models constrains the form of the physical laws that they might simulate. In particular, they propose that the models implement a particle description of the atmosphere, where the latent variables at each mesh point correspond to the position of a particle in the high dimensional latent space. They hypothesize that the movement of the particles follows a gradient flow in the latent space towards a minimum of a learned free energy functional. Analysis of the GraphCast and Aurora models show that they make changes on large spatial scales in the early processor layers and move to smaller scale with increasing layer depth, consistent with the gradient flow hypothesis. Paper: https://arxiv.org/abs/2605.23778

    How AI weather models reinvented physics

  7. Sep 13

    How AI predicts weather without physics

    Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical to understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here the authors present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying the tool to the graph neural network GraphCast, they present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool's ability to identify linear combinations of latent channels that appear to correspond to interpretable features. Paper: https://arxiv.org/abs/2604.20467

    How AI predicts weather without physics

  8. Sep 12

    How AI weather models actually think

    Artificial Intelligence (AI) weather models are improving rapidly, and their forecasts are already competitive with long-established traditional Numerical Weather Prediction (NWP). To build confidence in this new methodology, it is critical to understand how these predictions are generated. This is a huge challenge as these AI weather models remain largely black boxes. In other areas of Machine Learning (ML), mechanistic interpretability has emerged as a framework for understanding ML predictions by analysing the building blocks responsible for them. Here the authors present an open-source, highly adaptable tool which incorporates concepts from mechanistic interpretability. The tool organises internal latent representations from the model processor and allows for initial analyses, including cosine similarity and Principal Component Analysis (PCA), enabling the user to identify directions in latent space potentially associated with meteorological features. Applying the tool to the graph neural network GraphCast, they present preliminary case studies for mid-latitude synoptic-scale waves and specific humidity. These demonstrate the tool's ability to identify linear combinations of latent channels that appear to correspond to interpretable features. Paper: https://arxiv.org/abs/2604.20467

    How AI weather models actually think
See All (28)

About

The AIGS podcast tackling AI in earth system modeling and more.

Information

  • Creator
    The AIGS
  • Years Active
    2026
  • Episodes
    28
  • Rating
    Clean
  • Show Website
    The AIGS Pod