Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It.

A model can contain a decodable signal and not use it to produce its answer.

That sentence is the whole problem with interpretability, and it took me an embarrassingly long time to feel its weight. A linear probe recovers a concept from an activation. A sparse autoencoder isolates a direction that correlates with it. Neither shows that the direction did any work. The information is present. Whether it is load-bearing is a different question, and the two get conflated constantly.

Anthropic's Global Workspace paper is built around an instrument designed to separate them, and the instrument is more interesting to me than the findings. It is called the Jacobian lens.

What it replaces

The logit lens decodes a hidden state by pushing it through the model's unembedding matrix. Cheap, unsupervised, and brittle. In the workspace paper's framing, that amounts to setting the Jacobian to the identity: you assume the path from this layer to the output is a straight line, when it is not.

The tuned lens improves on that by training a small affine probe per block to match the output distribution. More reliable, and still correlational: it tells you what a layer's state resembles, not what the model does with it.

The Jacobian lens replaces the identity with the model's own averaged Jacobian from each layer to the final layer, computed over many prompts. That change is what makes its directions testable rather than merely readable.

Four operations, and only one of them matters for trust

READ projects a hidden state through the Jacobian, normalizes, and unembeds it into a vocabulary distribution.

WRITE adds a scaled lens vector into the residual stream.

PATCH swaps the coordinates of two lens vectors inside the activation.

ABLATE zeroes the projection of the residual stream onto the most active lens vectors across a chosen band of layers.

READ is the one that looks like the other lenses. The other three are the point. An interpretability claim earns trust when someone edits the internal quantity and observes the predicted behavior change, selectively.

The selectivity is not decoration. Swapping the internal pattern for "spider" to "ant" changes the model's arithmetic answer from eight to six. Swapping Soccer to Rugby flips the verbal report. A rhyming-couplet plan swap from Fight to Light shifts words that appear earlier in the line. In each case adjacent capabilities stay intact, which is what separates a real intervention from breaking the model and calling the damage a finding.

Two numbers that travel badly

This is where I want to slow down, because both of these get repeated without their scope, and I have seen both quoted flat.

The hundredfold connectivity. The sentence people cite is real: J-space patterns are read from and written to far more than ordinary patterns, "in some parts of the network by a factor of about a hundred." That phrasing is from Anthropic's plain-language research page. The technical paper reports the same property qualitatively, saying these vectors compose with downstream weights more broadly than other representational vectors. If you are citing the number, cite the overview, and keep "in some parts of the network" attached to it.

The ablation collapse. Remove J-space and multi-step reasoning drops to near zero. That is the headline, and it is accurate for that task. What travels less well is the other half of the same result: without its J-space the model still "speaks fluently, classifies sentiment, answers multiple-choice questions, and pulls facts out of passages roughly as before."

So the collapse is selective, not general. And there is a further wrinkle worth keeping: on grade-school math, chain-of-thought solving is substantially more robust to ablation than answering the same problems directly. The model externalizes its reasoning onto the page, and what is on the page does not need the workspace.

That last detail is the most interesting sentence in the whole area, and it almost never gets quoted.

The authors' limits, in their words

The paper is unusually forthright, and the limits are sharper than most summaries admit.

The lens only identifies vectors for concepts that are single tokens in the vocabulary. "San Francisco" and "machine learning" have no single unembedding direction. The limit is structural: the vocabulary indexes the lens.

Small integers fail to swap reliably, and the cause is unresolved. The paper leaves both hypotheses open: the model may compute over small integers outside the workspace, or its working representation of them may simply not align with the lens vectors. The evidence does not choose.

In roughly the first third of the model, readouts are noisy and largely uninterpretable. Whether that silence is genuine absence or lens degeneracy is explicitly left open.

And the summary judgment is theirs, not mine: the Jacobian lens is "an imperfect tool, which we believe only approximately and incompletely captures the model's underlying workspace structure."

There is also a circularity worth naming out loud. The lens is constructed to find verbalizable content, and then evaluated for further properties of that content.

Why any of this matters outside a weights lab

Here is the part I keep turning over.

An agent memory system has exactly the same problem, one layer up. You retrieve a fact and put it in the context. Did the model use it, or did it answer from its prior and leave your retrieved atom sitting there inert?

Retrieval metrics do not answer that. Hit rate says the record could be reached. Relevance scores say it resembled the query. Neither is the question. The question is whether the atom was load-bearing, and it is the same distinction the Jacobian lens was built to settle inside the network.

The instrument does not transfer. It needs weights, activations, and gradients. A production assistant is a black box: no weights, no activations, often not even per-token logprobs.

The obvious analogue is behavioral rather than mechanistic. Run the reader three times with the atom present, removed, and minimally corrupted, and judge the answer each time. If the atom entered the working set, the full condition differs from the other two in a predictable way. It is causal mediation applied to retrieval, and it would yield per-atom labels rather than a score: load-bearing, redundant, inert, harmful, contested.

I want to be careful about the status of that idea. It is a research question, not a method, and nobody ships it. The framing is mine and it is not a product.

The caveats are real, too. The removed arm is ambiguous: an atom can look ignored because it was unused, or because the answer was overdetermined by the model's prior and the atom simply had nothing left to contribute. The only mitigation I can think of is probing with facts the base model cannot know, which is more work than it sounds and which nobody has quantified.

So the honest summary is not a proposal. It is a gap.

Interpretability spent years learning that a decodable signal is not a used signal, and built an instrument that can tell the difference by intervention. Agent memory has not had that argument yet. We measure whether a fact can be found and call it done, and the distance between storing a fact and proving it was used is currently unmeasured.

Disclosure: I work on Mnemoverse, a memory engine for AI agents, so the last section is the one to weigh most carefully. Every claim about the Jacobian lens above comes from the paper or Anthropic's own research page, both linked, and the longer technical explainer is on the original page. Sources: the workspace paper, Anthropic's overview.