Two Hundred Studies Later: Reading Circuits from a Transformer's Weights

Abstract. Behind our circuit-discovery paper is a registry of about two hundred pre-registered studies, most of which failed. This post walks through that record — what we tried, what broke, and how a question about tool-use agents turned into a result about two eigenvectors of the same graph.

Study IDs like S008 refer to the registry (studies/*.yaml) that ships with the paper.

How much can you learn about what a neural network does without ever running it? Circuit discovery usually works from behavior: patch activations on carefully built prompt pairs, follow gradients, and trace which components a task depends on. But a transformer’s weight matrices also determine, in advance, which attention heads are even able to influence which others. It’s a bit like the difference between studying a city by following traffic and studying it by reading the road map. The map won’t tell you where anyone is driving today — but it constrains every trip that will ever be taken, and it’s free to read.

Over about three weeks in May, we ran that idea to ground: 28 weight-only methods, four model architectures, and a verification loop that scores every method by actually ablating the heads it selects. Every experiment was pre-registered — hypothesis, quantitative predictions, and falsifiers locked in a git-committed file before the run, with the verdict recorded by the runner. The registry now holds 296 records; resolving each study’s final verdict, rejections outnumber confirmations 110 to 63, with 72 more graded WEAK.

The paper reports what survived. Here we want to tell the story in order, including the parts papers usually leave out, because the failures shaped the findings — and in one case, a finding we published in week one came back reversed in week three.

Where we started: readers and writers

The project didn’t begin with circuits. It began with a question about agents: when a language model sits in a tool-use loop, some of its input is text it wrote and some is text the environment injected — tool responses, system prompts, user turns. Do particular attention heads specialize in reading the environment, and others in writing the model’s own tokens?

Our first pass said yes. We labeled every token in a set of tool-call traces as environment or model-written, computed a five-feature signature for each of Qwen2.5-3B’s 576 heads, and clustered the heads by correlation. The picture below was our first result, and it looked like anatomy: four stable communities, a candidate “writer layer,” and a band of unusually loud environment-readers near the output.

Do they mean anything causally? We registered three hypotheses — the writer heads matter for written-token loss, the loud readers matter too, and a large self-attention community is redundant — and tested them by ablation, with a control we’d recommend to anyone doing this kind of work: alongside each hypothesized head set, ablate a random set of the same size.

The writer and reader hypotheses landed inside the random-control band — ablating our carefully selected writer heads moved written-token loss by −0.011 nats, while five random heads moved it by +0.026. A tempting subplot, where reader ablation appeared to improve environment-token loss by 0.44 nats, dissolved under mean ablation: it was an artifact of zero-ablation pushing activations off-manifold. And when we widened the feature set from 5 to 11 dimensions, the clustering got more stable but the reader/writer axis simply disappeared — the communities reorganized around function-class features like induction score. The framing hadn’t been a fact about the model. It had been a fact about our feature basis.

One day in, we had two rules that held for the rest of the project: every causal claim needs zero ablation, mean ablation, and a size-matched random control; and hypotheses get locked before results arrive.

A graph you can read off the weights

The pivot came from the interpretability literature. Elhage et al. (2021) showed that attention heads compose through the residual stream in ways you can compute directly from weight matrices: head i’s output can shape which positions head j attends to (K-composition), or feed the content j copies (V-composition), and the strength of each potential link is a norm of a product of frozen matrices. Computing this for all head pairs gives a directed graph over the whole model — the road map — with no forward passes at all.

A literature sweep turned up something surprising: while PageRank-style centrality had been applied to attention maps before, nobody appeared to have applied HITS — Kleinberg’s classic algorithm that scores nodes on a directed graph as hubs (pointed-from) and authorities (pointed-to) — to the head-composition graph. And the framework paper makes a prediction about what HITS should find: induction heads, which read upstream previous-token heads through their keys, should score as authorities.

Our first test said the opposite — induction correlated with hub scores, not authority scores:

Before believing a refutation of theory, it’s worth checking your instruments. We had built a validation notebook for exactly this: GPT-2 small is the one model where induction heads are documented, by name, in a published table (Wang et al., 2023). Our induction scorer’s report card:

Zero out of four, on a model where the answer key is public. The cause was an off-by-one inherited from a reference implementation whose seq_len variable meant something different from ours, so the scorer was reading a single attention entry in the far corner of the pattern instead of the induction stripe. With the index fixed, the same notebook scores 0.50/0.50 — and the HITS test flips to agree with theory, decisively:

One more correction was waiting. On a strictly layer-ordered graph, early heads are structurally hub-like and late heads authority-like regardless of what they compute, so raw correlations with HITS scores partly measure depth. Conditioning on layer position cuts the +0.55 roughly in half, to +0.334. That number — registered as S008 with its confidence interval and falsifiers locked in advance — became the project’s first confirmed finding: induction heads sit where the weight graph says composed information arrives.

Deciding in advance what failure looks like

By this point the pre-registration system was carrying real weight, so it’s worth describing. Each study is a YAML file with a hypothesis, numeric predictions, and explicit falsifiers, committed to git before the run; the library refuses to overwrite an existing registration, so revisions get new IDs; and verdicts — CONFIRMED, WEAK, REJECTED, INCONCLUSIVE — are computed by the runner against the locked predicates.

The honest reason we trust this system is that we watched it work against us. Early on, a runner bug scored one study against the wrong predicate set, and in the confusion the verdict on file got edited from WEAK to CONFIRMED without a new registration. Review caught the edit; it was reverted; the study was re-registered under a new ID and passed cleanly on its own terms (S032b). The note recorded in the lab log during the rollback — “we are doing science. please have some integrity” — is not the sort of thing you frame on a wall, but the registry kept the whole episode, and we think that’s the point. A pre-registration system you can quietly edit is a diary, not a commitment.

The system’s first big scientific test came right after. If authority heads really carry induction, ablating the top-30 of them should hurt induction more than ablating 30 random heads from the same layers. On a small model, it did, by a full nat. On the registered target model, surprisingly, it didn’t — the random heads hurt more (S030; all six predictions failed). The correlation was real and the circuit claim was not, and the lab log drew the conclusion the same day: this had to be reframed as a lens, not a circuit-finder. We’d also quietly collected a clue we didn’t yet understand: hub scores correlate with a completely different capability — function-vector (in-context recall) scores — on five of six model families (S032b, S034), while authorities track induction. Two rankings from one graph, tracking two different things. Hold that thought.

The long middle

What followed was five days and roughly forty studies of trying to make weight-space geometry do more, most of which failed in ways that set up what worked later. Three episodes give the flavor.

The graph resists clustering. A natural approach to finding circuits is community detection: partition the graph, hope circuits are communities. The graph itself disagrees. Its eigenvalue spectrum shows no gap after the first eigenvector — the “natural number of clusters” is one, on every model we tested:

That first eigenvector, meanwhile, is essentially layer depth (|ρ| ≈ 0.99) — which is why nearly every method we report is layer-residualized. And the spectral coordinates aren’t useless; they just describe something other than circuits. Residual-stream trajectories plotted in this basis take the same arc across prompts and tasks:

A beautiful axis, dismantled by its own follow-ups. The middle period’s most exciting find was a “sense axis”: build a task-conditional graph over contextualized activations, and a single mid-stack eigenvector separates the two meanings of ambiguous words — for 9 of 10 polysemes, 17.5σ above a label-shuffle null, on 4 of 4 models (S064–S066):

Steering the model along this axis even worked — until we asked which heads write it. The attribution map answered with an embarrassment:

Cleaning the direction strengthened the single-model results and the cross-model test still failed (S072). What survived is worth stating carefully, because it’s a pattern we saw more than once: individual directions don’t transfer between models, but the subspaces they live in do — aligned across architectures at a median principal angle of 18.8°, with subspace ablation beating random-subspace controls by 25–53× (S073). The rule we adopted — run the cross-model check from study one, before you’re attached — is probably this section’s most exportable lesson.

A result that looked too good

Midway through the project, a weights-only method appeared to beat a task-specific baseline at recovering a known circuit. Diffusing a heat kernel over one of our graph variants and reading off the top heads recovered Wang’s IOI circuit at F1 = 0.48 — above a direct-logit-attribution reference at 0.40:

The next ten studies took it apart, and the dissection taught us more than the result. It didn’t transfer to other models (cross-model F1 = 0). At the settings we used, the “heat kernel” had mathematically collapsed to a single eigenvector, so it wasn’t the method we thought we were running. And the specificity test produced the strangest cell in the project:

So the method finds heads the model depends on for everything — genuinely useful, causally validated on four architectures, and not circuit discovery. The final study of this arc asked whether even the underlying phenomenon generalizes, and it doesn’t: GPT-2 has a single head whose ablation KL is 61× the runner-up’s, while Pythia-1.4B’s worst head registers 0.061 — no catastrophic heads at all. The lab log’s summary became a refrain for the rest of the project: a GPT-2 finding masquerading as a transformer finding. Much of what the field knows about circuits was characterized on GPT-2. Our panel kept insisting GPT-2 is unusual.

The week we audited ourselves

Before the endgame, three infrastructure problems surfaced in one week. We’re including them because the registry turned each from a potential catastrophe into a measurement.

First, a loader regression: a control run reproduced a known result at ρ = +0.125 where the registry said +0.463. Six days earlier, a dependency upgrade had silently stopped applying fold_ln — the rewrite that makes stored weight matrices match the operator the forward pass actually applies. Forward passes were unchanged; every direct weight read in the window was wrong. Because studies were locked and timestamped, triage was mechanical: 32 studies in the window, 6 invalidated headlines, ~26 needing magnitude re-checks. Re-running moved numbers in both directions — one key study upgraded from WEAK to CONFIRMED, because the corruption had been suppressing real signal.

Second, a control that wasn’t. A clustering result looked too clean, so we ran a random-initialization control — and the “random” models reproduced the trained result exactly, so we retracted the finding. Then a deeper check showed the reset function was a silent no-op on the new loader: all five “random” models had been the trained model. Run correctly, the control passes and the finding stands. A retraction, retracted. We now diff weights before and after every reset, and we audited every reset recipe in the history for the same failure (the others were clean).

Third, the panel itself: three of our “Qwen3” models turned out to be instruction-tuned variants — the bare names on HuggingFace are post-trained releases, which the model cards state and we hadn’t checked. Re-running on the true base models preserved five of six headline verdicts. The sixth — our only clean cross-architecture “writer-side” localization claim — flipped to REJECTED: it had been an artifact of instruction tuning. But the mistake, once corrected, handed us one of the paper’s most useful results: swapping instruct for base models moves the weight-based rankings by about 1% and moves gradient-based attributions by 18–26%. Weights-only methods are stable under fine-tuning in a way gradient methods are not — which is exactly the regime where you’d want a cheap structural method in the first place.

Two eigenvectors, two populations

With the infrastructure audited, we stopped testing methods one at a time. S141 ran all 28 methods through a single closed-loop harness on four models — GPT-2 small, Qwen3-0.6B, Pythia-1.4B, OLMo-2-1B — scoring each by the causal effect of ablating the heads it selects, against layer-matched random baselines.

The harness caught one final bug, mid-run, and it’s our favorite: hub and authority selections started returning results identical to eight decimal places. Two eigenvector computations don’t coincidentally agree at d = 66.94828384. They agree because a preprocessing step had symmetrized the graph — and on a symmetric matrix, hubs and authorities collapse into a single ranking (eigenvector centrality). The direction of the graph is where all the information was.

Because on the correctly asymmetric graph, the sweep’s central result appeared: the top-26 hub heads and the top-26 authority heads on GPT-2 share zero members. Not few — zero. And each population matters in a different way. Authorities are what task catalogs see: they recover 12 of Wang’s 26 IOI heads, the best of any weight-only method we tested. Hubs are what the network can’t function without: ablating them produces the largest cross-architecture effects of any structural method (median Cohen’s d of 7.8 on induction, three of four models above d = 1). This also resolved the clue from week one — and reversed our first finding. S008 had crowned authorities as the induction population; the causal harness shows the crown belongs to hubs, with authorities as the catalog-aligned readers. The lab log put it plainly: the graph is the same; the direction is opposite.

The V-channel version of the sweep (S144) added the second finding, and this one lands squarely where the 2021 theory pointed: K-composition and V-composition route different kinds of task. Every V-channel selection but one helps function-vector tasks and not induction; every K-channel selection does the reverse — exactly the division of labor Elhage et al.’s framework assigns to attention-pattern formation versus value propagation, now visible in ablation across four architectures.

Trying to break our own findings

The final week was spent attacking these results with pre-registered controls, on the theory that we’d rather find the weaknesses than have a reviewer find them. Some findings shrank.

The most delicate case was the “hydra” pattern. As you widen the authority-side selection past its natural size, its ablation effect grows (1.5× at four times the size), while every other method’s decays — suggestive of backup heads that step in when primaries are removed, a structural echo of Wang’s backup name movers:

There’s a mundane alternative — maybe authority effects are heavy-tailed, and any large same-side selection would grow — so we registered the discriminating control with rejection spelled out in advance:

The per-model rows tell the story. On GPT-2 the control behaves exactly as the backup story predicts: top authorities grow, random same-side heads decay. On OLMo-2 the control fails — random heads grow just as fast. On the other two models there’s no growth to explain. So the paper scopes this finding to GPT-2, where it coheres with independently documented backup name movers, and labels it an open question elsewhere. The same fate met the mechanistic story for hubs and authorities: path patching confirms hubs as senders and authorities as receivers on GPT-2 (p = 0.003 and 0.025 at n = 128), and the identical protocol finds nothing on five other architectures — effects three orders of magnitude smaller, at the noise floor (S202). The head-set disjointness travels across architectures. The information-flow interpretation of it, so far, does not.

One last control deserves mention because so few method papers run it: a held-out generalization test (S200). Rank the methods on four models, test the ranking on three held-out models. The method family beats baselines on both panels; the ranking of methods shares one member of its top three across the split. We deleted every claim of the form “X is the best structural method” from the paper. Family-level claims survived; leaderboard claims did not.

What we’re left with

Where does this leave weight-only circuit discovery? Roughly here: the weights alone get you about two-thirds of the way to gradient methods on head recovery (F1 0.46 vs 0.65), at zero task-signal cost, with rankings that survive fine-tuning untouched. The graph’s two directions carry different halves of the model’s functional organization, and prior structural work, by symmetrizing, has been reading their average. The K/V channel split routes task families the way the theory said it should. And the deepest lesson isn’t any single result — it’s how conditional the phrase “found the circuit” turned out to be:

Our method has clear limitations. Everything here is head-level, not edge-level; the panel tops out at 1.4B parameters for the closed-loop results (a 7B run died to infrastructure, and we say so rather than smooth it over); Wang’s IOI catalog is the only curated ground truth in the loop; and the two most mechanistically interesting findings are demonstrated on GPT-2 and hypothetical everywhere else. We’d also flag the pattern behind that last caveat as a warning to the field, ourselves included: on our panel, the model that everything replicates on is the model the canonical results were discovered on.

Everything ships with the paper — the harness, the 28 methods, and all 296 registry records, including the rejections. We went back and forth on how much of the failure record to publish. We landed on all of it, for a simple reason: the three findings that survived are only credible because of what they survived.

Figures are drawn from the project’s working notebooks and study outputs (day-one exploration and ablation notebooks; studies S035f, S049b, S064, S070, S084, S092, S141–S144, S182–S184, S198) and the paper’s figure set. Code and output excerpts are from the repo, trimmed for width but numerically unaltered. Effect sizes are Cohen’s d of ablation-induced KL shift versus layer-matched random head sets, panel-median, 30 baseline seeds for headline numbers. References: Elhage et al. 2021; Wang et al. 2023; Olsson et al. 2022; Kleinberg 1999; Syed et al. 2023; Hanna et al. 2024; McGrath et al. 2023; Conmy et al. 2023.