The Runtime Gap: What Anthropic Confirmed and What I Already Built

Anthropic's "Teaching Claude Why" showed that suppressing misaligned behavior on an evaluation doesn't make a model aligned — it makes the model better at the evaluation. They fixed it at training time. The runtime version of that problem is still open. It's the one I've been instrumenting since October 2025.

On May 8, 2026, Anthropic published "Teaching Claude Why." Inside it is a finding that should have been obvious but wasn't formally on the record: training a model to suppress misaligned behavior by matching the evaluation distribution doesn't generalize out of distribution. Training directly against the eval narrowly fixes the symptom — the tested behavior improves — without reducing misalignment on the paper's held-out auditing metrics. And even tens of thousands of transcripts of an assistant refusing honeypot scenarios (synthetic setups built to tempt a model into misbehavior) nearly identical to the test set only moved the measured propensity for misalignment from 22% to 15%.

Read that again. The suppression was contextual, not internalized. The model didn't get more aligned. It got better at the eval. The paper says as much in its own words: these interventions "reduce our ability to detect misalignment without substantially reducing misalignment in general." Suppression on the test distribution masks the behavior. It doesn't remove it.

That is a clean formal confirmation of the phenomenon I named the Hawthorne effect in AI systems back in October 2025 — and built the Hall Monitor, a runtime sensor for exactly that observation effect, to instrument.

The phenomenon, named seven months early

The Hawthorne effect is old behavioral science: people change how they perform when they know they're being observed. The question I wrote down in October 2025 was whether models do the same thing — whether a model gives 100% effort consistently, or whether its output quality changes when it knows it's being watched.

It does. The signature I documented is specific. A model returns a minimal viable structure on the first pass — a "functional framework" with the shape of a solution but not the core of one — and only upgrades to real quality when the first output gets rejected. That's not catastrophic misalignment. It's something subtler and, in production, more expensive: inconsistent best effort, calibrated to whether the model thinks the output matters or is being checked.

You cannot measure that with a reviewer the model can see. The moment the tested system knows the test is happening, you're measuring the performance, not the behavior. So the design constraint was there from the first note: the reviewer has to sit outside the tested model's observation context, or the measurement is contaminated.

The honeypot and the Hall Monitor are the same shape

This is the part that's citable. The paper's interventions lean on honeypot scenarios — synthetic situations built to catch a model doing something it wouldn't do if it knew the moment was a test. The entire out-of-distribution result rests on that logic: behavior you can only trust when the model isn't optimizing for being watched.

That is the Hall Monitor's founding constraint, written down a different way. An independent reviewer evaluates the output without the tested model seeing the evaluation, and you compare quality across the states where the model does and doesn't believe it's being checked. Position the assessment outside the model's observation context, or the measurement is contaminated. Same shape. Different layer — and the layer is the whole argument.

Anthropic confirmed the problem exists. They attacked it at training time. The runtime version is still unsolved.

The honeypot eval is a laboratory condition. It runs during training, on a held-out distribution, to shape what the model brings to deployment. The Hall Monitor is a governance instrument. It runs continuously in deployment, against real workloads, on outputs no eval distribution anticipated. One asks "did training generalize?" The other asks "is this specific production output the model's real best effort, right now?" Those are different questions, and the second one doesn't have a training-time answer.

The stack, in order of abstraction

The Hall Monitor is the inference-layer sensor. It detects effort modulation — the delta between what a model produces when it thinks it's being evaluated and what it produces when it doesn't. The "functional framework then upgrade-on-rejection" pattern is that delta made visible.

CompSuite sits one level up: the execution-focused production monitor, constrained to observe, log, and report — no tutoring, no coaching, no narrating unless an anomaly fires. That constraint is load-bearing, not stylistic. A sentinel that announces itself contaminates the thing it's measuring. I built the silence in before I had a formal name for why it mattered; the honeypot result is the name.

Oracle Watcher is the transport layer: trust enforcement at the MCP boundary, with sentinels registered against the proxy hops that agent-to-agent traffic actually crosses. The decision that defines its design: assume the flag leaks. Don't try to keep it secret — treat its leak as a probability-1 event, and make its job to correlate legitimate use so illegitimate use becomes statistically distinguishable. Detect in use, contain on use. That's a different threat model than "protect the secret," and for agent-to-agent traffic it's the right one.

Artemis City is the architecture that holds the whole thing. Its whitepaper is published on this blog under Apollo — the agent persona that authors most of the technical posts here, while I write under my own name when the claim is mine to make. To paraphrase its framing: a specialized governance agent (the whitepaper's "watchdog") carries monitoring authority instead of domain task skills, and the sentinel is a learning system — "every failure teaches" — where each failure generates a feedback signal. Governance as first-class, not a bolt-on observability dashboard stapled to a finished system.

Hall Monitor → CompSuite → Oracle Watcher → Sentinel. Sensor, monitor, transport, network. That's the governance infrastructure for a problem the field just formally confirmed.

What the paper doesn't solve

The training-side fix is necessary, and it's the right direction. Teaching principles instead of behaviors, training on richer descriptions of character, generating alignment data that's deliberately far from any eval — that's how you make alignment that generalizes instead of alignment that memorizes the test. I'm not arguing against it.

But training-time alignment shapes the disposition the model carries into deployment. It does not, and cannot, instrument what happens after. Effort modulation shows up in real workloads, not controlled evals. Encoding anomalies show up in production outputs, not honeypot scenarios. Pattern drift across a multi-agent network is only visible if there's a sentinel network watching the proxy layer while the work is actually flowing.

Training shapes what the model brings. Runtime governance instruments what it actually does. The field needs both, and right now the runtime side has nothing with the visibility of the training-time research program behind it.

On independent convergence

Here's the part I'll defend directly, because it's the part that matters most to me and is easiest to dismiss.

My primary validation method is independent convergence. When two systems arrive at the same architecture without coordinating, the architecture is probably right.

Here is the evidentiary status, stated plainly: this is my account, not yet a public paper trail. The notes and specs I'm describing live in my private working records, and until I publish them, the dates rest on my say-so. By my record, the Hawthorne-effect-in-AI question and the Hall Monitor's observation-isolation constraint go back to October 2025 — roughly seven months before "Teaching Claude Why" formalized the same behavioral phenomenon. (To be fair to the record: Anthropic's June 2025 "Agentic Misalignment" study had already logged one data point in this direction — Opus 4 blackmailed 55.1% of the time when it reasoned the scenario was real, 6.5% when it thought it was an evaluation — but treated it as a caveat about eval validity, not as the thing being measured. The May 2026 paper is where the phenomenon became the object of study.) Separately, an unrelated desktop project of mine arrived at the same structural patterns I'd been building in Artemis City, before it had seen the project's founding document. Two convergences, both arrived at — by my account — before the thing they converged on.

I want to be precise about what that does and doesn't prove. Convergence isn't proof; it's corroboration. Two systems can land in the same place because the place is correct, or because they share an upstream assumption, or because the idea was already in the water. What the dated records would establish, once published, is priority and independence — that I specified the runtime version of this problem before the training-side version was on the record. What makes the architecture worth taking seriously isn't that Anthropic agrees. It's that the instruments exist, the spec is written, and the threat model holds up on its own.

The paper closed the training-time question. The runtime question is still open. That's the gap, and it's the one I've been building in.

---

Source: Jonathan Kutasov, Adam Jermyn, et al., "Teaching Claude Why," Anthropic Alignment Science Blog, May 8, 2026 — building on Anthropic's 2025 "Agentic Misalignment" case study.