04 · D2 Decision Memo

Status: ✅ Decided. Option A, chip-mask schedule M3, ships. Parent: Document 03 (D2 Experiment Plan) — this memo is Step 5.6’s “Run matrix + memo” deliverable, and it closes the D2 row in Doc 02 §9’s decisions table. Evidence: results/d2sim-primary-m3-20260731-193030.csv + matching .log — the final, valid run (20 seeds, 50,000,000 events per run, 680 total runs, ~2h15m wall time). An earlier run (...151136) exists in the same directory and is void — see §4. Reading aid: every section with real numbers ends with a plain-terms recap. If those and the numbers ever disagree, the numbers win — the recaps exist to build intuition, not to define the result.

1. The verdict, in one sentence

Under every non-adversarial traffic pattern we tested, and under a deliberate attack designed to exploit its one structural weakness, Option A’s hash-entropy chip-mask eviction (schedule M3) behaves statistically indistinguishably from Option C (the published HeavyKeeper baseline) — and it’s cheaper, because it needs no random number generator. Per Doc 03 §4.3’s pre-decided tie-breaker, that means Option A ships.

🧭 In plain terms: we built two ways for the sketch to decide “who loses their spot when a newcomer needs one.” Option C is the well-established, published approach, but it needs a dice roll every time. Option A tries to get the same behavior out of leftover bits of a hash calculation we were already doing anyway — free, but unproven. This experiment’s whole job was to check: does Option A’s cheaper trick actually produce the same real-world outcomes as the dice-roll approach, or does it quietly do something worse? The answer, backed by 50 million simulated events × 20 independent trials × every traffic shape we could think to throw at it: yes, it holds up.

2. What was actually tested

Traffic shapes: three Zipfian skews (s=0.8, 1.1, 1.4 — how concentrated traffic is on a few “elephant” keys), a churn workload (keys constantly retiring and being replaced), and a burst workload (a brand-new key suddenly surging to dominant traffic).
Table pressure: three sizes per traffic shape — comfortable (generous headroom), tight (sized right at the expected elephant count), starved (deliberately undersized, so even legitimate elephants have to compete for room).
Seeds: 20 independent random trials per (traffic shape × pressure × option) combination — the minimum Doc 03 §4.1 pre-registered as enough to characterize Option C’s own natural run-to-run variation.
The adversarial clause: a separate scenario where an attacker deliberately injects keys engineered to bias Option A’s hash-derived lottery in their own favor — the one attack Option C is structurally immune to (it doesn’t use hash bits for its randomness at all) and Option A was specifically worried about.

That’s 5 traffic shapes × 3 pressure levels × 20 seeds × {Option A, Option C} = 600 runs for the main comparison, plus 80 more for the adversarial clause.

3. The evidence

3.1 Accuracy — perfect, no exceptions

elephant_err_p50 — how far off the sketch’s estimate is for the biggest, most important keys — never once fell outside Option C’s own range, across all 15 traffic-shape/pressure combinations. Zero misses out of 300 individual measurements.

🧭 In plain terms: for the one number that matters most — “does the sketch correctly track its most important keys” — Option A didn’t just pass, it was never even close to failing. This is the strongest possible signal that the cheaper trick isn’t cutting a safety corner.

3.2 Decision metrics — where the actual story is

Across the full 15-combination matrix, we checked 11 different measurements per combination (165 checks total): 157 landed squarely inside Option C’s own historical range. The remaining 8 were all the same one metric — how fast a new elephant gets a stable spot in the table — and all 8 were Option A being faster than Option C ever was, never slower.

Worked example, straight from the real data: for Zipfian traffic (skew 0.8) under the tightest table sizing, Option C’s 20 trials took anywhere from 18,604 to 34,061 events to stably admit a new elephant (its own natural range). Option A’s typical run: 17,266 events — faster than Option C’s single best trial. That’s not a fluke of one run; the same pattern showed up in 8 different combinations, always in the same direction.

Why: we also looked at a diagnostic (not part of the pass/fail decision) called eviction_rate — how often a contested slot actually changes hands. Option A’s rate ran consistently 7–8 percentage points higher than Option C’s, in every single one of the 15 combinations, including the fixed churn cells. The reason traces back to how the two schedules are built: for any weak, low-count occupant (the kind that dominates when a table is under real pressure), Option A’s schedule (M3) locks in a 100% chance of winning the challenge across a fairly wide range of weak counts, while Option C’s smooth mathematical curve only approaches 100% gradually — sitting anywhere from 93% down to 46% over that same range. Option A clears out weak occupants more decisively, which is exactly why new elephants get seated faster.

🧭 In plain terms: imagine two bouncers deciding whether to let a weak, half-hearted regular keep their spot at the bar when someone new wants it. Bouncer C rolls dice — usually says “leave,” but not always, and less so the stronger the regular seems. Bouncer A just looks at how weak the regular is and, past a certain point, always says “leave, no dice needed.” Bouncer A clears out the deadweight a little more reliably, so newcomers get seated a bit faster on average. That’s a feature, not a bug — and it’s exactly what the original design notes predicted would happen, in these words, before any code existed: “we chip near-dead occupants slightly more eagerly than C… it marginally speeds new-elephant admission. Logged as a known, benign divergence.”

3.3 Structural health — diagnostic only, explains the “why,” doesn’t gate the decision

Eviction rate, chip rate, and slot-churn rate (Doc 03 §2.3) never factor into pass/fail — they exist purely to explain accuracy differences when they show up. In this run they did their job: the eviction-rate gap above is the concrete mechanical explanation for the admission-time story in §3.2, not a mystery.

4. Two bugs found along the way — the honest part

The first full run of this matrix produced a “ship C” verdict. It was wrong, for two independent, since-fixed reasons. Both are logged in docs/epochsketch-03-d2-experiment-plan.md’s Deviations appendix as D1–D3; here’s the plain-language version.

4.1 The churn workload wasn’t actually churning

What we found: every single “churn” row in the first run was byte-for-byte identical to the equivalent “steady Zipfian” row at the same seed. Two supposedly different experiments produced the exact same numbers.

Why: the simulation has a repeating “advance time” signal that different pieces listen for. The churn workload’s own “rotate some of the population” step was listening for a different signal that the harness never actually sent. It’s like writing “swap in new background actors every scene” on a film set’s call sheet, and nobody ever picks the sheet up — so the same actors appear in every scene, forever.

Fix: the harness now checks whether a traffic generator has a “rotate” step and drives it on every tick. Verified with a test that specifically asserts churn and steady Zipfian now produce different numbers at the same seed — confirmed in the final run (9 of 11 measurements differ where they used to match exactly).

4.2 The original pass/fail rule was too strict to ever pass — even for two identical things

The original rule: every one of Option A’s 20 individual trial results, for every measurement, had to land somewhere inside the range Option C’s own 20 trials produced. No exceptions.

Why that’s broken, in plain terms: imagine 20 people measure the same table with a slightly wobbly ruler and you record the shortest and longest reading as “the acceptable range.” A 21st measurement of the same table, same ruler isn’t guaranteed to land inside that range — there’s a real chance it becomes a brand-new shortest or longest reading, purely because you’re now looking at 21 numbers instead of 20. Nothing about the table changed.

We proved this wasn’t hypothetical. We took Option C’s own 20 results and, for each one, asked “if this were a fresh 21st measurement instead, built from the other 19 — does it land inside their range?” Option C failed this test against itself in 12 out of 12 traffic-shape/pressure combinations. A rule that the reference answer can’t pass against its own repeated self isn’t measuring “is A different from C” — it’s just measuring “did we get unlucky with which values happened to be extreme,” which happens constantly by pure chance.

The fix: instead of requiring every individual trial to land in range, we now check whether Option A’s median (the middle value of its 20 trials) lands in Option C’s range. A single odd trial doesn’t sink the whole comparison anymore. We validated the new rule the same way we found the old one was broken — not by assuming it was fixed, but by testing it: ran 200 trials comparing two “clones” built from the same distribution (should basically always pass) — passed 200/200. Ran 200 trials with a real, deliberately-introduced difference baked in — failed 200/200. The new rule can tell “no real difference” from “real difference” reliably; the old one never could.

4.3 Some measurements have an obvious “better” direction — the rule now knows that

The problem this fixes: “how fast does a new elephant get seated” only has one good direction — faster is never bad. But the original rule couldn’t tell “Option A is worse” from “Option A is unexpectedly, harmlessly better” — it just saw “outside Option C’s range” and failed either way.

The fix: four specific measurements (admission time, mouse-count inflation rate, first-sighting miss rate, elephant recall) got tagged with a declared “better direction.” If Option A’s median lands outside Option C’s range specifically on the good side, it now passes — but it’s still printed out loud in the log as “passed because it’s better,” never silently folded in as if it were an ordinary in-range result. Every one of the 8 exceptions in §3.2 is tagged this way; none of them are hidden.

One deliberate limit, decided before re-running, not after seeing the result: the raw accuracy numbers (elephant/middle/mouse error) got no such allowance — a deviation there in either direction is treated as a real difference worth catching, since an estimate being biased in one direction isn’t obviously “better,” just different.

🧭 In plain terms, all of §4 together: we ran the real experiment once, it said “reject Option A” — and then, instead of accepting that at face value, we checked whether the test itself was trustworthy. It wasn’t, in two independent ways: one workload was silently broken, and the pass/fail rule was so strict that even the correct answer couldn’t pass it against itself. Both problems are fixed, both fixes are logged with the reasoning and the date, and both fixes were checked with their own separate proof before we trusted them — not just declared fixed and moved on.

5. A finding beyond pass/fail — flagged, not fully explained

Option C’s worst-case admission time under churn is dramatically more variable than under any other traffic shape. Every other traffic shape’s “slowest 5%” admission time for Option C sits somewhere in the 4,000–15,500 event range. Under churn at the tightest table sizing, Option C’s range is [7,834 → 321,111] events; at the most starved sizing, [9,306 → 2,205,784] — over two million events in its worst observed trial, roughly three orders of magnitude beyond every other traffic shape.

This doesn’t affect the verdict — Option A’s median admission time is still faster than Option C’s in these exact cells, so the gate correctly passes both. But it’s a real, substantial signal about Option C’s own tail behavior under population rotation that we don’t yet have a root-cause explanation for, and it deserves to be written up rather than quietly passed over. It’s a natural first input for THEORY.md item 1 (the admission-time bound).

6. Ideas raised during review — what happened to them

Add a variance/spread check alongside the median check. A real, correctly-identified gap: the median-only rule can’t distinguish “Option A is consistently a bit different” from “Option A is wildly unstable — half its runs fast, half catastrophically slow — that just happens to average out.” Deferred, not forgotten (tracked in ROADMAP.md) — any new gating rule needs the same clone-vs-shift calibration proof that caught §4.2’s bug before it can be trusted, and that hasn’t been built yet.

Use more formal-sounding statistical terminology in the write-up (e.g. “non-parametric median comparison,” “one-tailed non-inferiority criteria,” claiming the calibration test “guarantees zero false positives”). Considered and explicitly rejected. The suggested framing oversold what was actually done — our calibration is a Monte Carlo demonstration under specific tested conditions, not a formal guarantee with proven error bounds, and “non-inferiority testing” normally implies a pre-specified margin that our simpler rule doesn’t have. We’d rather the write-up stay accurate and slightly plain than sound more rigorous than it is.

7. What this decides

D2 (Doc 02 §9): Eviction mechanism → Option A, hash-entropy chip-mask eviction.
Chip-mask schedule: M3 (the primary candidate from Doc 03.2). It passed cleanly across the entire matrix, so the pre-registered fallback (M1) was never run — per the pre-decided scoping (“M3 vs C first, M1 only if M3 fails”), there was no need. M2 was eliminated on paper back in Doc 03.2 §6 and was never a candidate in this experiment at all.
§4.2’s keyed-hash requirement (Doc 02 §4.3, the designated defense if the entropy attack ever did defeat Option A) is not invoked — the plain recall-ratio test passed comfortably (0.9942 against a 0.95 threshold) without it.

8. Known gaps in this memo — stated plainly

No rendered CDF plots. Doc 03 §5.3 deliberately deprioritized plotting (“presentation is secondary, reproducibility is primary”), and we never built that layer. This memo presents p50/p99 summary tables instead of full distribution curves. internal/report.WriteCDFDump exists and can produce full quantile dumps if a future reader wants the actual curves — it just wasn’t run for this memo.
The variance/stability gap from §6 is real and open. We know the median-only rule has a blind spot; we haven’t yet built the check that would close it.
§5’s churn tail-variance finding is observed, not explained. We don’t yet have a mechanistic theory for why Option C’s admission tail specifically blows up under churn.

9. What happens next

Update docs/02-design-spec-v2.md §9, row D2, from “gated on simulation parity with C” to “decided: Option A (M3), see Document 04.”
THEORY.md items 1, 2, and 6 are now unblocked with real data (item 2 gains extra material from §5’s churn finding); items 3–5 remain genuinely separate future work.
ROADMAP.md tracks everything past this point — closing D2 is Part 1 of that list, not the end of the project.

Definition of done for this document

Per Doc 03’s own closing section: the D2 row in Doc 02 §9 flips from “recommended, gated” to decided, with this document as the evidence link. Done, as of this memo.