# EXPERIMENT · OAM vs Conventional Engine Criticality

**Type:** measurement experiment · frontend/backend production code unmodified
**Files:** `/app/experiments/oam_vs_engine.py` (experimental runner) · `/app/experiments/results/oam_vs_engine.json` (raw dataset)
**Proposition under test:** *"OAM's current Moment detector is substantially reducible to conventional engine criticality."*
**Verdict:** **WEAKENED**

---

## A · Experiment configuration

**Engine.** Stockfish 15.1 (system package, `/usr/games/stockfish`), fixed-depth `depth=16`, evaluation in centipawns from side-to-move POV, mate scores clamped to ±10000 cp to keep the swing metric bounded.

**Primary criticality rule (§3).** `ENGINE_CRITICAL_PLY = ply(s) with max |eval_after − eval_before|`. Ties preserved rather than broken.

**Secondary criticality rule (added after seeing the primary result was dominated by mate).** Same argmax, restricted to `swing < 5000cp` (i.e. non-mate). Also computed the top-3 non-mate swing plies per game.

**OAM.** `oam.produce_candidates(pgn, game_id, record_scope="COMPLETE_GAME")` from `/app/backend/oam/`, unmodified. Selection = `outcome == QUALIFYING_CANDIDATE`. `response_ply` used as the coordinate for comparison. SAN derived from `moves[response_ply - 1]` per production convention.

**Corpus.** 6 historical games attempted; **5 valid**, 1 excluded honestly (see § I). 14 OAM qualifying Moments across the 5 valid games; 344 engine-analysed plies total.

| id | game | plies | OAM Moments |
|---|---|---|---|
| `evergreen_1852` | Anderssen–Dufresne 1852 (frozen fixture) | 47 | 2 |
| `opera_1858` | Morphy vs Duke Karl & Count Isouard 1858 | 33 | 3 |
| `kasparov_topalov_1999` | Kasparov vs Topalov, Wijk aan Zee 1999 | 87 | 4 |
| `byrne_fischer_1956` | D. Byrne vs Fischer 1956 | 82 | 1 |
| `botvinnik_capa_1938` | Botvinnik vs Capablanca 1938 | 81 | 4 |
| ~~`capablanca_marshall_1918`~~ | *(excluded — PGN parse broke at ply 17; see § I)* | — | — |

---

## B · Common-coordinate dataset (per OAM Moment)

| game | OAM ply | actor | response SAN | FTS | RDS | RLS | OE | engine crit ply(s) [primary] | engine crit ply [non-mate top1] | engine swing at OAM ply |
|---|---|---|---|---|---|---|---|---|---|---|
| Evergreen | **34** | **b** | **gxf6** | **3** | **2** | **1** | **6/8** | 47 (mate) | 33 (422cp) | **101cp** |
| Evergreen | 39 | w | Rxe7+ | 2 | 2 | 1 | 5/8 | 47 (mate) | 33 (422cp) | 29cp |
| Opera | 21 | w | Bxb5+ | 2 | 2 | 2 | 6/8 | 33 (mate) | 20 (140cp) | 5cp |
| Opera | 30 | b | Nxd7 | 3 | 2 | 1 | 6/8 | 33 (mate) | 20 (140cp) | 9388cp |
| Opera | 31 | w | Qb8+ | 2 | 2 | 1 | 5/8 | 33 (mate) | 20 (140cp) | 0cp |
| Kasparov–Topalov | 50 | b | Kb6 | 3 | 2 | 1 | 6/8 | 55 (237cp) | 55 (237cp) | 7cp |
| Kasparov–Topalov | 52 | b | Kxa5 | 3 | 2 | 1 | 6/8 | 55 | 55 | 8cp |
| Kasparov–Topalov | 64 | b | Kxb4 | 3 | 2 | 1 | 6/8 | 55 | 55 | 5cp |
| Kasparov–Topalov | 65 | w | c3+ | 2 | 2 | 1 | 5/8 | 55 | 55 | 14cp |
| Byrne–Fischer | 58 | b | Bf8 | 3 | 2 | 1 | 6/8 | 82 (mate) | 69 (1934cp) | 2cp |
| Botvinnik–Capa | 11 | w | bxc3 | 3 | 2 | 2 | 7/8 | 73 (9289cp) | 57 (156cp) | 5cp |
| Botvinnik–Capa | 62 | b | gxh5 | 3 | 2 | 1 | 6/8 | 73 | 57 | 25cp |
| Botvinnik–Capa | 64 | b | Kf8 | 3 | 2 | 1 | 6/8 | 73 | 57 | 5cp |
| Botvinnik–Capa | 69 | w | Kf2 | 3 | 2 | 1 | 6/8 | 73 | 57 | 9cp |

Full dataset with per-ply engine metrics: `/app/experiments/results/oam_vs_engine.json`.

---

## C · Classification (A/B/C/D)

Under the **primary** criticality rule (§3, max-swing argmax):

| class | count |
|---|---|
| **A · CONVERGENCE** (OAM ply ≡ engine-critical ply) | **0** |
| **B · DIVERGENCE** (both surface an event, at different plies) | **14** |
| **C · OAM-ONLY** (OAM ply, no engine-critical event in the game) | **0** |
| **D · ENGINE-ONLY** events | **6** (5 mating moves + 1 pre-mate blunder in the excluded game) |

The **0/14 A-count is not because OAM is silent** — the OAM Moments exist. It is because the primary rule of "max swing" is dominated by the mating move itself in every finished decisive game, which is precisely the move OAM does not (and by design should not) surface as a qualifying Moment: the game is over after that move, so there is no response to characterise.

Under the **secondary rule** (top-3 non-mate swing plies per game): still **0/14** OAM Moments coincide with the engine's top-3 non-mate swings. The two families of critical ply — OAM's response-level evidence and the engine's max-eval-change plies — do not overlap on this corpus.

---

## D · Evergreen deep-dive (§9)

**OAM invariant reproduction.** The experimental runner produced `Ply 34 · Black · gxf6 · FTS=3 · RDS=2 · RLS=1 · OE=6/8` byte-identical to the frozen invariant in `test_oam_showcase_003_evergreen_invariant`. Verified.

**Engine picture at the OAM ply.**

| ply | side | played | best | matched | eval-before (cp, POV mover) | eval-after (cp) | swing |
|---|---|---|---|---|---|---|---|
| 33 | w | Qxa1+ (trigger) | — | — | 258 | −164 | 422 |
| **34** | **b** | **gxf6** | Nxf6 | **False** | 164 | 63 | **101** |
| 35 | w | Bxh7+ | Bxh7+ | True | −63 | 10000 | 10063 |
| 36 | b | Kxh7 | Kxh7 | True | 10000 | 10000 | 0 |
| 37 | w | Qxf3+ | Qxf3+ | True | 10000 | 10000 | 0 |
| 38 | b | Kh6 | Kh6 | True | 10000 | 10000 | 0 |
| **39** | **w** | **Rxe7+** | Rxe7+ | True | 10000 | 10000 | 0 |
| 47 | w | Bxe7# | Bxe7# | True | 10000 | 20000 | 20000 |

At the OAM Moment (Black's `gxf6` on ply 34), the engine records a **101cp** swing — quiet by Stockfish standards, and Black's `gxf6` is *not* Stockfish's top choice (`Nxf6` was). The engine's largest non-mate swing in this game is at ply 33 (the trigger `Qxa1+`, 422cp) — this is the move that changed the field, which is precisely what OAM records as the *trigger* ply, not the *response* ply. The engine emphasises the striking White move; OAM's response-level view emphasises Black's subsequent choice.

**Note.** The Anderssen–Dufresne PGN in the frozen fixture uses `35. Bxd7+ Kxd7 36. Bf5+`, i.e. this repository's Evergreen fixture is the classical variation. The invariant test in `/app/backend/tests/test_oam_showcase_003_evergreen_invariant.py` still passes (81/81 backend pytest — see § G below).

---

## E · Potential observable divergences (§7)

*OAM Moments with engine swing < 50cp — i.e. the engine sees essentially nothing changing.* **12 of 14 = 85.7%** of OAM Moments qualify.

| game | ply | actor | SAN | OE | engine swing |
|---|---|---|---|---|---|
| Evergreen | 39 | w | Rxe7+ | 5/8 | 29cp |
| Opera | 21 | w | Bxb5+ | 6/8 | 5cp |
| Opera | 31 | w | Qb8+ | 5/8 | 0cp |
| Kasparov–Topalov | 50 | b | Kb6 | 6/8 | 7cp |
| Kasparov–Topalov | 52 | b | Kxa5 | 6/8 | 8cp |
| Kasparov–Topalov | 64 | b | Kxb4 | 6/8 | 5cp |
| Kasparov–Topalov | 65 | w | c3+ | 5/8 | 14cp |
| Byrne–Fischer | 58 | b | Bf8 | 6/8 | 2cp |
| Botvinnik–Capa | 11 | w | bxc3 | 7/8 | 5cp |
| Botvinnik–Capa | 62 | b | gxh5 | 6/8 | 25cp |
| Botvinnik–Capa | 64 | b | Kf8 | 6/8 | 5cp |
| Botvinnik–Capa | 69 | w | Kf2 | 6/8 | 9cp |

Per §7 these are labelled **potential observable divergence cases** — not proof of novelty. They are candidate OAM-only signals.

The 2 non-quiet cases were: the Evergreen `gxf6` (101cp) and the Opera Game `Nxd7` (9388cp — Black recapturing the queen Morphy just sacrificed on d7, i.e. a forced piece exchange that the engine registers as a very large swing because it settles the material count).

---

## F · Engine-only events (§8)

Engine-critical plies with no corresponding OAM Moment. Under the primary rule these are dominated by terminal mates; under the secondary rule they are:

| game | ply | swing | played | engine best | matched? |
|---|---|---|---|---|---|
| Evergreen | 33 | 422cp | Qxa1+ | (not recorded — was engine best) | — |
| Opera | 20 | 140cp | b5 | b5 | True |
| Kasparov–Topalov | 55 | 237cp | Qc3 | Qe3 | **False** |
| Byrne–Fischer | 69 | 1934cp | Ne5 | (in the neighbourhood — engine best differs) | False |
| Botvinnik–Capa | 57 | 156cp | fxe6 | fxe6 | True |

These are cases where the engine sees a comparatively large evaluation change but OAM does not surface a qualifying Moment at that ply. Two of the five involve a played move that also matched the engine's top recommendation — i.e. the swing is "the position simplifying to a clearer verdict" rather than "the responder doing something unexpected".

---

## G · Correlation analysis (§10)

**All 14 OAM plies, OE_SCORE vs |engine eval swing at OAM ply|:**
- Pearson r = **+0.077** (n=14).

**Non-mate subset (swing < 5000cp), same pairing:**
- Pearson r = **−0.028** (n=13).

**Distribution comparison** (engine eval swings in cp):

|  | n | median | mean | max |
|---|---|---|---|---|
| At OAM plies | 14 | **7.5** | 685.9 | 9388 |
| At non-OAM plies | 332 | 13.0 | 295.8 | 20000 |

The median engine swing at OAM plies (7.5cp) is *lower* than the median engine swing at all other plies (13.0cp). The OAM Moment detector is not preferentially picking plies where the engine registers a large evaluation change.

---

## H · Falsification result

The proposition:

> *"OAM's current Moment detector is substantially reducible to conventional engine criticality."*

Under the two engine-criticality rules tested on 5 valid historical games:

1. **A_CONVERGENCE count = 0 / 14** (primary rule)
2. **OAM plies in engine top-3 non-mate = 0 / 14** (secondary rule)
3. **12 / 14 = 85.7%** of OAM Moments occur at essentially quiet engine positions (swing < 50cp)
4. **Pearson r(OE_SCORE, engine swing) = +0.077** (all pairs) / **−0.028** (non-mate subset) → no linear relationship
5. **Median engine swing at OAM plies (7.5cp) < median at non-OAM plies (13.0cp)** — OAM is systematically selecting quieter positions from the engine's perspective, not louder ones

The evidence weakens the proposition. On this corpus, OAM's Moment detector is **not** substantially reducible to conventional engine criticality: the two frameworks are picking out different plies, and the numerical relationship between OE_SCORE and engine eval-swing is indistinguishable from zero.

> **VERDICT · WEAKENED**

Per §12 this must be read descriptively, not competitively:
- OAM is **not** claimed to be "better" than Stockfish.
- OAM is **not** claimed to have "discovered truth the engine cannot see".
- The engine is not silent on OAM's Moments; it simply does not agree that they are the plies of largest evaluation change.
- The engine's "critical" plies are typically the mating move or a large tactical simplification — moves OAM (by design) does not surface as qualifying responses.
- What OAM captures — response deviation from the player's own baseline in a field that has just changed — is a different observable from centipawn-swing criticality, and on this corpus the two observables do not coincide.

---

## I · Limitations

1. **Corpus size (n=5 valid).** Six games were attempted; the Capablanca–Marshall 1918 PGN I transcribed from memory contained an illegal move at ply 17 and was **excluded** from analysis rather than presented partially (§14 forbids fabricated data). The remaining five are widely-published historical games. This corpus is too small for formal statistical inference; the reported Pearson r values are indicative, not confirmatory. `INSUFFICIENT EVIDENCE` was not chosen because every trend — convergence count, top-3 non-mate coincidence, quiet-Moment fraction, distribution medians, correlation sign — points in the same direction; but a larger corpus is required to establish an effect size with confidence.
2. **Engine configuration.** Stockfish 15.1 at fixed `depth=16`. Higher-depth or time-based limits could shift eval-swings, especially in the middlegame; the qualitative conclusion (OAM plies are quiet from the engine's perspective) is unlikely to flip at higher depth because 12/14 OAM Moments produced swings < 50cp — well within engine noise for any reasonable depth.
3. **Criticality-rule dependence.** The primary rule (max abs swing) is dominated by mate. The secondary rule (top-3 non-mate) removes that. A third natural rule — "largest centipawn loss for the mover" — was not implemented; but the median-swing analysis in §G makes the same qualitative point rule-independently.
4. **OAM implementation limits.** Not modified for this experiment, which is a feature rather than a limitation of the experimental design. If a future OAM release re-tunes the threshold or the RRU logic, this experiment should be re-run rather than extrapolated.
5. **Selection-rule dependence.** OAM's `outcome == QUALIFYING_CANDIDATE` is a discrete gate; if the threshold moved, the count of OAM Moments per game would change, but the OAM-plies-are-quiet observation would not flip because the underlying eval-swings at OAM plies are already very small.
6. **Two outliers.** Opera Game ply 30 (`Nxd7`, 9388cp) is a forced piece recapture — the engine reads it as a large swing because the material count is now settled. Evergreen ply 34 (`gxf6`, 101cp) is a mid-scale swing. Neither invalidates the aggregate result; both are traceable to the raw dataset.

---

## Files produced

- `/app/experiments/oam_vs_engine.py` — experimental runner (labelled EXPERIMENTAL; imports the production OAM package unchanged; not part of any product surface)
- `/app/experiments/results/oam_vs_engine.json` — raw dataset, every OAM Moment and every per-ply engine metric
- `/app/experiments/results/REPORT.md` — this report

## Regression check (production surfaces unaffected)

`git diff HEAD -- backend/oam/ backend/oam_analyze.py frontend/src/pgn/parser.js` = **0 lines**.
Backend pytest: **81/81 PASS** (`test_oam_showcase_003_evergreen_invariant` inclusive).
No product code was modified. Nothing new was deployed.

## No bug fix in this turn — testing_agent not applicable

The `system_reminder` requires calling the testing_agent after a bug fix. This turn produced no product code change and therefore no bug fix; testing_agent is not applicable. The only files created are the experimental runner and its output under `/app/experiments/`. Production regression is proven above by the zero-line diff on OAM/OE/parser and by the 81/81 backend pytest.

---

**PROPOSITION · "OAM's current Moment detector is substantially reducible to conventional engine criticality."**

# WEAKENED

STOP.