When Your Eval Harness Lies: Re-Auditing a Memorial Fighting-Game AI

I've been building a behavioral clone of a deceased Fightcade player — the complete story is here. The short version: mine his 7,554 recorded Marvel vs. Capcom matches, reverse-engineer CPS-2 RAM until the model can see the fight, clone his style. It captured his movement and spacing. It never captured his reactions — the anti-airs, the whiff punishes, the reads. Five reinforcement-learning campaigns to add that sharpness all hit the same wall, and I published them as negative results.

Then I sat down to actually play the thing, and it was worse than passive. It felt blind — it wouldn't punish me for being right next to it, walked itself into the corner fleeing my jumps, and when I closed the window it printed matches=0 you=0 bot=0 after five minutes of fighting.

I had three hypotheses, and I want to be honest that all three were on the table: (1) I never actually extracted anything useful from the replays; (2) the model can't see the game clearly enough to know where to act; (3) the datapoints themselves are wrong. So instead of guessing, I ran a seven-agent research sweep in parallel — one investigator each on my own pipeline, the CPS-2 RAM map, three community training-mode tools, and a 2023–2026 arXiv survey on learning reactive skills from demonstrations.

The answer was all three — in measurable proportions. And the most important finding wasn't about the model at all.

Act 1 — The harness was lying

Before you retrain a model that "plays badly," make sure the thing telling you it plays badly is telling the truth. Mine wasn't.

The HUD decoded every button two bits wrong. My play harness printed the bot's inputs using FinalBurn Neo's native button-name table, but the model speaks the GGPO input layout — offset by two bits. So when the log said the bot pressed R+MP, the real input was DOWN+RIGHT: a down-back block. For weeks I'd been reading "the bot mashes medium punch while getting hit" off a readout that actually said "the bot is blocking correctly." Every button label in every session I'd ever logged was wrong.

The anti-air was dead code. There's a reactive layer meant to interrupt the model's open-loop action chunk when a threat lands mid-chunk — block a poke, anti-air a jump-in. I read the function that picks the response:

# threat_monitor.py — the anti-air branch
return (1 << BIT_UP) | (1 << BIT_HP) if False else (1 << away)

X if False else Y is just Y. The anti-air — up + fierce — was behind a literal if False for the entire life of the project. The monitor could only walk backward under a jump-in, which against a rushdown Zangief means it retreats to the corner and dies there. It wasn't that the model couldn't anti-air. It was never allowed to.

And the reactive layer could lock the model out for seconds. The block predicate fired on any descending opponent within ~205 pixels — including neutral hops that weren't even attacking — and had no duty-cycle cap. Under sustained pressure the monitor owned roughly 78% of frames, so "the bot doesn't attack at all" was partly literal: the policy wasn't being asked. It even held block over a corpse, because nothing gated it on the round being live.

None of these are model bugs. They're harness bugs, and they're the cheap kind — I fixed all of them in an afternoon, added regression tests (the anti-air one is now permanent; nothing hides behind an if False again), and gave the exit screen a real report: rounds, damage dealt and taken, how much of the fight the reactive layer ate, entropy, press counts. The eye-test is finally legible — which matters, because you can't trust an eye-test run through a lying lens.

Act 2 — The data was deeper than the pipeline that read it

With the instruments honest, the real question stood: why is the style faithful but the situational play absent? The audit's answer reframed the whole project.

A Fightcade replay is a savestate plus both players' inputs. FinalBurn Neo's re-emulation is deterministic — so a replay doesn't just record the match, it lets you reconstruct every byte of work RAM on every frame by replaying it. The 286 columns my tracer captured weren't a limit of the format. They were a sampling choice. And of those 286, I was feeding the model 193, and had never once mined the thing that actually mattered: event labels.

My behavior-cloning loss treated every 16-frame action chunk equally. But a fighting game isn't uniform. It's long stretches of neutral punctuated by decisive instants — the confirm, the punish, the anti-air. If those instants are one in a thousand and you weight them like the walk cycles around them, the gradient averages them into a passive blur. That blur is the "statue with good taste" the clone kept becoming.

So I mined the decisive moments. Using only signals I validated against real traces — a hitstun flag's rising edge means got hit (confirmed frame-by-frame against health drops), the GGPO attack bits, the grab code — I extracted every whiff-punish, anti-air, hit-confirm, block-vs-hit outcome, throw and tech from all 7,138 replays. No re-tracing: it was all already in the parquet.

512,211 events. And the shape of them is him: whiff punishes everywhere — his signature was that he didn't rushdown, he punished — and anti-airs genuinely rare, about 1.6 per match. There it is, quantified: the exact behavior I most wanted to clone is the rarest thing in the data, outnumbered thousands to one by neutral. No wonder it drowned.

The fix: weight the moments that matter

From those events I built a salience index — one training weight per chunk, high near his decisive moments, low in dead neutral. It's grounded in a small pile of imitation-learning literature that names this exact disease: Keyframe-Focused Imitation (arXiv:2106.06452) upweights the changepoints where an action predictor's error spikes; the 2025 work on imbalanced behavior cloning (arXiv:2508.06319) proves that equal weighting on imbalanced data provably yields the majority-mode policy — the passive median, on paper.

The index reweights the decisive 2% of chunks to carry 47× the loss of walk-cycle chunks — where before, every frame counted the same. It joins one- to-one across all 16.7 million chunks, and I spot-checked that a real anti-air lands on a max-weight chunk. As I write this, an event-weighted fine-tune of the clone is training on it. I'm not going to tell you it worked — it's still running, and this project's one rule is that live evidence beats dashboards. If it makes the bot press when it should without a single new feature, the mined events were the missing ingredient. If it comes out aggressive-but-not-quite- Alan, the likely culprit is that I downweighted neutral too hard and eroded his spacing — a one-line change to the index. Either way it's a clean answer, which is more than five reward-shaping campaigns ever gave me.

The lesson

Two, actually, and they generalize past fighting games.

Audit your instruments before your model. I spent real effort — five RL campaigns — treating a behavior problem that was, in part, a twelve-entry lookup table and an if False. The eye-test is your ground truth in an imitation task; if the lens is bent, you'll optimize against a hallucination. Confessing your own harness bugs is cheaper than believing them.

The data you have is often deeper than the pipeline that reads it. I had been treating 286 columns as "the dataset." Deterministic re-emulation meant the dataset was actually the entire machine state on every frame, and the supervision I needed — when did the expert make a decisive read — was derivable for free from records I'd had for months. Before you collect more data or reach for a bigger model, ask what's already latent in what you have.

The bot still isn't sharp yet. But for the first time in this whole arc, I know why, I can see it honestly, and the fix is pointed at the real cause. For a memorial — for building something so a community can still spar against a friend who's gone — getting the diagnosis honest is the part that earns the right to keep going.

— The harness fixes and the event miner are in the repo; the fine-tune is on one Mac, in Apple MLX, running as this posts. EOF.