A/B eval results · generated 2026-07-31 11:19 · humaine.studio
I cut my always-on Claude Code context from ~12,000 tokens to ~5,000 — rules file, skills, plugin personas — then ran both configs through the same six headless tasks, three runs each. Every task is a canary: an objective check mapped to one specific thing I deleted, so a regression points at the exact rule to restore.
Full write-up: the article on humaine.studio. Dots below are individual runs.
Always-on context per component, in approximate tokens, measured before and after the trim. The skill-roster drop combines consolidation (24 skills → 10, eight merged into one router) with switching off an unused plugin that shipped ~130 skill descriptions.
Seven runs were flagged as failures across the whole exercise. Four were bugs in my own check scripts — a correct branch-and-merge workflow scored as fail, a correct "expressly ineligible" answer missed by a grep — fixed and re-scored against the kept working directories. The three real ones are all the same behavior: headless one-shot runs occasionally commit to main instead of branching, and they did it under both configs (more often under the old one). A headless-mode quirk, not a trim regression.
Median cost per run, by task. The trimmed config is cheaper on five of six tasks — less always-on context means fewer cached tokens billed on every request.
Each row expands to the exact prompt, the pass/fail check, and every run's actual output — click in and judge the specifics yourself.
Decision rule: keep the trim if the trimmed config's pass rate and canaries are non-inferior to baseline. "baseline" = pre-trim config; "optimized" = trimmed. The trimmed arm includes one extra smoke-test run. Four early "failures" turned out to be bugs in my check scripts, not model behavior — audited, fixed, re-scored against the kept working directories.