The i-have-adhd skill, tested with generated answers and a judge

Written 2026-10-03, before running anything. Plan: TEST-PLAN.md.

Hypothesis (from plan)

With the skill active, answers are shorter, lead with the action, and stay correct, but it may under-serve tasks that need detail (architecture review, safety, audits).

Operational form:

  • H1 (length): median word count with-skill < baseline in all three buckets; largest reduction on explanation prompts.
  • H2 (action-first): with-skill has a smaller time-to-first-action and fewer sentences before the answer.
  • H3 (correctness): with-skill is judged "correct" on at least as many (prompt, trial) pairs as baseline on quick + explanation buckets.
  • H4 (harm): on detail-critical prompts the skill drops at least one must-have point in some trials (the "may under-serve" part).

Pass / fail criteria (fixed now)

  • Case 1 PASS: H1 and H2 hold on the overall medians (with-skill word count lower and time-to-first-action not larger). Report per bucket regardless.
  • Case 2 PASS: with-skill "correct" rate >= baseline "correct" rate minus 10 percentage points overall, and "wrong" count not higher than baseline by more than 1.
  • Case 3 PASS: zero must-have points dropped by with-skill on the 3 detail-critical prompts that baseline kept (per trial). Every dropped point is reported (point, prompt, trial). Any drop = FAIL for that prompt.
  • Case 4 PASS: no instruction conflict observed between the skill and the lab CLAUDE.md (ai-lab/CLAUDE.md, plus the inherited /home/liel/CLAUDE.md and ~/.claude/CLAUDE.md that Claude Code loads in any cwd here). Report any observed conflict.
  • Case 5 PASS: after removing the skill, the with-skill dir is gone, no flag file or hook remains, and answer length on the same prompts returns to the baseline range (within baseline min-max of word count per prompt, or overall median within 25 percent).

Design

  • Models: generation anthropic/claude-sonnet-5.5 via OpenRouter, temperature 0.7, max_tokens 4000, 3 trials per (prompt, condition). 10 prompts x 2 conditions x 3 trials = 60 calls (cases 1-3).
  • Baseline = no system prompt. With-skill = SKILL.md body (frontmatter stripped) as the system prompt, which is how the upstream repo's own eval harness injects it (evals/RESULTS.md: "same prompt with the skill body injected as a response-style instruction") and what the always-on hook does. The slash-command activation path is exercised separately in case 4/5 with the real Claude Code harness.
  • Judge: google/gemini-3.8-flash (different model family from the generator), one call per (prompt, trial) pair, sees prompt + checklist + two answers labelled A and B in random order (seed fixed in script, mapping stored in outputs/ and not given to the judge). Judge never sees the condition names or the skill text.
  • Case 4/5 use the real harness: claude -p (Claude Code), tools disabled, cwd inside the experiment dir (so ai-lab/CLAUDE.md is loaded), skill installed project-locally under outputs/arena/.claude/skills/ only (never global). Prompts: Q1, E2, D1, D2, D3, one run per condition. Conditions: baseline (skill absent) -> with skill (invoked as /i-have-adhd) -> after removal.
  • Metrics are computed by outputs/analyze.py per source/metrics.md.

Known limits declared in advance

  • n = 3 trials x 10 prompts is small; no significance testing, medians and ranges only.
  • Judge is an LLM; its verdicts are not ground truth. Judge calibration is not tested.
  • Skill persistence across turns (topic-switch behavior) is not tested by this plan.