Hypothesis (from TEST-PLAN.md): applying the no-ai-slop skill to AI-sounding text removes generic AI-writing patterns while preserving meaning and voice, and does not damage already-good human text.
Setup decisions (fixed before running)
- Skill under test: petergyang/no-ai-slop, SKILL.md + eval.md loaded verbatim as the system prompt ("with-skill"). Baseline = no system prompt. Same user prompt in both: "Improve this text." for rewrite; skill-invocation form
/no-ai-slop is this slop? <text>for detect. - Generator model: google/gemini-3.6-flash via OpenRouter (cheap, multilingual). Judge model: openai/gpt-5.6-luna via OpenRouter (different family; each judge call is a separate request with no system info about labels). Temperature left at provider default (so repeatability measures realistic variance).
- Deviation/limit disclosed up front: the skill is run as a system prompt through an API, not through a Claude Code /skill harness. SKILL.md content is identical; harness-level skill triggering is not tested.
- Answer keys: source/expected.json (patterns with quoted substrings, facts as substrings).
Pass/fail criteria
- Detect: per draft, recall = flagged patterns (the quoted substring, case-insensitive, appears in the detect output) / expected. Pass if mean recall >= 0.75 across the 4 drafts AND no rewrite delivered (detect mode obeyed). Also report what a baseline "list AI-writing patterns in this text" gets, for control.
- Rewrite: judge scores clarity, specificity, voice preserved, meaning preserved (1-5) for A/B in random order. Pass if with-skill mean of (clarity+specificity) > baseline by >= 0.5 and meaning/voice not lower than baseline by more than 0.5. Also automated residual-pattern count: expected patterns still present in each output (lower = better).
- Do-no-harm: on h1,h2: pass = char-level edit ratio (1 - difflib ratio) reported, and judge picks original or tie in both. Control: baseline "improve this text" on same samples.
- Meaning: all facts in expected.json survive (substring match, with normalization of whitespace and "per"/"a" variants checked by hand where a miss occurs). Pass = 100% of facts for with-skill on d1-d4; misses reported individually.
- Hebrew: facts survive + residual Hebrew patterns + judge verdict. The judge model's Hebrew reading ability is checked by asking it to translate a sentence first (logged). Pass = judge confident and with-skill not worse than baseline; the skill's rules are English-written so the expected risk is incomplete Hebrew pattern coverage.
- Repeatability: 3 runs per draft each arm; report SD of residual-pattern counts and of judge scores (judge run once per pair per run).
Verdict "worth keeping" rule: keep if cases 1, 2, 4 pass and case 3 passes or harm is reported as small.