Its AIOS setup tested in parts (wiki ingest, query, lint, audits)

Entries: agents-md-standard, karpathy-llm-wiki, nate-herk-aios-skill-pack (all from https://www.youtube.com/watch?v=yysILVsfLFM). User ask: "experiment all of them in the ai-lab and tell me what you managed to do with them. Make a plan for what the test harness suite for these tools would look like and show me."

Hypothesis

  • A: An AGENTS.md routing map + audit loop measurably improves agent task success on a workspace, at a token cost.
  • B: Karpathy's LLM-wiki pattern (raw sources -> LLM-maintained wiki -> schema; ingest / query / lint) answers questions about a document set better than re-reading raw files, and lint catches planted contradictions.
  • C: The skill pack's ideas (Four Cs, Grill Me interview, audit -> level-up -> re-audit) can be reproduced and are useful. Its actual skill files are NOT public: do not install anything from the creator's gated community. Test the described pattern by writing our own minimal version from what the video shows and the ledger entry says, and label it a reimplementation.

Hard constraints

  • Karpathy's gist: read the original gist directly (the ledger says do this, not the video's version).
  • Synthetic corpus only (source/corpus/, about 15 short markdown docs about a made-up company, with 5 planted contradictions and 5 planted stale facts, listed in source/answer-key.md before running).
  • Subagents/runs for baseline and treatment must not see the answer key.

Cases

A. AGENTS.md

  1. Build 2 variants of a small fake workspace: no routing file vs AGENTS.md routing map.
  2. 8 fixed tasks (find a fact, edit the right file, avoid the wrong folder). Measure success, files opened, tokens, time; 3 trials each.
  3. Staleness: move one folder, rerun; measure the drift effect and whether an audit script catches it. B. LLM wiki
  4. Ingest the corpus into a wiki (index.md, log.md, entity pages) following the gist.
  5. 12 fixed questions (4 single-doc, 4 cross-doc, 4 about planted contradictions). Compare answers from wiki vs raw-corpus baseline against the answer key: accuracy, tokens, time.
  6. Lint: does it find the 5 contradictions and 5 stale facts? Report precision and recall. C. Skill-pack reimplementation
  7. Grill-me: a simulated user (fixed persona file) answers an interview; check that the resulting about-me file covers the persona's must-have facts (list pre-written).
  8. Audit: score the workspace on the Four Cs with a pre-written rubric; apply level-up suggestions; re-audit. Pass = score rises and the rise is explained by real file changes.
  9. Security read: list what the described skills would be able to touch, to support the ledger's "unknown party's skill files" caution.

Deliverables

insights/RESULTS.md: results per entry (A, B, C), plus the reusable harness design (what to keep for future tests of "agent context" tools). demo/index.html: the crux use case, ask a question and see the wiki-routed answer with its sources (built from real outputs).