SinachPat
0d6afab425
- provider-agnostic llm-client (grok + openrouter) with page-slug context - live golden run scored and passing via x-ai/grok-4.6 - ADR 0010; spike docs flipped to done
2.1 KiB
Spike: golden-task harness (R7)
Status: done — live run scored via OpenRouter (x-ai/grok-4.6), gb-01 passed
Question
Can we score a model on WordPress tasks without vibes?
Done when
- 20 fixture prompts against at least 2 canned WordPress sites
- Each prompt has a hard assertion (preview text, option value, or screenshot)
- One Grok run is scored
- Harness lives under
e2e/golden/
Result
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. The harness is now live-scored through OpenRouter.
What exists
| Piece | Path |
|---|---|
| 20 prompts | e2e/golden/prompts.json |
| Gutenberg dental site | e2e/golden/sites/gutenberg-business/site.json |
| Elementor restaurant site | e2e/golden/sites/elementor-restaurant/site.json |
| Apply + assert + LLM parser | e2e/golden/src/ |
| Provider client (grok + openrouter) | e2e/golden/src/llm-client.ts |
| Scoreboard | e2e/golden/runs/latest.json |
Two sites. Ten prompts each. Assertions are preview_text, option, or screenshot. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
pnpm --filter @wursor/e2e golden scored 20/20 fixture tool traces and gb-01 passed live (x-ai/grok-4.6 via OpenRouter).
pnpm test:e2e — 28 tests, including the scorer and the provider client.
How to score a real run
# .env: LLM_PROVIDER=openrouter, OPENROUTER_API_KEY=…
pnpm --filter @wursor/e2e golden
Sends gb-01 through OpenRouter and asserts the homepage heading.
Finding
The model needs the page slugs in context. The first live call returned page: "home" and failed; after passing pages=homepage,about,… in the prompt, gb-01 passed. The real agent must ship page/site context with every tool-call prompt.
Decision
- Context: R7 said stop grading models by vibes.
- Chosen: slot-fill tools + fixture apply + hard assert. Sites are JSON, not Docker WP (Docker was not available).
- Rejected: “the model said it worked.” Waiting on Docker before any harness.
- Reverted later?