pat/wursor
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

feat: add OpenRouter provider to golden harness, close Phase 0 gate

Browse Source 
- provider-agnostic llm-client (grok + openrouter) with page-slug context
- live golden run scored and passing via x-ai/grok-4.6
- ADR 0010; spike docs flipped to done
This commit is contained in:
SinachPat
2026-08-15 17:58:09 +01:00
parent f42549c451
commit 0d6afab425
9 changed files with 200 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
spikes/golden-task.md
+14 -8
View File
@@ -1,6 +1,6 @@
# Spike: golden-task harness (R7)
**Status:** partial — harness exists; live Grok run not scored (`XAI_API_KEY` unset)
**Status:** done — live run scored via OpenRouter (`x-ai/grok-4.6`), `gb-01` passed
## Question
@@ -15,7 +15,7 @@ Can we score a model on WordPress tasks without vibes?
## Result
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. No, if it means we have a Grok quality number. This machine has no `XAI_API_KEY`, so the live call was skipped.
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. The harness is now live-scored through OpenRouter.
### What exists
@@ -24,22 +24,28 @@ Yes, if “score” means: apply a tool call to a fixture and assert the new hea
| 20 prompts | `e2e/golden/prompts.json` |
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
| Apply + assert + Grok parser | `e2e/golden/src/` |
| Apply + assert + LLM parser | `e2e/golden/src/` |
| Provider client (grok + openrouter) | `e2e/golden/src/llm-client.ts` |
| Scoreboard | `e2e/golden/runs/latest.json` |
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces. Live Grok: skipped.
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces and **gb-01 passed live** (`x-ai/grok-4.6` via OpenRouter).
`pnpm test:e2e` — 21 tests, including the scorer.
`pnpm test:e2e` — 28 tests, including the scorer and the provider client.
### How to score a real Grok run
### How to score a real run
```bash
XAI_API_KEY=… pnpm --filter @wursor/e2e golden
# .env: LLM_PROVIDER=openrouter, OPENROUTER_API_KEY=…
pnpm --filter @wursor/e2e golden
```
That sends `gb-01` through `api.x.ai` and asserts the homepage heading.
Sends `gb-01` through OpenRouter and asserts the homepage heading.
### Finding
The model needs the page slugs in context. The first live call returned `page: "home"` and failed; after passing `pages=homepage,about,…` in the prompt, `gb-01` passed. The real agent must ship page/site context with every tool-call prompt.
### Decision