Compare commits

..
4 Commits
Author SHA1 Message Date
SinachPat 2f53b1901b ci: reference GITEA secret by its actual name
Mirror to GitHub / mirror (push) Canceled after 0s
2026-08-15 18:57:40 +01:00
SinachPat 6eeac23719 chore: update repo references after transfer to Wursor org 2026-08-15 18:10:03 +01:00
SinachPat 9a7058c85b feat: auto-load .env in golden harness; scaffold CI workflow 2026-08-15 18:05:37 +01:00
SinachPat 0d6afab425 feat: add OpenRouter provider to golden harness, close Phase 0 gate
- provider-agnostic llm-client (grok + openrouter) with page-slug context
- live golden run scored and passing via x-ai/grok-4.6
- ADR 0010; spike docs flipped to done
2026-08-15 17:58:09 +01:00
14 changed files with 240 additions and 29 deletions
+4 -1
View File
@@ -8,9 +8,12 @@ REDIS_URL=redis://localhost:6379
# Auth
SESSION_SECRET=replace-me
# LLM — Grok is the default adapter. Switch provider with LLM_PROVIDER.
# LLM — Grok is the default adapter. Switch provider with LLM_PROVIDER (grok | openrouter).
LLM_PROVIDER=grok
XAI_API_KEY=
OPENROUTER_API_KEY=
# Model sent when LLM_PROVIDER=openrouter (default x-ai/grok-4.6).
OPENROUTER_MODEL=x-ai/grok-4.6
LLM_FALLBACK_PROVIDER=
# Sandbox
+1 -1
View File
@@ -22,4 +22,4 @@ jobs:
- name: Reconcile with GitHub
run: bash scripts/mirror.sh
env:
MIRROR_URL: https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/SinachPat/wursor.git
MIRROR_URL: https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/Wursor/wursor.git
+27
View File
@@ -0,0 +1,27 @@
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 10.33.0
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm test
- run: pnpm lint
+1 -1
View File
@@ -22,4 +22,4 @@ jobs:
- name: Reconcile with Gitea
run: bash scripts/mirror.sh
env:
MIRROR_URL: https://x-access-token:${{ secrets.GITEA_TOKEN }}@git.weown.tools/pat/wursor.git
MIRROR_URL: https://x-access-token:${{ secrets.GITEA }}@git.weown.tools/pat/wursor.git
+1 -1
View File
@@ -12,7 +12,7 @@ The Agentic WordPress Management Platform
| **Date** | August 13, 2026 |
| **Author** | Patrick (Product Lead) |
| **Status** | Draft — Internal (non-technical-first pivot; Phase 0) |
| **Repo** | SinachPat/wursor |
| **Repo** | Wursor/wursor |
| **Classification** | Confidential |
| **Supersedes** | v1.3 (engineer-first; desktop shell; Tauri + Monaco) |
+2
View File
@@ -24,3 +24,5 @@ Rename the repository to `wursor`, set the description to the product tagline, c
- Remote is `https://github.com/SinachPat/wursor`; identity matches the product.
- Redirects from the old name are handled by GitHub automatically.
> **Follow-up (2026-08-15):** the repo was later transferred to the `Wursor` GitHub organization — canonical path is now `https://github.com/Wursor/wursor`.
@@ -0,0 +1,30 @@
# 10. Golden harness scores live runs through a provider-agnostic LLM client (OpenRouter first)
- **Status:** Accepted
- **Date:** 2026-08-15
## Context
Phase 0's golden-task spike was the last gate item, stuck at "partial" because no `XAI_API_KEY` was set and the harness hard-coded `api.x.ai`. The developer holds an OpenRouter key, and OpenRouter speaks the same OpenAI-compatible `chat/completions` shape (including tool-calling), so the live run could be unblocked without a Grok-specific key.
## Decision
Replace the hard-coded `grok-client.ts` with a provider-agnostic `llm-client.ts` that supports `grok` (x.ai) and `openrouter`, selected by `LLM_PROVIDER`. The OpenRouter default model is `x-ai/grok-4.6`. The harness now passes the site's page slugs into the prompt.
### Options considered
- Wait for an `XAI_API_KEY` and keep the hard-coded x.ai client.
- Hard-code OpenRouter, dropping the x.ai path.
- Provider-agnostic client with `grok` + `openrouter` (chosen).
### Rejected
- Wait for x.ai — blocks the gate on a key we don't have, for no technical reason.
- OpenRouter-only — the plan (IMPLEMENTATION §3) still names Grok the default adapter; keeping both providers matches that plan and costs one env switch.
## Consequences
- The live gate run is scored and passing (`gb-01` passed on `x-ai/grok-4.6`); the Phase 0 golden spike flips to done.
- `x-ai/grok-latest` is not a callable OpenRouter ID (returns 400); the pinned `x-ai/grok-4.6` works and supports `tool_choice: required`.
- A real finding: without the page-slug list in the prompt, the model guessed `page: "home"` and failed. Tool-call prompts must always carry site/page context. Sprint 3 must inherit this when it builds `api/src/agents/llm-client.ts`.
- `llm-client.ts` here is a spike; Sprint 3 will generalize it (streaming, circuit breaker, fallback) rather than promote this file verbatim.
+1
View File
@@ -23,6 +23,7 @@ Each ADR is a single file following the [Nygard format](https://cognitect.com/bl
| [0007](0007-media-proxy-not-copy.md) | Sandboxes proxy uploads; never copy the media library | Accepted |
| [0008](0008-empty-packages-not-stubs.md) | Workspace ships empty packages, not placeholder source | Accepted |
| [0009](0009-repo-rename.md) | Repository renamed originmain → wursor | Accepted |
| [0010](0010-openrouter-live-golden.md) | Golden harness scores live runs through a provider-agnostic LLM client (OpenRouter first) | Accepted |
## How to add one
+98
View File
@@ -0,0 +1,98 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { resolveProvider, callLlm } from '../src/llm-client.ts';
let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
fetchMock = vi.fn();
vi.stubGlobal('fetch', fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
});
describe('resolveProvider', () => {
it('maps grok to the x.ai endpoint and grok-3 model', () => {
expect(resolveProvider('grok')).toEqual({ baseUrl: 'https://api.x.ai/v1', model: 'grok-3' });
});
it('maps openrouter to the openrouter endpoint and the grok-4.6 model', () => {
expect(resolveProvider('openrouter')).toEqual({
baseUrl: 'https://openrouter.ai/api/v1',
model: 'x-ai/grok-4.6',
});
});
it('honors an explicit model override', () => {
expect(resolveProvider('openrouter', 'openai/gpt-4o')).toEqual({
baseUrl: 'https://openrouter.ai/api/v1',
model: 'openai/gpt-4o',
});
});
});
describe('callLlm', () => {
it('posts an OpenAI-shaped tool-call request to the grok endpoint', async () => {
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
await callLlm({
provider: 'grok',
apiKey: 'test-key',
prompt: 'Change the heading',
siteId: 'gutenberg-business',
builder: 'gutenberg',
});
const [url, init] = fetchMock.mock.calls[0] as [string, { headers: Record<string, string>; body: string }];
expect(url).toBe('https://api.x.ai/v1/chat/completions');
expect(init.headers.Authorization).toBe('Bearer test-key');
const body = JSON.parse(init.body) as { model: string; tool_choice: string; tools: Array<{ function: { name: string } }> };
expect(body.model).toBe('grok-3');
expect(body.tool_choice).toBe('required');
expect(body.tools[0]?.function.name).toBe('edit_heading');
});
it('adds OpenRouter identification headers and uses the openrouter endpoint', async () => {
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
await callLlm({
provider: 'openrouter',
apiKey: 'test-key',
prompt: 'Change the heading',
siteId: 'gutenberg-business',
builder: 'gutenberg',
});
const [url, init] = fetchMock.mock.calls[0] as [string, { headers: Record<string, string>; body: string }];
expect(url).toBe('https://openrouter.ai/api/v1/chat/completions');
expect(init.headers['HTTP-Referer']).toBeDefined();
expect(init.headers['X-Title']).toBeDefined();
expect(JSON.parse(init.body).model).toBe('x-ai/grok-4.6');
});
it('includes page slugs in the user message when provided', async () => {
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
await callLlm({
provider: 'grok',
apiKey: 'test-key',
prompt: 'Change the heading',
siteId: 'gutenberg-business',
builder: 'gutenberg',
pages: ['homepage', 'about', 'contact'],
});
const [, init] = fetchMock.mock.calls[0] as [string, { body: string }];
const body = JSON.parse(init.body) as { messages: Array<{ content: string }> };
expect(body.messages[1]?.content).toContain('pages=homepage,about,contact');
});
it('throws when the provider returns a non-ok response', async () => {
fetchMock.mockResolvedValue({ ok: false, status: 401 });
await expect(
callLlm({ provider: 'grok', apiKey: 'bad', prompt: 'x', siteId: 's', builder: 'b' }),
).rejects.toThrow('LLM HTTP 401');
});
});
+2 -2
View File
@@ -2,8 +2,8 @@
"fixturePassed": 20,
"fixtureTotal": 20,
"grokLive": {
"skipped": true,
"reason": "XAI_API_KEY not set"
"id": "gb-01",
"passed": true
},
"scores": [
{
@@ -1,21 +1,47 @@
import type { GrokResponse } from './types.ts';
const url = 'https://api.x.ai/v1/chat/completions';
export type LlmProvider = 'grok' | 'openrouter';
export async function callGrok(input: {
export type LlmConfig = {
baseUrl: string;
model: string;
};
const PROVIDERS: Record<LlmProvider, LlmConfig> = {
grok: { baseUrl: 'https://api.x.ai/v1', model: 'grok-3' },
openrouter: { baseUrl: 'https://openrouter.ai/api/v1', model: 'x-ai/grok-4.6' },
};
export function resolveProvider(provider: LlmProvider, model?: string): LlmConfig {
const config = PROVIDERS[provider];
return { baseUrl: config.baseUrl, model: model ?? config.model };
}
export async function callLlm(input: {
provider: LlmProvider;
apiKey: string;
model?: string;
prompt: string;
siteId: string;
builder: string;
pages?: string[];
}): Promise<GrokResponse> {
const response = await fetch(url, {
method: 'POST',
headers: {
const { baseUrl, model } = resolveProvider(input.provider, input.model);
const headers: Record<string, string> = {
Authorization: `Bearer ${input.apiKey}`,
'Content-Type': 'application/json',
},
};
if (input.provider === 'openrouter') {
headers['HTTP-Referer'] = 'https://wursor.dev';
headers['X-Title'] = 'Wursor golden harness';
}
const response = await fetch(`${baseUrl}/chat/completions`, {
method: 'POST',
headers,
body: JSON.stringify({
model: 'grok-3',
model,
messages: [
{
role: 'system',
@@ -24,7 +50,7 @@ export async function callGrok(input: {
},
{
role: 'user',
content: `site=${input.siteId} builder=${input.builder}\n${input.prompt}`,
content: `site=${input.siteId} builder=${input.builder}${input.pages ? ` pages=${input.pages.join(',')}` : ''}\n${input.prompt}`,
},
],
tools: [
@@ -71,7 +97,7 @@ export async function callGrok(input: {
});
if (!response.ok) {
throw new Error(`Grok HTTP ${response.status}`);
throw new Error(`LLM HTTP ${response.status}`);
}
return (await response.json()) as GrokResponse;
}
+22 -4
View File
@@ -1,17 +1,29 @@
import { mkdirSync, writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { loadEnvFile } from 'node:process';
import { fileURLToPath } from 'node:url';
import { detectBuilder } from './builder-detect.ts';
import { asGrokResponse, expectedCalls } from './expected-calls.ts';
import { callGrok } from './grok-client.ts';
import { callLlm, type LlmProvider } from './llm-client.ts';
import { loadPrompts } from './load-prompts.ts';
import { loadSite } from './load-site.ts';
import { scoreGrokResponse } from './score.ts';
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
const repoRoot = join(goldenRoot, '..', '..');
try {
loadEnvFile(join(repoRoot, '.env'));
} catch {
// no .env at repo root — live run will skip
}
function provider(): LlmProvider {
return process.env.LLM_PROVIDER === 'openrouter' ? 'openrouter' : 'grok';
}
function key(): string | undefined {
const value = process.env.XAI_API_KEY;
const value = provider() === 'openrouter' ? process.env.OPENROUTER_API_KEY : process.env.XAI_API_KEY;
return value !== undefined && value !== '' ? value : undefined;
}
@@ -39,11 +51,14 @@ if (apiKey !== undefined) {
}
const site = loadSite(prompt.site);
try {
const grok = await callGrok({
const grok = await callLlm({
provider: provider(),
apiKey,
model: provider() === 'openrouter' ? process.env.OPENROUTER_MODEL : undefined,
prompt: prompt.prompt,
siteId: site.id,
builder: detectBuilder(site),
pages: site.posts.map((post) => post.slug),
});
grokLive = {
id: prompt.id,
@@ -57,7 +72,10 @@ if (apiKey !== undefined) {
const report = {
fixturePassed: fixtureScores.filter((row) => row.passed).length,
fixtureTotal: fixtureScores.length,
grokLive: grokLive ?? { skipped: true, reason: 'XAI_API_KEY not set' },
grokLive: grokLive ?? {
skipped: true,
reason: provider() === 'openrouter' ? 'OPENROUTER_API_KEY not set' : 'XAI_API_KEY not set',
},
scores: fixtureScores,
};
+1 -1
View File
@@ -6,7 +6,7 @@ Throwaway fixtures and scripts are allowed. Product UI is not.
| Spike | File | Status |
|---|---|---|
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | partial — live Grok pending key |
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | done — live run scored via OpenRouter |
| Builder detect (R6 / R13) | [builder-detect.md](./builder-detect.md) | done |
| Pairing threat model (R9) | [pairing-threat-model.md](./pairing-threat-model.md) | done |
| Large-site mirror timing (R4) | [mirror-timing.md](./mirror-timing.md) | done — synthetic 2GB |
+14 -8
View File
@@ -1,6 +1,6 @@
# Spike: golden-task harness (R7)
**Status:** partial — harness exists; live Grok run not scored (`XAI_API_KEY` unset)
**Status:** done — live run scored via OpenRouter (`x-ai/grok-4.6`), `gb-01` passed
## Question
@@ -15,7 +15,7 @@ Can we score a model on WordPress tasks without vibes?
## Result
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. No, if it means we have a Grok quality number. This machine has no `XAI_API_KEY`, so the live call was skipped.
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. The harness is now live-scored through OpenRouter.
### What exists
@@ -24,22 +24,28 @@ Yes, if “score” means: apply a tool call to a fixture and assert the new hea
| 20 prompts | `e2e/golden/prompts.json` |
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
| Apply + assert + Grok parser | `e2e/golden/src/` |
| Apply + assert + LLM parser | `e2e/golden/src/` |
| Provider client (grok + openrouter) | `e2e/golden/src/llm-client.ts` |
| Scoreboard | `e2e/golden/runs/latest.json` |
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces. Live Grok: skipped.
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces and **gb-01 passed live** (`x-ai/grok-4.6` via OpenRouter).
`pnpm test:e2e` — 21 tests, including the scorer.
`pnpm test:e2e` — 28 tests, including the scorer and the provider client.
### How to score a real Grok run
### How to score a real run
```bash
XAI_API_KEY=… pnpm --filter @wursor/e2e golden
# .env: LLM_PROVIDER=openrouter, OPENROUTER_API_KEY=…
pnpm --filter @wursor/e2e golden
```
That sends `gb-01` through `api.x.ai` and asserts the homepage heading.
Sends `gb-01` through OpenRouter and asserts the homepage heading.
### Finding
The model needs the page slugs in context. The first live call returned `page: "home"` and failed; after passing `pages=homepage,about,…` in the prompt, `gb-01` passed. The real agent must ship page/site context with every tool-call prompt.
### Decision