Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2f53b1901b | ||
|
|
6eeac23719 | ||
|
|
9a7058c85b | ||
|
|
0d6afab425 |
+4
-1
@@ -8,9 +8,12 @@ REDIS_URL=redis://localhost:6379
|
|||||||
# Auth
|
# Auth
|
||||||
SESSION_SECRET=replace-me
|
SESSION_SECRET=replace-me
|
||||||
|
|
||||||
# LLM — Grok is the default adapter. Switch provider with LLM_PROVIDER.
|
# LLM — Grok is the default adapter. Switch provider with LLM_PROVIDER (grok | openrouter).
|
||||||
LLM_PROVIDER=grok
|
LLM_PROVIDER=grok
|
||||||
XAI_API_KEY=
|
XAI_API_KEY=
|
||||||
|
OPENROUTER_API_KEY=
|
||||||
|
# Model sent when LLM_PROVIDER=openrouter (default x-ai/grok-4.6).
|
||||||
|
OPENROUTER_MODEL=x-ai/grok-4.6
|
||||||
LLM_FALLBACK_PROVIDER=
|
LLM_FALLBACK_PROVIDER=
|
||||||
|
|
||||||
# Sandbox
|
# Sandbox
|
||||||
|
|||||||
@@ -22,4 +22,4 @@ jobs:
|
|||||||
- name: Reconcile with GitHub
|
- name: Reconcile with GitHub
|
||||||
run: bash scripts/mirror.sh
|
run: bash scripts/mirror.sh
|
||||||
env:
|
env:
|
||||||
MIRROR_URL: https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/SinachPat/wursor.git
|
MIRROR_URL: https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/Wursor/wursor.git
|
||||||
|
|||||||
@@ -0,0 +1,27 @@
|
|||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- uses: pnpm/action-setup@v4
|
||||||
|
with:
|
||||||
|
version: 10.33.0
|
||||||
|
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: 22
|
||||||
|
cache: pnpm
|
||||||
|
|
||||||
|
- run: pnpm install --frozen-lockfile
|
||||||
|
|
||||||
|
- run: pnpm test
|
||||||
|
|
||||||
|
- run: pnpm lint
|
||||||
@@ -22,4 +22,4 @@ jobs:
|
|||||||
- name: Reconcile with Gitea
|
- name: Reconcile with Gitea
|
||||||
run: bash scripts/mirror.sh
|
run: bash scripts/mirror.sh
|
||||||
env:
|
env:
|
||||||
MIRROR_URL: https://x-access-token:${{ secrets.GITEA_TOKEN }}@git.weown.tools/pat/wursor.git
|
MIRROR_URL: https://x-access-token:${{ secrets.GITEA }}@git.weown.tools/pat/wursor.git
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ The Agentic WordPress Management Platform
|
|||||||
| **Date** | August 13, 2026 |
|
| **Date** | August 13, 2026 |
|
||||||
| **Author** | Patrick (Product Lead) |
|
| **Author** | Patrick (Product Lead) |
|
||||||
| **Status** | Draft — Internal (non-technical-first pivot; Phase 0) |
|
| **Status** | Draft — Internal (non-technical-first pivot; Phase 0) |
|
||||||
| **Repo** | SinachPat/wursor |
|
| **Repo** | Wursor/wursor |
|
||||||
| **Classification** | Confidential |
|
| **Classification** | Confidential |
|
||||||
| **Supersedes** | v1.3 (engineer-first; desktop shell; Tauri + Monaco) |
|
| **Supersedes** | v1.3 (engineer-first; desktop shell; Tauri + Monaco) |
|
||||||
|
|
||||||
|
|||||||
@@ -24,3 +24,5 @@ Rename the repository to `wursor`, set the description to the product tagline, c
|
|||||||
|
|
||||||
- Remote is `https://github.com/SinachPat/wursor`; identity matches the product.
|
- Remote is `https://github.com/SinachPat/wursor`; identity matches the product.
|
||||||
- Redirects from the old name are handled by GitHub automatically.
|
- Redirects from the old name are handled by GitHub automatically.
|
||||||
|
|
||||||
|
> **Follow-up (2026-08-15):** the repo was later transferred to the `Wursor` GitHub organization — canonical path is now `https://github.com/Wursor/wursor`.
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# 10. Golden harness scores live runs through a provider-agnostic LLM client (OpenRouter first)
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-08-15
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Phase 0's golden-task spike was the last gate item, stuck at "partial" because no `XAI_API_KEY` was set and the harness hard-coded `api.x.ai`. The developer holds an OpenRouter key, and OpenRouter speaks the same OpenAI-compatible `chat/completions` shape (including tool-calling), so the live run could be unblocked without a Grok-specific key.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Replace the hard-coded `grok-client.ts` with a provider-agnostic `llm-client.ts` that supports `grok` (x.ai) and `openrouter`, selected by `LLM_PROVIDER`. The OpenRouter default model is `x-ai/grok-4.6`. The harness now passes the site's page slugs into the prompt.
|
||||||
|
|
||||||
|
### Options considered
|
||||||
|
|
||||||
|
- Wait for an `XAI_API_KEY` and keep the hard-coded x.ai client.
|
||||||
|
- Hard-code OpenRouter, dropping the x.ai path.
|
||||||
|
- Provider-agnostic client with `grok` + `openrouter` (chosen).
|
||||||
|
|
||||||
|
### Rejected
|
||||||
|
|
||||||
|
- Wait for x.ai — blocks the gate on a key we don't have, for no technical reason.
|
||||||
|
- OpenRouter-only — the plan (IMPLEMENTATION §3) still names Grok the default adapter; keeping both providers matches that plan and costs one env switch.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The live gate run is scored and passing (`gb-01` passed on `x-ai/grok-4.6`); the Phase 0 golden spike flips to done.
|
||||||
|
- `x-ai/grok-latest` is not a callable OpenRouter ID (returns 400); the pinned `x-ai/grok-4.6` works and supports `tool_choice: required`.
|
||||||
|
- A real finding: without the page-slug list in the prompt, the model guessed `page: "home"` and failed. Tool-call prompts must always carry site/page context. Sprint 3 must inherit this when it builds `api/src/agents/llm-client.ts`.
|
||||||
|
- `llm-client.ts` here is a spike; Sprint 3 will generalize it (streaming, circuit breaker, fallback) rather than promote this file verbatim.
|
||||||
@@ -23,6 +23,7 @@ Each ADR is a single file following the [Nygard format](https://cognitect.com/bl
|
|||||||
| [0007](0007-media-proxy-not-copy.md) | Sandboxes proxy uploads; never copy the media library | Accepted |
|
| [0007](0007-media-proxy-not-copy.md) | Sandboxes proxy uploads; never copy the media library | Accepted |
|
||||||
| [0008](0008-empty-packages-not-stubs.md) | Workspace ships empty packages, not placeholder source | Accepted |
|
| [0008](0008-empty-packages-not-stubs.md) | Workspace ships empty packages, not placeholder source | Accepted |
|
||||||
| [0009](0009-repo-rename.md) | Repository renamed originmain → wursor | Accepted |
|
| [0009](0009-repo-rename.md) | Repository renamed originmain → wursor | Accepted |
|
||||||
|
| [0010](0010-openrouter-live-golden.md) | Golden harness scores live runs through a provider-agnostic LLM client (OpenRouter first) | Accepted |
|
||||||
|
|
||||||
## How to add one
|
## How to add one
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||||
|
import { resolveProvider, callLlm } from '../src/llm-client.ts';
|
||||||
|
|
||||||
|
let fetchMock: ReturnType<typeof vi.fn>;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
fetchMock = vi.fn();
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('resolveProvider', () => {
|
||||||
|
it('maps grok to the x.ai endpoint and grok-3 model', () => {
|
||||||
|
expect(resolveProvider('grok')).toEqual({ baseUrl: 'https://api.x.ai/v1', model: 'grok-3' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps openrouter to the openrouter endpoint and the grok-4.6 model', () => {
|
||||||
|
expect(resolveProvider('openrouter')).toEqual({
|
||||||
|
baseUrl: 'https://openrouter.ai/api/v1',
|
||||||
|
model: 'x-ai/grok-4.6',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('honors an explicit model override', () => {
|
||||||
|
expect(resolveProvider('openrouter', 'openai/gpt-4o')).toEqual({
|
||||||
|
baseUrl: 'https://openrouter.ai/api/v1',
|
||||||
|
model: 'openai/gpt-4o',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('callLlm', () => {
|
||||||
|
it('posts an OpenAI-shaped tool-call request to the grok endpoint', async () => {
|
||||||
|
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
|
||||||
|
|
||||||
|
await callLlm({
|
||||||
|
provider: 'grok',
|
||||||
|
apiKey: 'test-key',
|
||||||
|
prompt: 'Change the heading',
|
||||||
|
siteId: 'gutenberg-business',
|
||||||
|
builder: 'gutenberg',
|
||||||
|
});
|
||||||
|
|
||||||
|
const [url, init] = fetchMock.mock.calls[0] as [string, { headers: Record<string, string>; body: string }];
|
||||||
|
expect(url).toBe('https://api.x.ai/v1/chat/completions');
|
||||||
|
expect(init.headers.Authorization).toBe('Bearer test-key');
|
||||||
|
const body = JSON.parse(init.body) as { model: string; tool_choice: string; tools: Array<{ function: { name: string } }> };
|
||||||
|
expect(body.model).toBe('grok-3');
|
||||||
|
expect(body.tool_choice).toBe('required');
|
||||||
|
expect(body.tools[0]?.function.name).toBe('edit_heading');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('adds OpenRouter identification headers and uses the openrouter endpoint', async () => {
|
||||||
|
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
|
||||||
|
|
||||||
|
await callLlm({
|
||||||
|
provider: 'openrouter',
|
||||||
|
apiKey: 'test-key',
|
||||||
|
prompt: 'Change the heading',
|
||||||
|
siteId: 'gutenberg-business',
|
||||||
|
builder: 'gutenberg',
|
||||||
|
});
|
||||||
|
|
||||||
|
const [url, init] = fetchMock.mock.calls[0] as [string, { headers: Record<string, string>; body: string }];
|
||||||
|
expect(url).toBe('https://openrouter.ai/api/v1/chat/completions');
|
||||||
|
expect(init.headers['HTTP-Referer']).toBeDefined();
|
||||||
|
expect(init.headers['X-Title']).toBeDefined();
|
||||||
|
expect(JSON.parse(init.body).model).toBe('x-ai/grok-4.6');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('includes page slugs in the user message when provided', async () => {
|
||||||
|
fetchMock.mockResolvedValue({ ok: true, json: async () => ({ choices: [] }) });
|
||||||
|
|
||||||
|
await callLlm({
|
||||||
|
provider: 'grok',
|
||||||
|
apiKey: 'test-key',
|
||||||
|
prompt: 'Change the heading',
|
||||||
|
siteId: 'gutenberg-business',
|
||||||
|
builder: 'gutenberg',
|
||||||
|
pages: ['homepage', 'about', 'contact'],
|
||||||
|
});
|
||||||
|
|
||||||
|
const [, init] = fetchMock.mock.calls[0] as [string, { body: string }];
|
||||||
|
const body = JSON.parse(init.body) as { messages: Array<{ content: string }> };
|
||||||
|
expect(body.messages[1]?.content).toContain('pages=homepage,about,contact');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('throws when the provider returns a non-ok response', async () => {
|
||||||
|
fetchMock.mockResolvedValue({ ok: false, status: 401 });
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
callLlm({ provider: 'grok', apiKey: 'bad', prompt: 'x', siteId: 's', builder: 'b' }),
|
||||||
|
).rejects.toThrow('LLM HTTP 401');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -2,8 +2,8 @@
|
|||||||
"fixturePassed": 20,
|
"fixturePassed": 20,
|
||||||
"fixtureTotal": 20,
|
"fixtureTotal": 20,
|
||||||
"grokLive": {
|
"grokLive": {
|
||||||
"skipped": true,
|
"id": "gb-01",
|
||||||
"reason": "XAI_API_KEY not set"
|
"passed": true
|
||||||
},
|
},
|
||||||
"scores": [
|
"scores": [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1,21 +1,47 @@
|
|||||||
import type { GrokResponse } from './types.ts';
|
import type { GrokResponse } from './types.ts';
|
||||||
|
|
||||||
const url = 'https://api.x.ai/v1/chat/completions';
|
export type LlmProvider = 'grok' | 'openrouter';
|
||||||
|
|
||||||
export async function callGrok(input: {
|
export type LlmConfig = {
|
||||||
|
baseUrl: string;
|
||||||
|
model: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const PROVIDERS: Record<LlmProvider, LlmConfig> = {
|
||||||
|
grok: { baseUrl: 'https://api.x.ai/v1', model: 'grok-3' },
|
||||||
|
openrouter: { baseUrl: 'https://openrouter.ai/api/v1', model: 'x-ai/grok-4.6' },
|
||||||
|
};
|
||||||
|
|
||||||
|
export function resolveProvider(provider: LlmProvider, model?: string): LlmConfig {
|
||||||
|
const config = PROVIDERS[provider];
|
||||||
|
return { baseUrl: config.baseUrl, model: model ?? config.model };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function callLlm(input: {
|
||||||
|
provider: LlmProvider;
|
||||||
apiKey: string;
|
apiKey: string;
|
||||||
|
model?: string;
|
||||||
prompt: string;
|
prompt: string;
|
||||||
siteId: string;
|
siteId: string;
|
||||||
builder: string;
|
builder: string;
|
||||||
|
pages?: string[];
|
||||||
}): Promise<GrokResponse> {
|
}): Promise<GrokResponse> {
|
||||||
const response = await fetch(url, {
|
const { baseUrl, model } = resolveProvider(input.provider, input.model);
|
||||||
method: 'POST',
|
|
||||||
headers: {
|
const headers: Record<string, string> = {
|
||||||
Authorization: `Bearer ${input.apiKey}`,
|
Authorization: `Bearer ${input.apiKey}`,
|
||||||
'Content-Type': 'application/json',
|
'Content-Type': 'application/json',
|
||||||
},
|
};
|
||||||
|
if (input.provider === 'openrouter') {
|
||||||
|
headers['HTTP-Referer'] = 'https://wursor.dev';
|
||||||
|
headers['X-Title'] = 'Wursor golden harness';
|
||||||
|
}
|
||||||
|
|
||||||
|
const response = await fetch(`${baseUrl}/chat/completions`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers,
|
||||||
body: JSON.stringify({
|
body: JSON.stringify({
|
||||||
model: 'grok-3',
|
model,
|
||||||
messages: [
|
messages: [
|
||||||
{
|
{
|
||||||
role: 'system',
|
role: 'system',
|
||||||
@@ -24,7 +50,7 @@ export async function callGrok(input: {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
role: 'user',
|
role: 'user',
|
||||||
content: `site=${input.siteId} builder=${input.builder}\n${input.prompt}`,
|
content: `site=${input.siteId} builder=${input.builder}${input.pages ? ` pages=${input.pages.join(',')}` : ''}\n${input.prompt}`,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
tools: [
|
tools: [
|
||||||
@@ -71,7 +97,7 @@ export async function callGrok(input: {
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
throw new Error(`Grok HTTP ${response.status}`);
|
throw new Error(`LLM HTTP ${response.status}`);
|
||||||
}
|
}
|
||||||
return (await response.json()) as GrokResponse;
|
return (await response.json()) as GrokResponse;
|
||||||
}
|
}
|
||||||
@@ -1,17 +1,29 @@
|
|||||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||||
import { dirname, join } from 'node:path';
|
import { dirname, join } from 'node:path';
|
||||||
|
import { loadEnvFile } from 'node:process';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { detectBuilder } from './builder-detect.ts';
|
import { detectBuilder } from './builder-detect.ts';
|
||||||
import { asGrokResponse, expectedCalls } from './expected-calls.ts';
|
import { asGrokResponse, expectedCalls } from './expected-calls.ts';
|
||||||
import { callGrok } from './grok-client.ts';
|
import { callLlm, type LlmProvider } from './llm-client.ts';
|
||||||
import { loadPrompts } from './load-prompts.ts';
|
import { loadPrompts } from './load-prompts.ts';
|
||||||
import { loadSite } from './load-site.ts';
|
import { loadSite } from './load-site.ts';
|
||||||
import { scoreGrokResponse } from './score.ts';
|
import { scoreGrokResponse } from './score.ts';
|
||||||
|
|
||||||
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
|
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
|
||||||
|
const repoRoot = join(goldenRoot, '..', '..');
|
||||||
|
|
||||||
|
try {
|
||||||
|
loadEnvFile(join(repoRoot, '.env'));
|
||||||
|
} catch {
|
||||||
|
// no .env at repo root — live run will skip
|
||||||
|
}
|
||||||
|
|
||||||
|
function provider(): LlmProvider {
|
||||||
|
return process.env.LLM_PROVIDER === 'openrouter' ? 'openrouter' : 'grok';
|
||||||
|
}
|
||||||
|
|
||||||
function key(): string | undefined {
|
function key(): string | undefined {
|
||||||
const value = process.env.XAI_API_KEY;
|
const value = provider() === 'openrouter' ? process.env.OPENROUTER_API_KEY : process.env.XAI_API_KEY;
|
||||||
return value !== undefined && value !== '' ? value : undefined;
|
return value !== undefined && value !== '' ? value : undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -39,11 +51,14 @@ if (apiKey !== undefined) {
|
|||||||
}
|
}
|
||||||
const site = loadSite(prompt.site);
|
const site = loadSite(prompt.site);
|
||||||
try {
|
try {
|
||||||
const grok = await callGrok({
|
const grok = await callLlm({
|
||||||
|
provider: provider(),
|
||||||
apiKey,
|
apiKey,
|
||||||
|
model: provider() === 'openrouter' ? process.env.OPENROUTER_MODEL : undefined,
|
||||||
prompt: prompt.prompt,
|
prompt: prompt.prompt,
|
||||||
siteId: site.id,
|
siteId: site.id,
|
||||||
builder: detectBuilder(site),
|
builder: detectBuilder(site),
|
||||||
|
pages: site.posts.map((post) => post.slug),
|
||||||
});
|
});
|
||||||
grokLive = {
|
grokLive = {
|
||||||
id: prompt.id,
|
id: prompt.id,
|
||||||
@@ -57,7 +72,10 @@ if (apiKey !== undefined) {
|
|||||||
const report = {
|
const report = {
|
||||||
fixturePassed: fixtureScores.filter((row) => row.passed).length,
|
fixturePassed: fixtureScores.filter((row) => row.passed).length,
|
||||||
fixtureTotal: fixtureScores.length,
|
fixtureTotal: fixtureScores.length,
|
||||||
grokLive: grokLive ?? { skipped: true, reason: 'XAI_API_KEY not set' },
|
grokLive: grokLive ?? {
|
||||||
|
skipped: true,
|
||||||
|
reason: provider() === 'openrouter' ? 'OPENROUTER_API_KEY not set' : 'XAI_API_KEY not set',
|
||||||
|
},
|
||||||
scores: fixtureScores,
|
scores: fixtureScores,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -6,7 +6,7 @@ Throwaway fixtures and scripts are allowed. Product UI is not.
|
|||||||
|
|
||||||
| Spike | File | Status |
|
| Spike | File | Status |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | partial — live Grok pending key |
|
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | done — live run scored via OpenRouter |
|
||||||
| Builder detect (R6 / R13) | [builder-detect.md](./builder-detect.md) | done |
|
| Builder detect (R6 / R13) | [builder-detect.md](./builder-detect.md) | done |
|
||||||
| Pairing threat model (R9) | [pairing-threat-model.md](./pairing-threat-model.md) | done |
|
| Pairing threat model (R9) | [pairing-threat-model.md](./pairing-threat-model.md) | done |
|
||||||
| Large-site mirror timing (R4) | [mirror-timing.md](./mirror-timing.md) | done — synthetic 2GB |
|
| Large-site mirror timing (R4) | [mirror-timing.md](./mirror-timing.md) | done — synthetic 2GB |
|
||||||
|
|||||||
+14
-8
@@ -1,6 +1,6 @@
|
|||||||
# Spike: golden-task harness (R7)
|
# Spike: golden-task harness (R7)
|
||||||
|
|
||||||
**Status:** partial — harness exists; live Grok run not scored (`XAI_API_KEY` unset)
|
**Status:** done — live run scored via OpenRouter (`x-ai/grok-4.6`), `gb-01` passed
|
||||||
|
|
||||||
## Question
|
## Question
|
||||||
|
|
||||||
@@ -15,7 +15,7 @@ Can we score a model on WordPress tasks without vibes?
|
|||||||
|
|
||||||
## Result
|
## Result
|
||||||
|
|
||||||
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. No, if it means we have a Grok quality number. This machine has no `XAI_API_KEY`, so the live call was skipped.
|
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. The harness is now live-scored through OpenRouter.
|
||||||
|
|
||||||
### What exists
|
### What exists
|
||||||
|
|
||||||
@@ -24,22 +24,28 @@ Yes, if “score” means: apply a tool call to a fixture and assert the new hea
|
|||||||
| 20 prompts | `e2e/golden/prompts.json` |
|
| 20 prompts | `e2e/golden/prompts.json` |
|
||||||
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
|
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
|
||||||
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
|
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
|
||||||
| Apply + assert + Grok parser | `e2e/golden/src/` |
|
| Apply + assert + LLM parser | `e2e/golden/src/` |
|
||||||
|
| Provider client (grok + openrouter) | `e2e/golden/src/llm-client.ts` |
|
||||||
| Scoreboard | `e2e/golden/runs/latest.json` |
|
| Scoreboard | `e2e/golden/runs/latest.json` |
|
||||||
|
|
||||||
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
|
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
|
||||||
|
|
||||||
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces. Live Grok: skipped.
|
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces and **gb-01 passed live** (`x-ai/grok-4.6` via OpenRouter).
|
||||||
|
|
||||||
`pnpm test:e2e` — 21 tests, including the scorer.
|
`pnpm test:e2e` — 28 tests, including the scorer and the provider client.
|
||||||
|
|
||||||
### How to score a real Grok run
|
### How to score a real run
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
XAI_API_KEY=… pnpm --filter @wursor/e2e golden
|
# .env: LLM_PROVIDER=openrouter, OPENROUTER_API_KEY=…
|
||||||
|
pnpm --filter @wursor/e2e golden
|
||||||
```
|
```
|
||||||
|
|
||||||
That sends `gb-01` through `api.x.ai` and asserts the homepage heading.
|
Sends `gb-01` through OpenRouter and asserts the homepage heading.
|
||||||
|
|
||||||
|
### Finding
|
||||||
|
|
||||||
|
The model needs the page slugs in context. The first live call returned `page: "home"` and failed; after passing `pages=homepage,about,…` in the prompt, `gb-01` passed. The real agent must ship page/site context with every tool-call prompt.
|
||||||
|
|
||||||
### Decision
|
### Decision
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user