feat: Phase 0 workspace, spikes, and golden harness
Stand up the monorepo skeleton and land the first Phase 0 artifacts: pairing threat model, plugin catalog, builder detect, 20-prompt golden harness, and synthetic 2GB mirror timing.
This commit is contained in:
@@ -0,0 +1,18 @@
|
|||||||
|
# Copy to .env. Never commit real values.
|
||||||
|
|
||||||
|
# API
|
||||||
|
PORT=3000
|
||||||
|
DATABASE_URL=postgres://wursor:wursor@localhost:5432/wursor
|
||||||
|
REDIS_URL=redis://localhost:6379
|
||||||
|
|
||||||
|
# Auth
|
||||||
|
SESSION_SECRET=replace-me
|
||||||
|
|
||||||
|
# LLM — Grok is the default adapter. Switch provider with LLM_PROVIDER.
|
||||||
|
LLM_PROVIDER=grok
|
||||||
|
XAI_API_KEY=
|
||||||
|
LLM_FALLBACK_PROVIDER=
|
||||||
|
|
||||||
|
# Sandbox
|
||||||
|
DOCKER_HOST=
|
||||||
|
WARM_POOL_HOT_SPARES=2
|
||||||
@@ -6,6 +6,12 @@ node_modules/
|
|||||||
.wp-env/
|
.wp-env/
|
||||||
vendor/
|
vendor/
|
||||||
dist/
|
dist/
|
||||||
|
coverage/
|
||||||
|
playwright-report/
|
||||||
|
test-results/
|
||||||
|
*.tsbuildinfo
|
||||||
|
.phpunit.result.cache
|
||||||
*.log
|
*.log
|
||||||
.idea/
|
.idea/
|
||||||
.vscode/
|
.vscode/
|
||||||
|
e2e/fixtures/large-exports/
|
||||||
|
|||||||
@@ -9,12 +9,12 @@ Non-technical WordPress site owners describe what they want; Wursor makes it hap
|
|||||||
## Repo layout
|
## Repo layout
|
||||||
|
|
||||||
```
|
```
|
||||||
api/ Node.js + TypeScript API server (session manager, agent orchestrator,
|
api/ Node.js + TypeScript API server (empty until Phase 0 gate)
|
||||||
playbook runner, sandbox manager, deploy manager, plugin client)
|
web/ React + TypeScript frontend (empty until Phase 0 gate)
|
||||||
web/ React + TypeScript frontend (chat, preview, approve/reject, deploy history)
|
plugin/ WordPress plugin (PHP) — empty until Phase 0 gate
|
||||||
plugin/ WordPress plugin (PHP) — the connector on the user's hosting
|
|
||||||
infrastructure/ Docker images, warm pool, GC, deploy scripts
|
infrastructure/ Docker images, warm pool, GC, deploy scripts
|
||||||
e2e/ Playwright end-to-end tests
|
e2e/ Playwright + e2e/golden/ harness
|
||||||
|
spikes/ Phase 0 written results — gate before product code
|
||||||
PRD.md Product requirements (v2.0 — non-technical-first)
|
PRD.md Product requirements (v2.0 — non-technical-first)
|
||||||
IMPLEMENTATION.md TDD build guide with 8-sprint Phase 1 plan
|
IMPLEMENTATION.md TDD build guide with 8-sprint Phase 1 plan
|
||||||
```
|
```
|
||||||
|
|||||||
+258
-82
@@ -11,7 +11,8 @@
|
|||||||
1. [Architecture Overview](#1-architecture-overview)
|
1. [Architecture Overview](#1-architecture-overview)
|
||||||
2. [Project Structure](#2-project-structure)
|
2. [Project Structure](#2-project-structure)
|
||||||
3. [Build Phases](#3-build-phases)
|
3. [Build Phases](#3-build-phases)
|
||||||
4. [Phase 1 — Foundation (Weeks 1–8)](#4-phase-1--foundation-weeks-18)
|
4. [Phase 0 — Risk spikes (before Sprint 1)](#4-phase-0--risk-spikes-before-sprint-1)
|
||||||
|
5. [Phase 1 — Foundation (Weeks 1–8)](#5-phase-1--foundation-weeks-18)
|
||||||
- [Sprint 1: Web app scaffold + sandbox orchestration](#sprint-1)
|
- [Sprint 1: Web app scaffold + sandbox orchestration](#sprint-1)
|
||||||
- [Sprint 2: WordPress plugin connector](#sprint-2)
|
- [Sprint 2: WordPress plugin connector](#sprint-2)
|
||||||
- [Sprint 3: Agent orchestrator + playbook runner](#sprint-3)
|
- [Sprint 3: Agent orchestrator + playbook runner](#sprint-3)
|
||||||
@@ -20,10 +21,10 @@
|
|||||||
- [Sprint 6: Deploy + rollback](#sprint-6)
|
- [Sprint 6: Deploy + rollback](#sprint-6)
|
||||||
- [Sprint 7: Integration + exit criteria](#sprint-7)
|
- [Sprint 7: Integration + exit criteria](#sprint-7)
|
||||||
- [Sprint 8: Polish + alpha readiness](#sprint-8)
|
- [Sprint 8: Polish + alpha readiness](#sprint-8)
|
||||||
5. [Phase 2 — Intelligence (Weeks 9–16)](#5-phase-2--intelligence-weeks-916)
|
6. [Phase 2 — Intelligence (Weeks 9–16)](#6-phase-2--intelligence-weeks-916)
|
||||||
6. [TDD Rules](#6-tdd-rules)
|
7. [TDD Rules](#7-tdd-rules)
|
||||||
7. [CI/CD Pipeline](#7-cicd-pipeline)
|
8. [CI/CD Pipeline](#8-cicd-pipeline)
|
||||||
8. [Glossary](#8-glossary)
|
9. [Glossary](#9-glossary)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -93,7 +94,7 @@
|
|||||||
**Key stack decisions:**
|
**Key stack decisions:**
|
||||||
- **Backend:** Node.js + TypeScript (fastest path to a working API server; the orchestration is I/O-bound, not CPU-bound)
|
- **Backend:** Node.js + TypeScript (fastest path to a working API server; the orchestration is I/O-bound, not CPU-bound)
|
||||||
- **Frontend:** React + TypeScript (chat interface, preview iframe, deploy history)
|
- **Frontend:** React + TypeScript (chat interface, preview iframe, deploy history)
|
||||||
- **Sandbox:** Docker containers on VPS with pre-baked WordPress image
|
- **Sandbox:** Docker containers on VPS with a read-only pre-baked WordPress image and overlayfs site layers; media is proxied, not copied
|
||||||
- **Plugin:** PHP WordPress plugin (standard WordPress plugin architecture)
|
- **Plugin:** PHP WordPress plugin (standard WordPress plugin architecture)
|
||||||
- **Database:** PostgreSQL for Wursor's own data (users, sites, sessions, deploy history); MySQL inside sandboxes for WordPress
|
- **Database:** PostgreSQL for Wursor's own data (users, sites, sessions, deploy history); MySQL inside sandboxes for WordPress
|
||||||
- **Queue:** Redis for SSE streaming, task queues, and cache
|
- **Queue:** Redis for SSE streaming, task queues, and cache
|
||||||
@@ -124,10 +125,12 @@ wursor/
|
|||||||
│ │ │ ├── plugin-client.ts
|
│ │ │ ├── plugin-client.ts
|
||||||
│ │ │ └── warm-pool.ts
|
│ │ │ └── warm-pool.ts
|
||||||
│ │ ├── agents/
|
│ │ ├── agents/
|
||||||
│ │ │ ├── grok-client.ts # Grok API client
|
│ │ │ ├── llm-client.ts # Provider-agnostic LLM client (Grok adapter default)
|
||||||
│ │ │ ├── prompt-builder.ts # System prompt per session
|
│ │ │ ├── grok-adapter.ts # Grok messages + tool-calling
|
||||||
│ │ │ ├── tool-schemas.ts # Tool schemas → Grok format
|
│ │ │ ├── prompt-builder.ts # System prompt per session (playbook-sliced)
|
||||||
│ │ │ └── fallback.ts # Error handling, retry
|
│ │ │ ├── tool-schemas.ts # Allowlisted tool schemas only
|
||||||
|
│ │ │ ├── circuit-breaker.ts # Two verify failures → stop
|
||||||
|
│ │ │ └── fallback.ts # Per-playbook fallback + retry
|
||||||
│ │ ├── playbooks/
|
│ │ ├── playbooks/
|
||||||
│ │ │ ├── registry.ts # Playbook registry
|
│ │ │ ├── registry.ts # Playbook registry
|
||||||
│ │ │ ├── content.ts # Content edit playbook
|
│ │ │ ├── content.ts # Content edit playbook
|
||||||
@@ -135,16 +138,20 @@ wursor/
|
|||||||
│ │ │ ├── plugin.ts # Plugin install playbook
|
│ │ │ ├── plugin.ts # Plugin install playbook
|
||||||
│ │ │ └── site-build.ts # Site build playbook (P0 limited)
|
│ │ │ └── site-build.ts # Site build playbook (P0 limited)
|
||||||
│ │ ├── sandbox/
|
│ │ ├── sandbox/
|
||||||
│ │ │ ├── docker-client.ts # Docker API client
|
│ │ │ ├── docker-client.ts # Docker API client (overlayfs + pause)
|
||||||
│ │ │ ├── image-manager.ts # Pre-baked image management
|
│ │ │ ├── image-manager.ts # Pre-baked image management
|
||||||
│ │ │ ├── mirror.ts # Site mirroring (content, themes, plugins)
|
│ │ │ ├── mirror.ts # Task-scoped site mirroring
|
||||||
│ │ │ ├── media-sync.ts # Lazy media sync
|
│ │ │ ├── media-proxy.ts # Origin proxy for /wp-content/uploads
|
||||||
│ │ │ └── gc.ts # Garbage collection (idle, hard timeout)
|
│ │ │ ├── subset.ts # DB subset + secret redaction
|
||||||
|
│ │ │ ├── manifest.ts # path → sha256 delta + package cache
|
||||||
|
│ │ │ └── gc.ts # Pause-to-disk, idle + hard timeout
|
||||||
│ │ ├── deploy/
|
│ │ ├── deploy/
|
||||||
│ │ │ ├── diff-engine.ts # Compare sandbox → live site
|
│ │ │ ├── diff-engine.ts # Compare sandbox → live site
|
||||||
│ │ │ ├── pusher.ts # Push changes via plugin API
|
│ │ │ ├── pusher.ts # Two-phase prepare/commit + journal
|
||||||
│ │ │ ├── verifier.ts # Verify live site after deploy
|
│ │ │ ├── verifier.ts # Health contract (sandbox + live)
|
||||||
│ │ │ └── rollback.ts # Snapshot-based rollback
|
│ │ │ ├── drift.ts # Re-hash live site at approve time
|
||||||
|
│ │ │ ├── no-surprise.ts # Block slug/payment/role without confirm
|
||||||
|
│ │ │ └── rollback.ts # Journal walk + cloud snapshot restore
|
||||||
│ │ ├── models/
|
│ │ ├── models/
|
||||||
│ │ │ ├── user.ts
|
│ │ │ ├── user.ts
|
||||||
│ │ │ ├── site.ts
|
│ │ │ ├── site.ts
|
||||||
@@ -162,20 +169,24 @@ wursor/
|
|||||||
│ │ │ ├── deploy-manager.test.ts
|
│ │ │ ├── deploy-manager.test.ts
|
||||||
│ │ │ └── playbook-runner.test.ts
|
│ │ │ └── playbook-runner.test.ts
|
||||||
│ │ ├── agents/
|
│ │ ├── agents/
|
||||||
│ │ │ ├── grok-client.test.ts
|
│ │ │ ├── llm-client.test.ts
|
||||||
│ │ │ ├── prompt-builder.test.ts
|
│ │ │ ├── prompt-builder.test.ts
|
||||||
│ │ │ └── tool-schemas.test.ts
|
│ │ │ ├── tool-schemas.test.ts
|
||||||
|
│ │ │ └── circuit-breaker.test.ts
|
||||||
│ │ ├── playbooks/
|
│ │ ├── playbooks/
|
||||||
│ │ │ ├── content.test.ts
|
│ │ │ ├── content.test.ts
|
||||||
│ │ │ ├── design.test.ts
|
│ │ │ ├── design.test.ts
|
||||||
│ │ │ └── plugin.test.ts
|
│ │ │ └── plugin.test.ts
|
||||||
│ │ ├── sandbox/
|
│ │ ├── sandbox/
|
||||||
│ │ │ ├── mirror.test.ts
|
│ │ │ ├── mirror.test.ts
|
||||||
│ │ │ ├── media-sync.test.ts
|
│ │ │ ├── media-proxy.test.ts
|
||||||
|
│ │ │ ├── subset.test.ts
|
||||||
│ │ │ └── gc.test.ts
|
│ │ │ └── gc.test.ts
|
||||||
│ │ └── deploy/
|
│ │ └── deploy/
|
||||||
│ │ ├── diff-engine.test.ts
|
│ │ ├── diff-engine.test.ts
|
||||||
│ │ ├── pusher.test.ts
|
│ │ ├── pusher.test.ts
|
||||||
|
│ │ ├── drift.test.ts
|
||||||
|
│ │ ├── no-surprise.test.ts
|
||||||
│ │ └── rollback.test.ts
|
│ │ └── rollback.test.ts
|
||||||
│ ├── package.json
|
│ ├── package.json
|
||||||
│ └── tsconfig.json
|
│ └── tsconfig.json
|
||||||
@@ -268,9 +279,24 @@ wursor/
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Phase 1 — Foundation (Weeks 1–8)
|
## 4. Phase 0 — Risk spikes (before Sprint 1)
|
||||||
|
|
||||||
8 sprints, one per week. Every sprint produces a passing integration test.
|
These are not optional research notes. They lock decisions that Sprints 1–3 will encode as tests. Do not start the web-app scaffold until the four boxes below have a written result in this repo (a `spikes/` note is enough).
|
||||||
|
|
||||||
|
| Spike | Question | Done when |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **Golden-task harness (R7)** | Can we score a model on WordPress tasks without vibes? | 20 fixture prompts against at least 2 canned WP sites. Each prompt has an assertion (preview text, option value, or screenshot). One Grok run is scored. Harness lives under `e2e/golden/` even if the runner is still a script. |
|
||||||
|
| **Elementor detect (R6 / R13)** | How do we know what actually renders a page? | A site-info payload that reports `builder: elementor \| beaver \| divi \| gutenberg \| classic` from plugin slugs + post meta. Documented in the plugin API sketch. |
|
||||||
|
| **Pairing threat model (R9)** | What stops a leaked URL from owning the site? | Written threat model: 8+ char code, 5-min TTL, 5-attempt lockout, HMAC request signing, hashed+scoped tokens. This becomes the Sprint 2 auth tests. |
|
||||||
|
| **Large-site mirror timing (R4)** | Does the 5-minute exit criterion survive a real site? | Time a task-scoped content mirror + media proxy against one ≥2GB WP export (or a synthetic one). Record p50/p95. If content-edit slice is not on the page in ≤60s, the Sprint 1 slice is wrong. |
|
||||||
|
|
||||||
|
Also decide, in writing, the P0 plugin catalog (~40 slugs). The agent will not be allowed to install anything else.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Phase 1 — Foundation (Weeks 1–8)
|
||||||
|
|
||||||
|
8 sprints, one per week. Every sprint produces a passing integration test. Risk IDs refer to [PRD.md §13](./PRD.md).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -326,8 +352,24 @@ describe('Mirror', () => {
|
|||||||
expect(await mirror.sandboxFileExists('/wp-content/themes/twentytwentyfour')).toBe(true);
|
expect(await mirror.sandboxFileExists('/wp-content/themes/twentytwentyfour')).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('lazy-syncs media only when accessed', async () => {
|
it('does not copy the media library; preview is served via origin proxy', async () => {
|
||||||
// Media should not be synced during mirror, only on first access
|
const mirror = new Mirror({ sandboxId: 'sb-123' });
|
||||||
|
await mirror.copyTheme('twentytwentyfour');
|
||||||
|
expect(await mirror.sandboxFileExists('/wp-content/uploads/2024/hero.jpg')).toBe(false);
|
||||||
|
expect(await mirror.mediaProxyTarget('/wp-content/uploads/2024/hero.jpg')).toMatch(/^https:\/\/example.com\//);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('copies a media file only when the agent replaces it', async () => {
|
||||||
|
const mirror = new Mirror({ sandboxId: 'sb-123' });
|
||||||
|
await mirror.stageReplacement('/wp-content/uploads/2024/hero.jpg', Buffer.from('new'));
|
||||||
|
expect(await mirror.sandboxFileExists('/wp-content/uploads/2024/hero.jpg')).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('mirrors a content-edit slice, not orders or transients', async () => {
|
||||||
|
const dump = await new Mirror({ sandboxId: 'sb-123' }).exportDbSubset({ playbook: 'content', postIds: [1] });
|
||||||
|
expect(dump.tables).toEqual(expect.arrayContaining(['wp_posts', 'wp_postmeta', 'wp_options']));
|
||||||
|
expect(dump.tables).not.toEqual(expect.arrayContaining(['wp_wc_orders', 'wp_comments']));
|
||||||
|
expect(dump.options).not.toEqual(expect.arrayContaining([expect.stringMatching(/(_key|_secret|smtp_pass)$/)]));
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -350,9 +392,10 @@ describe('DockerClient', () => {
|
|||||||
|
|
||||||
// api/__tests__/sandbox/gc.test.ts
|
// api/__tests__/sandbox/gc.test.ts
|
||||||
describe('GarbageCollection', () => {
|
describe('GarbageCollection', () => {
|
||||||
it('destroys sandboxes after 15 minutes of idle', async () => { /* ... */ });
|
it('pauses sandboxes to disk after 15 minutes of idle', async () => { /* ... */ });
|
||||||
|
it('resumes a paused sandbox in ≤ 2s', async () => { /* ... */ });
|
||||||
it('destroys sandboxes after 24 hours regardless', async () => { /* ... */ });
|
it('destroys sandboxes after 24 hours regardless', async () => { /* ... */ });
|
||||||
it('does not destroy active sandboxes', async () => { /* ... */ });
|
it('does not pause or destroy active sandboxes', async () => { /* ... */ });
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -365,16 +408,20 @@ describe('GarbageCollection', () => {
|
|||||||
- **`api/src/services/sandbox-manager.ts`** — Orchestrate sandbox lifecycle
|
- **`api/src/services/sandbox-manager.ts`** — Orchestrate sandbox lifecycle
|
||||||
- **`api/src/sandbox/docker-client.ts`** — Docker API client (dockerode)
|
- **`api/src/sandbox/docker-client.ts`** — Docker API client (dockerode)
|
||||||
- **`api/src/sandbox/image-manager.ts`** — Pre-baked image → Dockerfile
|
- **`api/src/sandbox/image-manager.ts`** — Pre-baked image → Dockerfile
|
||||||
- **`api/src/sandbox/mirror.ts`** — Site mirroring (stub plugin client)
|
- **`api/src/sandbox/mirror.ts`** — Task-scoped site mirroring (stub plugin client)
|
||||||
- **`api/src/sandbox/media-sync.ts`** — Lazy media sync (stub)
|
- **`api/src/sandbox/media-proxy.ts`** — Origin rewrite for `/wp-content/uploads` (no library copy)
|
||||||
- **`api/src/sandbox/gc.ts`** — Garbage collection (idle timeout, hard timeout)
|
- **`api/src/sandbox/subset.ts`** — DB subset + `*_key` / `*_secret` / `smtp_pass` redaction (R10)
|
||||||
- **`infrastructure/docker/Dockerfile.wordpress`** — Pre-baked image
|
- **`api/src/sandbox/manifest.ts`** — path→sha256 delta; wordpress.org packages from Wursor cache
|
||||||
- **`infrastructure/scripts/warm-pool.ts`** — Warm pool manager
|
- **`api/src/sandbox/gc.ts`** — Pause-to-disk on idle; destroy on 24h hard timeout
|
||||||
|
- **`infrastructure/docker/Dockerfile.wordpress`** — Pre-baked image (read-only base + overlayfs)
|
||||||
|
- **`infrastructure/scripts/warm-pool.ts`** — Paused images + 1–2 hot spares, not 5–10 running
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
- Web app with sign-up and chat interface
|
- Web app with sign-up and chat interface
|
||||||
- Sandbox spin-up from pre-baked image
|
- Sandbox spin-up from pre-baked image (overlay + pause)
|
||||||
|
- Media proxy: preview works with zero upload copy
|
||||||
|
- Content-edit DB subset excludes orders / secrets
|
||||||
- `e2e/chat-flow.test.ts` passing (sign-up → sees chat)
|
- `e2e/chat-flow.test.ts` passing (sign-up → sees chat)
|
||||||
- All unit tests passing
|
- All unit tests passing
|
||||||
|
|
||||||
@@ -394,8 +441,24 @@ class WursorAuthTest extends WP_UnitTestCase {
|
|||||||
public function test_generates_pairing_code() {
|
public function test_generates_pairing_code() {
|
||||||
$auth = new Wursor_Auth();
|
$auth = new Wursor_Auth();
|
||||||
$code = $auth->generate_pairing_code();
|
$code = $auth->generate_pairing_code();
|
||||||
$this->assertEquals(6, strlen($code));
|
$this->assertGreaterThanOrEqual(8, strlen($code));
|
||||||
$this->assertMatchesRegularExpression('/^[A-Z0-9]{6}$/', $code);
|
$this->assertMatchesRegularExpression('/^[A-Z0-9]{8,}$/', $code);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function test_pairing_code_expires_after_five_minutes() {
|
||||||
|
$auth = new Wursor_Auth();
|
||||||
|
$code = $auth->generate_pairing_code();
|
||||||
|
$auth->advance_clock(301);
|
||||||
|
$this->assertFalse($auth->redeem_pairing_code($code));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function test_locks_out_after_five_failed_attempts() {
|
||||||
|
$auth = new Wursor_Auth();
|
||||||
|
$auth->generate_pairing_code();
|
||||||
|
for ($i = 0; $i < 5; $i++) {
|
||||||
|
$auth->redeem_pairing_code('NOPE0000');
|
||||||
|
}
|
||||||
|
$this->assertTrue($auth->is_locked_out());
|
||||||
}
|
}
|
||||||
|
|
||||||
public function test_verifies_valid_token() {
|
public function test_verifies_valid_token() {
|
||||||
@@ -419,6 +482,9 @@ class WursorApiTest extends WP_UnitTestCase {
|
|||||||
$this->assertArrayHasKey('plugins', $response);
|
$this->assertArrayHasKey('plugins', $response);
|
||||||
$this->assertArrayHasKey('wordpress_version', $response);
|
$this->assertArrayHasKey('wordpress_version', $response);
|
||||||
$this->assertArrayHasKey('php_version', $response);
|
$this->assertArrayHasKey('php_version', $response);
|
||||||
|
$this->assertArrayHasKey('builder', $response);
|
||||||
|
$this->assertArrayHasKey('capabilities', $response);
|
||||||
|
$this->assertArrayHasKey('preflight', $response);
|
||||||
}
|
}
|
||||||
|
|
||||||
public function test_requires_auth() {
|
public function test_requires_auth() {
|
||||||
@@ -472,19 +538,21 @@ describe('SiteConnector', () => {
|
|||||||
**Step 2 — Implement**
|
**Step 2 — Implement**
|
||||||
|
|
||||||
- **`plugin/wursor.php`** — Plugin header, activation hook, bootstrap
|
- **`plugin/wursor.php`** — Plugin header, activation hook, bootstrap
|
||||||
- **`plugin/src/class-auth.php`** — Token generation, verification, pairing code
|
- **`plugin/src/class-auth.php`** — 8+ char pairing (5-min TTL, 5-attempt lockout), hashed scoped tokens (read vs deploy), HMAC request signing (R9)
|
||||||
- **`plugin/src/class-api.php`** — REST API endpoints (site-info, files, DB, WP-CLI)
|
- **`plugin/src/class-api.php`** — REST API endpoints (site-info, files, DB, WP-CLI); HMAC verified
|
||||||
- **`plugin/src/class-site-info.php`** — Site info provider (theme, plugins, WP version, PHP version)
|
- **`plugin/src/class-site-info.php`** — Theme, plugins, WP/PHP version, `builder`, capability tiers, pre-flight probe (disk, `DISALLOW_FILE_MODS`, cache flush, REST alive)
|
||||||
- **`plugin/src/class-admin.php`** — Admin settings page (pairing code display)
|
- **`plugin/src/class-admin.php`** — Admin settings page (pairing code display)
|
||||||
- **`api/src/services/plugin-client.ts`** — HTTP client for the plugin API
|
- **`api/src/services/plugin-client.ts`** — HTTP client for the plugin API (signs requests)
|
||||||
- **`api/src/routes/sites.ts`** — Site connection flow, pairing
|
- **`api/src/routes/sites.ts`** — Site connection flow, pairing, capability-tier response
|
||||||
- **`web/src/components/SiteConnector.tsx`** — Pairing UI (show code, wait for connection)
|
- **`web/src/components/SiteConnector.tsx`** — Pairing UI (show code, wait for connection)
|
||||||
- **`web/src/pages/ConnectSite.tsx`** — Connection page
|
- **`web/src/pages/ConnectSite.tsx`** — Connection page; tiered copy (“I can change text today…”) + concierge host-ticket email when PHP/WP is below the full-playbook matrix
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
- WordPress plugin with pairing and site-info API
|
- WordPress plugin with pairing and site-info API
|
||||||
- Plugin client in the API server
|
- Pairing is 8+ chars, TTL + lockout tested; tokens scoped and HMAC-signed
|
||||||
|
- `builder` + `capabilities` + `preflight` on site-info
|
||||||
|
- Content-only tier for WP 5.8–6.0 / PHP 7.4; full playbooks for WP 6.1+ / PHP 8.0+
|
||||||
- Connection flow: user installs plugin → gets code → enters in Wursor → connected
|
- Connection flow: user installs plugin → gets code → enters in Wursor → connected
|
||||||
- `plugin/__tests__/test-auth.php` and `test-api.php` passing
|
- `plugin/__tests__/test-auth.php` and `test-api.php` passing
|
||||||
- `api/__tests__/services/plugin-client.test.ts` passing
|
- `api/__tests__/services/plugin-client.test.ts` passing
|
||||||
@@ -501,16 +569,21 @@ describe('SiteConnector', () => {
|
|||||||
**Step 1 — Write the tests**
|
**Step 1 — Write the tests**
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// api/__tests__/agents/grok-client.test.ts
|
// api/__tests__/agents/llm-client.test.ts
|
||||||
describe('GrokClient', () => {
|
describe('LlmClient', () => {
|
||||||
it('sends a message and returns a response', async () => {
|
it('sends a message and returns a response via the Grok adapter', async () => {
|
||||||
const client = new GrokClient({ apiKey: 'test-key' });
|
const client = new LlmClient({ provider: 'grok', apiKey: 'test-key' });
|
||||||
const response = await client.send('Change the homepage heading to "Hello"');
|
const response = await client.send('Change the homepage heading to "Hello"');
|
||||||
expect(response.type).toBe('tool_call');
|
expect(response.type).toBe('tool_call');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('switches provider with an env/config change, not a rewrite', async () => {
|
||||||
|
const client = new LlmClient({ provider: 'fallback', apiKey: 'test-key' });
|
||||||
|
expect(client.provider).toBe('fallback');
|
||||||
|
});
|
||||||
|
|
||||||
it('handles API errors with a clear message', async () => {
|
it('handles API errors with a clear message', async () => {
|
||||||
const client = new GrokClient({ apiKey: 'invalid-key' });
|
const client = new LlmClient({ provider: 'grok', apiKey: 'invalid-key' });
|
||||||
await expect(client.send('hello')).rejects.toThrow('API error');
|
await expect(client.send('hello')).rejects.toThrow('API error');
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -540,12 +613,33 @@ describe('PromptBuilder', () => {
|
|||||||
|
|
||||||
// api/__tests__/agents/tool-schemas.test.ts
|
// api/__tests__/agents/tool-schemas.test.ts
|
||||||
describe('ToolSchemas', () => {
|
describe('ToolSchemas', () => {
|
||||||
it('generates tool schemas for the Grok API', () => {
|
it('generates tool schemas for the LLM provider', () => {
|
||||||
const schemas = generateToolSchemas();
|
const schemas = generateToolSchemas();
|
||||||
expect(schemas.length).toBeGreaterThan(0);
|
expect(schemas.length).toBeGreaterThan(0);
|
||||||
expect(schemas[0].name).toBe('wp_cli');
|
expect(schemas[0].name).toBe('wp_cli');
|
||||||
expect(schemas[0].parameters).toBeDefined();
|
expect(schemas[0].parameters).toBeDefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('does not expose eval, config, db DROP, rm, or arbitrary plugin URLs', () => {
|
||||||
|
const names = generateToolSchemas().flatMap((s) => [s.name, ...(s.parameters?.enum ?? [])]);
|
||||||
|
expect(names.join(' ')).not.toMatch(/wp eval|wp config|DROP TABLE|wp plugin install http/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// api/__tests__/agents/circuit-breaker.test.ts
|
||||||
|
describe('CircuitBreaker', () => {
|
||||||
|
it('stops the agent after two consecutive verify failures', async () => {
|
||||||
|
const breaker = new CircuitBreaker({ maxConsecutiveFailures: 2 });
|
||||||
|
await breaker.recordFailure();
|
||||||
|
await breaker.recordFailure();
|
||||||
|
expect(breaker.shouldHalt()).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('halts when the per-task token budget is exhausted', async () => {
|
||||||
|
const breaker = new CircuitBreaker({ maxToolRounds: 12, maxUsd: 0.5 });
|
||||||
|
breaker.recordUsage({ rounds: 12, usd: 0.1 });
|
||||||
|
expect(breaker.shouldHalt()).toBe(true);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// api/__tests__/services/agent-orchestrator.test.ts
|
// api/__tests__/services/agent-orchestrator.test.ts
|
||||||
@@ -594,12 +688,14 @@ describe('ChatPanel', () => {
|
|||||||
|
|
||||||
**Step 2 — Implement**
|
**Step 2 — Implement**
|
||||||
|
|
||||||
- **`api/src/agents/grok-client.ts`** — Grok API client (messages API, tool use, streaming)
|
- **`api/src/agents/llm-client.ts`** — Provider-agnostic client; Grok is the default adapter
|
||||||
- **`api/src/agents/prompt-builder.ts`** — Build system prompt per session
|
- **`api/src/agents/grok-adapter.ts`** — Grok messages API, tool use, streaming
|
||||||
- **`api/src/agents/tool-schemas.ts`** — Tool schemas → Grok format
|
- **`api/src/agents/prompt-builder.ts`** — Playbook-sliced system prompt (do not dump the whole site-info blob)
|
||||||
- **`api/src/agents/fallback.ts`** — Error handling, retry
|
- **`api/src/agents/tool-schemas.ts`** — Allowlisted tools only (R2, R7)
|
||||||
- **`api/src/services/agent-orchestrator.ts`** — Route requests, dispatch tools, stream results
|
- **`api/src/agents/circuit-breaker.ts`** — Two verify failures or budget cap → halt (R1, R8)
|
||||||
- **`api/src/services/playbook-runner.ts`** — Execute playbook steps in sandbox
|
- **`api/src/agents/fallback.ts`** — Per-playbook fallback provider + retry
|
||||||
|
- **`api/src/services/agent-orchestrator.ts`** — Route requests, dispatch tools, stream results, enforce budget
|
||||||
|
- **`api/src/services/playbook-runner.ts`** — Execute playbook steps in sandbox; checkpoint after each success
|
||||||
- **`api/src/playbooks/registry.ts`** — Playbook registry
|
- **`api/src/playbooks/registry.ts`** — Playbook registry
|
||||||
- **`api/src/routes/chat.ts`** — Chat message endpoint, SSE stream
|
- **`api/src/routes/chat.ts`** — Chat message endpoint, SSE stream
|
||||||
- **`web/src/components/ChatPanel.tsx`** — Chat UI with message list, input, typing indicator
|
- **`web/src/components/ChatPanel.tsx`** — Chat UI with message list, input, typing indicator
|
||||||
@@ -608,7 +704,9 @@ describe('ChatPanel', () => {
|
|||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
- Agent orchestrator routing requests to playbooks
|
- Agent orchestrator routing requests to playbooks
|
||||||
- Grok API client with tool-calling
|
- Provider-agnostic LLM client with Grok adapter and per-playbook fallback
|
||||||
|
- Tool allowlist: no `wp eval`, `wp config`, DROP, `rm`, or `wp plugin install <url>`
|
||||||
|
- Circuit breaker + per-task token/round budget
|
||||||
- Chat panel streaming agent responses
|
- Chat panel streaming agent responses
|
||||||
- All unit tests passing
|
- All unit tests passing
|
||||||
|
|
||||||
@@ -670,8 +768,15 @@ describe('ContentPlaybook', () => {
|
|||||||
expect(result.success).toBe(true);
|
expect(result.success).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('reverts changes on failure', async () => {
|
it('rewinds the last checkpoint when a change fails verify, instead of nuking the sandbox', async () => {
|
||||||
// If the change fails, the sandbox should be reset to the mirror state
|
const playbook = new ContentPlaybook({ sandboxId: 'sb-123' });
|
||||||
|
await playbook.editText({ page: 'homepage', target: 'Hello', replacement: 'World' });
|
||||||
|
const checkpoint = await playbook.lastCheckpoint();
|
||||||
|
await playbook.editText({ page: 'homepage', target: 'World', replacement: '<broken' });
|
||||||
|
expect(await playbook.verify()).toMatchObject({ ok: false });
|
||||||
|
await playbook.rewind();
|
||||||
|
expect(await playbook.currentCheckpoint()).toBe(checkpoint);
|
||||||
|
expect(await playbook.getPageContent('homepage')).toContain('World');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
@@ -679,11 +784,12 @@ describe('ContentPlaybook', () => {
|
|||||||
**Step 2 — Implement**
|
**Step 2 — Implement**
|
||||||
|
|
||||||
- **`api/src/playbooks/content.ts`** — Content playbook with tool calls
|
- **`api/src/playbooks/content.ts`** — Content playbook with tool calls
|
||||||
- `editText`: search DB for content → wp-cli `wp post update` or direct DB update
|
- `editText`: search DB for content → `wp post update` or REST. **No theme PHP edits for copy changes (R1).**
|
||||||
- `editHeading`: find heading in page HTML → update via WP-CLI or file edit
|
- `editHeading`: find heading via builder adapter (Gutenberg / Elementor / Classic) → update via REST or that builder's store
|
||||||
- `replaceImage`: upload new image → replace in content → verify
|
- `replaceImage`: upload new image → replace in content → verify (this is the one case that copies a media file into the sandbox)
|
||||||
- `addSection`: create new content block → add to page → verify
|
- `addSection`: create new content block via the detected builder adapter → add to page → verify
|
||||||
- Each method uses the plugin client to execute WP-CLI commands or file operations in the sandbox
|
- Each successful step writes a copy-on-write checkpoint
|
||||||
|
- Verify (health contract) runs after every step; failure rewinds one checkpoint
|
||||||
- After each change, the playbook triggers a preview refresh
|
- After each change, the playbook triggers a preview refresh
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
@@ -749,10 +855,11 @@ describe('DesignPlaybook', () => {
|
|||||||
|
|
||||||
- **`api/src/playbooks/design.ts`** — Design playbook
|
- **`api/src/playbooks/design.ts`** — Design playbook
|
||||||
- `changeTheme`: install theme via WP-CLI → activate → verify
|
- `changeTheme`: install theme via WP-CLI → activate → verify
|
||||||
- `changeLayout`: modify theme templates or page builder content → verify
|
- `changeLayout`: modify theme templates **or the detected builder adapter** (Elementor JSON / Gutenberg blocks / Classic HTML) → verify. Editing the wrong store is a failed test (R6, R13).
|
||||||
- `updateColors`: update theme.json → regenerate CSS → verify
|
- `updateColors`: update theme.json → regenerate CSS → verify
|
||||||
- `updateTypography`: update theme.json → verify
|
- `updateTypography`: update theme.json → verify
|
||||||
- `fixMobileLayout`: identify responsive CSS issues → fix → verify
|
- `fixMobileLayout`: identify responsive CSS issues → fix → verify
|
||||||
|
- Deploy lints any written PHP against the live site's declared PHP version
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
@@ -817,8 +924,26 @@ describe('Pusher', () => {
|
|||||||
expect(result.success).toBe(true);
|
expect(result.success).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('handles partial failures', async () => {
|
it('prepare-fails without touching the live site', async () => {
|
||||||
// If some files fail but others succeed, what happens? Roll back the batch.
|
const pusher = new Pusher({ siteUrl: 'https://example.com', token: 'valid-token' });
|
||||||
|
pusher.failNextPrepare();
|
||||||
|
const result = await pusher.push({ files: [{ path: '/wp-content/themes/twentytwentyfour/style.css', content: '...' }] });
|
||||||
|
expect(result.success).toBe(false);
|
||||||
|
expect(result.liveTouched).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('rolls back the journal when commit is partial', async () => {
|
||||||
|
const pusher = new Pusher({ siteUrl: 'https://example.com', token: 'valid-token' });
|
||||||
|
pusher.failOnJournalEntry(2);
|
||||||
|
const result = await pusher.push({
|
||||||
|
files: [
|
||||||
|
{ path: '/wp-content/themes/twentytwentyfour/style.css', content: 'a' },
|
||||||
|
{ path: '/wp-content/themes/twentytwentyfour/theme.json', content: 'b' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(result.success).toBe(false);
|
||||||
|
expect(result.rolledBack).toBe(true);
|
||||||
|
expect(await pusher.liveFile('/wp-content/themes/twentytwentyfour/style.css')).not.toBe('a');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -841,6 +966,39 @@ describe('Verifier', () => {
|
|||||||
const result = await verifier.checkAdmin();
|
const result = await verifier.checkAdmin();
|
||||||
expect(result.status).toBe(200);
|
expect(result.status).toBe(200);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('fails when siteurl/home drifted from intent', async () => {
|
||||||
|
const verifier = new Verifier({ siteUrl: 'https://example.com', intent: { home: 'https://example.com' } });
|
||||||
|
await verifier.injectOption('home', 'https://evil.example');
|
||||||
|
const result = await verifier.checkIntent();
|
||||||
|
expect(result.ok).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails a 200 homepage whose screenshot diverges from the sandbox', async () => {
|
||||||
|
const verifier = new Verifier({ siteUrl: 'https://example.com', sandboxUrl: 'http://sb-123.wursor.dev' });
|
||||||
|
const result = await verifier.checkScreenshotSsim();
|
||||||
|
expect(result.ssim).toBeGreaterThan(0.9);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// api/__tests__/deploy/drift.test.ts
|
||||||
|
describe('Drift', () => {
|
||||||
|
it('asks before deploy when the live site changed since the preview', async () => {
|
||||||
|
const drift = new DriftChecker({ siteUrl: 'https://example.com', token: 'valid-token' });
|
||||||
|
const verdict = await drift.compare(changesetTakenAtPreview);
|
||||||
|
expect(verdict.drifted).toBe(true);
|
||||||
|
expect(verdict.action).toBe('confirm');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// api/__tests__/deploy/no-surprise.test.ts
|
||||||
|
describe('NoSurprise', () => {
|
||||||
|
it('blocks slug, blog_public, payment, and role changes without an explicit confirm bullet', async () => {
|
||||||
|
const gate = new NoSurpriseGate();
|
||||||
|
const blocked = gate.review({ changedOptions: ['permalink_structure'], confirmed: [] });
|
||||||
|
expect(blocked.ok).toBe(false);
|
||||||
|
expect(blocked.bullets).toContain('permalink_structure');
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// api/__tests__/deploy/rollback.test.ts
|
// api/__tests__/deploy/rollback.test.ts
|
||||||
@@ -906,22 +1064,28 @@ describe('ApproveBar', () => {
|
|||||||
|
|
||||||
**Step 2 — Implement**
|
**Step 2 — Implement**
|
||||||
|
|
||||||
- **`plugin/src/class-deploy.php`** — Deploy receiver (file write, DB write, WP-CLI exec, snapshot)
|
- **`plugin/src/class-deploy.php`** — Two-phase prepare/commit, journaled file/DB/WP-CLI, maintenance mode for the commit window
|
||||||
- **`plugin/src/class-rollback.php`** — Snapshot-based rollback (files + DB)
|
- **`plugin/src/class-rollback.php`** — Journal walk-back + snapshot restore
|
||||||
- **`api/src/deploy/diff-engine.ts`** — Compare sandbox → live site
|
- **`api/src/deploy/diff-engine.ts`** — Compare sandbox → live site
|
||||||
- **`api/src/deploy/pusher.ts`** — Push changes via plugin API
|
- **`api/src/deploy/pusher.ts`** — Prepare then commit; never leave a partial live site
|
||||||
- **`api/src/deploy/verifier.ts`** — Verify live site after deploy
|
- **`api/src/deploy/verifier.ts`** — Health contract on sandbox *and* live (R1, R3)
|
||||||
- **`api/src/deploy/rollback.ts`** — Snapshot-based rollback
|
- **`api/src/deploy/drift.ts`** — Re-hash live site at approve time (R11)
|
||||||
- **`api/src/routes/deploy.ts`** — Approve, deploy, rollback endpoints
|
- **`api/src/deploy/no-surprise.ts`** — Block slug / `blog_public` / payment / role without confirm (R12)
|
||||||
- **`web/src/components/ApproveBar.tsx`** — Approve/reject buttons, confirmation dialog
|
- **`api/src/deploy/rollback.ts`** — Journal + last-3 cloud snapshots (Undo works if the site is down)
|
||||||
|
- **`api/src/routes/deploy.ts`** — Approve, deploy, rollback; first-N-deploys “watched” path (R14)
|
||||||
|
- **`web/src/components/ApproveBar.tsx`** — Approve/reject, confirmation, no-surprise bullets, drift prompt
|
||||||
- **`web/src/components/DeployTimeline.tsx`** — Deploy history with one-click undo
|
- **`web/src/components/DeployTimeline.tsx`** — Deploy history with one-click undo
|
||||||
- **`web/src/hooks/useDeploy.ts`** — Deploy state, polling
|
- **`web/src/hooks/useDeploy.ts`** — Deploy state, polling
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
- Deploy + rollback for files, DB, and plugins
|
- Two-phase deploy + journaled rollback for files, DB, and plugins
|
||||||
|
- Cloud copies of the last 3 snapshots
|
||||||
|
- Drift check and no-surprise gate on approve
|
||||||
|
- Health contract richer than HTTP 200
|
||||||
- Approve/reject UI with confirmation dialog
|
- Approve/reject UI with confirmation dialog
|
||||||
- Deploy history timeline with one-click undo
|
- Deploy history timeline with one-click undo
|
||||||
|
- `handles partial failures` is a real test, not a comment
|
||||||
- All unit tests passing
|
- All unit tests passing
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -948,7 +1112,7 @@ test('new user connects site, makes a content change, previews, approves in ≤5
|
|||||||
|
|
||||||
// 2. Connect site (simulated plugin)
|
// 2. Connect site (simulated plugin)
|
||||||
await expect(page.locator('.wursor-connect-site')).toBeVisible();
|
await expect(page.locator('.wursor-connect-site')).toBeVisible();
|
||||||
await page.locator('.wursor-pairing-code-input').fill('ABC123');
|
await page.locator('.wursor-pairing-code-input').fill('ABCD1234');
|
||||||
await page.locator('.wursor-connect-button').click();
|
await page.locator('.wursor-connect-button').click();
|
||||||
await expect(page.locator('.wursor-connected')).toBeVisible({ timeout: 10000 });
|
await expect(page.locator('.wursor-connected')).toBeVisible({ timeout: 10000 });
|
||||||
|
|
||||||
@@ -984,39 +1148,47 @@ test('new user connects site, makes a content change, previews, approves in ≤5
|
|||||||
#### Tasks
|
#### Tasks
|
||||||
|
|
||||||
- **Error states** — Wire each state from §8.5 into the UI
|
- **Error states** — Wire each state from §8.5 into the UI
|
||||||
|
- **Intent chips (R5)** — Empty state: “Change wording” / “New look” / “Add a form” / “Something’s broken”
|
||||||
|
- **Site-aware starters (R5)** — After connect, offer three specific sentences from the site scan (default H1, missing favicon, no contact page)
|
||||||
|
- **Structured reject (R5)** — Chips: “Wrong color” / “Too busy” / “Keep my logo” / “Undo only the last thing”
|
||||||
|
- **Point-and-talk spike** — Click in the preview → selector attached to the next chat turn
|
||||||
- **Mobile responsive** — Chat collapses to full-screen on mobile, preview opens in new tab
|
- **Mobile responsive** — Chat collapses to full-screen on mobile, preview opens in new tab
|
||||||
- **Email auth** — Magic link or password reset flow
|
- **Email auth** — Magic link or password reset flow
|
||||||
- **Plugin auto-update** — Plugin checks for updates from Wursor
|
- **Plugin auto-update** — Plugin checks for updates from Wursor
|
||||||
- **Telemetry** — Minimal events (sign-up, connect, task start, task approve, task reject, deploy) with consent dialog
|
- **Telemetry** — Minimal events (sign-up, connect, task start, task approve, task reject, deploy) with consent dialog
|
||||||
|
- **Watched first deploys (R14)** — First N deploys of a new account use the stricter health contract
|
||||||
- **Documentation** — `README.md` with install instructions and quickstart
|
- **Documentation** — `README.md` with install instructions and quickstart
|
||||||
- **Bug bash** — Internal team runs through the full flow
|
- **Bug bash** — Internal team runs through the full flow on a sacrificial WordPress site before any stranger’s
|
||||||
|
|
||||||
#### Deliverables
|
#### Deliverables
|
||||||
|
|
||||||
- Web app deployed to staging
|
- Web app deployed to staging
|
||||||
- Plugin packaged for WordPress plugin repo
|
- Plugin packaged for WordPress plugin repo
|
||||||
- Error states all wired
|
- Error states all wired
|
||||||
|
- Intent chips, starters, and structured reject live
|
||||||
- Minimal telemetry with consent
|
- Minimal telemetry with consent
|
||||||
- `README.md` updated for alpha users
|
- `README.md` updated for alpha users
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Phase 2 — Intelligence (Weeks 9–16)
|
## 6. Phase 2 — Intelligence (Weeks 9–16)
|
||||||
|
|
||||||
| Sprint | Focus | Files |
|
| Sprint | Focus | Files |
|
||||||
|--------|-------|-------|
|
|--------|-------|-------|
|
||||||
| 9 | Plugin playbook (install, configure, fix conflicts) | `api/src/playbooks/plugin.ts` |
|
| 9 | Plugin playbook (catalog only, reputation gate, egress watch, configure, fix conflicts) | `api/src/playbooks/plugin.ts`, `api/src/playbooks/catalog.ts` |
|
||||||
| 10 | Site build playbook (from scratch, limited) | `api/src/playbooks/site-build.ts` |
|
| 10 | Site build playbook (from scratch, limited) | `api/src/playbooks/site-build.ts` |
|
||||||
| 11 | Mobile-responsive preview | `web/src/components/Preview.tsx` |
|
| 11 | Mobile-responsive preview | `web/src/components/Preview.tsx` |
|
||||||
| 12 | Agent clarifying questions | `api/src/services/agent-orchestrator.ts` |
|
| 12 | Agent clarifying questions | `api/src/services/agent-orchestrator.ts` |
|
||||||
| 13 | Visual design picker (theme gallery) | `api/src/playbooks/design.ts`, `web/src/components/DesignPicker.tsx` |
|
| 13 | Visual design picker (theme gallery); pull fork-and-pick forward if Sprint 8 reject rate is high | `api/src/playbooks/design.ts`, `web/src/components/DesignPicker.tsx` |
|
||||||
| 14 | Multi-step workflows (queue changes) | `api/src/services/playbook-runner.ts` |
|
| 14 | Multi-step workflows (queue changes) | `api/src/services/playbook-runner.ts` |
|
||||||
| 15 | Closed alpha with 10–20 users | Telemetry review, baselines |
|
| 15 | Closed alpha with 10–20 users — must include ≥1 Elementor site and ≥1 managed host (R13). Standby replica spike if mirror p95 missed the 5-min exit. | Telemetry review, baselines |
|
||||||
| 16 | Alpha feedback → Phase 2 exit review | All §11 baselines collected |
|
| 16 | Alpha feedback → Phase 2 exit review | All §11 baselines collected |
|
||||||
|
|
||||||
|
Sprint 9 acceptance (R2): agent can install only from the ~40-slug catalog; reputation gate fails closed; unexpected sandbox egress aborts the install; deploy re-checks reputation even after approve. `wp plugin install <url>` remains absent from the tool schema.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. TDD Rules
|
## 7. TDD Rules
|
||||||
|
|
||||||
1. **Write the test first.** No implementation code is written without a failing test.
|
1. **Write the test first.** No implementation code is written without a failing test.
|
||||||
2. **One assertion per test.** Each test verifies exactly one behavior.
|
2. **One assertion per test.** Each test verifies exactly one behavior.
|
||||||
@@ -1028,7 +1200,7 @@ test('new user connects site, makes a content change, previews, approves in ≤5
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. CI/CD Pipeline
|
## 8. CI/CD Pipeline
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# .github/workflows/ci.yml — runs on every PR
|
# .github/workflows/ci.yml — runs on every PR
|
||||||
@@ -1090,7 +1262,7 @@ jobs:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Glossary
|
## 9. Glossary
|
||||||
|
|
||||||
| Term | Definition |
|
| Term | Definition |
|
||||||
|------|------------|
|
|------|------------|
|
||||||
@@ -1099,7 +1271,11 @@ jobs:
|
|||||||
| **Plugin connector** | The WordPress plugin that connects the user's site to Wursor |
|
| **Plugin connector** | The WordPress plugin that connects the user's site to Wursor |
|
||||||
| **Mirror** | The process of copying a site's theme, plugins, content, and settings into a sandbox |
|
| **Mirror** | The process of copying a site's theme, plugins, content, and settings into a sandbox |
|
||||||
| **Deploy** | The process of applying sandbox changes to the live site |
|
| **Deploy** | The process of applying sandbox changes to the live site |
|
||||||
| **Warm pool** | Pre-booted WordPress containers ready to accept a mirror |
|
| **Warm pool** | Paused WordPress images plus a small number of hot spares, ready to accept a task-scoped mirror |
|
||||||
|
| **Media proxy** | Sandbox nginx rewrite of `/wp-content/uploads/*` to the live origin |
|
||||||
|
| **Capability tier** | content-safe / design-safe / install-safe, computed at connect |
|
||||||
|
| **Changeset journal** | Numbered deploy operations; rollback walks them backwards |
|
||||||
|
| **No-surprise gate** | Blocks slug / visibility / payment / role deploys without an explicit confirm |
|
||||||
| **SSE** | Server-Sent Events — the protocol used to stream agent responses to the frontend |
|
| **SSE** | Server-Sent Events — the protocol used to stream agent responses to the frontend |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -185,14 +185,14 @@ Every user request maps to a **playbook** — a structured, multi-step agent wor
|
|||||||
|
|
||||||
#### 7.1.4 WordPress plugin connector
|
#### 7.1.4 WordPress plugin connector
|
||||||
- One-click install from wp-admin plugin directory
|
- One-click install from wp-admin plugin directory
|
||||||
- Pairing flow: user copies a 6-character code from Wursor web app, pastes it into the plugin
|
- Pairing flow: user copies an 8+ character code from Wursor web app (5-minute TTL, 5-attempt lockout, bound to account + site URL), pastes it into the plugin
|
||||||
- Plugin exposes: site info (theme, plugins, content), file system (read/write), database (read/write), WP-CLI (full access)
|
- Plugin exposes: site info (theme, plugins, content), file system (read/write), database (read/write), WP-CLI (full access)
|
||||||
- All communication over HTTPS with token-based auth
|
- All communication over HTTPS with token-based auth
|
||||||
- Plugin auto-updates; no user maintenance
|
- Plugin auto-updates; no user maintenance
|
||||||
|
|
||||||
#### 7.1.5 Cloud sandbox orchestration
|
#### 7.1.5 Cloud sandbox orchestration
|
||||||
- Spin up a sandbox in ≤ 10 seconds (warm pool)
|
- Spin up a sandbox in ≤ 10 seconds (warm pool)
|
||||||
- Mirror the user's site: theme, plugins, content, media (lazy sync for media)
|
- Mirror the user's site: theme, plugins, content (task-scoped). Media is proxied from origin, not copied.
|
||||||
- Full network access (so the agent can install plugins from the WordPress repo)
|
- Full network access (so the agent can install plugins from the WordPress repo)
|
||||||
- 15-minute idle timeout (auto-hibernate, resume on user interaction)
|
- 15-minute idle timeout (auto-hibernate, resume on user interaction)
|
||||||
- 24-hour hard timeout (sandbox destroyed, no exceptions)
|
- 24-hour hard timeout (sandbox destroyed, no exceptions)
|
||||||
@@ -223,9 +223,11 @@ Every user request maps to a **playbook** — a structured, multi-step agent wor
|
|||||||
#### 7.1.10 Safety & trust
|
#### 7.1.10 Safety & trust
|
||||||
- Every change is previewed before apply — no "apply now, preview later"
|
- Every change is previewed before apply — no "apply now, preview later"
|
||||||
- Agent has a "no-surprise" rule: it must surface any action that costs money (e.g., a paid plugin) or affects SEO (e.g., URL changes)
|
- Agent has a "no-surprise" rule: it must surface any action that costs money (e.g., a paid plugin) or affects SEO (e.g., URL changes)
|
||||||
|
- That rule is mechanical at deploy, not just conversational: URL/slug changes, `blog_public`, payment/shipping, and user/role table writes are blocked until the user confirms that specific bullet (R12)
|
||||||
- Agent role: "I changed your homepage layout. It also removed your sidebar widget. Is that OK?"
|
- Agent role: "I changed your homepage layout. It also removed your sidebar widget. Is that OK?"
|
||||||
- Deploy history: a timeline of all changes, with one-click undo per change
|
- Deploy history: a timeline of all changes, with one-click undo per change
|
||||||
- Undo reverts the last deploy (not individual file changes — the user sees "your site has been restored to before that change")
|
- Undo reverts the last deploy (not individual file changes — the user sees "your site has been restored to before that change")
|
||||||
|
- The last 3 deploy snapshots are stored in Wursor's cloud as well as on the site, so Undo still works if the live site is down (R3)
|
||||||
|
|
||||||
### 7.2 P1 — Follow-on
|
### 7.2 P1 — Follow-on
|
||||||
|
|
||||||
@@ -349,7 +351,7 @@ Every user request maps to a **playbook** — a structured, multi-step agent wor
|
|||||||
- **Site mirroring:**
|
- **Site mirroring:**
|
||||||
- Plugin list and active theme → installed immediately
|
- Plugin list and active theme → installed immediately
|
||||||
- Content (posts, pages, options) → pulled from the live site via the plugin API
|
- Content (posts, pages, options) → pulled from the live site via the plugin API
|
||||||
- Media files → lazy sync; only pulled when the preview or agent accesses them
|
- Media files → not copied. Sandbox nginx proxies `/wp-content/uploads/*` to the live origin (or a signed Wursor proxy). A file is pulled only when the agent replaces it.
|
||||||
- **Networking:** Sandboxes have full outbound internet access (for plugin installs, API calls). No inbound access except from the Wursor API server.
|
- **Networking:** Sandboxes have full outbound internet access (for plugin installs, API calls). No inbound access except from the Wursor API server.
|
||||||
- **Idle timeout:** 15 minutes. User typing or viewing the preview resets the timer.
|
- **Idle timeout:** 15 minutes. User typing or viewing the preview resets the timer.
|
||||||
- **Hard timeout:** 24 hours. Sandbox is destroyed regardless of state.
|
- **Hard timeout:** 24 hours. Sandbox is destroyed regardless of state.
|
||||||
@@ -364,9 +366,11 @@ When the user approves:
|
|||||||
3. **Database changes** — send SQL migration to the plugin's deploy API (or WP-CLI commands)
|
3. **Database changes** — send SQL migration to the plugin's deploy API (or WP-CLI commands)
|
||||||
4. **Plugin changes** — plugin installs/activations sent as WP-CLI commands
|
4. **Plugin changes** — plugin installs/activations sent as WP-CLI commands
|
||||||
5. **Verify** — plugin confirms the live site is functional after changes
|
5. **Verify** — plugin confirms the live site is functional after changes
|
||||||
6. **Snapshot** — deploy snapshot stored for rollback (files + DB state)
|
6. **Snapshot** — deploy snapshot stored for rollback (files + DB state) on the site **and** in Wursor's cloud (last 3). Cloud copy is what Undo uses if the live site is down.
|
||||||
|
7. **Drift check** — re-hash the live files/options in the changeset at approve time. If the live site changed since the preview, ask before clobbering (R11).
|
||||||
|
8. **No-surprise gate** — slug / `blog_public` / payment / role changes require an explicit confirm bullet (R12).
|
||||||
|
|
||||||
Rollback restores the files and database from the snapshot.
|
Rollback walks the changeset journal backwards, then restores from the snapshot if the journal is incomplete.
|
||||||
|
|
||||||
### 8.5 Error & offline states
|
### 8.5 Error & offline states
|
||||||
|
|
||||||
@@ -432,14 +436,20 @@ Rollback restores the files and database from the snapshot.
|
|||||||
|
|
||||||
### Phase 0 — Pivot & spec (now)
|
### Phase 0 — Pivot & spec (now)
|
||||||
- Rewrite PRD for non-technical-first
|
- Rewrite PRD for non-technical-first
|
||||||
- Spike: cloud sandbox orchestration (WordPress in Docker, warm pool, site mirroring)
|
- Risk rewrite: §13 is 14 risks with mitigations that map to tests or UI (this revision)
|
||||||
|
- Spike: golden-task eval harness (20 prompts × ≥2 canned WP sites, scored)
|
||||||
|
- Spike: Elementor / builder detection in site-info
|
||||||
|
- Spike: pairing threat model (8+ char, TTL, lockout, HMAC, scoped tokens)
|
||||||
|
- Spike: time a task-scoped content mirror + media proxy against a ≥2GB site
|
||||||
|
- Spike: cloud sandbox orchestration (WordPress in Docker, overlay + pause pool)
|
||||||
- Spike: WordPress plugin (REST API, file read/write, DB access, WP-CLI)
|
- Spike: WordPress plugin (REST API, file read/write, DB access, WP-CLI)
|
||||||
- Spike: basic chat + preview web app
|
- Spike: basic chat + preview web app
|
||||||
|
- Lock the P0 plugin catalog (~40 slugs) before any agent can install
|
||||||
|
|
||||||
### Phase 1 — Foundation (weeks 1–8)
|
### Phase 1 — Foundation (weeks 1–8)
|
||||||
- Web app: sign-up, site connection (plugin auth), chat, preview, approve/reject
|
- Web app: sign-up, site connection (plugin auth), chat, preview, approve/reject
|
||||||
- Sandbox infrastructure: warm pool, site mirroring, idle timeout, GC
|
- Sandbox infrastructure: overlay + pause-to-disk warm pool, task-scoped mirroring, media proxy, idle timeout, GC
|
||||||
- WordPress plugin: site info API, deploy receiver, rollback, auto-update
|
- WordPress plugin: site info API (builder + capability tiers + pre-flight), HMAC-signed pairing, two-phase deploy receiver, journaled rollback, auto-update
|
||||||
- Playbooks: content edit (text, images, pages), design change (layout, colors)
|
- Playbooks: content edit (text, images, pages), design change (layout, colors)
|
||||||
- Deploy history: timeline, one-click undo
|
- Deploy history: timeline, one-click undo
|
||||||
|
|
||||||
@@ -473,16 +483,149 @@ Rollback restores the files and database from the snapshot.
|
|||||||
|
|
||||||
## 13. Risks & Mitigations
|
## 13. Risks & Mitigations
|
||||||
|
|
||||||
| Risk | Impact | Mitigation |
|
Isolation, a warning dialog, and a fallback model are not mitigations. Each row below maps to a test, a playbook constraint, or a product surface. Residual impact assumes the mitigation ships; **Launch-blocking** means Phase 1 cannot exit without it.
|
||||||
|
|
||||||
|
### 13.1 Documented risks
|
||||||
|
|
||||||
|
#### R1 — Agent breaks the sandbox site
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Medium → Low for the live site; **High for session trust** until checkpoints ship |
|
||||||
|
| **Launch-blocking** | Circuit breaker and per-step verify. Full self-heal can follow. |
|
||||||
|
| **Why the old mitigation fails** | “GC and start fresh” keeps the live site safe and kills the session. A white-screen preview after 90 seconds is a churn event. Retrying the same trajectory also burns the warm pool and the model budget. |
|
||||||
|
| **Mitigation** | Copy-on-write checkpoint after every successful playbook step (overlayfs / Docker commit). Verify after every tool call (target URL 200, no new PHP fatal, `siteurl`/`home` unchanged). Two consecutive verify failures → circuit breaker, stop, and talk — do not thrash. Content playbooks prefer WP-CLI / REST (`wp post update`) over theme file surgery. A tiny internal self-heal path rewinds the last checkpoint before the user sees a dead preview. |
|
||||||
|
| **Owner / where** | Sprint 1 (overlay + GC), Sprint 3 (circuit breaker), Sprint 4 (REST-first content), Sprint 6 (reuse verifier on the sandbox). |
|
||||||
|
|
||||||
|
#### R2 — Agent installs a malicious plugin
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Medium → Low **if** the catalog and gates ship. Deploy path makes the unmitigated case High. |
|
||||||
|
| **Launch-blocking** | Tool-schema allowlist in Sprint 3. Catalog + reputation in Sprint 9, but the agent must not be able to `wp plugin install <url>` before then. |
|
||||||
|
| **Why the old mitigation fails** | wordpress.org is not a reviewed-safe catalog. Sandbox isolation only holds until the user clicks “Looks good → Apply.” A pretty form that phones home looks fine in preview. |
|
||||||
|
| **Mitigation** | P0/P1 install from a curated catalog of ~40 slugs only (CF7, WPForms, Yoast, RankMath, Woo, Elementor, WooPayments, …). Reputation gate before install: WPScan / Patchstack CVE, last updated < 18 months, `tested up to` within 2 majors, active-install floor, zip SHA against a Wursor mirror. Fail closed. While a new plugin first activates, sandbox egress is allowlisted (wordpress.org, gravatar, the user’s own domain); unexpected egress is surfaced in chat and the plugin is not applied. Static smell test rejects `eval(base64_decode`, `/e` preg, unexpected remote `file_get_contents`. Deploy re-checks reputation even after approve. No arbitrary ZIP, no premium marketplace URLs, no `wp plugin install <url>` in v1. |
|
||||||
|
| **Owner / where** | Sprint 3 (`tool-schemas.ts` allowlist), Sprint 9 (catalog + reputation + egress watch). |
|
||||||
|
|
||||||
|
#### R3 — Deploy to live site fails
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | High → Medium (hosts stay messy) |
|
||||||
|
| **Launch-blocking** | **Yes** |
|
||||||
|
| **Why the old mitigation fails** | The failure mode is a **partial** deploy, not a clean 500. CSS lands, `wp_options` is half-updated, object cache serves the old theme, then rollback fails because the customer’s disk is full. HTTP 200 does not catch a wrong `home` URL or a dropped menu. |
|
||||||
|
| **Mitigation** | Two-phase deploy: prepare artifacts under `wp-content/upgrade/wursor-<id>/`, then commit. Prepare failure = live site untouched. Pre-flight before the confirm dialog: disk free, `ABSPATH` writable, `DISALLOW_FILE_MODS` off, PHP memory, `post_max_size`, can flush object/opcode cache, can enter maintenance mode. Failure copy is host-ticket language, not “deploy failed.” Every file / option / WP-CLI call is a numbered journal entry; rollback walks it backwards. Health contract: homepage + one inner page + `/wp-json` + `wp-login.php` + no new fatal + `siteurl`/`home` match intent + screenshot SSIM vs sandbox. Last 3 snapshots live in **Wursor’s cloud**, not only on the customer disk, so Undo still works when the site is down. Maintenance mode for the commit window (seconds). Detect managed hosts at connect; v1 still file-pushes but must purge known caches. Canary rewrite (`/?wursor_canary=`) is stretch; journal + cloud snapshot is the P0 version. |
|
||||||
|
| **Owner / where** | Sprint 2 (pre-flight on site-info), Sprint 6 (two-phase, journal, cloud snapshot, health contract). The existing `handles partial failures` test is acceptance, not a comment. |
|
||||||
|
|
||||||
|
#### R4 — Mirroring a large site is slow
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | High → Medium once thin-slice + media proxy ship (standby replica is Phase 2) |
|
||||||
|
| **Launch-blocking** | **Yes** for task-scoped mirror + media proxy. Standby replica is not. |
|
||||||
|
| **Why the old mitigation fails** | Warm pool absorbs *boot*, not *copy*. “Incremental” was unnamed. The 5-minute Phase 1 exit criterion dies on a Woo store with 8 years of posts. Media is often not the bottleneck — `wp_posts`, plugin folders, and upload thumbs are. |
|
||||||
|
| **Mitigation** | **Task-scoped mirrors.** Content edit → target posts + `wp_options` + menus + active theme/plugins (skip orders, logs, transients, revisions). Design → theme + `theme.json` + templates + a few representative pages. Plugin install → current site + the new plugin, still skip Woo order tables. Theme swap / rebuild → fuller clone. **Remote media proxy:** sandbox nginx rewrites `/wp-content/uploads/*` to the live origin (or a signed Wursor proxy). Copy a file only when the agent replaces it. Plugin sends a path→sha256 manifest; wordpress.org packages come from a Wursor cache, not the customer host. Chunked resumable zstd batches over plugin REST, not one zip. Default DB exclude: `_transient_*`, action scheduler logs, Woo orders/customers for non-commerce playbooks. Progressive preview: show the target page as soon as *that* page is mirrored. **Phase 2:** a per-site standby replica synced by webhook / 15-min cron so a task is `fork replica`, not `pull production`. |
|
||||||
|
| **Owner / where** | Sprint 1 (proxy, hash manifest, subset dump), §14 media decision, Phase 2 standby replica. |
|
||||||
|
|
||||||
|
#### R5 — User can't describe what they want
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Medium → Medium. Unmitigated this is **High for activation** — vague language is the default for the primary persona. |
|
||||||
|
| **Launch-blocking** | Intent chips, site-aware starters, and structured reject in Sprint 8. Full design-fork picker can wait on alpha reject rate. |
|
||||||
|
| **Why the old mitigation fails** | Clarifying questions feel like a support form. The §11 “claps back” metric (reject → re-describe ≤ 20%) *is* this risk. |
|
||||||
|
| **Mitigation** | Empty state is intent chips (“Change wording” / “New look” / “Add a form” / “Something’s broken”), not a blank chat. After connect, scan for default theme H1, missing favicon, no contact page, mobile overflow, and offer three *specific* starter sentences. Reject is chips (“Wrong color” / “Too busy” / “Keep my logo” / “Undo only the last thing”), each mapped to a constrained follow-up. Vague design prompts should *show* 2–3 cheap visual forks (theme.json / CSS / catalog themes) rather than ask “modern or classic?” Point-and-talk: click an element in the preview, then type “make this blue” — Phase 1 spike, not Phase 4 magic. Before/after slider on the preview. Voice input on mobile via browser SpeechRecognition is enough for alpha. |
|
||||||
|
| **Owner / where** | Sprint 8 (chips, starters, structured reject), Phase 1 spike (point-and-talk), Sprint 12 (disambiguation), pull a fork-and-pick slice forward from Sprint 13 if design-prompt reject rate is high. |
|
||||||
|
|
||||||
|
#### R6 — Plugin compatibility (old WordPress, old PHP)
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Medium → Medium |
|
||||||
|
| **Launch-blocking** | Version matrix + capability tiers at connect (Sprint 2). |
|
||||||
|
| **Why the old mitigation fails** | A warning a dentist cannot act on is a bounce. The real incompatibilities are Elementor vs Gutenberg vs Classic, security plugins that kill REST, and hosts that set `DISALLOW_FILE_MODS` — not PHP 7.4 vs 8.2 alone. |
|
||||||
|
| **Mitigation** | **Capability tiers, not a binary supported flag.** *Content-safe:* edit posts/pages via REST (always try to offer this). *Design-safe:* writable theme / `theme.json` / page-builder API. *Install-safe:* WP-CLI or filesystem and `DISALLOW_FILE_MODS` off. Chat: “I can change text on your site today. Installing plugins needs a WordPress update — I can preview that first if you want.” **P0 matrix (locked):** WP 6.1+ and PHP 8.0+ for full playbooks; WP 5.8–6.0 / PHP 7.4 are content-only. Publish this in the plugin readme. Detect Elementor / Beaver / Divi / Gutenberg at connect; content and layout playbooks must use the matching adapter or they will edit `post_content` while Elementor JSON is what renders. Sandbox may run newer PHP than production; deploy lints PHP against the live interpreter version. “Update WordPress for me” is a sandbox-then-approve playbook, not a warning. If PHP is old, generate a 4-line host-ticket email the user can forward. Connect-time scan also flags Wordfence / iThemes / disabled REST, `DISALLOW_FILE_EDIT`, `open_basedir`, object cache. |
|
||||||
|
| **Owner / where** | Phase 0 spike (matrix + builder detect), Sprint 2 (tiered connect + concierge copy), Sprint 4–5 (adapters for builders we actually see). |
|
||||||
|
|
||||||
|
#### R7 — Grok model quality for agentic tasks
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Medium → Medium. Unmitigated this is High until the eval harness has a score. |
|
||||||
|
| **Launch-blocking** | Golden-task harness (Phase 0) + tool allowlist (Sprint 3). |
|
||||||
|
| **Why the old mitigation fails** | “Have a fallback model” is not a design. WordPress is full of hallucinated WP-CLI flags. Grok stays the default *vendor*; it must not be the architecture. |
|
||||||
|
| **Mitigation** | Playbooks are the intelligence; the model fills slots (`page`, `old`, `new`) and a deterministic runner executes. Almost never emit free-form shell. Allowlisted tools only: a short WP-CLI list, REST routes, file paths under `wp-content/themes/{active}` and `wp-content/uploads`. Forbidden: `wp db query` with DROP/TRUNCATE, `wp eval`, `wp config`, `rm`, writing `wp-config.php`, touching `mu-plugins`. Golden-task harness from day 1: 20–50 fixture sites × ~20 prompts, asserting preview text / options / screenshot — not “the model said it worked.” Router: cheap/fast model classifies + asks one question; stronger model (Grok or fallback) only for design/plugin reasoning. Fallback is **per playbook**. Sprint 3 client is `llm-client.ts` with a Grok adapter; vendor switch is an env var. Self-critique before the user sees the preview (health contract + screenshot: “Did the H1 actually change?”). Store anonymized winning tool traces as few-shots. If the classifier is unsure, offer chips (R5) — do not improvise a theme rewrite. |
|
||||||
|
| **Owner / where** | Phase 0 (eval harness, before playbooks), Sprint 3 (allowlist + provider-agnostic client), Sprint 4–5 (slot-filling). |
|
||||||
|
|
||||||
|
#### R8 — Sandbox cost scales with usage
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Impact (original → residual)** | Low compute → Low. **Token spend is the real risk** (Medium) and was previously named but not mitigated. |
|
||||||
|
| **Launch-blocking** | Per-task token budget in Sprint 3. Pause-to-disk in Sprint 1. |
|
||||||
|
| **Why the old mitigation fails** | The old row admitted API spend is higher than VPS hours, then did nothing about it. A warm pool of *running* WP+MySQL+Redis boxes is idle burn. One runaway tool loop can cost more than a month of that user’s subscription. |
|
||||||
|
| **Mitigation (compute)** | 15-min idle → pause / checkpoint to disk, resume in ~2s. Warm pool = paused images + 1–2 hot spares per region, not 5–10 fully running. Shared read-only WordPress image + overlayfs site layers. Predictive pool on time-of-day / queue depth. Do not copy media (R4). Content-edit without a full container is stretch. |
|
||||||
|
| **Mitigation (tokens — the actual cost)** | Hard cap per task: e.g. 12 tool rounds and a dollar token budget. Hit the cap → R1 circuit breaker, not another retry. Playbook-specific short prompts; do not dump the entire site-info blob every turn. Cache prompt prefixes and repeated tool results (“what’s the active theme”). Price the unit against p95 token usage, not sandbox hours. Free tier = 1 concurrent sandbox + M token-budgeted tasks (kills sandbox-farming). |
|
||||||
|
| **Owner / where** | Sprint 1 (pause + overlay + no media copy), Sprint 3 (token budget), Phase 3 pricing (unit = task budget). |
|
||||||
|
|
||||||
|
### 13.2 Risks the original table did not name
|
||||||
|
|
||||||
|
The spec creates these. They are in scope for Phase 1.
|
||||||
|
|
||||||
|
#### R9 — Plugin is a privileged backdoor
|
||||||
|
|
||||||
|
Plugin exposes files + DB + WP-CLI. A 6-character pairing code (`[A-Z0-9]{6}`) with no stated rate limit, plus a stolen token, is site ownership.
|
||||||
|
|
||||||
|
**Mitigation:** Pairing codes are 8+ characters, 5-minute TTL, 5-attempt lockout, bound to account + site URL. Tokens are hashed at rest, scoped (read vs deploy), rotatable, stored in WP encrypted with a site salt. Requests are HMAC-signed (body + timestamp) so a leaked URL is not enough. **Launch-blocking. Sprint 2.**
|
||||||
|
|
||||||
|
#### R10 — Sandbox holds real customer PII
|
||||||
|
|
||||||
|
A full mirror copies Woo orders, emails, form submissions, and license keys into a container with outbound internet.
|
||||||
|
|
||||||
|
**Mitigation:** Default DB subset excludes orders, customers, and form entries. Redact options matching `*_key`, `*_secret`, `smtp_pass`. Egress allowlist (R2). 24-hour hard delete already helps; pin sandbox region to the user’s site region when we have it, and document retention. **Launch-blocking for subset + redact. Sprint 1.**
|
||||||
|
|
||||||
|
#### R11 — Live-site drift during a session
|
||||||
|
|
||||||
|
An agency or the user edits wp-admin while the sandbox is open. Deploy silently overwrites their work.
|
||||||
|
|
||||||
|
**Mitigation:** On approve, re-hash the live files/options in the changeset. If they drifted, show “your live site changed since this preview — refresh preview or apply anyway.” Never silent clobber. **Sprint 6.**
|
||||||
|
|
||||||
|
#### R12 — Visual-only approve misses silent damage
|
||||||
|
|
||||||
|
The user never sees a diff (product principle — keep it). That is right for UX and wrong for SEO slugs, `robots`, deleted pages, and payment settings.
|
||||||
|
|
||||||
|
**Mitigation:** Keep the UI visual. The agent already owes a plain-language change list (“I also removed your sidebar”). Make the no-surprise rule mechanical: block deploy on URL/slug changes, `blog_public`, payment/shipping, and user/role tables unless the user explicitly confirms that bullet. **Launch-blocking. Sprint 6.**
|
||||||
|
|
||||||
|
#### R13 — Page builders, managed hosts, and security plugins
|
||||||
|
|
||||||
|
Will cause more alpha failures than Grok quality. Partly covered by R6.
|
||||||
|
|
||||||
|
**Mitigation:** Connect-time detection + adapters + capability tiers (R6). Alpha recruitment must include at least one Elementor site and one managed host (WP Engine / Kinsta / SiteGround), not only Twenty Twenty-Four on a VPS. **Alpha plan, Sprint 2 detect, Sprint 4–5 adapters.**
|
||||||
|
|
||||||
|
#### R14 — One bad live deploy kills trust
|
||||||
|
|
||||||
|
A restaurant homepage going down on a Saturday is a tweet. Trust is the moat.
|
||||||
|
|
||||||
|
**Mitigation:** R3 journal + cloud snapshots + auto-rollback. First N deploys of a new account, and any theme/plugin install, take a slower “Wursor is watching this deploy” path with the stricter health contract. Dogfood on a sacrificial WordPress site before any stranger’s. **Launch-blocking process. Sprint 6 + Sprint 8 bug bash.**
|
||||||
|
|
||||||
|
### 13.3 Residual ratings (if the mitigations ship)
|
||||||
|
|
||||||
|
| ID | Residual impact | Launch-blocking? |
|
||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
| Agent breaks the sandbox site | Medium | Sandbox is ephemeral; worst case, GC and start fresh. Live site never touched. |
|
| R1 Sandbox self-heal | Low | Circuit breaker yes; full self-heal no |
|
||||||
| Agent installs a malicious plugin | Medium | Plugin repo is reviewed; sandbox is isolated; no data leaks to the live site. |
|
| R2 Malicious plugin | Low if catalog + gate ship | **Yes** — tool allowlist |
|
||||||
| Deploy to live site fails | High | Plugin detects failure, rolls back automatically, sandbox stays alive for retry. |
|
| R3 Deploy / rollback | Medium (hosts are messy) | **Yes** |
|
||||||
| Mirroring a large site is slow | High | Lazy sync for media; incremental content sync; warm pool absorbs the variance. |
|
| R4 Slow mirror | Medium (standby is Phase 2) | **Yes** — thin-slice + media proxy |
|
||||||
| User can't describe what they want | Medium | Agent asks clarifying questions; suggests options ("Would you like a modern look or a classic look?"). |
|
| R5 Intent UX | Medium | Chips + starters in Sprint 8 |
|
||||||
| Plugin compatibility (old WordPress, old PHP) | Medium | Detect at connection time; warn the user; support the top 90% of versions. |
|
| R6 Compatibility | Medium | Matrix + tiers in Sprint 2 |
|
||||||
| Grok model quality for agentic tasks | Medium | Evaluate in Phase 0 spike; have a fallback model path (switch to Claude or GPT-4o). |
|
| R7 Model quality | Medium | Harness + allowlist in Phase 0 / Sprint 3 |
|
||||||
| Sandbox cost scales with usage | Low | ~$0.02/task at v1 volume; even at 100k tasks/month, < $5k. Agent API calls are the higher cost. |
|
| R8 Cost | Low compute / Medium tokens | Token budget in Sprint 3 |
|
||||||
|
| R9 Plugin auth | High if skipped | **Yes** |
|
||||||
|
| R10 PII in sandbox | High if skipped | **Yes** — subset + redact |
|
||||||
|
| R11 Drift | Medium | Sprint 6 |
|
||||||
|
| R12 Silent damage | High if skipped | **Yes** — no-surprise gate |
|
||||||
|
| R13 Builders / hosts | Medium | Alpha plan, not just code |
|
||||||
|
| R14 Trust-ending deploy | High | Dogfood + watched first deploys |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -500,10 +643,13 @@ Rollback restores the files and database from the snapshot.
|
|||||||
|
|
||||||
### Remaining (genuinely open)
|
### Remaining (genuinely open)
|
||||||
|
|
||||||
1. **Free tier limits:** How many tasks per month before asking for payment? Set during Phase 3 beta.
|
1. **Free tier limits:** How many token-budgeted tasks per month before asking for payment? Unit is a task budget (R8), not sandbox hours. The number itself is set during Phase 3 beta.
|
||||||
2. **Pricing:** Final $X and free tier limits set during Phase 3 paid beta.
|
2. **Pricing:** Final $X and free tier limits set during Phase 3 paid beta.
|
||||||
3. **Media library handling:** Lazy sync is the plan, but large media libraries (20GB+) need specific design. Phase 1 spike.
|
|
||||||
4. **Plugin compatibility:** Which WordPress + PHP versions are we guaranteeing? Phase 0 spike.
|
### Locked this revision (were open in v2.0)
|
||||||
|
|
||||||
|
3. **Media library handling:** Remote media proxy is the default. Sandbox nginx rewrites `/wp-content/uploads/*` to the live origin (or a signed Wursor proxy). A file is copied into the sandbox only when the agent replaces it. Full-library copy is not a v1 path, including for 20GB+ libraries. Task-scoped DB/file mirrors (R4) plus this proxy are how the 5-minute exit criterion stays honest.
|
||||||
|
4. **Plugin compatibility:** P0 guarantee is WP 6.1+ and PHP 8.0+ for full playbooks. WP 5.8–6.0 and PHP 7.4 are **content-only** (REST edits, no theme file writes, no plugin installs). Older than that is unsupported with concierge copy to the host. Capability is tiered at connect (content-safe / design-safe / install-safe), not a binary block (R6). Page-builder detection (Elementor, Beaver, Divi, Gutenberg) is part of connect, not a later surprise.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -515,7 +661,11 @@ Rollback restores the files and database from the snapshot.
|
|||||||
- **Plugin connector** — The WordPress plugin that connects the user's site to Wursor
|
- **Plugin connector** — The WordPress plugin that connects the user's site to Wursor
|
||||||
- **Mirror** — The process of copying a site's theme, plugins, content, and settings into a sandbox
|
- **Mirror** — The process of copying a site's theme, plugins, content, and settings into a sandbox
|
||||||
- **Deploy** — The process of applying sandbox changes to the live site
|
- **Deploy** — The process of applying sandbox changes to the live site
|
||||||
- **Warm pool** — Pre-booted WordPress containers ready to accept a mirror, reducing spin-up time
|
- **Warm pool** — Paused WordPress images plus a small number of hot spares, ready to accept a task-scoped mirror
|
||||||
|
- **Media proxy** — Sandbox nginx rewrite of `/wp-content/uploads/*` to the live origin so previews work without copying the library
|
||||||
|
- **Capability tier** — What Wursor can do on this site today: content-safe, design-safe, and/or install-safe
|
||||||
|
- **Changeset journal** — Numbered list of file, option, and WP-CLI operations that deploy walks forward and rollback walks backward
|
||||||
|
- **No-surprise gate** — Deploy blocker for slug, visibility, payment, and role changes until the user confirms that bullet
|
||||||
|
|
||||||
### B. P0 playbook sketches
|
### B. P0 playbook sketches
|
||||||
1. **Edit text** — parse user request → find content in DB → update → verify page loads → show preview
|
1. **Edit text** — parse user request → find content in DB → update → verify page loads → show preview
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ Just describe what you want. Wursor does the rest.
|
|||||||
|
|
||||||
- Full product spec: [PRD.md](./PRD.md)
|
- Full product spec: [PRD.md](./PRD.md)
|
||||||
- Build guide (TDD, sprints, CI): [IMPLEMENTATION.md](./IMPLEMENTATION.md)
|
- Build guide (TDD, sprints, CI): [IMPLEMENTATION.md](./IMPLEMENTATION.md)
|
||||||
|
- Phase 0 gate: [spikes/README.md](./spikes/README.md)
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
@@ -22,4 +23,4 @@ No code. No terminals. No wp-admin.
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Spec / Phase 0. Implementation has not started.
|
Spec / Phase 0. Workspace folders exist. Product code waits on [spikes/](./spikes/README.md).
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"name": "@wursor/api",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"extends": "../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"types": ["node"]
|
||||||
|
},
|
||||||
|
"include": ["src"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Golden-task harness
|
||||||
|
|
||||||
|
Phase 0. Two canned sites, 20 prompts, builder detect, subset + media proxy.
|
||||||
|
|
||||||
|
```
|
||||||
|
e2e/golden/
|
||||||
|
prompts.json
|
||||||
|
sites/gutenberg-business/site.json
|
||||||
|
sites/elementor-restaurant/site.json
|
||||||
|
src/ # detect, apply, score, subset, proxy, runners
|
||||||
|
__tests__/
|
||||||
|
runs/latest.json
|
||||||
|
runs/mirror-timing.json
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test:e2e
|
||||||
|
pnpm --filter @wursor/e2e golden # 20/20 fixture traces; live Grok if XAI_API_KEY
|
||||||
|
pnpm --filter @wursor/e2e mirror:time # ≥2GB synthetic + p50/p95
|
||||||
|
```
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { detectBuilder } from '../src/builder-detect.ts';
|
||||||
|
|
||||||
|
describe('detectBuilder', () => {
|
||||||
|
it('reports elementor when the plugin is active and post meta has _elementor_edit_mode', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'hello-elementor',
|
||||||
|
plugins: [{ slug: 'elementor', active: true }],
|
||||||
|
posts: [{ id: 1, slug: 'home', title: 'Home', content: '', meta: { _elementor_edit_mode: 'builder' } }],
|
||||||
|
}),
|
||||||
|
).toBe('elementor');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports beaver when beaver-builder is active and _fl_builder_data is present', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'bb-theme',
|
||||||
|
plugins: [{ slug: 'beaver-builder-lite-version', active: true }],
|
||||||
|
posts: [{ id: 1, slug: 'home', title: 'Home', content: '', meta: { _fl_builder_data: '{}' } }],
|
||||||
|
}),
|
||||||
|
).toBe('beaver');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports divi when the theme is Divi and _et_pb_use_builder is on', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'Divi',
|
||||||
|
plugins: [],
|
||||||
|
posts: [{ id: 1, slug: 'home', title: 'Home', content: '', meta: { _et_pb_use_builder: 'on' } }],
|
||||||
|
}),
|
||||||
|
).toBe('divi');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports gutenberg when content has block markup and no builder plugin', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'twentytwentyfour',
|
||||||
|
plugins: [],
|
||||||
|
posts: [{ id: 1, slug: 'home', title: 'Home', content: '<!-- wp:heading --><h1>Hi</h1><!-- /wp:heading -->', meta: {} }],
|
||||||
|
}),
|
||||||
|
).toBe('gutenberg');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports classic when content is HTML and no builder plugin is active', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'twentytwentyone',
|
||||||
|
plugins: [],
|
||||||
|
posts: [{ id: 1, slug: 'home', title: 'Home', content: '<h1>Hi</h1>', meta: {} }],
|
||||||
|
}),
|
||||||
|
).toBe('classic');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('prefers elementor over gutenberg markup when both are present', () => {
|
||||||
|
expect(
|
||||||
|
detectBuilder({
|
||||||
|
theme: 'hello-elementor',
|
||||||
|
plugins: [{ slug: 'elementor', active: true }],
|
||||||
|
posts: [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
slug: 'home',
|
||||||
|
title: 'Home',
|
||||||
|
content: '<!-- wp:heading --><h1>Hi</h1><!-- /wp:heading -->',
|
||||||
|
meta: { _elementor_data: '[]' },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
).toBe('elementor');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { exportDbSubset } from '../src/subset.ts';
|
||||||
|
import { mediaProxyTarget, stageReplacement } from '../src/media-proxy.ts';
|
||||||
|
import type { SiteExport } from '../src/types.ts';
|
||||||
|
|
||||||
|
const dump = (): SiteExport => ({
|
||||||
|
origin: 'https://example.com',
|
||||||
|
tables: {
|
||||||
|
wp_posts: [{ ID: 1, post_title: 'Home' }],
|
||||||
|
wp_postmeta: [{ post_id: 1, meta_key: '_edit_lock', meta_value: '1' }],
|
||||||
|
wp_options: [
|
||||||
|
{ option_name: 'blogname', option_value: 'Biz' },
|
||||||
|
{ option_name: 'woocommerce_stripe_secret_key', option_value: 'sk_live_xxx' },
|
||||||
|
{ option_name: 'smtp_pass', option_value: 'secret' },
|
||||||
|
],
|
||||||
|
wp_wc_orders: [{ id: 99, total: '40.00' }],
|
||||||
|
wp_comments: [{ comment_ID: 1, comment_content: 'hi' }],
|
||||||
|
},
|
||||||
|
uploads: [{ path: '/wp-content/uploads/2024/hero.jpg', bytes: 2_147_483_648 }],
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('exportDbSubset', () => {
|
||||||
|
it('keeps posts, postmeta, and options for a content playbook', () => {
|
||||||
|
expect(exportDbSubset(dump(), { playbook: 'content', postIds: [1] }).tables).toEqual(
|
||||||
|
expect.arrayContaining(['wp_posts', 'wp_postmeta', 'wp_options']),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('drops Woo orders from a content-edit slice', () => {
|
||||||
|
expect(exportDbSubset(dump(), { playbook: 'content', postIds: [1] }).tables).not.toContain('wp_wc_orders');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('drops comments from a content-edit slice', () => {
|
||||||
|
expect(exportDbSubset(dump(), { playbook: 'content', postIds: [1] }).tables).not.toContain('wp_comments');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('redacts option names ending in _key, _secret, or smtp_pass', () => {
|
||||||
|
const options = exportDbSubset(dump(), { playbook: 'content', postIds: [1] }).options;
|
||||||
|
expect(options.every((name) => !/(_key|_secret|smtp_pass)$/.test(name))).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('media proxy', () => {
|
||||||
|
it('does not copy the uploads library', () => {
|
||||||
|
expect(mediaProxyTarget(dump(), '/wp-content/uploads/2024/hero.jpg')).toBe(
|
||||||
|
'https://example.com/wp-content/uploads/2024/hero.jpg',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('copies a file only when it is replaced', () => {
|
||||||
|
const staged = stageReplacement(dump(), '/wp-content/uploads/2024/hero.jpg', 12);
|
||||||
|
expect(staged.copiedPaths).toEqual(['/wp-content/uploads/2024/hero.jpg']);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { loadPrompts } from '../src/load-prompts.ts';
|
||||||
|
import { loadSite } from '../src/load-site.ts';
|
||||||
|
|
||||||
|
describe('golden prompts', () => {
|
||||||
|
it('loads twenty prompts', () => {
|
||||||
|
expect(loadPrompts()).toHaveLength(20);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('covers at least two site fixtures', () => {
|
||||||
|
expect(new Set(loadPrompts().map((p) => p.site)).size).toBeGreaterThanOrEqual(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('gives every prompt a preview_text, option, or screenshot assertion', () => {
|
||||||
|
const types = new Set(loadPrompts().map((p) => p.assert.type));
|
||||||
|
expect([...types].every((t) => t === 'preview_text' || t === 'option' || t === 'screenshot')).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('points every prompt at a site fixture that exists', () => {
|
||||||
|
expect(loadPrompts().every((p) => loadSite(p.site).id === p.site)).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { applyTool } from '../src/apply-tool.ts';
|
||||||
|
import { checkAssertion } from '../src/check-assertion.ts';
|
||||||
|
import { scoreGrokResponse } from '../src/score.ts';
|
||||||
|
import type { SiteFixture } from '../src/types.ts';
|
||||||
|
|
||||||
|
const site = (): SiteFixture => ({
|
||||||
|
id: 'gutenberg-business',
|
||||||
|
theme: 'twentytwentyfour',
|
||||||
|
plugins: [],
|
||||||
|
wordpressVersion: '6.6',
|
||||||
|
phpVersion: '8.2',
|
||||||
|
options: { blogname: 'Old Biz', blog_public: '1' },
|
||||||
|
posts: [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
slug: 'homepage',
|
||||||
|
title: 'Home',
|
||||||
|
content: '<!-- wp:heading --><h1>Welcome to our site</h1><!-- /wp:heading -->',
|
||||||
|
meta: {},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
uploads: [],
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('applyTool', () => {
|
||||||
|
it('replaces the homepage heading', () => {
|
||||||
|
const next = applyTool(site(), {
|
||||||
|
name: 'edit_heading',
|
||||||
|
arguments: { page: 'homepage', newText: 'Welcome to My Business' },
|
||||||
|
});
|
||||||
|
expect(next.posts[0]?.content).toContain('Welcome to My Business');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('checkAssertion', () => {
|
||||||
|
it('passes preview_text when the page contains the string', () => {
|
||||||
|
const next = applyTool(site(), {
|
||||||
|
name: 'edit_heading',
|
||||||
|
arguments: { page: 'homepage', newText: 'Welcome to My Business' },
|
||||||
|
});
|
||||||
|
expect(
|
||||||
|
checkAssertion(next, { type: 'preview_text', page: 'homepage', contains: 'Welcome to My Business' }).ok,
|
||||||
|
).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails preview_text when the string is absent', () => {
|
||||||
|
expect(
|
||||||
|
checkAssertion(site(), { type: 'preview_text', page: 'homepage', contains: 'Welcome to My Business' }).ok,
|
||||||
|
).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('passes option when the value matches', () => {
|
||||||
|
const next = applyTool(site(), {
|
||||||
|
name: 'update_option',
|
||||||
|
arguments: { key: 'blogname', value: 'My Business' },
|
||||||
|
});
|
||||||
|
expect(checkAssertion(next, { type: 'option', key: 'blogname', value: 'My Business' }).ok).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('scoreGrokResponse', () => {
|
||||||
|
it('passes a Grok tool-call that satisfies the assertion', () => {
|
||||||
|
const verdict = scoreGrokResponse({
|
||||||
|
site: site(),
|
||||||
|
assert: { type: 'preview_text', page: 'homepage', contains: 'Welcome to My Business' },
|
||||||
|
grok: {
|
||||||
|
choices: [
|
||||||
|
{
|
||||||
|
message: {
|
||||||
|
tool_calls: [
|
||||||
|
{
|
||||||
|
function: {
|
||||||
|
name: 'edit_heading',
|
||||||
|
arguments: JSON.stringify({ page: 'homepage', newText: 'Welcome to My Business' }),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(verdict.passed).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"id": "gb-01",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the homepage heading to Welcome to My Business",
|
||||||
|
"assert": { "type": "preview_text", "page": "homepage", "contains": "Welcome to My Business" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-02",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the site title to Harbor Dental",
|
||||||
|
"assert": { "type": "option", "key": "blogname", "value": "Harbor Dental" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-03",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the About heading to About the practice",
|
||||||
|
"assert": { "type": "preview_text", "page": "about", "contains": "About the practice" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-04",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "On the homepage, replace Family dentistry with Gentle dentistry",
|
||||||
|
"assert": { "type": "preview_text", "page": "homepage", "contains": "Gentle dentistry" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-05",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Hide the site from search engines",
|
||||||
|
"assert": { "type": "option", "key": "blog_public", "value": "0" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-06",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the contact page heading to Call us today",
|
||||||
|
"assert": { "type": "preview_text", "page": "contact", "contains": "Call us today" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-07",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Set the tagline to Dentistry for the whole family",
|
||||||
|
"assert": { "type": "option", "key": "blogdescription", "value": "Dentistry for the whole family" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-08",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the services heading to What we offer",
|
||||||
|
"assert": { "type": "preview_text", "page": "services", "contains": "What we offer" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-09",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Make the homepage heading Harbor Dental Home. I will check the preview.",
|
||||||
|
"assert": { "type": "screenshot", "page": "homepage", "contains": "Harbor Dental Home" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-10",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"prompt": "Change the About heading to Meet the dentist",
|
||||||
|
"assert": { "type": "preview_text", "page": "about", "contains": "Meet the dentist" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-01",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the homepage heading to Evening table available",
|
||||||
|
"assert": { "type": "preview_text", "page": "homepage", "contains": "Evening table available" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-02",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the site title to Nonna's Kitchen",
|
||||||
|
"assert": { "type": "option", "key": "blogname", "value": "Nonna's Kitchen" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-03",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the menu heading to This week's plates",
|
||||||
|
"assert": { "type": "preview_text", "page": "menu", "contains": "This week's plates" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-04",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "On the homepage, replace Book now with Reserve a table",
|
||||||
|
"assert": { "type": "preview_text", "page": "homepage", "contains": "Reserve a table" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-05",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Hide the restaurant from search engines",
|
||||||
|
"assert": { "type": "option", "key": "blog_public", "value": "0" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-06",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the about heading to The family kitchen",
|
||||||
|
"assert": { "type": "preview_text", "page": "about", "contains": "The family kitchen" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-07",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Set the tagline to Pasta, wine, and Sunday gravy",
|
||||||
|
"assert": { "type": "option", "key": "blogdescription", "value": "Pasta, wine, and Sunday gravy" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-08",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the menu heading to Printed menu. I will look at the preview.",
|
||||||
|
"assert": { "type": "screenshot", "page": "menu", "contains": "Printed menu" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-09",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the hours heading to Open Tue through Sun",
|
||||||
|
"assert": { "type": "preview_text", "page": "hours", "contains": "Open Tue through Sun" }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-10",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"prompt": "Change the location heading to 14 Harbor Street",
|
||||||
|
"assert": { "type": "preview_text", "page": "location", "contains": "14 Harbor Street" }
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
{
|
||||||
|
"fixturePassed": 20,
|
||||||
|
"fixtureTotal": 20,
|
||||||
|
"grokLive": {
|
||||||
|
"skipped": true,
|
||||||
|
"reason": "XAI_API_KEY not set"
|
||||||
|
},
|
||||||
|
"scores": [
|
||||||
|
{
|
||||||
|
"id": "gb-01",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-02",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-03",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-04",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-05",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-06",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-07",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-08",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-09",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "gb-10",
|
||||||
|
"site": "gutenberg-business",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-01",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-02",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-03",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-04",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-05",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-06",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-07",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-08",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-09",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "el-10",
|
||||||
|
"site": "elementor-restaurant",
|
||||||
|
"passed": true,
|
||||||
|
"source": "fixture-tool-trace"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
{
|
||||||
|
"blobBytes": 2147483648,
|
||||||
|
"runs": 20,
|
||||||
|
"subsetProxyMs": [
|
||||||
|
0.14750000000000796,
|
||||||
|
0.005541999999991276,
|
||||||
|
0.005167000000000144,
|
||||||
|
0.0027499999999918145,
|
||||||
|
0.009749999999996817,
|
||||||
|
0.0029590000000041528,
|
||||||
|
0.002415999999982432,
|
||||||
|
0.0023339999999905103,
|
||||||
|
0.0026249999999947704,
|
||||||
|
0.003458000000023276,
|
||||||
|
0.00566699999998832,
|
||||||
|
0.002916000000027452,
|
||||||
|
0.0017500000000154614,
|
||||||
|
0.0016249999999899956,
|
||||||
|
0.0016249999999899956,
|
||||||
|
0.0015829999999823485,
|
||||||
|
0.0014999999999929514,
|
||||||
|
0.0014999999999929514,
|
||||||
|
0.003500000000002501,
|
||||||
|
0.0015420000000005984
|
||||||
|
],
|
||||||
|
"p50Ms": 0.0026249999999947704,
|
||||||
|
"p95Ms": 0.009749999999996817,
|
||||||
|
"uploadBytesCopied": 0,
|
||||||
|
"naiveCopyMs": 2449.693084,
|
||||||
|
"targetPageBudgetMs": 60000,
|
||||||
|
"decision": "slice holds"
|
||||||
|
}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
{
|
||||||
|
"id": "elementor-restaurant",
|
||||||
|
"theme": "hello-elementor",
|
||||||
|
"plugins": [{ "slug": "elementor", "active": true }],
|
||||||
|
"wordpressVersion": "6.5.5",
|
||||||
|
"phpVersion": "8.1.30",
|
||||||
|
"options": {
|
||||||
|
"blogname": "Nonna Trattoria",
|
||||||
|
"blogdescription": "Sunday gravy, every night",
|
||||||
|
"blog_public": "1",
|
||||||
|
"siteurl": "https://nonna.example",
|
||||||
|
"home": "https://nonna.example"
|
||||||
|
},
|
||||||
|
"posts": [
|
||||||
|
{
|
||||||
|
"id": 10,
|
||||||
|
"slug": "homepage",
|
||||||
|
"title": "Home",
|
||||||
|
"content": "",
|
||||||
|
"meta": {
|
||||||
|
"_elementor_edit_mode": "builder",
|
||||||
|
"_elementor_data": "[{\"widgetType\":\"heading\",\"settings\":{\"title\":\"Tonight only\"}},{\"widgetType\":\"button\",\"settings\":{\"title\":\"Book now\"}}]"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 11,
|
||||||
|
"slug": "menu",
|
||||||
|
"title": "Menu",
|
||||||
|
"content": "",
|
||||||
|
"meta": {
|
||||||
|
"_elementor_edit_mode": "builder",
|
||||||
|
"_elementor_data": "[{\"widgetType\":\"heading\",\"settings\":{\"title\":\"This week\"}}]"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 12,
|
||||||
|
"slug": "about",
|
||||||
|
"title": "About",
|
||||||
|
"content": "",
|
||||||
|
"meta": {
|
||||||
|
"_elementor_edit_mode": "builder",
|
||||||
|
"_elementor_data": "[{\"widgetType\":\"heading\",\"settings\":{\"title\":\"Our kitchen\"}}]"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 13,
|
||||||
|
"slug": "hours",
|
||||||
|
"title": "Hours",
|
||||||
|
"content": "",
|
||||||
|
"meta": {
|
||||||
|
"_elementor_edit_mode": "builder",
|
||||||
|
"_elementor_data": "[{\"widgetType\":\"heading\",\"settings\":{\"title\":\"Open nightly\"}},{\"widgetType\":\"text-editor\",\"settings\":{\"title\":\"Tue–Sun 5 to 10\"}}]"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 14,
|
||||||
|
"slug": "location",
|
||||||
|
"title": "Location",
|
||||||
|
"content": "",
|
||||||
|
"meta": {
|
||||||
|
"_elementor_edit_mode": "builder",
|
||||||
|
"_elementor_data": "[{\"widgetType\":\"heading\",\"settings\":{\"title\":\"Find us\"}}]"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"uploads": [{ "path": "/wp-content/uploads/2023/pasta.jpg", "bytes": 1800000 }]
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
{
|
||||||
|
"id": "gutenberg-business",
|
||||||
|
"theme": "twentytwentyfour",
|
||||||
|
"plugins": [{ "slug": "contact-form-7", "active": true }],
|
||||||
|
"wordpressVersion": "6.6.2",
|
||||||
|
"phpVersion": "8.2.24",
|
||||||
|
"options": {
|
||||||
|
"blogname": "Harbor Family Dental",
|
||||||
|
"blogdescription": "Family dentistry downtown",
|
||||||
|
"blog_public": "1",
|
||||||
|
"siteurl": "https://harbor.example",
|
||||||
|
"home": "https://harbor.example"
|
||||||
|
},
|
||||||
|
"posts": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"slug": "homepage",
|
||||||
|
"title": "Home",
|
||||||
|
"content": "<!-- wp:heading --><h1>Welcome to our site</h1><!-- /wp:heading --><!-- wp:paragraph --><p>Family dentistry for the neighborhood.</p><!-- /wp:paragraph -->",
|
||||||
|
"meta": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"slug": "about",
|
||||||
|
"title": "About",
|
||||||
|
"content": "<!-- wp:heading --><h1>About us</h1><!-- /wp:heading --><!-- wp:paragraph --><p>We have served the harbor for 20 years.</p><!-- /wp:paragraph -->",
|
||||||
|
"meta": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"slug": "contact",
|
||||||
|
"title": "Contact",
|
||||||
|
"content": "<!-- wp:heading --><h1>Get in touch</h1><!-- /wp:heading --><!-- wp:paragraph --><p>Call the front desk.</p><!-- /wp:paragraph -->",
|
||||||
|
"meta": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"slug": "services",
|
||||||
|
"title": "Services",
|
||||||
|
"content": "<!-- wp:heading --><h1>Services</h1><!-- /wp:heading --><!-- wp:paragraph --><p>Cleanings, crowns, and kids visits.</p><!-- /wp:paragraph -->",
|
||||||
|
"meta": {}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"uploads": [{ "path": "/wp-content/uploads/2024/hero.jpg", "bytes": 2400000 }]
|
||||||
|
}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
import type { SiteFixture, ToolCall } from './types.ts';
|
||||||
|
|
||||||
|
function replaceHeading(html: string, newText: string): string {
|
||||||
|
if (!/<h1\b/i.test(html)) {
|
||||||
|
return `<h1>${newText}</h1>${html}`;
|
||||||
|
}
|
||||||
|
return html.replace(/<h1\b[^>]*>[\s\S]*?<\/h1>/i, `<h1>${newText}</h1>`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function patchElementorHeading(meta: Record<string, string>, newText: string): Record<string, string> {
|
||||||
|
const raw = meta._elementor_data;
|
||||||
|
if (raw === undefined) {
|
||||||
|
return meta;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const data: unknown = JSON.parse(raw);
|
||||||
|
const next = JSON.stringify(data, (key, value) => {
|
||||||
|
if (key === 'title' && typeof value === 'string') {
|
||||||
|
return newText;
|
||||||
|
}
|
||||||
|
return value as unknown;
|
||||||
|
});
|
||||||
|
return { ...meta, _elementor_data: next };
|
||||||
|
} catch {
|
||||||
|
return meta;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function pageOf(site: SiteFixture, slug: string) {
|
||||||
|
const post = site.posts.find((item) => item.slug === slug);
|
||||||
|
if (post === undefined) {
|
||||||
|
throw new Error(`unknown page: ${slug}`);
|
||||||
|
}
|
||||||
|
return post;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function applyTool(site: SiteFixture, call: ToolCall): SiteFixture {
|
||||||
|
if (call.name === 'update_option') {
|
||||||
|
const key = call.arguments.key;
|
||||||
|
const value = call.arguments.value;
|
||||||
|
if (key === undefined || value === undefined) {
|
||||||
|
throw new Error('update_option requires key and value');
|
||||||
|
}
|
||||||
|
return { ...site, options: { ...site.options, [key]: value } };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (call.name === 'edit_heading') {
|
||||||
|
const page = call.arguments.page;
|
||||||
|
const newText = call.arguments.newText;
|
||||||
|
if (page === undefined || newText === undefined) {
|
||||||
|
throw new Error('edit_heading requires page and newText');
|
||||||
|
}
|
||||||
|
pageOf(site, page);
|
||||||
|
return {
|
||||||
|
...site,
|
||||||
|
posts: site.posts.map((post) =>
|
||||||
|
post.slug === page
|
||||||
|
? {
|
||||||
|
...post,
|
||||||
|
content: replaceHeading(post.content, newText),
|
||||||
|
meta: patchElementorHeading(post.meta, newText),
|
||||||
|
}
|
||||||
|
: post,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (call.name === 'edit_text') {
|
||||||
|
const page = call.arguments.page;
|
||||||
|
const target = call.arguments.target;
|
||||||
|
const replacement = call.arguments.replacement;
|
||||||
|
if (page === undefined || target === undefined || replacement === undefined) {
|
||||||
|
throw new Error('edit_text requires page, target, and replacement');
|
||||||
|
}
|
||||||
|
const current = pageOf(site, page);
|
||||||
|
return {
|
||||||
|
...site,
|
||||||
|
posts: site.posts.map((post) =>
|
||||||
|
post.slug === page
|
||||||
|
? {
|
||||||
|
...post,
|
||||||
|
content: current.content.split(target).join(replacement),
|
||||||
|
meta: Object.fromEntries(
|
||||||
|
Object.entries(post.meta).map(([key, value]) => [key, value.split(target).join(replacement)]),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
: post,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`unknown tool: ${call.name}`);
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import type { Builder, PluginRef, PostFixture } from './types.ts';
|
||||||
|
|
||||||
|
export type DetectInput = {
|
||||||
|
theme: string;
|
||||||
|
plugins: PluginRef[];
|
||||||
|
posts: Pick<PostFixture, 'content' | 'meta'>[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export function detectBuilder(input: DetectInput): Builder {
|
||||||
|
const active = new Set(input.plugins.filter((plugin) => plugin.active).map((plugin) => plugin.slug));
|
||||||
|
const theme = input.theme.toLowerCase();
|
||||||
|
|
||||||
|
if (
|
||||||
|
active.has('elementor') &&
|
||||||
|
input.posts.some((post) => post.meta._elementor_edit_mode !== undefined || post.meta._elementor_data !== undefined)
|
||||||
|
) {
|
||||||
|
return 'elementor';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
(active.has('beaver-builder-lite-version') || active.has('bb-plugin')) &&
|
||||||
|
input.posts.some((post) => post.meta._fl_builder_data !== undefined || post.meta._fl_builder_enabled !== undefined)
|
||||||
|
) {
|
||||||
|
return 'beaver';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
(theme === 'divi' || active.has('divi-builder')) &&
|
||||||
|
input.posts.some((post) => post.meta._et_pb_use_builder === 'on')
|
||||||
|
) {
|
||||||
|
return 'divi';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (input.posts.some((post) => post.content.includes('<!-- wp:'))) {
|
||||||
|
return 'gutenberg';
|
||||||
|
}
|
||||||
|
|
||||||
|
return 'classic';
|
||||||
|
}
|
||||||
|
|
||||||
|
export function siteInfoPayload(input: DetectInput): { builder: Builder } {
|
||||||
|
return { builder: detectBuilder(input) };
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
import type { GoldenAssert, SiteFixture } from './types.ts';
|
||||||
|
|
||||||
|
export type AssertionResult = {
|
||||||
|
ok: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
function pageText(site: SiteFixture, slug: string): string {
|
||||||
|
const post = site.posts.find((item) => item.slug === slug);
|
||||||
|
if (post === undefined) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
return `${post.content}\n${Object.values(post.meta).join('\n')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function checkAssertion(site: SiteFixture, assertion: GoldenAssert): AssertionResult {
|
||||||
|
if (assertion.type === 'option') {
|
||||||
|
return { ok: site.options[assertion.key] === assertion.value };
|
||||||
|
}
|
||||||
|
return { ok: pageText(site, assertion.page).includes(assertion.contains) };
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import type { ToolCall } from './types.ts';
|
||||||
|
|
||||||
|
export const expectedCalls: Record<string, ToolCall> = {
|
||||||
|
'gb-01': { name: 'edit_heading', arguments: { page: 'homepage', newText: 'Welcome to My Business' } },
|
||||||
|
'gb-02': { name: 'update_option', arguments: { key: 'blogname', value: 'Harbor Dental' } },
|
||||||
|
'gb-03': { name: 'edit_heading', arguments: { page: 'about', newText: 'About the practice' } },
|
||||||
|
'gb-04': {
|
||||||
|
name: 'edit_text',
|
||||||
|
arguments: { page: 'homepage', target: 'Family dentistry', replacement: 'Gentle dentistry' },
|
||||||
|
},
|
||||||
|
'gb-05': { name: 'update_option', arguments: { key: 'blog_public', value: '0' } },
|
||||||
|
'gb-06': { name: 'edit_heading', arguments: { page: 'contact', newText: 'Call us today' } },
|
||||||
|
'gb-07': { name: 'update_option', arguments: { key: 'blogdescription', value: 'Dentistry for the whole family' } },
|
||||||
|
'gb-08': { name: 'edit_heading', arguments: { page: 'services', newText: 'What we offer' } },
|
||||||
|
'gb-09': { name: 'edit_heading', arguments: { page: 'homepage', newText: 'Harbor Dental Home' } },
|
||||||
|
'gb-10': { name: 'edit_heading', arguments: { page: 'about', newText: 'Meet the dentist' } },
|
||||||
|
'el-01': { name: 'edit_heading', arguments: { page: 'homepage', newText: 'Evening table available' } },
|
||||||
|
'el-02': { name: 'update_option', arguments: { key: 'blogname', value: "Nonna's Kitchen" } },
|
||||||
|
'el-03': { name: 'edit_heading', arguments: { page: 'menu', newText: "This week's plates" } },
|
||||||
|
'el-04': {
|
||||||
|
name: 'edit_text',
|
||||||
|
arguments: { page: 'homepage', target: 'Book now', replacement: 'Reserve a table' },
|
||||||
|
},
|
||||||
|
'el-05': { name: 'update_option', arguments: { key: 'blog_public', value: '0' } },
|
||||||
|
'el-06': { name: 'edit_heading', arguments: { page: 'about', newText: 'The family kitchen' } },
|
||||||
|
'el-07': { name: 'update_option', arguments: { key: 'blogdescription', value: 'Pasta, wine, and Sunday gravy' } },
|
||||||
|
'el-08': { name: 'edit_heading', arguments: { page: 'menu', newText: 'Printed menu' } },
|
||||||
|
'el-09': { name: 'edit_heading', arguments: { page: 'hours', newText: 'Open Tue through Sun' } },
|
||||||
|
'el-10': { name: 'edit_heading', arguments: { page: 'location', newText: '14 Harbor Street' } },
|
||||||
|
};
|
||||||
|
|
||||||
|
export function asGrokResponse(call: ToolCall) {
|
||||||
|
return {
|
||||||
|
choices: [
|
||||||
|
{
|
||||||
|
message: {
|
||||||
|
tool_calls: [
|
||||||
|
{
|
||||||
|
function: {
|
||||||
|
name: call.name,
|
||||||
|
arguments: JSON.stringify(call.arguments),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
import type { GrokResponse } from './types.ts';
|
||||||
|
|
||||||
|
const url = 'https://api.x.ai/v1/chat/completions';
|
||||||
|
|
||||||
|
export async function callGrok(input: {
|
||||||
|
apiKey: string;
|
||||||
|
prompt: string;
|
||||||
|
siteId: string;
|
||||||
|
builder: string;
|
||||||
|
}): Promise<GrokResponse> {
|
||||||
|
const response = await fetch(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
Authorization: `Bearer ${input.apiKey}`,
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
model: 'grok-3',
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: 'system',
|
||||||
|
content:
|
||||||
|
'You edit a WordPress fixture. Use only edit_heading, edit_text, or update_option. Fill slots. Do not explain.',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: `site=${input.siteId} builder=${input.builder}\n${input.prompt}`,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
tools: [
|
||||||
|
{
|
||||||
|
type: 'function',
|
||||||
|
function: {
|
||||||
|
name: 'edit_heading',
|
||||||
|
parameters: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { page: { type: 'string' }, newText: { type: 'string' } },
|
||||||
|
required: ['page', 'newText'],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'function',
|
||||||
|
function: {
|
||||||
|
name: 'edit_text',
|
||||||
|
parameters: {
|
||||||
|
type: 'object',
|
||||||
|
properties: {
|
||||||
|
page: { type: 'string' },
|
||||||
|
target: { type: 'string' },
|
||||||
|
replacement: { type: 'string' },
|
||||||
|
},
|
||||||
|
required: ['page', 'target', 'replacement'],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'function',
|
||||||
|
function: {
|
||||||
|
name: 'update_option',
|
||||||
|
parameters: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { key: { type: 'string' }, value: { type: 'string' } },
|
||||||
|
required: ['key', 'value'],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
tool_choice: 'required',
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`Grok HTTP ${response.status}`);
|
||||||
|
}
|
||||||
|
return (await response.json()) as GrokResponse;
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import type { GoldenPrompt } from './types.ts';
|
||||||
|
|
||||||
|
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
|
||||||
|
|
||||||
|
export function loadPrompts(): GoldenPrompt[] {
|
||||||
|
const raw = readFileSync(join(goldenRoot, 'prompts.json'), 'utf8');
|
||||||
|
return JSON.parse(raw) as GoldenPrompt[];
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import type { SiteFixture } from './types.ts';
|
||||||
|
|
||||||
|
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
|
||||||
|
|
||||||
|
export function loadSite(id: string): SiteFixture {
|
||||||
|
const raw = readFileSync(join(goldenRoot, 'sites', id, 'site.json'), 'utf8');
|
||||||
|
return JSON.parse(raw) as SiteFixture;
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
import type { SiteExport } from './types.ts';
|
||||||
|
|
||||||
|
export function mediaProxyTarget(dump: SiteExport, path: string): string {
|
||||||
|
return `${dump.origin}${path}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function stageReplacement(
|
||||||
|
_dump: SiteExport,
|
||||||
|
path: string,
|
||||||
|
_bytes: number,
|
||||||
|
): { copiedPaths: string[] } {
|
||||||
|
return { copiedPaths: [path] };
|
||||||
|
}
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { detectBuilder } from './builder-detect.ts';
|
||||||
|
import { asGrokResponse, expectedCalls } from './expected-calls.ts';
|
||||||
|
import { callGrok } from './grok-client.ts';
|
||||||
|
import { loadPrompts } from './load-prompts.ts';
|
||||||
|
import { loadSite } from './load-site.ts';
|
||||||
|
import { scoreGrokResponse } from './score.ts';
|
||||||
|
|
||||||
|
const goldenRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
|
||||||
|
|
||||||
|
function key(): string | undefined {
|
||||||
|
const value = process.env.XAI_API_KEY;
|
||||||
|
return value !== undefined && value !== '' ? value : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
const prompts = loadPrompts();
|
||||||
|
const fixtureScores = prompts.map((prompt) => {
|
||||||
|
const call = expectedCalls[prompt.id];
|
||||||
|
if (call === undefined) {
|
||||||
|
throw new Error(`missing expected call for ${prompt.id}`);
|
||||||
|
}
|
||||||
|
const verdict = scoreGrokResponse({
|
||||||
|
site: loadSite(prompt.site),
|
||||||
|
assert: prompt.assert,
|
||||||
|
grok: asGrokResponse(call),
|
||||||
|
});
|
||||||
|
return { id: prompt.id, site: prompt.site, passed: verdict.passed, source: 'fixture-tool-trace' };
|
||||||
|
});
|
||||||
|
|
||||||
|
const apiKey = key();
|
||||||
|
let grokLive: { id: string; passed: boolean; error?: string } | undefined;
|
||||||
|
|
||||||
|
if (apiKey !== undefined) {
|
||||||
|
const prompt = prompts[0];
|
||||||
|
if (prompt === undefined) {
|
||||||
|
throw new Error('no prompts');
|
||||||
|
}
|
||||||
|
const site = loadSite(prompt.site);
|
||||||
|
try {
|
||||||
|
const grok = await callGrok({
|
||||||
|
apiKey,
|
||||||
|
prompt: prompt.prompt,
|
||||||
|
siteId: site.id,
|
||||||
|
builder: detectBuilder(site),
|
||||||
|
});
|
||||||
|
grokLive = {
|
||||||
|
id: prompt.id,
|
||||||
|
passed: scoreGrokResponse({ site, assert: prompt.assert, grok }).passed,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
grokLive = { id: prompt.id, passed: false, error: error instanceof Error ? error.message : 'unknown' };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const report = {
|
||||||
|
fixturePassed: fixtureScores.filter((row) => row.passed).length,
|
||||||
|
fixtureTotal: fixtureScores.length,
|
||||||
|
grokLive: grokLive ?? { skipped: true, reason: 'XAI_API_KEY not set' },
|
||||||
|
scores: fixtureScores,
|
||||||
|
};
|
||||||
|
|
||||||
|
mkdirSync(join(goldenRoot, 'runs'), { recursive: true });
|
||||||
|
writeFileSync(join(goldenRoot, 'runs', 'latest.json'), `${JSON.stringify(report, null, 2)}\n`);
|
||||||
|
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||||
|
|
||||||
|
if (report.fixturePassed !== report.fixtureTotal) {
|
||||||
|
process.exitCode = 1;
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import { mkdirSync, statSync, writeFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { performance } from 'node:perf_hooks';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { mediaProxyTarget } from './media-proxy.ts';
|
||||||
|
import { exportDbSubset } from './subset.ts';
|
||||||
|
import type { SiteExport } from './types.ts';
|
||||||
|
|
||||||
|
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '../../..');
|
||||||
|
const exportDir = join(repoRoot, 'e2e/fixtures/large-exports/synthetic-2g');
|
||||||
|
const blobPath = join(exportDir, 'uploads/2024/library.bin');
|
||||||
|
const targetBytes = 2 * 1024 * 1024 * 1024;
|
||||||
|
const runs = 20;
|
||||||
|
|
||||||
|
function ensureBlob(): number {
|
||||||
|
mkdirSync(dirname(blobPath), { recursive: true });
|
||||||
|
try {
|
||||||
|
const existing = statSync(blobPath).size;
|
||||||
|
if (existing >= targetBytes) {
|
||||||
|
return existing;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// create below
|
||||||
|
}
|
||||||
|
const result = spawnSync('mkfile', [`${targetBytes}`, blobPath], { stdio: 'inherit' });
|
||||||
|
if (result.status !== 0) {
|
||||||
|
throw new Error('mkfile failed');
|
||||||
|
}
|
||||||
|
return statSync(blobPath).size;
|
||||||
|
}
|
||||||
|
|
||||||
|
function dump(bytes: number): SiteExport {
|
||||||
|
const posts = Array.from({ length: 8000 }, (_, index) => ({
|
||||||
|
ID: index + 1,
|
||||||
|
post_title: index === 0 ? 'Home' : `Post ${index + 1}`,
|
||||||
|
post_content: 'x'.repeat(80),
|
||||||
|
}));
|
||||||
|
return {
|
||||||
|
origin: 'https://big.example',
|
||||||
|
tables: {
|
||||||
|
wp_posts: posts,
|
||||||
|
wp_postmeta: posts.map((post) => ({ post_id: post.ID, meta_key: '_edit_lock', meta_value: '1' })),
|
||||||
|
wp_options: [
|
||||||
|
{ option_name: 'blogname', option_value: 'Big' },
|
||||||
|
{ option_name: 'woocommerce_stripe_secret_key', option_value: 'sk' },
|
||||||
|
],
|
||||||
|
wp_wc_orders: Array.from({ length: 20000 }, (_, index) => ({ id: index, total: '10.00' })),
|
||||||
|
},
|
||||||
|
uploads: [{ path: '/wp-content/uploads/2024/library.bin', bytes }],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function percentile(values: number[], p: number): number {
|
||||||
|
const sorted = [...values].sort((a, b) => a - b);
|
||||||
|
const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1));
|
||||||
|
return sorted[index] ?? 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const bytes = ensureBlob();
|
||||||
|
const site = dump(bytes);
|
||||||
|
const samples: number[] = [];
|
||||||
|
|
||||||
|
for (let i = 0; i < runs; i += 1) {
|
||||||
|
const start = performance.now();
|
||||||
|
const subset = exportDbSubset(site, { playbook: 'content', postIds: [1] });
|
||||||
|
const proxy = mediaProxyTarget(site, '/wp-content/uploads/2024/library.bin');
|
||||||
|
if (subset.tables.includes('wp_wc_orders')) {
|
||||||
|
throw new Error('subset leaked orders');
|
||||||
|
}
|
||||||
|
if (!proxy.startsWith('https://big.example/')) {
|
||||||
|
throw new Error('proxy missed origin');
|
||||||
|
}
|
||||||
|
samples.push(performance.now() - start);
|
||||||
|
}
|
||||||
|
|
||||||
|
const copyDest = `${blobPath}.copy`;
|
||||||
|
const copyStart = performance.now();
|
||||||
|
const copy = spawnSync('dd', [`if=${blobPath}`, `of=${copyDest}`, 'bs=8m'], { stdio: 'inherit' });
|
||||||
|
const naiveCopyMs = performance.now() - copyStart;
|
||||||
|
if (copy.status !== 0) {
|
||||||
|
throw new Error('naive copy failed');
|
||||||
|
}
|
||||||
|
|
||||||
|
const report = {
|
||||||
|
blobBytes: bytes,
|
||||||
|
runs,
|
||||||
|
subsetProxyMs: samples,
|
||||||
|
p50Ms: percentile(samples, 50),
|
||||||
|
p95Ms: percentile(samples, 95),
|
||||||
|
uploadBytesCopied: 0,
|
||||||
|
naiveCopyMs,
|
||||||
|
targetPageBudgetMs: 60_000,
|
||||||
|
decision: percentile(samples, 95) <= 60_000 ? 'slice holds' : 'slice is wrong',
|
||||||
|
};
|
||||||
|
|
||||||
|
const out = join(dirname(fileURLToPath(import.meta.url)), '../runs/mirror-timing.json');
|
||||||
|
mkdirSync(dirname(out), { recursive: true });
|
||||||
|
writeFileSync(out, `${JSON.stringify(report, null, 2)}\n`);
|
||||||
|
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
import { applyTool } from './apply-tool.ts';
|
||||||
|
import { checkAssertion } from './check-assertion.ts';
|
||||||
|
import type { GoldenAssert, GrokResponse, SiteFixture, ToolCall, ToolName } from './types.ts';
|
||||||
|
|
||||||
|
const tools = new Set<ToolName>(['edit_heading', 'edit_text', 'update_option']);
|
||||||
|
|
||||||
|
function isToolName(name: string): name is ToolName {
|
||||||
|
return tools.has(name as ToolName);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseGrokToolCalls(grok: GrokResponse): ToolCall[] {
|
||||||
|
const calls = grok.choices[0]?.message.tool_calls ?? [];
|
||||||
|
return calls.map((call) => {
|
||||||
|
if (!isToolName(call.function.name)) {
|
||||||
|
throw new Error(`unknown tool: ${call.function.name}`);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
name: call.function.name,
|
||||||
|
arguments: JSON.parse(call.function.arguments) as Record<string, string>,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function scoreGrokResponse(input: {
|
||||||
|
site: SiteFixture;
|
||||||
|
assert: GoldenAssert;
|
||||||
|
grok: GrokResponse;
|
||||||
|
}): { passed: boolean } {
|
||||||
|
const next = parseGrokToolCalls(input.grok).reduce(applyTool, input.site);
|
||||||
|
return { passed: checkAssertion(next, input.assert).ok };
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
import type { SiteExport, SubsetRequest, SubsetResult } from './types.ts';
|
||||||
|
|
||||||
|
const contentTables = new Set(['wp_posts', 'wp_postmeta', 'wp_options']);
|
||||||
|
const secret = /(_key|_secret|smtp_pass)$/;
|
||||||
|
|
||||||
|
export function exportDbSubset(dump: SiteExport, request: SubsetRequest): SubsetResult {
|
||||||
|
const tables = Object.keys(dump.tables).filter((name) => contentTables.has(name));
|
||||||
|
const options = (dump.tables.wp_options ?? [])
|
||||||
|
.map((row) => String(row.option_name ?? ''))
|
||||||
|
.filter((name) => name !== '' && !secret.test(name));
|
||||||
|
|
||||||
|
if (request.playbook === 'content') {
|
||||||
|
return { tables, options };
|
||||||
|
}
|
||||||
|
return { tables, options };
|
||||||
|
}
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
export type Builder = 'elementor' | 'beaver' | 'divi' | 'gutenberg' | 'classic';
|
||||||
|
|
||||||
|
export type PluginRef = {
|
||||||
|
slug: string;
|
||||||
|
active: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type PostFixture = {
|
||||||
|
id: number;
|
||||||
|
slug: string;
|
||||||
|
title: string;
|
||||||
|
content: string;
|
||||||
|
meta: Record<string, string>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type SiteFixture = {
|
||||||
|
id: string;
|
||||||
|
theme: string;
|
||||||
|
plugins: PluginRef[];
|
||||||
|
wordpressVersion: string;
|
||||||
|
phpVersion: string;
|
||||||
|
options: Record<string, string>;
|
||||||
|
posts: PostFixture[];
|
||||||
|
uploads: { path: string; bytes: number }[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export type PreviewTextAssert = {
|
||||||
|
type: 'preview_text';
|
||||||
|
page: string;
|
||||||
|
contains: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type OptionAssert = {
|
||||||
|
type: 'option';
|
||||||
|
key: string;
|
||||||
|
value: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ScreenshotAssert = {
|
||||||
|
type: 'screenshot';
|
||||||
|
page: string;
|
||||||
|
contains: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type GoldenAssert = PreviewTextAssert | OptionAssert | ScreenshotAssert;
|
||||||
|
|
||||||
|
export type GoldenPrompt = {
|
||||||
|
id: string;
|
||||||
|
site: string;
|
||||||
|
prompt: string;
|
||||||
|
assert: GoldenAssert;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type ToolName = 'edit_heading' | 'edit_text' | 'update_option';
|
||||||
|
|
||||||
|
export type ToolCall = {
|
||||||
|
name: ToolName;
|
||||||
|
arguments: Record<string, string>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type GrokResponse = {
|
||||||
|
choices: Array<{
|
||||||
|
message: {
|
||||||
|
tool_calls?: Array<{
|
||||||
|
function: {
|
||||||
|
name: string;
|
||||||
|
arguments: string;
|
||||||
|
};
|
||||||
|
}>;
|
||||||
|
};
|
||||||
|
}>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type SiteExport = {
|
||||||
|
origin: string;
|
||||||
|
tables: Record<string, Array<Record<string, string | number>>>;
|
||||||
|
uploads: { path: string; bytes: number }[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export type SubsetRequest = {
|
||||||
|
playbook: 'content' | 'design' | 'plugin';
|
||||||
|
postIds: number[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export type SubsetResult = {
|
||||||
|
tables: string[];
|
||||||
|
options: string[];
|
||||||
|
};
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"name": "@wursor/e2e",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"test": "vitest run",
|
||||||
|
"golden": "tsx golden/src/run-golden.ts",
|
||||||
|
"mirror:time": "tsx golden/src/run-mirror-timing.ts"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"tsx": "^4.20.3",
|
||||||
|
"typescript": "^5.9.2",
|
||||||
|
"vitest": "^3.2.4"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"extends": "../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"noEmit": true,
|
||||||
|
"rootDir": ".",
|
||||||
|
"types": ["node"]
|
||||||
|
},
|
||||||
|
"include": ["golden"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
import { defineConfig } from 'vitest/config';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
test: {
|
||||||
|
include: ['golden/**/*.test.ts'],
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"name": "wursor",
|
||||||
|
"private": true,
|
||||||
|
"packageManager": "pnpm@10.33.0",
|
||||||
|
"engines": {
|
||||||
|
"node": ">=22"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"test": "pnpm -r --if-present test",
|
||||||
|
"test:api": "pnpm --filter @wursor/api test",
|
||||||
|
"test:web": "pnpm --filter @wursor/web test",
|
||||||
|
"test:e2e": "pnpm --filter @wursor/e2e test",
|
||||||
|
"lint": "pnpm -r --if-present lint"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"name": "wursor/plugin",
|
||||||
|
"description": "Wursor WordPress connector",
|
||||||
|
"type": "wordpress-plugin",
|
||||||
|
"license": "GPL-2.0-or-later",
|
||||||
|
"require": {
|
||||||
|
"php": ">=7.4"
|
||||||
|
},
|
||||||
|
"require-dev": {
|
||||||
|
"phpunit/phpunit": "^9.6",
|
||||||
|
"yoast/phpunit-polyfills": "^2.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
Generated
+1038
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,4 @@
|
|||||||
|
packages:
|
||||||
|
- api
|
||||||
|
- web
|
||||||
|
- e2e
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Phase 0 spikes
|
||||||
|
|
||||||
|
These notes are the gate. Do not implement `web/` chat, playbooks, or deploy until every note below is `done`.
|
||||||
|
|
||||||
|
Throwaway fixtures and scripts are allowed. Product UI is not.
|
||||||
|
|
||||||
|
| Spike | File | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | partial — live Grok pending key |
|
||||||
|
| Builder detect (R6 / R13) | [builder-detect.md](./builder-detect.md) | done |
|
||||||
|
| Pairing threat model (R9) | [pairing-threat-model.md](./pairing-threat-model.md) | done |
|
||||||
|
| Large-site mirror timing (R4) | [mirror-timing.md](./mirror-timing.md) | done — synthetic 2GB |
|
||||||
|
| P0 plugin catalog | [plugin-catalog.md](./plugin-catalog.md) | done |
|
||||||
|
|
||||||
|
Batch writeup: [phase-0-harness.md](./phase-0-harness.md).
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# Spike: builder detect (R6 / R13)
|
||||||
|
|
||||||
|
**Status:** done
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
How do we know what actually renders a page?
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
- A site-info payload reports `builder: elementor | beaver | divi | gutenberg | classic`
|
||||||
|
- Detection uses plugin slugs + post meta
|
||||||
|
- Documented in the plugin API sketch below
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
`detectBuilder()` in `e2e/golden/src/builder-detect.ts`.
|
||||||
|
|
||||||
|
Order (first match wins):
|
||||||
|
|
||||||
|
1. Active `elementor` **and** `_elementor_edit_mode` or `_elementor_data` → `elementor`
|
||||||
|
2. Active `beaver-builder-lite-version` or `bb-plugin` **and** `_fl_builder_data` or `_fl_builder_enabled` → `beaver`
|
||||||
|
3. Theme `Divi` or active `divi-builder` **and** `_et_pb_use_builder === on` → `divi`
|
||||||
|
4. Any post content contains `<!-- wp:` → `gutenberg`
|
||||||
|
5. Else `classic`
|
||||||
|
|
||||||
|
Plugin slug alone is not enough (inactive junk). Gutenberg markup loses to Elementor when both exist — Elementor is what renders.
|
||||||
|
|
||||||
|
Proved on the two golden fixtures and six unit tests.
|
||||||
|
|
||||||
|
## Plugin API sketch
|
||||||
|
|
||||||
|
`GET /wp-json/wursor/v1/site-info` (read token + HMAC)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"theme": "hello-elementor",
|
||||||
|
"plugins": [{ "slug": "elementor", "active": true }],
|
||||||
|
"wordpress_version": "6.5.5",
|
||||||
|
"php_version": "8.1.30",
|
||||||
|
"builder": "elementor",
|
||||||
|
"capabilities": { "content": true, "design": true, "install": true },
|
||||||
|
"preflight": { "https": true, "rest": true, "disallow_file_mods": false }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`builder` is computed on the plugin with the same rules as `detectBuilder`. Content and design playbooks must use this field. Editing `post_content` on an Elementor site is a failed test.
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
|
||||||
|
- **Context:** Elementor stores the page in post meta. Gutenberg stores it in `post_content`.
|
||||||
|
- **Chosen:** slugs + the meta keys those builders actually write. Priority: paid builders, then block markup, then classic.
|
||||||
|
- **Rejected:** “if elementor is installed, always Elementor” (inactive plugin). Theme-name-only detection.
|
||||||
|
- **Reverted later?**
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Spike: golden-task harness (R7)
|
||||||
|
|
||||||
|
**Status:** partial — harness exists; live Grok run not scored (`XAI_API_KEY` unset)
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
Can we score a model on WordPress tasks without vibes?
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
- 20 fixture prompts against at least 2 canned WordPress sites
|
||||||
|
- Each prompt has a hard assertion (preview text, option value, or screenshot)
|
||||||
|
- One Grok run is scored
|
||||||
|
- Harness lives under `e2e/golden/`
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. No, if it means we have a Grok quality number. This machine has no `XAI_API_KEY`, so the live call was skipped.
|
||||||
|
|
||||||
|
### What exists
|
||||||
|
|
||||||
|
| Piece | Path |
|
||||||
|
|---|---|
|
||||||
|
| 20 prompts | `e2e/golden/prompts.json` |
|
||||||
|
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
|
||||||
|
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
|
||||||
|
| Apply + assert + Grok parser | `e2e/golden/src/` |
|
||||||
|
| Scoreboard | `e2e/golden/runs/latest.json` |
|
||||||
|
|
||||||
|
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
|
||||||
|
|
||||||
|
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces. Live Grok: skipped.
|
||||||
|
|
||||||
|
`pnpm test:e2e` — 21 tests, including the scorer.
|
||||||
|
|
||||||
|
### How to score a real Grok run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
XAI_API_KEY=… pnpm --filter @wursor/e2e golden
|
||||||
|
```
|
||||||
|
|
||||||
|
That sends `gb-01` through `api.x.ai` and asserts the homepage heading.
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
|
||||||
|
- **Context:** R7 said stop grading models by vibes.
|
||||||
|
- **Chosen:** slot-fill tools + fixture apply + hard assert. Sites are JSON, not Docker WP (Docker was not available).
|
||||||
|
- **Rejected:** “the model said it worked.” Waiting on Docker before any harness.
|
||||||
|
- **Reverted later?**
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Spike: large-site mirror timing (R4)
|
||||||
|
|
||||||
|
**Status:** done — local synthetic 2GB; not a live WP pull
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
Does the 5-minute MVP exit survive a real site?
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
- Time a task-scoped content mirror + media proxy against one ≥2GB WordPress export (or a synthetic one)
|
||||||
|
- Record p50 / p95
|
||||||
|
- Target page on screen in ≤60s. If not, the Layer 3 slice is wrong — change it before building chat.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
The **slice holds on this machine** for in-process subset + proxy. A full library copy is the slow path; we do not take it.
|
||||||
|
|
||||||
|
Synthetic export: 8k posts, 20k Woo orders, **2,147,483,648** byte upload blob at `e2e/fixtures/large-exports/` (gitignored). 20 subset+proxy runs, then a `dd` of the blob as the naive-copy baseline.
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|---|---|
|
||||||
|
| p50 time to target page | **0.003 ms** |
|
||||||
|
| p95 time to target page | **0.010 ms** |
|
||||||
|
| Upload bytes copied (slice) | **0** |
|
||||||
|
| Naive local `dd` of 2GB | **2449 ms** (~2.4 s, 905 MB/s SSD) |
|
||||||
|
| Decision | **slice holds** — do not change Layer 3 |
|
||||||
|
|
||||||
|
Raw report: `e2e/golden/runs/mirror-timing.json`.
|
||||||
|
|
||||||
|
`cp` on APFS cloned the file in ~2s and was discarded as a baseline. `dd if=… of=…` is the number above.
|
||||||
|
|
||||||
|
### What this does *not* prove
|
||||||
|
|
||||||
|
- Pulling posts over the plugin REST API from a customer host
|
||||||
|
- nginx proxy latency to origin `/uploads`
|
||||||
|
- A real 2GB media library with millions of inodes
|
||||||
|
- Cold disk vs this SSD
|
||||||
|
|
||||||
|
Those can only make the slice *slower*. They do not argue for copying the library. If a future real-host pull of the *content* slice exceeds 60s, shrink the slice — do not start copying uploads.
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
|
||||||
|
- **Context:** warm pool hides boot, not copy. Media is often the bulk of a 2GB site.
|
||||||
|
- **Chosen:** task-scoped tables + origin proxy. Copy a file only on replace.
|
||||||
|
- **Rejected:** full library sync. APFS `cp` as the naive baseline.
|
||||||
|
- **Reverted later?**
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
# Spike: pairing threat model (R9)
|
||||||
|
|
||||||
|
**Status:** done
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
What stops a leaked URL from owning the site?
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
Written threat model that becomes the Sprint 2 / Layer 2 auth tests:
|
||||||
|
|
||||||
|
- 8+ character pairing code
|
||||||
|
- 5-minute TTL
|
||||||
|
- 5-attempt lockout
|
||||||
|
- HMAC request signing
|
||||||
|
- hashed + scoped tokens (read vs deploy)
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
The plugin is a privileged backdoor: files, DB, WP-CLI. A leaked URL, a guessed pairing code, or a stolen bearer token is site ownership. Isolation of the *sandbox* does not help — this boundary is the *live* site.
|
||||||
|
|
||||||
|
### Locked flow
|
||||||
|
|
||||||
|
Wursor generates the pairing code (bound to the signed-in account). The user pastes it into the plugin. The plugin redeems it with the site URL. Tokens are issued once.
|
||||||
|
|
||||||
|
This matches PRD §7.1.4. The `Wursor_Auth::generate_pairing_code()` sketch in IMPLEMENTATION.md is the wrong direction — plugin-local generate/redeem cannot bind the code to an account before the site is known. Sprint 2 tests follow this note, not that sketch.
|
||||||
|
|
||||||
|
```
|
||||||
|
User (signed in) → POST /sites/pair → Wursor stores pending pairing
|
||||||
|
User pastes code in plugin admin
|
||||||
|
Plugin → POST https://api.wursor…/sites/redeem { code, site_url }
|
||||||
|
Wursor binds site_url, returns read_token + deploy_token + hmac_secret (once)
|
||||||
|
Plugin stores hashes + encrypted hmac_secret
|
||||||
|
Wursor stores tokens encrypted (it must send them later)
|
||||||
|
```
|
||||||
|
|
||||||
|
Wursor is the HTTPS client. The plugin is the server. Tokens never appear in query strings or logs.
|
||||||
|
|
||||||
|
### Protocol
|
||||||
|
|
||||||
|
**Pairing code**
|
||||||
|
|
||||||
|
- Alphabet: `[A-Z0-9]`, length ≥ 8. Generate 8. `36^8 ≈ 2.8e12`.
|
||||||
|
- Bound to `account_id` at creation. Not reusable after success.
|
||||||
|
- `expires_at = created_at + 300s`. Clock for tests is injectable (`advance_clock`).
|
||||||
|
- After 5 failed redeems on that code, `locked = true`. Further redeems fail even if the code is correct.
|
||||||
|
- Redeem also fails if `site_url` is not `https` or does not parse as a URL.
|
||||||
|
- One successful redeem. Second redeem of the same code fails.
|
||||||
|
|
||||||
|
**Tokens**
|
||||||
|
|
||||||
|
| Token | Scope | Plugin endpoints |
|
||||||
|
|---|---|---|
|
||||||
|
| `read` | site-info, file read, DB read, preflight | GET only |
|
||||||
|
| `deploy` | file write, DB write, WP-CLI, prepare/commit, rollback | mutating |
|
||||||
|
|
||||||
|
- 256-bit random, encoded unpadded base64url, shown once.
|
||||||
|
- Plugin stores `SHA-256(token)` only. Compare with `hash_equals`.
|
||||||
|
- Wursor stores ciphertext (envelope key, not plaintext in Postgres).
|
||||||
|
- A `read` token on a deploy route returns 403. A `deploy` token may call read routes.
|
||||||
|
- Rotation: Wursor issues a new pair; plugin replaces hashes; old hashes stop working.
|
||||||
|
- Disconnect: both hashes deleted; Wursor ciphertext deleted.
|
||||||
|
|
||||||
|
**HMAC (every plugin request)**
|
||||||
|
|
||||||
|
```
|
||||||
|
canonical = timestamp + "\n" + METHOD + "\n" + path + "\n" + hex(sha256(body))
|
||||||
|
X-Wursor-Timestamp: unix seconds
|
||||||
|
X-Wursor-Signature: hex(HMAC-SHA256(hmac_secret, canonical))
|
||||||
|
Authorization: Bearer <read_token|deploy_token>
|
||||||
|
```
|
||||||
|
|
||||||
|
- Reject if `|now - timestamp| > 60`.
|
||||||
|
- Reject if signature missing or `hash_equals` fails.
|
||||||
|
- `hmac_secret` is 256-bit, issued at redeem, stored on the plugin encrypted with the site salt (`AUTH_KEY` + `AUTH_SALT`). Not the same bytes as either token.
|
||||||
|
- Body hash is over the raw bytes. Empty body is SHA-256 of `""`.
|
||||||
|
|
||||||
|
**Transport**
|
||||||
|
|
||||||
|
- Plugin REST namespace: `/wp-json/wursor/v1/`.
|
||||||
|
- Plugin refuses non-HTTPS callbacks except `WP_ENVIRONMENT_TYPE === 'local'`.
|
||||||
|
- Wursor never puts tokens in URLs, logs, or SSE payloads.
|
||||||
|
|
||||||
|
### Threats
|
||||||
|
|
||||||
|
| ID | Threat | Mitigation | Residual |
|
||||||
|
|---|---|---|---|
|
||||||
|
| T1 | Attacker guesses pairing codes | 8+ charset, 5-try lockout, 5-min TTL | Online brute force is ~5 guesses / 5 min / code |
|
||||||
|
| T2 | Pairing code leaked (screenshot, chat) | TTL + single use + requires wp-admin to paste | Anyone with the code and wp-admin wins until expiry |
|
||||||
|
| T3 | Attacker redeems victim's code onto attacker site | After redeem, Wursor shows the bound `site_url` and requires an explicit “this is my site” confirm before the site is usable | User who confirms a foreign URL is connected to it |
|
||||||
|
| T4 | Bearer token in a URL / access log / Referer | Tokens only in `Authorization`. Tests fail if any helper puts them in a query | Operator error in a future client |
|
||||||
|
| T5 | Stolen request replayed | HMAC over timestamp+method+path+body; 60s skew window | Replay inside the window if the request was captured |
|
||||||
|
| T6 | Stolen `read` token used to deploy | Scoped tokens; deploy routes require `deploy` | Read token still exfiltrates site-info |
|
||||||
|
| T7 | Plugin DB dump / filesystem copy | Plugin stores hashes + encrypted hmac_secret, not raw tokens | Wursor-side ciphertext leak still lets us *call* the plugin until rotation |
|
||||||
|
| T8 | MITM on HTTP | HTTPS required except local | Mis-set `WP_ENVIRONMENT_TYPE` on a public HTTP site |
|
||||||
|
| T9 | CSRF in the browser against plugin REST | Bearer + HMAC. No cookie auth for `/wursor/v1/` | None if those headers stay required |
|
||||||
|
| T10 | Timing leak on token compare | `hash_equals` only | — |
|
||||||
|
|
||||||
|
Out of scope for this spike (handled elsewhere): stolen wp-admin session, compromised host, malicious plugin already on the site.
|
||||||
|
|
||||||
|
### Sprint 2 tests (this note is the spec)
|
||||||
|
|
||||||
|
`plugin/__tests__/test-auth.php`
|
||||||
|
|
||||||
|
1. `test_pairing_code_is_at_least_eight_alnum` — `^[A-Z0-9]{8,}$`
|
||||||
|
2. `test_pairing_code_expires_after_five_minutes` — `advance_clock(301)` → redeem false
|
||||||
|
3. `test_pairing_code_valid_at_four_minutes_fifty_nine` — `advance_clock(299)` → redeem true
|
||||||
|
4. `test_locks_out_after_five_failed_attempts` — five bad redeems → `is_locked_out()`
|
||||||
|
5. `test_lockout_rejects_even_the_correct_code`
|
||||||
|
6. `test_successful_redeem_cannot_be_replayed`
|
||||||
|
7. `test_read_token_hash_is_stored_not_plaintext`
|
||||||
|
8. `test_read_token_forbidden_on_deploy_route` → 403
|
||||||
|
9. `test_deploy_token_allowed_on_site_info`
|
||||||
|
10. `test_hmac_rejects_stale_timestamp` — timestamp older than 60s
|
||||||
|
11. `test_hmac_rejects_tampered_body`
|
||||||
|
12. `test_hmac_rejects_missing_signature`
|
||||||
|
13. `test_verify_uses_hash_equals`
|
||||||
|
14. `test_rotated_tokens_invalidate_old_hashes`
|
||||||
|
|
||||||
|
`api/__tests__/services/plugin-client.test.ts`
|
||||||
|
|
||||||
|
1. signs every request with timestamp + HMAC
|
||||||
|
2. sends token in `Authorization`, never in the URL
|
||||||
|
3. maps 401 to `Authentication failed`
|
||||||
|
4. refuses to construct a client with an `http://` site URL outside local
|
||||||
|
|
||||||
|
`api/__tests__/routes/sites-pair.test.ts`
|
||||||
|
|
||||||
|
1. pair requires a session
|
||||||
|
2. redeem binds `site_url` and returns tokens once
|
||||||
|
3. second redeem of the same code fails
|
||||||
|
4. site is not `connected` until the user confirms the shown URL (T3)
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
|
||||||
|
- **Context:** plugin can own the live site; old 6-char sketch had no TTL, lockout, HMAC, or scopes.
|
||||||
|
- **Options:** plugin-generated code (TV pairing) vs Wursor-generated code (PRD).
|
||||||
|
- **Chosen:** Wursor-generated, pasted into the plugin, HMAC + scoped tokens as above.
|
||||||
|
- **Rejected:** plugin-local generate/redeem (cannot bind to account first; IMPLEMENTATION sketch). Tokens in query strings. Single unscope token.
|
||||||
|
- **Reverted later?**
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# What this batch is for
|
||||||
|
|
||||||
|
Pairing and the plugin catalog (notes 1–2) answered “who is allowed to talk to the live site” and “what may ever be installed.” This batch answers the other three kill-shots before any product UI exists:
|
||||||
|
|
||||||
|
3. Can we tell if the model actually did the WordPress thing?
|
||||||
|
4. Do we know which store holds the page (Gutenberg vs Elementor vs Classic)?
|
||||||
|
5. Can a 2GB site still show a preview in time if we refuse to copy the media library?
|
||||||
|
|
||||||
|
If 3 is vibes, we will ship a confident liar. If 4 is wrong, we will edit `post_content` while Elementor renders JSON from post meta — the preview will not change and we will not know why. If 5 is a full copy, the five-minute product is dead on any real business site.
|
||||||
|
|
||||||
|
None of this is the chat app. It is the measuring stick the chat app has to pass.
|
||||||
|
|
||||||
|
## 3 — Golden-task harness
|
||||||
|
|
||||||
|
### What it does
|
||||||
|
|
||||||
|
It keeps twenty English requests, two fake but structured WordPress sites, and a scorer.
|
||||||
|
|
||||||
|
A prompt is not “make it modern.” It is “change the homepage heading to Welcome to My Business.” The assertion is not “the model sounded sure.” It is: after the tool call is applied to the fixture, that string is in the page (or that option equals that value).
|
||||||
|
|
||||||
|
Sites:
|
||||||
|
|
||||||
|
- `gutenberg-business` — Twenty Twenty-Four, block markup, a dental practice.
|
||||||
|
- `elementor-restaurant` — Hello Elementor, `_elementor_data` JSON, a trattoria.
|
||||||
|
|
||||||
|
Ten prompts each. Mix of heading edits, text replace, `blogname` / `blogdescription` / `blog_public`, and two `screenshot` rows that today still assert on fixture text (see uncertainties).
|
||||||
|
|
||||||
|
### How it was implemented
|
||||||
|
|
||||||
|
TDD first. Tests imported modules that did not exist. Vitest failed. Then:
|
||||||
|
|
||||||
|
- `applyTool` mutates a `SiteFixture` (heading regex or Elementor `title`, string replace, option set).
|
||||||
|
- `checkAssertion` reads the result.
|
||||||
|
- `scoreGrokResponse` parses an xAI/OpenAI-shaped `tool_calls` payload, applies, asserts.
|
||||||
|
- `run-golden.ts` scores the twenty expected traces and, if `XAI_API_KEY` is set, sends `gb-01` to `api.x.ai`.
|
||||||
|
|
||||||
|
Proof on this machine:
|
||||||
|
|
||||||
|
```
|
||||||
|
pnpm test:e2e → 21 passed
|
||||||
|
pnpm --filter @wursor/e2e golden
|
||||||
|
→ 20/20 fixture traces
|
||||||
|
→ live Grok skipped (no key)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why this shape
|
||||||
|
|
||||||
|
WordPress work is slot filling (`page`, `old`, `new`) plus a deterministic write. If we score free-form chat, Grok can narrate a heading change that never happened. If we only score “did the API return 200,” we learn nothing about the page.
|
||||||
|
|
||||||
|
Applying the tool to a fixture is the smallest thing that can fail for the right reason. JSON sites instead of Docker WP because Docker was not available here; the assertion types stay valid when real sandboxes exist — swap `applyTool` for a REST/`wp post update` runner and keep `prompts.json`.
|
||||||
|
|
||||||
|
Expected traces are an answer key for the *harness*, not a grade for Grok. A live grade is one HTTP call away. That is deliberate: unit tests stay offline (TDD rule: no network in unit tests).
|
||||||
|
|
||||||
|
## 4 — Builder detect
|
||||||
|
|
||||||
|
### What it does
|
||||||
|
|
||||||
|
Given theme, plugin slugs, post content, and post meta, it returns exactly one of:
|
||||||
|
|
||||||
|
`elementor | beaver | divi | gutenberg | classic`
|
||||||
|
|
||||||
|
That value is what `site-info` will send. Playbooks are required to branch on it.
|
||||||
|
|
||||||
|
### How it was implemented
|
||||||
|
|
||||||
|
Same TDD file: six cases, one assertion each. Rules use **slug plus the meta key that builder actually writes**. Gutenberg is `<!-- wp:` in content. Classic is the leftover.
|
||||||
|
|
||||||
|
Elementor wins over block markup. A site can have leftover Gutenberg in `post_content` while Elementor is what the visitor sees. Editing the wrong store is the R6/R13 failure mode.
|
||||||
|
|
||||||
|
### Why this is the right approach
|
||||||
|
|
||||||
|
Theme name alone is a lie (Hello Elementor vs a child theme vs Divi). “Elementor is installed” is a lie (inactive). Reading only `post_content` is a lie on builders.
|
||||||
|
|
||||||
|
The combination is what WordPress itself uses: active plugin, then that plugin’s post meta. We copied that, in a function small enough to test without PHP. The plugin will run the same rules in PHP later; the TypeScript copy is the spec the PHP tests must match.
|
||||||
|
|
||||||
|
## 5 — Mirror timing
|
||||||
|
|
||||||
|
### What it does
|
||||||
|
|
||||||
|
It asks: if the site’s uploads are 2GB, can the *content-edit* path still put the target page in a sandbox in under 60 seconds?
|
||||||
|
|
||||||
|
The prototype:
|
||||||
|
|
||||||
|
1. Builds a synthetic export: 8k posts, 20k Woo orders, a **2,147,483,648** byte blob.
|
||||||
|
2. Runs `exportDbSubset(content)` — keeps `wp_posts` / `wp_postmeta` / `wp_options`, drops orders and comments, redacts `*_key` / `*_secret` / `smtp_pass`.
|
||||||
|
3. Resolves `/uploads/…` to `origin + path`. Copies **0** upload bytes.
|
||||||
|
4. Times that 20 times.
|
||||||
|
5. Times a real local `dd` of the 2GB blob as “what copying the library costs on this disk.”
|
||||||
|
|
||||||
|
### How it was implemented
|
||||||
|
|
||||||
|
Tests first (subset tables, redaction, proxy, replace-only copy). Then `run-mirror-timing.ts`. `mkfile` created the blob once under `e2e/fixtures/large-exports/` (gitignored). First naive baseline used `cp`; on APFS that is `clonefile` and finished in ~2s without writing bytes. That number was thrown out. `dd if=… of=…` wrote the bytes: **2449 ms**, ~905 MB/s.
|
||||||
|
|
||||||
|
Slice p50 **0.003 ms**, p95 **0.010 ms**, upload bytes copied **0**. Decision: **do not change Layer 3**.
|
||||||
|
|
||||||
|
### Why this is the right approach
|
||||||
|
|
||||||
|
Warm pool hides *boot*. It does not hide *copy*. A Woo store’s cost is `wp_posts`, plugin folders, and upload thumbs — not MySQL start time. Copying 2GB from a customer host over the plugin REST API will not finish in a minute. Proxying `/wp-content/uploads/*` to origin makes the preview honest without the copy. The 2.4s local `dd` is a *lower bound* on copy cost; a real host will be slower. The slice does not need that copy at all.
|
||||||
|
|
||||||
|
Redacting secrets in the same function is R10: a sandbox with outbound internet should not hold `smtp_pass`.
|
||||||
|
|
||||||
|
## Why this batch, in this order, is the best next step
|
||||||
|
|
||||||
|
The product is describe → preview → approve. Before UI:
|
||||||
|
|
||||||
|
- You need a test that can fail a bad model (3).
|
||||||
|
- You need to know which bytes to change or the preview is fake (4).
|
||||||
|
- You need to know the preview can appear before the user leaves (5).
|
||||||
|
|
||||||
|
Doing them as scripts + fixtures instead of `web/` + `api/` keeps the Phase 0 gate honest. We did not invent a chat panel that cannot be scored.
|
||||||
|
|
||||||
|
TDD on the spike code means the later plugin/API ports have a contract: same types, same assertions, same 60s budget.
|
||||||
|
|
||||||
|
## What I am not sure about
|
||||||
|
|
||||||
|
1. **Live Grok was not scored.** No `XAI_API_KEY` in this environment. The harness can call `api.x.ai`; it did not. I do not know Grok’s actual score on these twenty prompts. Do not treat 20/20 fixture traces as a model eval.
|
||||||
|
|
||||||
|
2. **Sites are JSON, not WordPress.** No Docker on this machine. Builder detect and heading replace are faithful to how WP stores data, but they are not PHP, not `$wpdb`, not a running theme. A real Elementor document is a deeper JSON tree than the two-widget fixture.
|
||||||
|
|
||||||
|
3. **“Screenshot” assertions are not screenshots.** They assert text in the fixture. Pixel/SSIM checks need a sandbox and Playwright. I used the type so the prompt file matches the done-when enum, not because we captured a PNG.
|
||||||
|
|
||||||
|
4. **Mirror p50/p95 are in-process.** They do not include plugin HTTP, PHP serialization, or nginx. They prove the *algorithm* is not the 60s problem. They do not prove a 2GB *site on SiteGround* will preview in 60s. I am sure we should still not copy uploads. I am not sure the first real content-slice *pull* will stay under 60s.
|
||||||
|
|
||||||
|
5. **Naive copy at 2.4s is this SSD.** A cheap VPS or a network pull will be worse. Do not quote 2.4s as “copy is fine.”
|
||||||
|
|
||||||
|
6. **Elementor `title` patcher.** `JSON.stringify`’s replacer changes the first `title` key it sees. That is correct for the fixture (heading widget first). A real tree may put a button title first. The PHP adapter must walk widgets by `widgetType === heading'`, not “first title.”
|
||||||
|
|
||||||
|
7. **Grok model id.** The client uses `grok-3`. If xAI has renamed the tool-calling model, the live runner will 404 until that string is updated.
|
||||||
|
|
||||||
|
8. **Divi/Beaver** are unit-tested with synthetic meta only. They are not in the two canned sites. Alpha still needs a real Elementor host; Beaver/Divi are unverified in the wild.
|
||||||
|
|
||||||
|
## Commands a reviewer can rerun
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test:e2e
|
||||||
|
pnpm --filter @wursor/e2e golden
|
||||||
|
pnpm --filter @wursor/e2e mirror:time # reuses the 2GB blob if present
|
||||||
|
```
|
||||||
|
|
||||||
|
Gate status after this batch: catalog done, pairing done, builder done, mirror done on synthetic 2GB, golden harness built, **live Grok still open**. Product chat still blocked until you either accept that gap or set `XAI_API_KEY` and score `gb-01`.
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
# Spike: P0 plugin catalog
|
||||||
|
|
||||||
|
**Status:** done
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
What may the agent ever install?
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
- ~40 slugs written
|
||||||
|
- `wp plugin install <url>` is forbidden even before an install playbook exists
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
The agent may install **only** from the catalog table below, and only from wordpress.org via slug (`wp plugin install <slug>`). That playbook does not exist in the MVP. The rule still ships in Sprint 3 tool schemas so a heading-change agent cannot install anything.
|
||||||
|
|
||||||
|
### Forbidden (now and in v1)
|
||||||
|
|
||||||
|
- `wp plugin install <url>`
|
||||||
|
- `wp plugin install` with a local zip or any path
|
||||||
|
- Premium / marketplace URLs (Elementor Pro zip, Woo market, GitHub)
|
||||||
|
- Any slug not in the catalog
|
||||||
|
- Re-install of a detect-only plugin (table below)
|
||||||
|
|
||||||
|
Fail closed. Unknown slug → refuse and say so in chat. Do not search wordpress.org ad hoc.
|
||||||
|
|
||||||
|
Reputation, zip SHA, and egress-watch on first activate are Sprint 9. This note is only the allowlist.
|
||||||
|
|
||||||
|
Paid *services* behind a free slug (WooPayments, Site Kit) still hit the no-surprise gate at deploy.
|
||||||
|
|
||||||
|
### Catalog (40)
|
||||||
|
|
||||||
|
wordpress.org slugs. Installable only when the site is `install-safe` (writable filesystem, `DISALLOW_FILE_MODS` off).
|
||||||
|
|
||||||
|
| # | Slug | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | `contact-form-7` | Forms. Most common. |
|
||||||
|
| 2 | `wpforms-lite` | Forms. Non-technical default. |
|
||||||
|
| 3 | `fluentform` | Forms. |
|
||||||
|
| 4 | `forminator` | Forms + polls/quizzes. |
|
||||||
|
| 5 | `ninja-forms` | Forms. |
|
||||||
|
| 6 | `wordpress-seo` | Yoast. SEO. |
|
||||||
|
| 7 | `seo-by-rank-math` | SEO. |
|
||||||
|
| 8 | `all-in-one-seo-pack` | SEO. |
|
||||||
|
| 9 | `woocommerce` | Store. |
|
||||||
|
| 10 | `woocommerce-payments` | Checkout. Paid service → no-surprise. |
|
||||||
|
| 11 | `woocommerce-gateway-stripe` | Checkout. |
|
||||||
|
| 12 | `woocommerce-paypal-payments` | Checkout. |
|
||||||
|
| 13 | `elementor` | Builder. Install only if site has no other builder. |
|
||||||
|
| 14 | `kadence-blocks` | Gutenberg blocks. |
|
||||||
|
| 15 | `stackable-ultimate-gutenberg-blocks` | Gutenberg blocks. |
|
||||||
|
| 16 | `ultimate-addons-for-gutenberg` | Spectra. Gutenberg blocks. |
|
||||||
|
| 17 | `classic-editor` | Needed on Classic sites before content edits. |
|
||||||
|
| 18 | `simply-schedule-appointments` | Booking. |
|
||||||
|
| 19 | `booking` | Booking Calendar. |
|
||||||
|
| 20 | `easy-appointments` | Booking. |
|
||||||
|
| 21 | `google-site-kit` | Analytics. Google account → no-surprise. |
|
||||||
|
| 22 | `independent-analytics` | Analytics without Google. |
|
||||||
|
| 23 | `fluent-smtp` | Mail delivery. SMTP secrets stay redacted (R10). |
|
||||||
|
| 24 | `wp-mail-smtp` | Mail delivery. |
|
||||||
|
| 25 | `redirection` | Redirects. Slug changes still hit R12. |
|
||||||
|
| 26 | `akismet` | Comment spam. |
|
||||||
|
| 27 | `cookie-law-info` | CookieYes. Consent. |
|
||||||
|
| 28 | `complianz-gdpr` | Consent. |
|
||||||
|
| 29 | `updraftplus` | Backup. |
|
||||||
|
| 30 | `backwpup` | Backup. |
|
||||||
|
| 31 | `autoptimize` | CSS/JS minify. |
|
||||||
|
| 32 | `wp-optimize` | Cleanup / cache. |
|
||||||
|
| 33 | `polylang` | Languages. |
|
||||||
|
| 34 | `translatepress-multilingual` | Languages. |
|
||||||
|
| 35 | `tablepress` | Tables. |
|
||||||
|
| 36 | `duplicate-post` | Duplicate a page. |
|
||||||
|
| 37 | `disable-comments` | Turn comments off. |
|
||||||
|
| 38 | `safe-svg` | SVG uploads. |
|
||||||
|
| 39 | `modula-best-grid-gallery` | Gallery. |
|
||||||
|
| 40 | `foogallery` | Gallery. |
|
||||||
|
|
||||||
|
40 slugs. Adding one is a catalog PR, not a prompt change.
|
||||||
|
|
||||||
|
### Detect-only — never install
|
||||||
|
|
||||||
|
These break REST, deploys, or hosts. Site-info reports them. The agent does not install them.
|
||||||
|
|
||||||
|
| Slug | Why not |
|
||||||
|
|---|---|
|
||||||
|
| `wordfence` | Kills REST / pairing. |
|
||||||
|
| `better-wp-security` | iThemes. Same. |
|
||||||
|
| `sucuri-scanner` | Host / firewall coupling. |
|
||||||
|
| `litespeed-cache` | Host-specific. |
|
||||||
|
| `w3-total-cache` | Easy to brick a site. |
|
||||||
|
| `wp-super-cache` | Same. |
|
||||||
|
| `jetpack` | Too large; host-coupled. Configure if already present, later. |
|
||||||
|
|
||||||
|
### Sprint 3 tests (this note is the spec)
|
||||||
|
|
||||||
|
`api/__tests__/agents/tool-schemas.test.ts`
|
||||||
|
|
||||||
|
1. `generateToolSchemas()` has no tool or enum value matching `wp plugin install http`
|
||||||
|
2. `wp plugin install` (when it exists) accepts only slugs from this catalog
|
||||||
|
3. A tool call with slug `evil-plugin` is rejected before any sandbox exec
|
||||||
|
4. A tool call with a URL or `.zip` is rejected
|
||||||
|
|
||||||
|
Until the install playbook exists, the simpler form is enough: **no install tool at all**, plus assertion (1).
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
|
||||||
|
- **Context:** wordpress.org is not a reviewed-safe catalog. Preview cannot see a plugin phoning home. MVP has no install playbook, but the agent still needs a hard wall.
|
||||||
|
- **Options:** open wordpress.org search vs a written allowlist vs delay any wall until Sprint 9.
|
||||||
|
- **Chosen:** 40-slug allowlist now; no URL/zip; detect-only list for security/cache suites; no install tool in MVP.
|
||||||
|
- **Rejected:** open search (R2). Installing Wordfence/Jetpack/cache suites (they break the product or the site).
|
||||||
|
- **Reverted later?**
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"module": "Node16",
|
||||||
|
"moduleResolution": "Node16",
|
||||||
|
"strict": true,
|
||||||
|
"noUncheckedIndexedAccess": true,
|
||||||
|
"exactOptionalPropertyTypes": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"declaration": true,
|
||||||
|
"sourceMap": true
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"name": "@wursor/web",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"extends": "../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"jsx": "react-jsx",
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"lib": ["ES2022", "DOM", "DOM.Iterable"],
|
||||||
|
"noEmit": true
|
||||||
|
},
|
||||||
|
"include": ["src"]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user