feat: Phase 0 workspace, spikes, and golden harness
Stand up the monorepo skeleton and land the first Phase 0 artifacts: pairing threat model, plugin catalog, builder detect, 20-prompt golden harness, and synthetic 2GB mirror timing.
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Phase 0 spikes
|
||||
|
||||
These notes are the gate. Do not implement `web/` chat, playbooks, or deploy until every note below is `done`.
|
||||
|
||||
Throwaway fixtures and scripts are allowed. Product UI is not.
|
||||
|
||||
| Spike | File | Status |
|
||||
|---|---|---|
|
||||
| Golden-task harness (R7) | [golden-task.md](./golden-task.md) | partial — live Grok pending key |
|
||||
| Builder detect (R6 / R13) | [builder-detect.md](./builder-detect.md) | done |
|
||||
| Pairing threat model (R9) | [pairing-threat-model.md](./pairing-threat-model.md) | done |
|
||||
| Large-site mirror timing (R4) | [mirror-timing.md](./mirror-timing.md) | done — synthetic 2GB |
|
||||
| P0 plugin catalog | [plugin-catalog.md](./plugin-catalog.md) | done |
|
||||
|
||||
Batch writeup: [phase-0-harness.md](./phase-0-harness.md).
|
||||
@@ -0,0 +1,54 @@
|
||||
# Spike: builder detect (R6 / R13)
|
||||
|
||||
**Status:** done
|
||||
|
||||
## Question
|
||||
|
||||
How do we know what actually renders a page?
|
||||
|
||||
## Done when
|
||||
|
||||
- A site-info payload reports `builder: elementor | beaver | divi | gutenberg | classic`
|
||||
- Detection uses plugin slugs + post meta
|
||||
- Documented in the plugin API sketch below
|
||||
|
||||
## Result
|
||||
|
||||
`detectBuilder()` in `e2e/golden/src/builder-detect.ts`.
|
||||
|
||||
Order (first match wins):
|
||||
|
||||
1. Active `elementor` **and** `_elementor_edit_mode` or `_elementor_data` → `elementor`
|
||||
2. Active `beaver-builder-lite-version` or `bb-plugin` **and** `_fl_builder_data` or `_fl_builder_enabled` → `beaver`
|
||||
3. Theme `Divi` or active `divi-builder` **and** `_et_pb_use_builder === on` → `divi`
|
||||
4. Any post content contains `<!-- wp:` → `gutenberg`
|
||||
5. Else `classic`
|
||||
|
||||
Plugin slug alone is not enough (inactive junk). Gutenberg markup loses to Elementor when both exist — Elementor is what renders.
|
||||
|
||||
Proved on the two golden fixtures and six unit tests.
|
||||
|
||||
## Plugin API sketch
|
||||
|
||||
`GET /wp-json/wursor/v1/site-info` (read token + HMAC)
|
||||
|
||||
```json
|
||||
{
|
||||
"theme": "hello-elementor",
|
||||
"plugins": [{ "slug": "elementor", "active": true }],
|
||||
"wordpress_version": "6.5.5",
|
||||
"php_version": "8.1.30",
|
||||
"builder": "elementor",
|
||||
"capabilities": { "content": true, "design": true, "install": true },
|
||||
"preflight": { "https": true, "rest": true, "disallow_file_mods": false }
|
||||
}
|
||||
```
|
||||
|
||||
`builder` is computed on the plugin with the same rules as `detectBuilder`. Content and design playbooks must use this field. Editing `post_content` on an Elementor site is a failed test.
|
||||
|
||||
### Decision
|
||||
|
||||
- **Context:** Elementor stores the page in post meta. Gutenberg stores it in `post_content`.
|
||||
- **Chosen:** slugs + the meta keys those builders actually write. Priority: paid builders, then block markup, then classic.
|
||||
- **Rejected:** “if elementor is installed, always Elementor” (inactive plugin). Theme-name-only detection.
|
||||
- **Reverted later?**
|
||||
@@ -0,0 +1,49 @@
|
||||
# Spike: golden-task harness (R7)
|
||||
|
||||
**Status:** partial — harness exists; live Grok run not scored (`XAI_API_KEY` unset)
|
||||
|
||||
## Question
|
||||
|
||||
Can we score a model on WordPress tasks without vibes?
|
||||
|
||||
## Done when
|
||||
|
||||
- 20 fixture prompts against at least 2 canned WordPress sites
|
||||
- Each prompt has a hard assertion (preview text, option value, or screenshot)
|
||||
- One Grok run is scored
|
||||
- Harness lives under `e2e/golden/`
|
||||
|
||||
## Result
|
||||
|
||||
Yes, if “score” means: apply a tool call to a fixture and assert the new heading/option. No, if it means we have a Grok quality number. This machine has no `XAI_API_KEY`, so the live call was skipped.
|
||||
|
||||
### What exists
|
||||
|
||||
| Piece | Path |
|
||||
|---|---|
|
||||
| 20 prompts | `e2e/golden/prompts.json` |
|
||||
| Gutenberg dental site | `e2e/golden/sites/gutenberg-business/site.json` |
|
||||
| Elementor restaurant site | `e2e/golden/sites/elementor-restaurant/site.json` |
|
||||
| Apply + assert + Grok parser | `e2e/golden/src/` |
|
||||
| Scoreboard | `e2e/golden/runs/latest.json` |
|
||||
|
||||
Two sites. Ten prompts each. Assertions are `preview_text`, `option`, or `screenshot`. Screenshot here means “the fixture page text must contain X” — not a PNG/SSIM check.
|
||||
|
||||
`pnpm --filter @wursor/e2e golden` scored **20/20** fixture tool traces. Live Grok: skipped.
|
||||
|
||||
`pnpm test:e2e` — 21 tests, including the scorer.
|
||||
|
||||
### How to score a real Grok run
|
||||
|
||||
```bash
|
||||
XAI_API_KEY=… pnpm --filter @wursor/e2e golden
|
||||
```
|
||||
|
||||
That sends `gb-01` through `api.x.ai` and asserts the homepage heading.
|
||||
|
||||
### Decision
|
||||
|
||||
- **Context:** R7 said stop grading models by vibes.
|
||||
- **Chosen:** slot-fill tools + fixture apply + hard assert. Sites are JSON, not Docker WP (Docker was not available).
|
||||
- **Rejected:** “the model said it worked.” Waiting on Docker before any harness.
|
||||
- **Reverted later?**
|
||||
@@ -0,0 +1,47 @@
|
||||
# Spike: large-site mirror timing (R4)
|
||||
|
||||
**Status:** done — local synthetic 2GB; not a live WP pull
|
||||
|
||||
## Question
|
||||
|
||||
Does the 5-minute MVP exit survive a real site?
|
||||
|
||||
## Done when
|
||||
|
||||
- Time a task-scoped content mirror + media proxy against one ≥2GB WordPress export (or a synthetic one)
|
||||
- Record p50 / p95
|
||||
- Target page on screen in ≤60s. If not, the Layer 3 slice is wrong — change it before building chat.
|
||||
|
||||
## Result
|
||||
|
||||
The **slice holds on this machine** for in-process subset + proxy. A full library copy is the slow path; we do not take it.
|
||||
|
||||
Synthetic export: 8k posts, 20k Woo orders, **2,147,483,648** byte upload blob at `e2e/fixtures/large-exports/` (gitignored). 20 subset+proxy runs, then a `dd` of the blob as the naive-copy baseline.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| p50 time to target page | **0.003 ms** |
|
||||
| p95 time to target page | **0.010 ms** |
|
||||
| Upload bytes copied (slice) | **0** |
|
||||
| Naive local `dd` of 2GB | **2449 ms** (~2.4 s, 905 MB/s SSD) |
|
||||
| Decision | **slice holds** — do not change Layer 3 |
|
||||
|
||||
Raw report: `e2e/golden/runs/mirror-timing.json`.
|
||||
|
||||
`cp` on APFS cloned the file in ~2s and was discarded as a baseline. `dd if=… of=…` is the number above.
|
||||
|
||||
### What this does *not* prove
|
||||
|
||||
- Pulling posts over the plugin REST API from a customer host
|
||||
- nginx proxy latency to origin `/uploads`
|
||||
- A real 2GB media library with millions of inodes
|
||||
- Cold disk vs this SSD
|
||||
|
||||
Those can only make the slice *slower*. They do not argue for copying the library. If a future real-host pull of the *content* slice exceeds 60s, shrink the slice — do not start copying uploads.
|
||||
|
||||
### Decision
|
||||
|
||||
- **Context:** warm pool hides boot, not copy. Media is often the bulk of a 2GB site.
|
||||
- **Chosen:** task-scoped tables + origin proxy. Copy a file only on replace.
|
||||
- **Rejected:** full library sync. APFS `cp` as the naive baseline.
|
||||
- **Reverted later?**
|
||||
@@ -0,0 +1,141 @@
|
||||
# Spike: pairing threat model (R9)
|
||||
|
||||
**Status:** done
|
||||
|
||||
## Question
|
||||
|
||||
What stops a leaked URL from owning the site?
|
||||
|
||||
## Done when
|
||||
|
||||
Written threat model that becomes the Sprint 2 / Layer 2 auth tests:
|
||||
|
||||
- 8+ character pairing code
|
||||
- 5-minute TTL
|
||||
- 5-attempt lockout
|
||||
- HMAC request signing
|
||||
- hashed + scoped tokens (read vs deploy)
|
||||
|
||||
## Result
|
||||
|
||||
The plugin is a privileged backdoor: files, DB, WP-CLI. A leaked URL, a guessed pairing code, or a stolen bearer token is site ownership. Isolation of the *sandbox* does not help — this boundary is the *live* site.
|
||||
|
||||
### Locked flow
|
||||
|
||||
Wursor generates the pairing code (bound to the signed-in account). The user pastes it into the plugin. The plugin redeems it with the site URL. Tokens are issued once.
|
||||
|
||||
This matches PRD §7.1.4. The `Wursor_Auth::generate_pairing_code()` sketch in IMPLEMENTATION.md is the wrong direction — plugin-local generate/redeem cannot bind the code to an account before the site is known. Sprint 2 tests follow this note, not that sketch.
|
||||
|
||||
```
|
||||
User (signed in) → POST /sites/pair → Wursor stores pending pairing
|
||||
User pastes code in plugin admin
|
||||
Plugin → POST https://api.wursor…/sites/redeem { code, site_url }
|
||||
Wursor binds site_url, returns read_token + deploy_token + hmac_secret (once)
|
||||
Plugin stores hashes + encrypted hmac_secret
|
||||
Wursor stores tokens encrypted (it must send them later)
|
||||
```
|
||||
|
||||
Wursor is the HTTPS client. The plugin is the server. Tokens never appear in query strings or logs.
|
||||
|
||||
### Protocol
|
||||
|
||||
**Pairing code**
|
||||
|
||||
- Alphabet: `[A-Z0-9]`, length ≥ 8. Generate 8. `36^8 ≈ 2.8e12`.
|
||||
- Bound to `account_id` at creation. Not reusable after success.
|
||||
- `expires_at = created_at + 300s`. Clock for tests is injectable (`advance_clock`).
|
||||
- After 5 failed redeems on that code, `locked = true`. Further redeems fail even if the code is correct.
|
||||
- Redeem also fails if `site_url` is not `https` or does not parse as a URL.
|
||||
- One successful redeem. Second redeem of the same code fails.
|
||||
|
||||
**Tokens**
|
||||
|
||||
| Token | Scope | Plugin endpoints |
|
||||
|---|---|---|
|
||||
| `read` | site-info, file read, DB read, preflight | GET only |
|
||||
| `deploy` | file write, DB write, WP-CLI, prepare/commit, rollback | mutating |
|
||||
|
||||
- 256-bit random, encoded unpadded base64url, shown once.
|
||||
- Plugin stores `SHA-256(token)` only. Compare with `hash_equals`.
|
||||
- Wursor stores ciphertext (envelope key, not plaintext in Postgres).
|
||||
- A `read` token on a deploy route returns 403. A `deploy` token may call read routes.
|
||||
- Rotation: Wursor issues a new pair; plugin replaces hashes; old hashes stop working.
|
||||
- Disconnect: both hashes deleted; Wursor ciphertext deleted.
|
||||
|
||||
**HMAC (every plugin request)**
|
||||
|
||||
```
|
||||
canonical = timestamp + "\n" + METHOD + "\n" + path + "\n" + hex(sha256(body))
|
||||
X-Wursor-Timestamp: unix seconds
|
||||
X-Wursor-Signature: hex(HMAC-SHA256(hmac_secret, canonical))
|
||||
Authorization: Bearer <read_token|deploy_token>
|
||||
```
|
||||
|
||||
- Reject if `|now - timestamp| > 60`.
|
||||
- Reject if signature missing or `hash_equals` fails.
|
||||
- `hmac_secret` is 256-bit, issued at redeem, stored on the plugin encrypted with the site salt (`AUTH_KEY` + `AUTH_SALT`). Not the same bytes as either token.
|
||||
- Body hash is over the raw bytes. Empty body is SHA-256 of `""`.
|
||||
|
||||
**Transport**
|
||||
|
||||
- Plugin REST namespace: `/wp-json/wursor/v1/`.
|
||||
- Plugin refuses non-HTTPS callbacks except `WP_ENVIRONMENT_TYPE === 'local'`.
|
||||
- Wursor never puts tokens in URLs, logs, or SSE payloads.
|
||||
|
||||
### Threats
|
||||
|
||||
| ID | Threat | Mitigation | Residual |
|
||||
|---|---|---|---|
|
||||
| T1 | Attacker guesses pairing codes | 8+ charset, 5-try lockout, 5-min TTL | Online brute force is ~5 guesses / 5 min / code |
|
||||
| T2 | Pairing code leaked (screenshot, chat) | TTL + single use + requires wp-admin to paste | Anyone with the code and wp-admin wins until expiry |
|
||||
| T3 | Attacker redeems victim's code onto attacker site | After redeem, Wursor shows the bound `site_url` and requires an explicit “this is my site” confirm before the site is usable | User who confirms a foreign URL is connected to it |
|
||||
| T4 | Bearer token in a URL / access log / Referer | Tokens only in `Authorization`. Tests fail if any helper puts them in a query | Operator error in a future client |
|
||||
| T5 | Stolen request replayed | HMAC over timestamp+method+path+body; 60s skew window | Replay inside the window if the request was captured |
|
||||
| T6 | Stolen `read` token used to deploy | Scoped tokens; deploy routes require `deploy` | Read token still exfiltrates site-info |
|
||||
| T7 | Plugin DB dump / filesystem copy | Plugin stores hashes + encrypted hmac_secret, not raw tokens | Wursor-side ciphertext leak still lets us *call* the plugin until rotation |
|
||||
| T8 | MITM on HTTP | HTTPS required except local | Mis-set `WP_ENVIRONMENT_TYPE` on a public HTTP site |
|
||||
| T9 | CSRF in the browser against plugin REST | Bearer + HMAC. No cookie auth for `/wursor/v1/` | None if those headers stay required |
|
||||
| T10 | Timing leak on token compare | `hash_equals` only | — |
|
||||
|
||||
Out of scope for this spike (handled elsewhere): stolen wp-admin session, compromised host, malicious plugin already on the site.
|
||||
|
||||
### Sprint 2 tests (this note is the spec)
|
||||
|
||||
`plugin/__tests__/test-auth.php`
|
||||
|
||||
1. `test_pairing_code_is_at_least_eight_alnum` — `^[A-Z0-9]{8,}$`
|
||||
2. `test_pairing_code_expires_after_five_minutes` — `advance_clock(301)` → redeem false
|
||||
3. `test_pairing_code_valid_at_four_minutes_fifty_nine` — `advance_clock(299)` → redeem true
|
||||
4. `test_locks_out_after_five_failed_attempts` — five bad redeems → `is_locked_out()`
|
||||
5. `test_lockout_rejects_even_the_correct_code`
|
||||
6. `test_successful_redeem_cannot_be_replayed`
|
||||
7. `test_read_token_hash_is_stored_not_plaintext`
|
||||
8. `test_read_token_forbidden_on_deploy_route` → 403
|
||||
9. `test_deploy_token_allowed_on_site_info`
|
||||
10. `test_hmac_rejects_stale_timestamp` — timestamp older than 60s
|
||||
11. `test_hmac_rejects_tampered_body`
|
||||
12. `test_hmac_rejects_missing_signature`
|
||||
13. `test_verify_uses_hash_equals`
|
||||
14. `test_rotated_tokens_invalidate_old_hashes`
|
||||
|
||||
`api/__tests__/services/plugin-client.test.ts`
|
||||
|
||||
1. signs every request with timestamp + HMAC
|
||||
2. sends token in `Authorization`, never in the URL
|
||||
3. maps 401 to `Authentication failed`
|
||||
4. refuses to construct a client with an `http://` site URL outside local
|
||||
|
||||
`api/__tests__/routes/sites-pair.test.ts`
|
||||
|
||||
1. pair requires a session
|
||||
2. redeem binds `site_url` and returns tokens once
|
||||
3. second redeem of the same code fails
|
||||
4. site is not `connected` until the user confirms the shown URL (T3)
|
||||
|
||||
### Decision
|
||||
|
||||
- **Context:** plugin can own the live site; old 6-char sketch had no TTL, lockout, HMAC, or scopes.
|
||||
- **Options:** plugin-generated code (TV pairing) vs Wursor-generated code (PRD).
|
||||
- **Chosen:** Wursor-generated, pasted into the plugin, HMAC + scoped tokens as above.
|
||||
- **Rejected:** plugin-local generate/redeem (cannot bind to account first; IMPLEMENTATION sketch). Tokens in query strings. Single unscope token.
|
||||
- **Reverted later?**
|
||||
@@ -0,0 +1,140 @@
|
||||
# What this batch is for
|
||||
|
||||
Pairing and the plugin catalog (notes 1–2) answered “who is allowed to talk to the live site” and “what may ever be installed.” This batch answers the other three kill-shots before any product UI exists:
|
||||
|
||||
3. Can we tell if the model actually did the WordPress thing?
|
||||
4. Do we know which store holds the page (Gutenberg vs Elementor vs Classic)?
|
||||
5. Can a 2GB site still show a preview in time if we refuse to copy the media library?
|
||||
|
||||
If 3 is vibes, we will ship a confident liar. If 4 is wrong, we will edit `post_content` while Elementor renders JSON from post meta — the preview will not change and we will not know why. If 5 is a full copy, the five-minute product is dead on any real business site.
|
||||
|
||||
None of this is the chat app. It is the measuring stick the chat app has to pass.
|
||||
|
||||
## 3 — Golden-task harness
|
||||
|
||||
### What it does
|
||||
|
||||
It keeps twenty English requests, two fake but structured WordPress sites, and a scorer.
|
||||
|
||||
A prompt is not “make it modern.” It is “change the homepage heading to Welcome to My Business.” The assertion is not “the model sounded sure.” It is: after the tool call is applied to the fixture, that string is in the page (or that option equals that value).
|
||||
|
||||
Sites:
|
||||
|
||||
- `gutenberg-business` — Twenty Twenty-Four, block markup, a dental practice.
|
||||
- `elementor-restaurant` — Hello Elementor, `_elementor_data` JSON, a trattoria.
|
||||
|
||||
Ten prompts each. Mix of heading edits, text replace, `blogname` / `blogdescription` / `blog_public`, and two `screenshot` rows that today still assert on fixture text (see uncertainties).
|
||||
|
||||
### How it was implemented
|
||||
|
||||
TDD first. Tests imported modules that did not exist. Vitest failed. Then:
|
||||
|
||||
- `applyTool` mutates a `SiteFixture` (heading regex or Elementor `title`, string replace, option set).
|
||||
- `checkAssertion` reads the result.
|
||||
- `scoreGrokResponse` parses an xAI/OpenAI-shaped `tool_calls` payload, applies, asserts.
|
||||
- `run-golden.ts` scores the twenty expected traces and, if `XAI_API_KEY` is set, sends `gb-01` to `api.x.ai`.
|
||||
|
||||
Proof on this machine:
|
||||
|
||||
```
|
||||
pnpm test:e2e → 21 passed
|
||||
pnpm --filter @wursor/e2e golden
|
||||
→ 20/20 fixture traces
|
||||
→ live Grok skipped (no key)
|
||||
```
|
||||
|
||||
### Why this shape
|
||||
|
||||
WordPress work is slot filling (`page`, `old`, `new`) plus a deterministic write. If we score free-form chat, Grok can narrate a heading change that never happened. If we only score “did the API return 200,” we learn nothing about the page.
|
||||
|
||||
Applying the tool to a fixture is the smallest thing that can fail for the right reason. JSON sites instead of Docker WP because Docker was not available here; the assertion types stay valid when real sandboxes exist — swap `applyTool` for a REST/`wp post update` runner and keep `prompts.json`.
|
||||
|
||||
Expected traces are an answer key for the *harness*, not a grade for Grok. A live grade is one HTTP call away. That is deliberate: unit tests stay offline (TDD rule: no network in unit tests).
|
||||
|
||||
## 4 — Builder detect
|
||||
|
||||
### What it does
|
||||
|
||||
Given theme, plugin slugs, post content, and post meta, it returns exactly one of:
|
||||
|
||||
`elementor | beaver | divi | gutenberg | classic`
|
||||
|
||||
That value is what `site-info` will send. Playbooks are required to branch on it.
|
||||
|
||||
### How it was implemented
|
||||
|
||||
Same TDD file: six cases, one assertion each. Rules use **slug plus the meta key that builder actually writes**. Gutenberg is `<!-- wp:` in content. Classic is the leftover.
|
||||
|
||||
Elementor wins over block markup. A site can have leftover Gutenberg in `post_content` while Elementor is what the visitor sees. Editing the wrong store is the R6/R13 failure mode.
|
||||
|
||||
### Why this is the right approach
|
||||
|
||||
Theme name alone is a lie (Hello Elementor vs a child theme vs Divi). “Elementor is installed” is a lie (inactive). Reading only `post_content` is a lie on builders.
|
||||
|
||||
The combination is what WordPress itself uses: active plugin, then that plugin’s post meta. We copied that, in a function small enough to test without PHP. The plugin will run the same rules in PHP later; the TypeScript copy is the spec the PHP tests must match.
|
||||
|
||||
## 5 — Mirror timing
|
||||
|
||||
### What it does
|
||||
|
||||
It asks: if the site’s uploads are 2GB, can the *content-edit* path still put the target page in a sandbox in under 60 seconds?
|
||||
|
||||
The prototype:
|
||||
|
||||
1. Builds a synthetic export: 8k posts, 20k Woo orders, a **2,147,483,648** byte blob.
|
||||
2. Runs `exportDbSubset(content)` — keeps `wp_posts` / `wp_postmeta` / `wp_options`, drops orders and comments, redacts `*_key` / `*_secret` / `smtp_pass`.
|
||||
3. Resolves `/uploads/…` to `origin + path`. Copies **0** upload bytes.
|
||||
4. Times that 20 times.
|
||||
5. Times a real local `dd` of the 2GB blob as “what copying the library costs on this disk.”
|
||||
|
||||
### How it was implemented
|
||||
|
||||
Tests first (subset tables, redaction, proxy, replace-only copy). Then `run-mirror-timing.ts`. `mkfile` created the blob once under `e2e/fixtures/large-exports/` (gitignored). First naive baseline used `cp`; on APFS that is `clonefile` and finished in ~2s without writing bytes. That number was thrown out. `dd if=… of=…` wrote the bytes: **2449 ms**, ~905 MB/s.
|
||||
|
||||
Slice p50 **0.003 ms**, p95 **0.010 ms**, upload bytes copied **0**. Decision: **do not change Layer 3**.
|
||||
|
||||
### Why this is the right approach
|
||||
|
||||
Warm pool hides *boot*. It does not hide *copy*. A Woo store’s cost is `wp_posts`, plugin folders, and upload thumbs — not MySQL start time. Copying 2GB from a customer host over the plugin REST API will not finish in a minute. Proxying `/wp-content/uploads/*` to origin makes the preview honest without the copy. The 2.4s local `dd` is a *lower bound* on copy cost; a real host will be slower. The slice does not need that copy at all.
|
||||
|
||||
Redacting secrets in the same function is R10: a sandbox with outbound internet should not hold `smtp_pass`.
|
||||
|
||||
## Why this batch, in this order, is the best next step
|
||||
|
||||
The product is describe → preview → approve. Before UI:
|
||||
|
||||
- You need a test that can fail a bad model (3).
|
||||
- You need to know which bytes to change or the preview is fake (4).
|
||||
- You need to know the preview can appear before the user leaves (5).
|
||||
|
||||
Doing them as scripts + fixtures instead of `web/` + `api/` keeps the Phase 0 gate honest. We did not invent a chat panel that cannot be scored.
|
||||
|
||||
TDD on the spike code means the later plugin/API ports have a contract: same types, same assertions, same 60s budget.
|
||||
|
||||
## What I am not sure about
|
||||
|
||||
1. **Live Grok was not scored.** No `XAI_API_KEY` in this environment. The harness can call `api.x.ai`; it did not. I do not know Grok’s actual score on these twenty prompts. Do not treat 20/20 fixture traces as a model eval.
|
||||
|
||||
2. **Sites are JSON, not WordPress.** No Docker on this machine. Builder detect and heading replace are faithful to how WP stores data, but they are not PHP, not `$wpdb`, not a running theme. A real Elementor document is a deeper JSON tree than the two-widget fixture.
|
||||
|
||||
3. **“Screenshot” assertions are not screenshots.** They assert text in the fixture. Pixel/SSIM checks need a sandbox and Playwright. I used the type so the prompt file matches the done-when enum, not because we captured a PNG.
|
||||
|
||||
4. **Mirror p50/p95 are in-process.** They do not include plugin HTTP, PHP serialization, or nginx. They prove the *algorithm* is not the 60s problem. They do not prove a 2GB *site on SiteGround* will preview in 60s. I am sure we should still not copy uploads. I am not sure the first real content-slice *pull* will stay under 60s.
|
||||
|
||||
5. **Naive copy at 2.4s is this SSD.** A cheap VPS or a network pull will be worse. Do not quote 2.4s as “copy is fine.”
|
||||
|
||||
6. **Elementor `title` patcher.** `JSON.stringify`’s replacer changes the first `title` key it sees. That is correct for the fixture (heading widget first). A real tree may put a button title first. The PHP adapter must walk widgets by `widgetType === heading'`, not “first title.”
|
||||
|
||||
7. **Grok model id.** The client uses `grok-3`. If xAI has renamed the tool-calling model, the live runner will 404 until that string is updated.
|
||||
|
||||
8. **Divi/Beaver** are unit-tested with synthetic meta only. They are not in the two canned sites. Alpha still needs a real Elementor host; Beaver/Divi are unverified in the wild.
|
||||
|
||||
## Commands a reviewer can rerun
|
||||
|
||||
```bash
|
||||
pnpm test:e2e
|
||||
pnpm --filter @wursor/e2e golden
|
||||
pnpm --filter @wursor/e2e mirror:time # reuses the 2GB blob if present
|
||||
```
|
||||
|
||||
Gate status after this batch: catalog done, pairing done, builder done, mirror done on synthetic 2GB, golden harness built, **live Grok still open**. Product chat still blocked until you either accept that gap or set `XAI_API_KEY` and score `gb-01`.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Spike: P0 plugin catalog
|
||||
|
||||
**Status:** done
|
||||
|
||||
## Question
|
||||
|
||||
What may the agent ever install?
|
||||
|
||||
## Done when
|
||||
|
||||
- ~40 slugs written
|
||||
- `wp plugin install <url>` is forbidden even before an install playbook exists
|
||||
|
||||
## Result
|
||||
|
||||
The agent may install **only** from the catalog table below, and only from wordpress.org via slug (`wp plugin install <slug>`). That playbook does not exist in the MVP. The rule still ships in Sprint 3 tool schemas so a heading-change agent cannot install anything.
|
||||
|
||||
### Forbidden (now and in v1)
|
||||
|
||||
- `wp plugin install <url>`
|
||||
- `wp plugin install` with a local zip or any path
|
||||
- Premium / marketplace URLs (Elementor Pro zip, Woo market, GitHub)
|
||||
- Any slug not in the catalog
|
||||
- Re-install of a detect-only plugin (table below)
|
||||
|
||||
Fail closed. Unknown slug → refuse and say so in chat. Do not search wordpress.org ad hoc.
|
||||
|
||||
Reputation, zip SHA, and egress-watch on first activate are Sprint 9. This note is only the allowlist.
|
||||
|
||||
Paid *services* behind a free slug (WooPayments, Site Kit) still hit the no-surprise gate at deploy.
|
||||
|
||||
### Catalog (40)
|
||||
|
||||
wordpress.org slugs. Installable only when the site is `install-safe` (writable filesystem, `DISALLOW_FILE_MODS` off).
|
||||
|
||||
| # | Slug | Why |
|
||||
|---|---|---|
|
||||
| 1 | `contact-form-7` | Forms. Most common. |
|
||||
| 2 | `wpforms-lite` | Forms. Non-technical default. |
|
||||
| 3 | `fluentform` | Forms. |
|
||||
| 4 | `forminator` | Forms + polls/quizzes. |
|
||||
| 5 | `ninja-forms` | Forms. |
|
||||
| 6 | `wordpress-seo` | Yoast. SEO. |
|
||||
| 7 | `seo-by-rank-math` | SEO. |
|
||||
| 8 | `all-in-one-seo-pack` | SEO. |
|
||||
| 9 | `woocommerce` | Store. |
|
||||
| 10 | `woocommerce-payments` | Checkout. Paid service → no-surprise. |
|
||||
| 11 | `woocommerce-gateway-stripe` | Checkout. |
|
||||
| 12 | `woocommerce-paypal-payments` | Checkout. |
|
||||
| 13 | `elementor` | Builder. Install only if site has no other builder. |
|
||||
| 14 | `kadence-blocks` | Gutenberg blocks. |
|
||||
| 15 | `stackable-ultimate-gutenberg-blocks` | Gutenberg blocks. |
|
||||
| 16 | `ultimate-addons-for-gutenberg` | Spectra. Gutenberg blocks. |
|
||||
| 17 | `classic-editor` | Needed on Classic sites before content edits. |
|
||||
| 18 | `simply-schedule-appointments` | Booking. |
|
||||
| 19 | `booking` | Booking Calendar. |
|
||||
| 20 | `easy-appointments` | Booking. |
|
||||
| 21 | `google-site-kit` | Analytics. Google account → no-surprise. |
|
||||
| 22 | `independent-analytics` | Analytics without Google. |
|
||||
| 23 | `fluent-smtp` | Mail delivery. SMTP secrets stay redacted (R10). |
|
||||
| 24 | `wp-mail-smtp` | Mail delivery. |
|
||||
| 25 | `redirection` | Redirects. Slug changes still hit R12. |
|
||||
| 26 | `akismet` | Comment spam. |
|
||||
| 27 | `cookie-law-info` | CookieYes. Consent. |
|
||||
| 28 | `complianz-gdpr` | Consent. |
|
||||
| 29 | `updraftplus` | Backup. |
|
||||
| 30 | `backwpup` | Backup. |
|
||||
| 31 | `autoptimize` | CSS/JS minify. |
|
||||
| 32 | `wp-optimize` | Cleanup / cache. |
|
||||
| 33 | `polylang` | Languages. |
|
||||
| 34 | `translatepress-multilingual` | Languages. |
|
||||
| 35 | `tablepress` | Tables. |
|
||||
| 36 | `duplicate-post` | Duplicate a page. |
|
||||
| 37 | `disable-comments` | Turn comments off. |
|
||||
| 38 | `safe-svg` | SVG uploads. |
|
||||
| 39 | `modula-best-grid-gallery` | Gallery. |
|
||||
| 40 | `foogallery` | Gallery. |
|
||||
|
||||
40 slugs. Adding one is a catalog PR, not a prompt change.
|
||||
|
||||
### Detect-only — never install
|
||||
|
||||
These break REST, deploys, or hosts. Site-info reports them. The agent does not install them.
|
||||
|
||||
| Slug | Why not |
|
||||
|---|---|
|
||||
| `wordfence` | Kills REST / pairing. |
|
||||
| `better-wp-security` | iThemes. Same. |
|
||||
| `sucuri-scanner` | Host / firewall coupling. |
|
||||
| `litespeed-cache` | Host-specific. |
|
||||
| `w3-total-cache` | Easy to brick a site. |
|
||||
| `wp-super-cache` | Same. |
|
||||
| `jetpack` | Too large; host-coupled. Configure if already present, later. |
|
||||
|
||||
### Sprint 3 tests (this note is the spec)
|
||||
|
||||
`api/__tests__/agents/tool-schemas.test.ts`
|
||||
|
||||
1. `generateToolSchemas()` has no tool or enum value matching `wp plugin install http`
|
||||
2. `wp plugin install` (when it exists) accepts only slugs from this catalog
|
||||
3. A tool call with slug `evil-plugin` is rejected before any sandbox exec
|
||||
4. A tool call with a URL or `.zip` is rejected
|
||||
|
||||
Until the install playbook exists, the simpler form is enough: **no install tool at all**, plus assertion (1).
|
||||
|
||||
### Decision
|
||||
|
||||
- **Context:** wordpress.org is not a reviewed-safe catalog. Preview cannot see a plugin phoning home. MVP has no install playbook, but the agent still needs a hard wall.
|
||||
- **Options:** open wordpress.org search vs a written allowlist vs delay any wall until Sprint 9.
|
||||
- **Chosen:** 40-slug allowlist now; no URL/zip; detect-only list for security/cache suites; no install tool in MVP.
|
||||
- **Rejected:** open search (R2). Installing Wordfence/Jetpack/cache suites (they break the product or the site).
|
||||
- **Reverted later?**
|
||||
Reference in New Issue
Block a user