Dashboard/Crawler-Factory/Engineering-Specs/11-frontend-configurator.md
Crawler-Factory

Engineering-Specs/11-frontend-configurator.md

Crawler Factory: Spec 11: Frontend Configurator (CategoryConfigurator + sub-components) Implementation Plan

wzxhzdk:46 For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build the per-pathGroup configurator drawer (CategoryConfigurator) and its three sub-components (SamplePreview, ConfigEditor, TestCrawlResults) so a human operator can review samples, edit the recordExtractor, regenerate it with feedback, run sandbox + real test crawls, and commit a per-content-domain crawler: strictly human-in-the-loop, no silent commits.

Architecture: A right-side shadcn Sheet drawer at 70% viewport width, opened from CategoryReview (Spec 10) when the user clicks Configure Selected. On open, the drawer fires POST /api/factory/sample then POST /api/factory/generate-extractor to populate samples + initial extractor. Three vertically-stacked sub-sections render the artifacts (samples on top, editor in the middle, test results at bottom), and three CTAs at the footer drive the workflow: Run sandbox test → Run REAL test crawl → Commit + Create Crawler. The Commit CTA is enabled only when both sandbox and real test results are green for the active pathGroup. This component cluster is render-only: it does NOT manage which pathGroup is active (that's Spec 12 orchestrator). It accepts a pathGroup prop and emits onCommit upward.

Tech Stack: TypeScript (strict), React 18, shadcn/ui (Sheet, Form, Input, Textarea, Switch, Button, Card, Tabs, Collapsible), react-hook-form + zod resolver, lucide-react icons, framer-motion (existing in repo) for subtle reveals, vitest + @testing-library/react for tests.

Depends on: - Spec 01 (lib/factory/dss.ts, lib/factory/types.ts: PathGroup, ContentDomain, DssEntry) - Spec 09 (src/hooks/factory/useFactorySession.ts: pulls session by id; not used directly here, but the configurator commits a blueprint that the session will later persist via Spec 12 orchestrator) - Spec 10 (src/components/admin/factory/DomainBadge.tsx: reused inside the drawer header) - Backend endpoints from Spec 08: POST /api/factory/sample, POST /api/factory/generate-extractor, POST /api/factory/test-sandbox, POST /api/factory/test-real. These are called via fetch() inside CategoryConfigurator: no new hook layer is added; the calls are imperative actions tied to user gestures.

Consumed by: Spec 12 (CrawlerFactory orchestrator): wires CategoryConfigurator open/close state, passes the active pathGroup, and on onCommit calls POST /api/factory/create-crawler.

Plan source: Projects/Crawler-Factory/00-Plan.md §5b (drawer ASCII mockup), §7 Task 27 (CategoryConfigurator), §19e (User-in-the-loop policy: no silent commits, every CTA explicit click).

File structure

Path Responsibility
src/components/admin/factory/CategoryConfigurator.tsx Drawer shell. Owns: open lifecycle, sample fetch, extractor fetch, sandbox + real test invocation, commit handler. Composes the three sub-components.
src/components/admin/factory/SamplePreview.tsx Render-only. Lists sample URLs with collapsible "view HTML"; renders detected selector ✓ field rows; renders schema.org JSON-LD blob if present.
src/components/admin/factory/ConfigEditor.tsx Form (indexName, renderJavaScript, rateLimit, pathsToMatch) + monospace Textarea for recordExtractor JS + "Regenerate from samples" button + free-form feedback Input (Enter submits regen with feedback).
src/components/admin/factory/TestCrawlResults.tsx Two-tab view (Sandbox / Real). Each tab: per-sample row with ✓/✗, record count, field-coverage bars (one per DSS-required field), "show first record JSON" expand. Run-test buttons live here.
src/components/admin/factory/__tests__/CategoryConfigurator.test.tsx Drawer behavior: open/close, fetches on open, regenerate-with-feedback re-fetches, commit-button gating.
src/components/admin/factory/__tests__/SamplePreview.test.tsx Render-only: sample list, expand-collapse, JSON-LD presence/absence.
src/components/admin/factory/__tests__/ConfigEditor.test.tsx Form changes call onChangeConfig; textarea changes call onChangeExtractor; Enter on feedback input fires onUserFeedback.
src/components/admin/factory/__tests__/TestCrawlResults.test.tsx Field-coverage percentages render correctly; tab switching; first-record JSON expand.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. All component props are explicit TypeScript interfaces. Zod is not used inside the component layer (validation lives at the API boundary, Spec 08), but the prop types are derived from Spec 01 zod-inferred types (PathGroup, ContentDomain). No any. Use T | null, not T | undefined, for "absent" semantic.
  2. Console.error/log only inside catch. No stray console.log in render paths. Errors caught from fetch() are surfaced to the user via toast (useToast from src/components/ui/use-toast) AND logged to console.
  3. Try/catch every fetch. Every fetch() call wrapped. On non-2xx, parse response.json() to surface { error } if present, fall back to response.statusText. Always preserve loading state symmetry (finally clears loading flag).
  4. Functions ≤20 lines, ≤3 params. Render helpers extracted as named local functions when a JSX block exceeds ~30 lines.
  5. Docstring on every exported component. JSDoc comment explaining purpose + key prop semantics. Comment WHY non-obvious choices exist.
  6. No raw dicts crossing module boundaries. Props use the PathGroup, Sample, StructureAnalysis, SandboxResult, RealResult, CrawlerConfig types: defined either in Spec 01 (lib/factory/types.ts) or, for the strictly UI-only result shapes, in a small local types.ts co-located in src/components/admin/factory/.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit. Use @testing-library/react's render, screen, fireEvent, waitFor. Mock fetch globally with vi.fn() per test.
  8. Mock external services in unit tests. global.fetch is mocked in every test for CategoryConfigurator. SamplePreview, ConfigEditor, TestCrawlResults are rendered with controlled props: no fetch.
  9. Naming. Components: PascalCase.tsx. Tests: same name with .test.tsx. Local hooks/helpers: camelCase. Types: PascalCase. Tailwind utility classes from existing /admin aesthetic.
  10. Conventional commits. feat(factory):, test(factory):. One sub-component per commit (per Task 27 directive: "Commit each sub-component separately").

Task 0: Define shared UI-only types

Files: - Create: src/components/admin/factory/types.ts

These types describe shapes that flow between the configurator and its sub-components but are not zod-validated themselves (they are derived from Spec 01 zod schemas at the API boundary upstream).

  • [ ] Step 0.1: Create src/components/admin/factory/types.ts

&

  • [ ] Step 0.2: Run tsc --noEmit to confirm clean

>

Expected: clean.

  • [ ] Step 0.3: Commit

>

Task 1: SamplePreview: render-only sample list + detected structure + JSON-LD

Files: - Create: src/components/admin/factory/SamplePreview.tsx - Create: src/components/admin/factory/__tests__/SamplePreview.test.tsx

  • [ ] Step 1.1: Write failing test: empty samples

Create src/components/admin/factory/__tests__/SamplePreview.test.tsx:

<

  • [ ] Step 1.2: Run test, expect FAIL

wzxhzdk:4

Expected: FAIL: Cannot find module '../SamplePreview'.

  • [ ] Step 1.3: Implement SamplePreview.tsx

Create src/components/admin/factory/SamplePreview.tsx:

wzxhzdk:5

  • [ ] Step 1.4: Run, expect PASS for empty-state test

wzxhzdk:6

Expected: 1 test passes.

  • [ ] Step 1.5: Add tests for sample list, expand HTML, structure rows, JSON-LD

Append to src/components/admin/factory/__tests__/SamplePreview.test.tsx:

wzxhzdk:7

  • [ ] Step 1.6: Run all SamplePreview tests, expect PASS

wzxhzdk:8

Expected: 6 tests pass.

  • [ ] Step 1.7: Run tsc --noEmit

wzxhzdk:9

Expected: clean.

  • [ ] Step 1.8: Commit

wzxhzdk:10

Task 2: ConfigEditor: form + extractor textarea + regen with feedback

Files: - Create: src/components/admin/factory/ConfigEditor.tsx - Create: src/components/admin/factory/__tests__/ConfigEditor.test.tsx

  • [ ] Step 2.1: Write failing test: form fields render with default values

Create src/components/admin/factory/__tests__/ConfigEditor.test.tsx:

wzxhzdk:11

  • [ ] Step 2.2: Run, expect FAIL

wzxhzdk:12

Expected: FAIL: Cannot find module '../ConfigEditor'.

  • [ ] Step 2.3: Implement ConfigEditor.tsx

Create src/components/admin/factory/ConfigEditor.tsx:

wzxhzdk:13

  • [ ] Step 2.4: Run form-render tests, expect PASS

wzxhzdk:14

Expected: 3 tests pass.

  • [ ] Step 2.5: Add tests for change handlers, regenerate, feedback Enter

Append to src/components/admin/factory/__tests__/ConfigEditor.test.tsx:

wzxhzdk:15

  • [ ] Step 2.6: Run all ConfigEditor tests, expect PASS

wzxhzdk:16

Expected: 10 tests pass.

  • [ ] Step 2.7: Run tsc --noEmit

wzxhzdk:17

Expected: clean.

  • [ ] Step 2.8: Commit

wzxhzdk:18

Task 3: TestCrawlResults: Sandbox/Real tabs with field-coverage bars

Files: - Create: src/components/admin/factory/TestCrawlResults.tsx - Create: src/components/admin/factory/__tests__/TestCrawlResults.test.tsx

  • [ ] Step 3.1: Write failing test: empty state, both tabs

Create src/components/admin/factory/__tests__/TestCrawlResults.test.tsx:

wzxhzdk:19

  • [ ] Step 3.2: Run, expect FAIL

wzxhzdk:20

Expected: FAIL: module not found.

  • [ ] Step 3.3: Implement TestCrawlResults.tsx

Create src/components/admin/factory/TestCrawlResults.tsx:

wzxhzdk:21

  • [ ] Step 3.4: Run empty-state tests, expect PASS

wzxhzdk:22

Expected: 3 tests pass.

  • [ ] Step 3.5: Add tests for results rendering, coverage bars, JSON expand

Append to src/components/admin/factory/__tests__/TestCrawlResults.test.tsx:

wzxhzdk:23

  • [ ] Step 3.6: Run all TestCrawlResults tests, expect PASS

wzxhzdk:24

Expected: 12 tests pass.

  • [ ] Step 3.7: Run tsc --noEmit

wzxhzdk:25

Expected: clean.

  • [ ] Step 3.8: Commit

wzxhzdk:26

Task 4: CategoryConfigurator: drawer shell, fetches, commit gating

Files: - Create: src/components/admin/factory/CategoryConfigurator.tsx - Create: src/components/admin/factory/__tests__/CategoryConfigurator.test.tsx

The drawer is the only component in this cluster that talks to the network. It owns: - open lifecycle (parent-controlled via open prop) - Initial sample fetch (POST /api/factory/sample) - Initial extractor generation (POST /api/factory/generate-extractor): runs immediately after samples land - Regeneration (with optional user feedback): re-fires POST /api/factory/generate-extractor - Sandbox run (POST /api/factory/test-sandbox) - Real run (POST /api/factory/test-real) - Commit handler: calls onCommit({ recordExtractor, crawlerConfig }) upward; gated until both sandbox + real are green for this pathGroup

Per §19e the Commit CTA is never silently auto-fired: it is enabled when conditions are met but always requires explicit user click.

  • [ ] Step 4.1: Write failing test: drawer renders title and closed-state

Create src/components/admin/factory/__tests__/CategoryConfigurator.test.tsx:

wzxhzdk:27

  • [ ] Step 4.2: Run, expect FAIL

wzxhzdk:28

Expected: FAIL: module not found.

  • [ ] Step 4.3: Implement CategoryConfigurator.tsx

Create src/components/admin/factory/CategoryConfigurator.tsx:

wzxhzdk:29

  • [ ] Step 4.4: Run the closed-state test, expect PASS

wzxhzdk:30

Expected: 1 test passes.

  • [ ] Step 4.5: Add test: drawer fires sample + extractor fetches on open

Append to src/components/admin/factory/__tests__/CategoryConfigurator.test.tsx:

wzxhzdk:31

  • [ ] Step 4.6: Run, expect PASS

wzxhzdk:32

Expected: 3 tests pass.

  • [ ] Step 4.7: Add test: regenerate-with-feedback re-fetches extractor with feedback in body

Append:

wzxhzdk:33

  • [ ] Step 4.8: Run, expect PASS

wzxhzdk:34

Expected: 4 tests pass.

  • [ ] Step 4.9: Add tests: Commit button gating (disabled until sandbox + real green)

Append:

wzxhzdk:35

  • [ ] Step 4.10: Run all CategoryConfigurator tests, expect PASS

wzxhzdk:36

Expected: 8 tests pass.

  • [ ] Step 4.11: Run tsc --noEmit

wzxhzdk:37

Expected: clean.

  • [ ] Step 4.12: Add test: onClose called when drawer overlay/X dismissed

Append:

wzxhzdk:38

  • [ ] Step 4.13: Run, expect PASS

wzxhzdk:39

Expected: 9 tests pass.

  • [ ] Step 4.14: Commit

wzxhzdk:40

Task 5: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 5.1: Run all factory frontend tests

wzxhzdk:41

Expected: 27 tests pass total (5 SamplePreview after merging, 10 ConfigEditor, 12 TestCrawlResults, 9 CategoryConfigurator): actual numbers from this spec: 6 + 10 + 12 + 9 = 37. Adjust counts if you split tests differently.

  • [ ] Step 5.2: Run full project test suite

wzxhzdk:42

Expected: full baseline still GREEN; new factory frontend tests pass.

  • [ ] Step 5.3: Run full tsc

wzxhzdk:43

Expected: clean.

  • [ ] Step 5.4: Smoke test in browser (manual, optional)

wzxhzdk:44

Then navigate to http://localhost:5173/admin/factory, exercise a fake session if available. (Drawer wiring lives in Spec 12; a standalone smoke is fine here using devtools to render CategoryConfigurator against mocked fetches.)

  • [ ] Step 5.5: Mark Spec 11 done in Status.md

In the vault Projects/Crawler-Factory/Status.md:

wzxhzdk:45

Acceptance criteria: Spec 11 done means:

  1. ✅ src/components/admin/factory/types.ts exports Sample, StructureAnalysis, SandboxResult, RealResult, CrawlerConfig and re-exports ContentDomain + PathGroup from lib/factory/types.
  2. ✅ src/components/admin/factory/SamplePreview.tsx renders sample list (collapsible HTML view), detected-structure rows (selector ✓ field), and JSON-LD (only when present). Mono font for selectors and JSON.
  3. ✅ src/components/admin/factory/ConfigEditor.tsx renders editable form (indexName default algoliacentral_&lt;contentDomain&gt;, renderJavaScript Switch, rateLimit numeric default 8, pathsToMatch comma-separated), monospace 24-row Textarea for recordExtractor, "Regenerate from samples" Button, free-form feedback Input that submits on Enter.
  4. ✅ src/components/admin/factory/TestCrawlResults.tsx renders Sandbox + Real tabs, per-sample rows with ✓/✗, record count, field-coverage bars (one per DSS searchable attribute, percentage 0–100% clamped), expandable first-record JSON.
  5. ✅ src/components/admin/factory/CategoryConfigurator.tsx renders shadcn Sheet side='right' at 70% width, title {contentDomain} | {pathGroup.pattern}. On open: POST /api/factory/sample then POST /api/factory/generate-extractor. Footer holds three CTAs: Run sandbox / Run REAL / Commit. Commit enabled only when both sandbox + real results are all-green for this pathGroup.
  6. ✅ Per §19e (no silent commits): Commit button is gated AND requires explicit user click, never auto-fires.
  7. ✅ All four components have unit tests with @testing-library/react. Coverage:
    • drawer renders title and opens/closes
    • sample fetch fires on open
    • extractor fetch fires after samples
    • regenerate-with-feedback re-fetches extractor with userFeedback in body
    • field-coverage bars render correct percentage widths
    • commit button gating proven across all four states (initial, sandbox-only, real-only, both)
    • explicit-click requirement proven (does not auto-commit)
  8. ✅ tsc --noEmit clean.
  9. ✅ Per CodingSOPs: every component has JSDoc, every fetch has try/catch + toast on failure, no any, props are typed interfaces.
  10. ✅ Each sub-component committed separately (per Task 27 directive).

Out of scope (handled by other specs)

  • Backend endpoints (/api/factory/sample, /generate-extractor, /test-sandbox, /test-real, /create-crawler) → Spec 08
  • Wiring CategoryConfigurator open/close from CategoryReview → Spec 12 (CrawlerFactory orchestrator)
  • The actual call to POST /api/factory/create-crawler on commit → Spec 12 (orchestrator owns it; this spec only emits onCommit upward)
  • LiveCrawlMonitor block that follows commit → Spec 12
  • DSS / zod types: already shipped in Spec 01
  • DomainBadge reuse: already shipped in Spec 10 (consumers may import it, but the configurator does not require it visually since the title carries the contentDomain in mono text)
  • Hooks useFactorySession, useDiscoveryStream, useCrawlProgress: Spec 09; this spec uses raw fetch because the calls are imperative actions tied to user gestures, not declarative session reads

wzxhzdk:48 Spec 08 follow-up flag (B11): The POST /api/factory/generate-extractor endpoint must return { recordExtractor, crawlerConfig, analysis: StructureAnalysis }: the analysis field (Spec 05's StructureAnalysis shape verbatim) is required so SamplePreview can render selectors / missing / confidence without a second round-trip. Spec 08's current contract returns only { recordExtractor, crawlerConfig }. Tracked as a Spec 08 follow-up; Spec 11 consumes it as soon as it lands.