Dashboard/Crawler-Factory/Engineering-Specs/12-frontend-orchestrator.md
Crawler-Factory

Engineering-Specs/12-frontend-orchestrator.md

Crawler Factory: Spec 12: Frontend Live Monitor + Orchestrator Implementation Plan

wzxhzdk:40 For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the live-crawl progress panel and the top-level FSM-driving orchestrator that wires every prior frontend cluster (Spec 09 hooks, Spec 10 discovery + category review + RollingLog, Spec 11 category configurator drawer) into a single resumable URL-driven flow at /admin/factory.

Architecture: LiveCrawlMonitor is a pure presentational component: one stacked block per active crawler, each block driven by useCrawlProgress(crawlerId) from Spec 09 (SSE polling against /api/factory/crawl-progress). CrawlerFactory is the FSM orchestrator: it reads ?session=<id> from the URL, loads session state via useFactorySession, switches the left column to the component matching session.status (null|discovering|categorized|configuring|crawling|done|error), and always renders the persistent RollingLog on the right column from the session's stored log entries. URL-as-state means the user can refresh, share, or come back tomorrow and resume exactly where they left off. The page shell AdminFactory.tsx (placeholder from Spec 09) is replaced with <CrawlerFactory />.

Tech Stack: React 18 + TypeScript (strict), react-router-dom v6 (useSearchParams for URL state), shadcn UI (Progress, Sheet, Card), framer-motion (existing /admin aesthetic: fade-up on mount), TanStack Query (already wired by useFactorySession), @testing-library/react + vitest for tests, MSW-free testing (mock the hooks directly per Spec 09 pattern).

Depends on: - Spec 09: hooks useFactorySession, useDiscoveryStream, useCrawlProgress; placeholder AdminFactory.tsx page shell - Spec 10: RollingLog, DomainBadge, DiscoveryPanel, CategoryReview - Spec 11: CategoryConfigurator drawer - Spec 08: /api/factory/session (CRUD) and /api/factory/crawl-progress (SSE) endpoints - Spec 01: FactorySession, PathGroup types from lib/factory/types.ts

Consumed by: - Spec 13: bundle + smoke test cluster (E2E flow validation, vault sync, docs)

Plan source: Projects/Crawler-Factory/00-Plan.md §2 (FSM), §5b (page layout), §7 Tasks 28, 29.

File structure

Path Responsibility
src/components/admin/factory/LiveCrawlMonitor.tsx Stacked progress blocks: one per active crawler. Pure presentation; subscribes via useCrawlProgress(crawlerId) per block.
src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx RTL tests: renders one block per crawler; renders header + progress + counters per block; handles empty list.
src/components/admin/factory/CrawlerFactory.tsx Top-level FSM orchestrator. Reads ?session=<id> from URL, loads session, switches left-column component on session.status, always renders RollingLog on right column.
src/components/admin/factory/__tests__/CrawlerFactory.test.tsx RTL tests: renders correct component per status; URL ?session=<id> resumes correctly; null → discovering → categorized → configuring → crawling → done flow.
src/pages/AdminFactory.tsx Modify: replace empty container from Spec 09 with <CrawlerFactory />. Header + framer-motion shell stays.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, UIUXDesignSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (already done by hooks reading API); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, for absent/sentinel values.
  2. No console.log. UI-side logging goes through the shared lib/utils/logger import only when a real signal is needed (errors caught at the boundary). Component bodies do not log routine state.
  3. Try/catch every fallible call. API errors surfaced from hooks are caught and routed to the error FSM branch: never silently swallowed.
  4. Functions ≤20 lines, ≤3 params. Sub-render helpers split out when a JSX block exceeds 20 lines or props exceed 3.
  5. Docstring on every exported component. One-line JSDoc explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. Component props are typed against FactorySession, PathGroup from lib/factory/types.ts.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock hooks in unit tests. vi.mock('@/hooks/factory/useFactorySession') etc.: no real network, no real timers.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for components and types, UPPER_SNAKE_CASE for module constants. Files: PascalCase.tsx (components), kebab-case.ts (utils). Tests: same name as subject + .test.tsx.
  10. Aesthetic conformance. Match /admin aesthetic exactly (Plan §5a): bg-[#0a0a0f], border-white/10, font-black uppercase tracking-tighter headers, mono font for IDs/URLs/timestamps, framer-motion fade-up on section mount.
  11. Accessibility. Every interactive element keyboard-reachable; ARIA labels on icon-only buttons; progress bars have aria-valuenow/aria-valuemin/aria-valuemax.
  12. URL is the source of truth. Session state is ?session=<id> query param. No client-only state for the session id. Resume on refresh works because the URL drives the session load.
  13. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.

Task 1: LiveCrawlMonitor: failing test for empty crawler list

Files: - Create: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

LiveCrawlMonitor receives crawlerIds: Array<{crawlerId, name, contentDomain, indexName}>. With an empty array it renders an empty-state message. With N entries it renders N stacked blocks. Each block reads its own progress via useCrawlProgress(crawlerId) (Spec 09) and shows: header (name + crawlerId mono + DomainBadge + indexName), shadcn Progress bar (0–100%), URLs/sec ticker, success/failed/ignored counters.

  • [ ] Step 1.1: Create the test file with empty-state test

Create src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx:

&

  • [ ] Step 1.2: Run, expect FAIL

wzxhzdk:1

Expected: FAIL: Cannot find module '../LiveCrawlMonitor' (component not yet implemented).

  • [ ] Step 1.3: Implement LiveCrawlMonitor minimally to pass the empty-state test

Create src/components/admin/factory/LiveCrawlMonitor.tsx:

wzxhzdk:2

  • [ ] Step 1.4: Run empty-state test, expect PASS

wzxhzdk:3

Expected: 1 test passes.

  • [ ] Step 1.5: Commit

wzxhzdk:4

Task 2: LiveCrawlMonitor: render one block per crawler

Files: - Modify: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

  • [ ] Step 2.1: Append failing test for "renders one block per crawler"

Append to src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx (inside the same describe):

wzxhzdk:5

  • [ ] Step 2.2: Run, expect PASS

The implementation from Task 1 already covers these: running them confirms.

wzxhzdk:6

Expected: 4 tests pass.

  • [ ] Step 2.3: Run tsc --noEmit

wzxhzdk:7

Expected: clean.

  • [ ] Step 2.4: Commit

wzxhzdk:8

Task 3: CrawlerFactory orchestrator: scaffold + null-session test

Files: - Create: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx - Create: src/components/admin/factory/CrawlerFactory.tsx

CrawlerFactory is the FSM driver. It reads ?session=<id> from the URL via useSearchParams. It loads session state via useFactorySession(sessionId). It picks the left-column component based on session.status:

status left column notes
null (no session id in URL) <DiscoveryPanel /> On done event from useDiscoveryStream, set ?session=<id> and continue.
'discovering' <DiscoveryPanel /> with active stream
'categorized' <CategoryReview /> On "Configure Selected", queue pathGroups + open drawer one at a time.
'configuring' <CategoryReview /> + <CategoryConfigurator /> overlay On commit, advance to next pathGroup; when queue empty, transition to 'crawling'.
'crawling' <LiveCrawlMonitor />
'done' <LiveCrawlMonitor /> + "Start new session" button
'error' error panel + retry/restart options

The right column ALWAYS renders <RollingLog /> driven by session log entries. Layout: 2-column grid (left 60%, right 40%).

  • [ ] Step 3.1: Write failing test for "no session id → renders DiscoveryPanel"

Create src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:

wzxhzdk:9

  • [ ] Step 3.2: Run, expect FAIL

wzxhzdk:10

Expected: FAIL: Cannot find module '../CrawlerFactory'.

  • [ ] Step 3.3: Implement CrawlerFactory minimally: null-session branch only

Create src/components/admin/factory/CrawlerFactory.tsx:

wzxhzdk:11

  • [ ] Step 3.4: Run null-session test, expect PASS

wzxhzdk:12

Expected: 1 test passes.

  • [ ] Step 3.5: Run tsc --noEmit

wzxhzdk:13

Expected: clean.

  • [ ] Step 3.6: Commit

wzxhzdk:14

Task 4: CrawlerFactory: discovering / categorized / configuring branches

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 4.1: Append failing tests for discovering, categorized, configuring

Append to src/components/admin/factory/__tests__/CrawlerFactory.test.tsx (inside the same describe):

wzxhzdk:15

Add at the top of the file (under the imports, above describe):

wzxhzdk:16

  • [ ] Step 4.2: Install @testing-library/user-event if not already present

wzxhzdk:17

Expected: prints a version (already installed) OR installs and prints 14.5.x.

  • [ ] Step 4.3: Run, expect PASS for the new tests

wzxhzdk:18

Expected: 4 tests pass total (null-session + discovering + categorized + configuring queue).

  • [ ] Step 4.4: Run tsc --noEmit

wzxhzdk:19

Expected: clean.

  • [ ] Step 4.5: Commit

wzxhzdk:20

Task 5: CrawlerFactory: crawling / done / error branches

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 5.1: Append failing tests for crawling, done, error, loading

Append to the same describe in src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:

wzxhzdk:21

  • [ ] Step 5.2: Remove vi.mock('../LiveCrawlMonitor') from this test file

The "crawling" test asserts on the real LiveCrawlMonitor (so data-testid="live-crawl-monitor" is the real component's id). The current mock list in this file does NOT mock LiveCrawlMonitor: confirm by re-reading the imports section. If you accidentally added one, remove it. The real LiveCrawlMonitor uses the already-mocked useCrawlProgress hook, so no network is hit.

  • [ ] Step 5.3: Run all CrawlerFactory tests, expect PASS

wzxhzdk:22

Expected: 9 tests pass (null + discovering + categorized + configuring + crawling + done + error + loading + hook-error).

  • [ ] Step 5.4: Run tsc --noEmit

wzxhzdk:23

Expected: clean.

  • [ ] Step 5.5: Commit

wzxhzdk:24

Task 6: CrawlerFactory: URL ?session=<id> resume + discovery handoff

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 6.1: Append failing tests for URL resume + discovery → URL handoff

Append to the same describe:

wzxhzdk:25

Add to the imports at the top of the test file (next to MemoryRouter):

wzxhzdk:26

  • [ ] Step 6.2: Run, expect PASS

wzxhzdk:27

Expected: 12 tests pass total.

  • [ ] Step 6.3: Run tsc --noEmit

wzxhzdk:28

Expected: clean.

  • [ ] Step 6.4: Commit

wzxhzdk:29

Task 7: Wire CrawlerFactory into AdminFactory page shell

Files: - Modify: src/pages/AdminFactory.tsx

The Spec 09 placeholder is an empty container under the page header. Replace its body with <CrawlerFactory />.

  • [ ] Step 7.1: Read the current AdminFactory.tsx to confirm its shape

wzxhzdk:30

Expected: a default-exported page component with header + framer-motion shell, body roughly <div className="container mx-auto..."><!-- placeholder --></div>. (Spec 09 left this empty container intentionally.)

  • [ ] Step 7.2: Replace the empty container body with <CrawlerFactory />

Open src/pages/AdminFactory.tsx. Inside the page's main content <div> (the one currently empty per Spec 09), import and render <CrawlerFactory />. Keep the existing header, framer-motion fade-up wrapper, and <NodeNetwork /> background untouched.

wzxhzdk:31

(If the existing container in your codebase uses different Tailwind classes, keep them: only swap the children.)

  • [ ] Step 7.3: Add the page-level smoke test

Create src/pages/__tests__/AdminFactory.test.tsx:

wzxhzdk:32

  • [ ] Step 7.4: Run the page test, expect PASS

wzxhzdk:33

Expected: 1 test passes.

  • [ ] Step 7.5: Run all factory frontend tests + tsc

wzxhzdk:34

Expected: all factory frontend tests pass; tsc clean.

  • [ ] Step 7.6: Commit

wzxhzdk:35

Task 8: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 8.1: Run the full project test suite

wzxhzdk:36

Expected: prior baseline tests + new factory tests, all GREEN. New tests added by this spec: ~17 (4 LiveCrawlMonitor + 12 CrawlerFactory + 1 AdminFactory page).

  • [ ] Step 8.2: Run full tsc

wzxhzdk:37

Expected: clean.

  • [ ] Step 8.3: Manual smoke (optional, only if dev server running)

wzxhzdk:38

Then open http://localhost:5173/admin/factory. Expect: header + DiscoveryPanel + RollingLog. URL should have no query param yet.

Refresh with ?session=does-not-exist. Expect: ErrorPanel rendered (or LoadingPanel if the API takes a moment), with a "Restart" button that clears the URL param.

  • [ ] Step 8.4: Mark Spec 12 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

wzxhzdk:39

Acceptance criteria: Spec 12 done means:

  1. ✅ src/components/admin/factory/LiveCrawlMonitor.tsx exists, exports LiveCrawlMonitor and types LiveCrawlMonitorProps, LiveCrawlMonitorCrawler
  2. ✅ Empty crawlerIds renders empty-state copy (No active crawls)
  3. ✅ Non-empty crawlerIds renders one block per crawler with: name, mono crawlerId, DomainBadge, indexName, shadcn Progress bar with proper aria attributes, URLs/sec, success/failed/ignored counters
  4. ✅ Each block subscribes to useCrawlProgress(crawlerId) (one hook call per crawlerId)
  5. ✅ src/components/admin/factory/CrawlerFactory.tsx exists and exports CrawlerFactory
  6. ✅ Reads ?session=<id> via useSearchParams and passes to useFactorySession(sessionId)
  7. ✅ FSM rendering verified by tests: null/discovering → DiscoveryPanel; categorized → CategoryReview; configuring → CategoryReview + CategoryConfigurator drawer; crawling → LiveCrawlMonitor; done → LiveCrawlMonitor + Restart; error → ErrorPanel; isLoading → LoadingPanel
  8. ✅ When discovery completes (useDiscoveryStream emits sessionId), URL is updated to ?session=<id> so refresh resumes correctly
  9. ✅ Configurator queue: clicking "Configure Selected" with N pathGroups walks them one at a time; each commit advances the queue
  10. ✅ Right column ALWAYS renders <RollingLog />; layout is 2-column grid (left ~60%, right ~40%) on lg breakpoint
  11. ✅ src/pages/AdminFactory.tsx mounts <CrawlerFactory /> (Spec 09 placeholder replaced)
  12. ✅ All new tests pass (~17 added by this spec)
  13. ✅ Full project test suite still GREEN
  14. ✅ tsc --noEmit clean
  15. ✅ Per CodingSOPs: every component has docstring, no console.log, no any, components ≤20-line functions where practical, mock-only-hooks unit tests
  16. ✅ Per UIUXDesignSOP / Plan §5a: aesthetic matches /admin (bg-[#0a0a0f], white/10 borders, font-black uppercase tracking-tighter, mono for IDs/URLs, framer-motion fade-up)

Out of scope (handled by other specs)

  • DiscoveryPanel, CategoryReview, RollingLog, DomainBadge implementations → Spec 10
  • CategoryConfigurator drawer + sub-components (SamplePreview, ConfigEditor, TestCrawlResults) → Spec 11
  • useFactorySession, useDiscoveryStream, useCrawlProgress hooks + page shell AdminFactory.tsx (without CrawlerFactory mount) → Spec 09
  • /api/factory/session and /api/factory/crawl-progress endpoints → Spec 08
  • FactorySession, PathGroup zod schemas → Spec 01
  • E2E browser smoke test (real backend hit) → Spec 13
  • Bundle build (scripts/bundle-api.mjs) + vault sync + Status.md toggle for the whole factory → Spec 13
  • The /admin/factory route registration in src/App.tsx → Spec 09 (already added there per Task 22)