Dashboard/Crawler-Factory/Engineering-Specs/10-frontend-discovery.md
Crawler-Factory

Engineering-Specs/10-frontend-discovery.md

Crawler Factory: Spec 10: Frontend Discovery Flow Implementation Plan

wzxhzdk:34 For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the four discovery-phase UI components: RollingLog, DomainBadge, DiscoveryPanel, CategoryReview: that drive the first half of the Crawler Factory user flow. Together they let an operator paste a URL, watch streamed discovery events, and select pathGroups to configure.

Architecture: Pure presentation + form components. No backend calls live in these components: they consume props/hooks already built by Spec 09 (useDiscoveryStream). The shell page (AdminFactory.tsx) and configurator drawer (Spec 11) are out of scope. Visual conventions match the existing /admin aesthetic: dark indigo background, framer-motion fade-up, mono fonts for IDs, color-coded badges. shadcn primitives (scroll-area, card, checkbox, input, button, form, badge) are the building blocks: no custom CSS beyond Tailwind utility classes.

Tech Stack: React 18 (function components, hooks), TypeScript strict, framer-motion, react-hook-form 7 + @hookform/resolvers/zod, zod 3.25, shadcn UI (Radix primitives), Tailwind, lucide-react icons, vitest + @testing-library/react + jsdom.

Depends on: - Spec 01 (ContentDomain type from lib/factory/types.ts) - Spec 09 (useDiscoveryStream hook + LogEntry shape from lib/factory/types.ts) - Existing shadcn components in src/components/ui/

Consumed by: - Spec 11 (CategoryConfigurator reads selectedPathGroupIds from CategoryReview.onConfigureSelected) - Spec 12 (LiveCrawlMonitor reuses RollingLog as a read-only persistent log) - Spec 13 (smoke test verifies the discovery flow end-to-end)

Plan source: Projects/Crawler-Factory/00-Plan.md §5 (UI specs: page layout, panels), §7 Tasks 24, 25, 26. §19c for traffic-light thresholds. Visual conventions referenced from src/pages/Admin.tsx and src/components/admin/HealthHUD.tsx.

File structure

Path Responsibility
src/components/admin/factory/RollingLog.tsx Auto-scrolling structured log viewer (shadcn ScrollArea). Stateless presentation only.
src/components/admin/factory/DomainBadge.tsx Pill badge: content domain name, confidence percent, traffic-light dot. Stateless.
src/components/admin/factory/DiscoveryPanel.tsx Step 1 panel: URL input form (react-hook-form + zod), submit, status line, embedded RollingLog.
src/components/admin/factory/CategoryReview.tsx Step 2 panel: pathGroup tree grouped by content domain, checkboxes, manual reclassify dropdown, "Configure Selected (N)" CTA.
src/components/admin/factory/__tests__/RollingLog.test.tsx Auto-scroll + level-color tests.
src/components/admin/factory/__tests__/DomainBadge.test.tsx Domain color + traffic-light threshold tests.
src/components/admin/factory/__tests__/DiscoveryPanel.test.tsx Form validation + submit tests (mocked hook).
src/components/admin/factory/__tests__/CategoryReview.test.tsx Grouping + selection + CTA tests.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime form validation; tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful (e.g., detectedDomain: ContentDomain | null per Spec 01 PathGroupSchema).
  2. No console.log. Top of every component file: import { createLogger } from '@/lib/utils/logger'; and const log = createLogger('factory:ui:rolling-log'); (or appropriate context). Wrap any error path with log.error({ event, ... }, 'short_msg'): never raw console.error from training priors. The logger module is already shipped (src/lib/utils/logger.ts via path alias @/lib/utils/logger): confirm path before importing; if the alias resolves to lib/utils/logger.ts use the relative path that works in your codebase build.
  3. Try/catch every fallible call inside event handlers. Form submit handlers wrap discover() in try/catch and log on rejection.
  4. Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object. Components receive a single typed Props object.
  5. Docstring on every exported component. Single line above the export explaining purpose. Comment WHY, not WHAT, for non-obvious logic (e.g., the auto-scroll ref pattern, the traffic-light threshold math).
  6. No raw dicts crossing module boundaries. Props are typed interfaces; arrays of LogEntry and PathGroup come from the Spec 01 zod-validated types.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. No live network in unit tests. useDiscoveryStream is mocked via vi.mock. Real SSE lives behind useDiscoveryStream in Spec 09 and is integration-tested there.
  9. React rules. - Functional components only. - Hooks at top level, never inside conditionals. - useEffect deps arrays explicit; ESLint react-hooks/exhaustive-deps must stay clean. - No inline any. No as unknown as X casts. - Each component file exports exactly one component (plus any helper types): no barrel files.
  10. Naming. TypeScript: camelCase for vars/functions, PascalCase for components/types, UPPER_SNAKE_CASE for module constants. Files: PascalCase.tsx for components (matches existing HealthHUD.tsx, BugTracker.tsx convention). Tests: same name with .test.tsx.
  11. Conventional commits. feat(factory):, test(factory):, chore(factory):, etc. One logical change per commit.
  12. Accessibility. All form controls have associated <label> (use shadcn FormLabel). Buttons disabled when invalid. Live regions (aria-live="polite") on the status line + log container so screen readers announce updates.
  13. No frontend may render an emoji. Use lucide-react icons (consistent with Admin.tsx).

Visual convention reminders (from src/pages/Admin.tsx + src/components/admin/HealthHUD.tsx)

  • Background: bg-[#0a0a0f], text white, borders border-white/10.
  • Section headers: text-3xl font-black uppercase tracking-tighter with indigo accent badge (bg-indigo-500/10 text-indigo-400 border border-indigo-500/30).
  • Section subheaders: text-sm font-bold text-white/40 uppercase tracking-widest, flanked by h-px bg-white/10 flex-1 dividers.
  • Cards: bg-white/5 border border-white/10 rounded-2xl backdrop-blur-md.
  • Mono font: font-mono for URLs, IDs, timestamps.
  • Badges: bg-{color}-500/20 text-{color}-400 border border-{color}-500/30 rounded-md px-2 py-1 text-[10px] font-bold uppercase tracking-widest.
  • Animations: framer-motion initial={{ opacity: 0, y: 20 }} animate={{ opacity: 1, y: 0 }} transition={{ duration: 0.5 }} on top-level sections.

Task 24: RollingLog: auto-scrolling structured log viewer

Files: - Create: src/components/admin/factory/RollingLog.tsx - Create: src/components/admin/factory/__tests__/RollingLog.test.tsx

RollingLog is a pure presentation component: takes an array of LogEntry, renders them inside shadcn ScrollArea, and auto-scrolls to the bottom whenever the list grows. Each entry shows a monospace timestamp, a colored level badge, and the message. It is reused by DiscoveryPanel (Step 1) and later by LiveCrawlMonitor (Spec 12).

LogEntry is imported from lib/factory/types.ts (Spec 01):

## Executive Team — [Company]
| Name | Title |
|---|---|
| Jane Smith | CEO |
  • [ ] Step 24.1: Write failing test for empty-state render

Create src/components/admin/factory/__tests__/RollingLog.test.tsx:

User: /scout-leadership apple.com
  │
  ▼
[1] Health check — GET /health
  → if DOWN: print startup command, exit
  │
  ▼
[2] Map domain — POST /map {"url": "https://apple.com", "max_pages": 200}
  → get urls[] list (may take 10–45s)
  │
  ▼
[3] Filter urls[] for leadership patterns
  → rank: /leadership > /executives > /team > /management > /about/* variants
  → if no match: log "no leadership URL found", proceed to fallback
  │
  ▼
[4a] Scrape best-match URL — POST /scrape {"url": "...", "use_js": true}
  → if markdown word_count < 100: go to [4b]
  → if scrape fails (success=false): go to fallback
  │
[4b] Retry with raw_html — POST /scrape {"url": "...", "use_js": true, "formats": ["raw_html"]}
  → parse HTML for card patterns (h3/h4 names, role/title spans)
  │
  ▼
[5] Structure output
  → Build markdown table: | Name | Title | Source |
  → If LinkedIn links found in ScrapeResponse.links: add | LinkedIn |
  → Attach source label per row
  │
  ▼
[6] Write file: ./{company-slug}-leadership.md
  │
  ▼
[7] Print to chat:
  ## Leadership Team — Apple (apple.com)
  | Name | Title | Source |
  |------|-------|--------|
  | Tim Cook | CEO | apple.com/leadership |
  ...
  Source: https://www.apple.com/leadership/
  Scraped: 2026-05-05T14:32:00Z | Duration: 38s
  • [ ] Step 24.2: Run, expect FAIL
[3] Filter returns no match
  │
  ▼
[F1] Attempt common fallbacks: /about, /company, /about-us
     → scrape and scan for executive mentions
  │
  ▼
[F2] If execs found in fallback:
     → return table with WARNING banner: "Data from [fallback URL], not a dedicated leadership page"
  │
  ▼
[F3] If nothing found:
     → return: "No leadership page found for apple.com. URLs checked: [list]. Try: apple.com/leadership manually."

Expected: FAIL: Cannot find module '../RollingLog'.

  • [ ] Step 24.3: Implement RollingLog.tsx with empty state

Create src/components/admin/factory/RollingLog.tsx:

<

  • [ ] Step 24.4: Run empty-state test, expect PASS

wzxhzdk:4

Expected: 1 test pass.

  • [ ] Step 24.5: Add tests for level colors + auto-scroll

Append to src/components/admin/factory/__tests__/RollingLog.test.tsx:

wzxhzdk:5

  • [ ] Step 24.6: Run all RollingLog tests, expect PASS

wzxhzdk:6

Expected: 7 tests pass.

  • [ ] Step 24.7: Run tsc --noEmit

wzxhzdk:7

Expected: clean.

  • [ ] Step 24.8: Commit

wzxhzdk:8

Task 25: DomainBadge: color-coded pill with confidence + traffic light

Files: - Create: src/components/admin/factory/DomainBadge.tsx - Create: src/components/admin/factory/__tests__/DomainBadge.test.tsx

DomainBadge shows the detected content domain on a pill, the confidence as a small monospace percentage, and a traffic-light dot keyed to §19c thresholds:

Confidence Dot color Class
≥ 0.85 green bg-emerald-400
0.65–0.84 yellow bg-yellow-400
< 0.65 amber bg-amber-500

Per-domain pill colors (from §7 Task 24):

Domain Color
marketing indigo
support red
education emerald
technical blue
customer-stories purple
product-catalog amber
events pink
legal slate
social fuchsia
  • [ ] Step 25.1: Write failing tests

Create src/components/admin/factory/__tests__/DomainBadge.test.tsx:

wzxhzdk:9

  • [ ] Step 25.2: Run, expect FAIL

wzxhzdk:10

Expected: FAIL: Cannot find module '../DomainBadge'.

  • [ ] Step 25.3: Implement DomainBadge.tsx

Create src/components/admin/factory/DomainBadge.tsx:

wzxhzdk:11

  • [ ] Step 25.4: Run all DomainBadge tests, expect PASS

wzxhzdk:12

Expected: 12 tests pass.

  • [ ] Step 25.5: Run tsc --noEmit

wzxhzdk:13

Expected: clean.

  • [ ] Step 25.6: Commit

wzxhzdk:14

Task 26: DiscoveryPanel: URL form + status line + embedded log

Files: - Create: src/components/admin/factory/DiscoveryPanel.tsx - Create: src/components/admin/factory/__tests__/DiscoveryPanel.test.tsx

DiscoveryPanel is the Step 1 layout (per §5b page diagram): a 40/60 two-column card. Panel A holds the URL input, Submit, and a status line. Panel B holds an embedded RollingLog driven by useDiscoveryStream from Spec 09.

Hook contract assumed from Spec 09 (already shipped: do NOT redefine):

wzxhzdk:15

If Spec 09 has not yet shipped, the implementation below mocks the hook in tests and the runtime imports remain valid because the import path resolves to the (separately built) Spec 09 module. Do not write Spec 09's body inside this spec.

  • [ ] Step 26.1: Write failing test for URL validation

Create src/components/admin/factory/__tests__/DiscoveryPanel.test.tsx:

wzxhzdk:16

  • [ ] Step 26.2: Run, expect FAIL

wzxhzdk:17

Expected: FAIL: Cannot find module '../DiscoveryPanel'.

  • [ ] Step 26.3: Implement DiscoveryPanel.tsx

Create src/components/admin/factory/DiscoveryPanel.tsx:

wzxhzdk:18

  • [ ] Step 26.4: Run all DiscoveryPanel tests, expect PASS

wzxhzdk:19

Expected: 8 tests pass.

  • [ ] Step 26.5: Run tsc --noEmit

wzxhzdk:20

Expected: clean. If useDiscoveryStream is not yet shipped (Spec 09 pending), tsc will fail on the import: block this task on Spec 09 completion.

  • [ ] Step 26.6: Commit

wzxhzdk:21

Task 27: CategoryReview: pathGroup tree, selection, manual reclassify, CTA

Files: - Create: src/components/admin/factory/CategoryReview.tsx - Create: src/components/admin/factory/__tests__/CategoryReview.test.tsx

CategoryReview renders the post-discovery tree (per §5b second diagram). Top-level rows are content domains. Each row is collapsible (chevron icon flips). Inside, each pathGroup row contains:

  • shadcn Checkbox (controlled) for selection
  • path pattern (mono font)
  • URL count
  • DomainBadge with confidence
  • "show samples" toggle (revealing up to 3 sample URLs)
  • "reclassify" dropdown (shadcn Select) listing all 9 content domains for manual override

The bottom CTA Configure Selected (N) is enabled only when ≥1 row is selected. Clicking calls props.onConfigureSelected(selectedPathGroupIds).

klzzwxh:0080 shape (already shipped from Spec 01 klzzwxh:0081):

wzxhzdk:22

This component owns local UI state for selection and manual domain override: both are edited optimistically, then bubbled up via onSelectionChange and onDomainOverride so the parent (Spec 13 CrawlerFactory.tsx) can persist into the session. The pathGroups prop remains the source of truth between renders.

  • [ ] Step 27.1: Write failing tests

Create src/components/admin/factory/__tests__/CategoryReview.test.tsx:

wzxhzdk:23

  • [ ] Step 27.2: Run, expect FAIL

wzxhzdk:24

Expected: FAIL: Cannot find module '../CategoryReview'.

  • [ ] Step 27.3: Implement CategoryReview.tsx

Create src/components/admin/factory/CategoryReview.tsx:

wzxhzdk:25

  • [ ] Step 27.4: Run all CategoryReview tests, expect PASS

wzxhzdk:26

Expected: 7 tests pass.

  • [ ] Step 27.5: Run tsc --noEmit

wzxhzdk:27

Expected: clean.

  • [ ] Step 27.6: Commit

wzxhzdk:28

Task 28: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 28.1: Run the full project test suite

wzxhzdk:29

Expected: existing baseline tests + 4 new factory UI test files (~34 new test cases), all GREEN.

  • [ ] Step 28.2: Run full tsc

wzxhzdk:30

Expected: clean.

  • [ ] Step 28.3: Run lint

wzxhzdk:31

Expected: clean. ESLint react-hooks/exhaustive-deps and @typescript-eslint/no-explicit-any must pass.

  • [ ] Step 28.4: Visual smoke (manual, post-merge)

Boot the dev shell:

wzxhzdk:32

Navigate to /admin/factory once Spec 13 (AdminFactory.tsx) is mounted. Confirm:

  • The two-column DiscoveryPanel renders with the dark indigo aesthetic.
  • The URL input rejects non-https values and shows the inline error.
  • An empty RollingLog shows the "no events yet" hint.
  • A populated CategoryReview groups pathGroups by content domain, the chevron flips on collapse, and the "Configure Selected (0)" button is disabled.

Note: this manual smoke is documented here for completeness; it depends on Spec 09 + Spec 13. Do not block the commit on it.

  • [ ] Step 28.5: Mark Spec 10 done in vault Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

wzxhzdk:33

Acceptance criteria: Spec 10 done means:

  1. src/components/admin/factory/RollingLog.tsx exists, exports RollingLog, accepts { entries, maxHeight? }, auto-scrolls on new entry, color-codes level badges (info=indigo, warn=amber, error=red), and renders monospace timestamps.
  2. src/components/admin/factory/DomainBadge.tsx exists, exports DomainBadge, accepts { contentDomain, confidence }, renders one of nine domain colors, monospace integer percentage, and a green/yellow/amber traffic-light dot per §19c thresholds (≥0.85 / 0.65–0.84 / wzxhzdk:50.65).
  3. src/components/admin/factory/DiscoveryPanel.tsx exists, exports DiscoveryPanel, takes no required props, owns a react-hook-form + zod URL form (https only), invokes useDiscoveryStream.discover(url) on submit, shows status line ("Awaiting URL." → "Discovering... N URLs found, M categorized." → "Error: …"), and embeds RollingLog.
  4. src/components/admin/factory/CategoryReview.tsx exists, exports CategoryReview, accepts { pathGroups, onConfigureSelected, onDomainOverride? }, groups rows by detected content domain (collapsible per group), renders shadcn Checkbox, DomainBadge, sample-URL toggle (≤3 URLs), reclassify dropdown (9 options), and a "Configure Selected (N)" CTA enabled only when ≥1 row is selected.
  5. All four components apply the /admin visual conventions: bg-[#0a0a0f]-compatible cards (bg-white/5 border-white/10 rounded-2xl backdrop-blur-md), uppercase tracked-widest section headers, mono fonts for IDs/URLs/timestamps, indigo accent for primary actions.
  6. ~34 new vitest cases pass, full project test suite stays GREEN, tsc --noEmit clean, npm run lint clean.
  7. Per CodingSOPs: every component file has a logger, every public component has a docstring, no any, no console.log, all hooks at top-level with explicit deps. All form controls have associated &lt;label&gt;. All live regions use aria-live="polite".

Out of scope (handled by other specs)

  • The useDiscoveryStream hook implementation: Spec 09 (mocked here).
  • The useFactorySession + useCrawlProgress hooks: Spec 09.
  • CategoryConfigurator drawer (Step 3: Configure): Spec 11.
  • LiveCrawlMonitor (Step 4: Live crawl): Spec 12 (but it reuses RollingLog from this spec).
  • CrawlerFactory.tsx orchestrator + FSM driver: Spec 13.
  • AdminFactory.tsx page shell + route registration: Spec 13.
  • Backend SSE endpoints (/api/factory/discover, etc.): Specs 06–08.
  • Persisting selectedPathGroupIds to the session record: Spec 13 orchestrator.
  • Persisting manual reclassify overrides into the session: Spec 13 orchestrator wires onDomainOverride to useFactorySession.update.
  • Visual smoke against a live API: Spec 13 manual verification.