Scout Leadership Intel — Research + Technical Context

Created: 2026-05-05

Sources

• Scout skill/command file: ~/.claude/commands/scout.md
• Scout API source: /Users/arijitchowdhury/AI-Development/Scout/scout/core/types.py
• Scout project memory: ~/.claude/projects/-Users-arijitchowdhury-AI-Development-PIP/memory/project_scout_crawler.md
• Scout eval data: ~/.claude/scout-skill-workspace/iteration-2/eval-1-exec-team/

Scout Platform — What It CAN Do

Scout exposes 5 POST endpoints at http://localhost:8421, all authenticated via X-API-Key: dev-key:

Core integration pattern (Map → Filter → Scrape): 1. POST /map {"url": "https://company.com", "max_pages": 200} → discover all URLs 2. Filter urls[] for leadership patterns 3. POST /scrape {"url": "<best_match>", "use_js": true} → markdown content 4. If sparse: retry with {"formats": ["raw_html"]} → parse HTML card patterns

ScrapeRequest shape:

// (Already shipped — do not redefine)
export const LogEntrySchema = z.object({
  ts: z.number().int().positive(),
  level: z.enum(['info', 'warn', 'error']),
  message: z.string().min(1),
  meta: z.record(z.unknown()).optional(),
});
export type LogEntry = z.infer<typeof LogEntrySchema>;

ScrapeResponse fields relevant to this feature: - markdown: str — primary extraction target - raw_html: str — fallback for JS-card pages - links: list[str] — for LinkedIn URL harvesting - metadata.crawled_at: str — source timestamp for attribution - success: bool + error: str — for failure reporting - duration_ms: int — for latency tracking

ExtractRequest shape (LLM or CSS):

/**
 * RollingLog tests — verify auto-scroll on new entry, level badge colors,
 * empty-state message. No backend; no SSE; pure presentation.
 */
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, act } from '@testing-library/react';
import { RollingLog } from '../RollingLog';
import type { LogEntry } from '@/../lib/factory/types';

// jsdom does not implement scrollIntoView — stub it.
beforeEach(() => {
  Element.prototype.scrollIntoView = vi.fn();
});

const entry = (level: LogEntry['level'], ts: number, message: string): LogEntry => ({
  level,
  ts,
  message,
});

describe('RollingLog', () => {
  it('renders empty-state hint when entries are empty', () => {
    render(<RollingLog entries={[]} />);
    expect(screen.getByText(/no events yet/i)).toBeInTheDocument();
  });
});

Scout Platform — What It CANNOT Do

• Not a general-purpose scraper — optimised for 5 specific intel types only
• No parallel crawls — single-instance; sequential only
• No access to authenticated or login-gated pages
• ATS SPA job boards (Workday, Greenhouse React): JS XHR interception gap
• /extract with LLM requires LLM_API_KEY in Scout .env; CSS fallback works without it
• localhost URLs cannot be fetched via WebFetch (sandbox restriction — use curl/bash)

Real-World Performance (Eval Data)

From eval-1-exec-team on algolia.com (iteration 2): - 75% pass rate on exec extraction (3/4 assertions passed) - Map-first is mandatory — standard leadership URL paths can 404 - formats: ["raw_html"] is the reliable path for JS-heavy card layouts - Exec data from press releases (fallback) may be stale vs. current leadership page - Leadership URL patterns vary: /about/leadership, /about/team, /company/management — no single correct path

Scout Status

• Feature complete as of 2026-05-04
• 78/78 unit tests passing
• Source: /Users/arijitchowdhury/AI-Development/Scout/
• Start command: cd /Users/arijitchowdhury/AI-Development/Scout && python3 -m uvicorn scout.api.main:app --host 0.0.0.0 --port 8421
• Built on Crawl4AI (Apache 2.0) + Playwright