Dashboard/Algolia-Search-Audit/Refactor-Architecture.md
Algolia-Search-Audit

Refactor-Architecture.md

Algolia Search Audit — Refactor Architecture

wzxhzdk:3 The target architecture for the refactored audit pipeline. Based on Anthropic's agent harness patterns (AgentHarnessPatterns).

Status: Planning (2026-04-08) Decision: Python CLI harness + Pydantic schemas + codified deterministic logic

Design Principles

  1. Context isolation — each module runs in its own claude -p subprocess (fresh 200K context)
  2. Schema-first — every module output validated by Pydantic before the harness marks it complete
  3. Codify what you can — if a step is deterministic math/rules, it's Python, not LLM
  4. Files are the API — modules communicate only through .json files in the workspace
  5. The harness is not an LLM — orchestration, gating, retry, and progress tracking are pure Python
  6. Resume-from-failure — JSONL progress log allows --resume to skip completed modules

Target Architecture

topic_id            (stable id, e.g. ga_g2_sci_forces)
state, grade, subject
county / district   (metadata; pacing source)
standard_code       (e.g. S2P2)  + standard_text
title               (kid-friendly display name)
learning_objectives[]
key_vocabulary[]
misconceptions[]
pacing_hint?        (per-county sequence/timing, optional)
source_url[]        (authoritative citation)
status              (draft | validated | published)
validated_by, validated_at
researched_at, catalog_version

Module Execution Model (New)

Each module now has a clear execution model:

Type 1: Pure Python (no LLM)

Script runs, produces .json, harness validates against Pydantic, done.

Module Script Schema
1B techstack collect-techstack.py + deterministic status classification TechStack
1C traffic collect-traffic.py TrafficData
1H hiring collect-hiring.py + classify-roles.py HiringSignals
1I social collect-social.py + keyword relevance scoring SocialSignals
1J news collect-news.py + keyword categorization NewsSignals
3A business case (numbers) calculate-roi.py BusinessCase
4A factcheck (mechanical) factcheck-mechanical.py FactcheckMechanical

Type 2: Python Script + LLM Enrichment

Script runs first (API calls, data collection), then claude -p enriches the output.

Module Script LLM Prompt Schema
1A company collect-company.py company-enrich.txt CompanyContext
1D competitors collect-competitors.py competitors-enrich.txt Competitors
1E financial (public) collect-financials.py + collect-sec-data.py financial-public-enrich.txt FinancialProfile
1F financial (private) collect-financials.py --private financial-private-enrich.txt FinancialProfile
1G investor collect-investor.py + collect-sec-data.py (shared) investor-enrich.txt InvestorIntel
1K partner collect-partners.py (NEW) partner enrichment (if needed) PartnerIntel
1L industry collect-industry.py industry-enrich.txt IndustryIntel

Type 3: Pure LLM (irreducibly creative)

claude -p with full prompt, reads specific input files, writes output.

Module LLM Prompt Schema
2 queries queries.txt TestQueries
L2 browser browser.txt BrowserFindings
3A business case (narrative) business-case-narrative.txt (merged with calc output)
3B sales plays sales-plays.txt SalesPlays
3C report report-assembly.txt AuditData
3D ABX campaign abx-campaign.txt ABXCampaign
4B factcheck (narrative) factcheck-narrative.txt FactcheckNarrative

Harness Flow

>

Key Differences from Current Architecture

Aspect Current Refactored
Orchestration Claude Code Skill tool (inline, shared context) Python audit-harness.py (subprocess, isolated contexts)
Schema validation None Pydantic model per module, validated before marking complete
ROI calculation LLM inference calculate-roi.py (Python)
Report data source .md prose files .json structured files only
Partner intel output .md only .md + .json
ABX data storage Mutates audit-data.json Separate abx-data.json
SEC/earnings fetch Duplicated (1E + 1G) Shared collect-sec-data.py
Factcheck All 20 dims via LLM 15 dims Python + 5 dims LLM
Algolia customers Scraped per audit Static algolia_customers.json
Progress tracking Inconsistent JSONL Harness-managed, per-module cost/timing
Retry logic None (manual re-run) 3 attempts per module with backoff
Resume Start over --resume reads JSONL, skips completed
Observability None Rich terminal dashboard + JSONL audit trail

Distributable Package

For sharing with others (AEs, SEs, partners):

>

No Claude Code skills needed. No Skill tool. Just Python + claude CLI.

Sprint Plan

Sprint 1: Foundation (harness + schemas + codification)

  • audit-harness.py core (workspace, progress, args, resume)
  • All 20 Pydantic schemas
  • Wire calculate-roi.py, create factcheck-mechanical.py
  • Create collect-sec-data.py (shared 1E + 1G)
  • Create collect-partners.py (tech stack cross-ref)
  • Static datasets (algolia_customers.json, buyer_tier_keywords.json)

Sprint 2: Skill prompts + enrichment

  • Write all prompts/*.txt files
  • Review + fix each skill's LLM enrichment instructions
  • Add JSON output where missing (partner, queries, browser findings, sales plays)
  • Fix field name inconsistencies across all skills

Sprint 3: Report pipeline

  • Switch report to read .json exclusively
  • Wire generate-audit-data.py + calculate-score.py
  • Fix capability matrix dynamic competitor mapping
  • Separate ABX output from audit-data.json

Sprint 4: Terminal UI + testing

  • Rich progress dashboard
  • Per-module cost/timing tracking
  • End-to-end smoke test on a known domain
  • Documentation
  • AgentHarnessPatterns — Anthropic's harness patterns (source of key patterns)
  • Index — project overview
  • Module-Catalog — per-module detail
  • Known-Issues — problems this refactor fixes