Crawler Factory: Spec 02: Sitemap Discovery + URL Streaming Implementation Plan

wzxhzdk:49 For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build the streaming sitemap discovery layer: walkSitemap async iterator (uncapped, gzip-aware, hreflang-aware, cycle-safe), discoverSitemaps (robots.txt + fallback chain), groupByPathPrefix (streaming pathGroup builder), and estimateScope (1–2s pre-flight probe). Together these turn an arbitrary root URL into a paginated stream of URLs and a heuristic pathGroup list: without ever holding all URLs in memory.

Architecture: Pure I/O + parsing layer. Three independent modules: sitemap.ts (network + XML), path-grouper.ts (in-memory aggregation), scope-estimator.ts (single-shot probe). All three are streaming where the data shape allows: walkSitemap is async function* yielding string[] batches of 1000; groupByPathPrefix is async function* yielding PathGroup objects so a 10M-URL site never materializes a single array. fast-xml-parser parses both klzzwxh:0018 and klzzwxh:0019 shapes; the walker treats EITHER as valid at any depth (cycle-prevention via klzzwxh:0020 of fetched URLs: no depth cap). The walker fetches sub-sitemaps in parallel batches of 8. Files klzzwxh:00215MB use streaming/chunked parse paths to avoid OOM on Vercel functions (1024MB ceiling).

Tech Stack: TypeScript (strict), fast-xml-parser 4.5+ (added in Spec 01), native fetch + DecompressionStream('gzip') (no extra deps), zod 3.25 for boundary validation, vitest for tests, pino logger via lib/utils/logger.ts.

Depends on: Spec 01 (uses LogEntrySchema, PathGroupSchema, ContentDomainEnum from lib/factory/types.ts; fast-xml-parser is already installed and externalized). Consumed by: Spec 03 (sampler reads URLs from walkSitemap output written into url_shard records), Spec 04 (classifier consumes pathGroups from groupByPathPrefix), Spec 11 (frontend wizard reads estimateScope for the pre-flight UI).

Plan source: Projects/Crawler-Factory/00-Plan.md §3a (seeder-list discovery best practices), §3a-bis (pre-flight scope estimation), §7 Tasks 3 + 4, §14b (Sitemap & robots.txt adoption: Web Almanac 2024), §14f (cross-vertical site coverage), §16l (massive product+language silo sites: Microsoft 1M-10M URLs), §19a (first-touch protocol), §19f principles 3 + 9.

---

File structure

Path	Responsibility
lib/factory/sitemap.ts	walkSitemap(url) async iterator yielding URL batches of 1000; discoverSitemaps(rootUrl) reads robots.txt + fallback chain; recursive sub-sitemap fetch with cycle prevention, gzip decompression, hreflang extraction, parallel batching (8 concurrent), streaming SAX-style parse for files >5MB
lib/factory/path-grouper.ts	Streaming groupByPathPrefix(urlSource) async iterator emitting PathGroup objects; first-non-empty-segment heuristic; merges groups <3 URLs into "other"; sample retention capped at 10 URLs per group
lib/factory/scope-estimator.ts	estimateScope(rootUrl): fast probe (≤2s typical): GET robots.txt, GET top sitemap, count &lt;sitemap&gt; children if sitemapindex, detect language splits, return {urlEstimateRange, subSitemapCount, languagesDetected, expectedDuration, transport}
lib/factory/__tests__/sitemap.test.ts	Sitemap walker tests: robots.txt parse, fallback chain, sitemapindex recursion, cycle-self-reference, gzipped sitemap, 50K-URL urlset, depth-3 nested index (Microsoft pattern), hreflang extraction
lib/factory/__tests__/path-grouper.test.ts	Path-grouper tests: basic grouping, merge-small-into-other, sample cap, streaming behavior with large input
lib/factory/__tests__/scope-estimator.test.ts	Scope-estimator tests: flat urlset estimate, sitemapindex with 531 children (Apple Autopush pattern), language detection from hreflang and URL paths, WAF transport flag

---

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

1. Two-layer type safety. zod for runtime boundary validation (every public API + file I/O); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful.
2. Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:sitemap'); (or factory:path-grouper, factory:scope-estimator). No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
3. Try/catch every fallible call. Every fetch, every XML parse, every DecompressionStream operation is wrapped. Catch specific errors first, generic last. Always log before rethrow with full context (url, bytes, depth, parent).
4. Functions ≤20 lines, ≤3 params. More than 3 params → typed config object. Walker recursion happens via a private helper that takes a single WalkContext config object.
5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
6. No raw dicts crossing module boundaries. walkSitemap yields string[] (URLs are typed at the union level); groupByPathPrefix yields zod-validated PathGroup; estimateScope returns a zod-validated ScopeEstimate.
7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
8. Mock external services in unit tests. Use vitest's vi.mock to stub global fetch. Fixture XML lives inline in the test file as template strings: not separate files. Real network calls live in integration tests (Spec 13).
9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
10. Conventional commits. feat(factory):, test(factory):, one logical change per commit.
11. Streaming everywhere. walkSitemap is async function*, never returns Promise<string[]>. groupByPathPrefix consumes AsyncIterable<string[]> and yields PathGroup objects. No code path may materialize all URLs in a single array (Microsoft Learn = 10M URLs; one such array would OOM Vercel).
12. No depth cap, no URL cap. Cycle prevention via Set<string> of fetched sitemap URLs. The walker MUST handle depth-3+ nested sitemap-indexes (Microsoft pattern, §16l). The walker MUST handle a single 50,000-URL urlset (sitemap-protocol max).

---

Task 1: discoverSitemaps: robots.txt + fallback chain

Files: - Create: lib/factory/sitemap.ts (initial version: only discoverSitemaps) - Create: lib/factory/__tests__/sitemap.test.ts

discoverSitemaps(rootUrl) returns a list of sitemap URLs to walk. Per §19a step 2 + §3a step 1–2: 1. GET ${rootUrl}/robots.txt with User-Agent: AlgoliaCentralFactory/1.0, timeout 8s. 2. Parse Sitemap: directives (RFC 9309: case-insensitive directive name, full URL value). 3. If robots.txt has zero Sitemap: lines OR returns 404, fall through to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml. HEAD-then-GET each in order; first 200 wins. 4. If all four fail, return [] and log a warning. Caller decides what to do (Spec 11 surfaces "no sitemap: supply manual seeds" UI).

• [ ] Step 1.1: Write failing test for robots.txt with one Sitemap directive

Create lib/factory/__tests__/sitemap.test.ts:

&

• [ ] Step 1.2: Run, expect FAIL

>

Expected: FAIL: Cannot find module '../sitemap'.

• [ ] Step 1.3: Implement initial lib/factory/sitemap.ts with discoverSitemaps only

Create lib/factory/sitemap.ts:

>

• [ ] Step 1.4: Run, expect PASS

<

Expected: 2 tests pass.

• [ ] Step 1.5: Add tests for fallback chain + 404 robots.txt

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:4

• [ ] Step 1.6: Run, expect PASS

wzxhzdk:5

Expected: 7 tests pass.

• [ ] Step 1.7: Run tsc --noEmit

wzxhzdk:6

Expected: clean.

• [ ] Step 1.8: Commit

wzxhzdk:7

---

Task 2: walkSitemap: async iterator with recursion + cycle prevention

Files: - Modify: lib/factory/sitemap.ts (add walkSitemap) - Modify: lib/factory/__tests__/sitemap.test.ts (add walker tests)

walkSitemap(url) is an async function* yielding string[] batches of up to 1000 URLs. It MUST handle: - Plain <urlset> files (yield batches of 1000). - <sitemapindex> files (recursively follow children, parallel-fetched 8 at a time). - Self-referencing or cyclic indexes (visited-set; each URL fetched at most once). - No depth cap. Microsoft pattern: 3 levels of nesting (root → product → language). - Mixed sitemaps (some children urlsets, some indexes: both walked in the same batch).

• [ ] Step 2.1: Write failing test: flat urlset with 3 URLs

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:8

• [ ] Step 2.2: Run, expect FAIL

wzxhzdk:9

Expected: FAIL: walkSitemap is not exported.

• [ ] Step 2.3: Implement walkSitemap for flat urlsets first

Append to lib/factory/sitemap.ts:

wzxhzdk:10

• [ ] Step 2.4: Run flat-urlset test, expect PASS

wzxhzdk:11

Expected: 1 test passes.

• [ ] Step 2.5: Add test for sitemapindex with two children

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:12

• [ ] Step 2.6: Run, expect PASS

wzxhzdk:13

Expected: 1 test passes.

• [ ] Step 2.7: Add cycle-prevention test (self-referencing sitemapindex)

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:14

• [ ] Step 2.8: Run, expect PASS

wzxhzdk:15

Expected: 1 test passes.

• [ ] Step 2.9: Commit walker core

wzxhzdk:16

---

Task 3: walkSitemap: gzip support + hreflang alternates + depth-3 nesting

Files: - Modify: lib/factory/__tests__/sitemap.test.ts

The walker code already includes gzip + hreflang logic from Task 2.3. This task adds the regression tests that prove those code paths work.

• [ ] Step 3.1: Write failing test: gzipped .xml.gz sitemap

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:17

• [ ] Step 3.2: Run, expect PASS (gzip path is already implemented)

wzxhzdk:18

Expected: 2 tests pass.

• [ ] Step 3.3: Write failing test: hreflang alternate extraction

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:19

• [ ] Step 3.4: Run, expect PASS

wzxhzdk:20

Expected: 1 test passes.

• [ ] Step 3.5: Write failing test: depth-3 nested sitemap-index (Microsoft pattern)

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:21

• [ ] Step 3.6: Run, expect PASS

wzxhzdk:22

Expected: 1 test passes.

• [ ] Step 3.7: Write failing test: 50K-URL urlset (sitemap-protocol max, no cap)

Append to lib/factory/__tests__/sitemap.test.ts:

wzxhzdk:23

• [ ] Step 3.8: Run, expect PASS

wzxhzdk:24

Expected: 1 test passes (may take a few seconds).

• [ ] Step 3.9: Run all sitemap tests + tsc

wzxhzdk:25

Expected: all sitemap tests pass; tsc clean.

• [ ] Step 3.10: Commit

wzxhzdk:26

---

Task 4: groupByPathPrefix: streaming pathGroup builder

Files: - Create: lib/factory/path-grouper.ts - Create: lib/factory/__tests__/path-grouper.test.ts

groupByPathPrefix(urlSource) is an async function* that: - Consumes an AsyncIterable<string[]> (the same shape walkSitemap yields). - Groups URLs by their first non-empty path segment (e.g., /blog/x and /blog/y → group /blog/*). - Each yielded PathGroup conforms to PathGroupSchema from lib/factory/types.ts. - Caps sampleUrls at 10 per group (to keep groups under Algolia's 100KB record limit when persisted). - After the source is exhausted, merges any group with <3 URLs into a single other group and yields that last.

This module is in-memory but bounded: only the per-group counts and capped samples (10 URLs × N groups) live in RAM. For Microsoft Learn (10M URLs, ~50 path roots), that's ~500 sample URLs total: trivial.

• [ ] Step 4.1: Write failing test: basic grouping by first segment

Create lib/factory/__tests__/path-grouper.test.ts:

wzxhzdk:27

• [ ] Step 4.2: Run, expect FAIL

wzxhzdk:28

Expected: FAIL: Cannot find module '../path-grouper'.

• [ ] Step 4.3: Implement lib/factory/path-grouper.ts

Create lib/factory/path-grouper.ts:

wzxhzdk:29

• [ ] Step 4.4: Run, expect PASS for basic grouping tests

wzxhzdk:30

Expected: 2 tests pass.

• [ ] Step 4.5: Add tests for small-group merge + sample cap + streaming behavior

Append to lib/factory/__tests__/path-grouper.test.ts:

wzxhzdk:31

• [ ] Step 4.6: Run all path-grouper tests

wzxhzdk:32

Expected: all tests pass.

• [ ] Step 4.7: Run tsc --noEmit

wzxhzdk:33

Expected: clean.

• [ ] Step 4.8: Commit

wzxhzdk:34

---

Task 5: estimateScope: pre-flight scope probe

Files: - Create: lib/factory/scope-estimator.ts - Create: lib/factory/__tests__/scope-estimator.test.ts

estimateScope(rootUrl) is the §3a-bis / §19a pre-flight probe. ≤2s typical. It does NOT walk the full sitemap: it returns a fast estimate so the wizard UI can ask the user "This site has ~531 sub-sitemaps and an estimated 100K-500K URLs. Proceed?" before committing to a full discovery.

Algorithm: 1. Call discoverSitemaps(rootUrl) to get the top sitemap URL(s). 2. For each top sitemap, fetch + parse (NOT recursive: only the top file). 3. If urlset: count its <url> entries → that's the URL count for this branch. Multiply by num-of-locales-detected if hreflang alternates exist. 4. If sitemapindex: count <sitemap> children. Multiply by an average urlset size estimate (default 5000) for a low/high range. 5. Detect language splits via: - URL pathnames (/en-us/, /de-de/, /fr/, etc.: ISO 639-1 + region pattern) - hreflang attributes on the first leaf urlset (only if we already fetched one) 6. Compute expectedDuration = subSitemapCount / 8 (concurrency) × 1.5s avg for sitemapindex case; 5s flat for urlset case. 7. Set transport: 'fetch' | 'playwright' | 'unknown'. If robots.txt or top sitemap returned 403 / timed out, transport = 'playwright'. Otherwise 'fetch'.

The estimator returns a zod-validated ScopeEstimate for direct rendering by the wizard.

• [ ] Step 5.1: Write failing test: flat urlset scope estimate

Create lib/factory/__tests__/scope-estimator.test.ts:

wzxhzdk:35

• [ ] Step 5.2: Run, expect FAIL

wzxhzdk:36

Expected: FAIL: Cannot find module '../scope-estimator'.

• [ ] Step 5.3: Implement lib/factory/scope-estimator.ts

Create lib/factory/scope-estimator.ts:

wzxhzdk:37

• [ ] Step 5.4: Run urlset test, expect PASS

wzxhzdk:38

Expected: 1 test passes.

• [ ] Step 5.5: Add tests for sitemapindex, language detection, and WAF transport

Append to lib/factory/__tests__/scope-estimator.test.ts:

wzxhzdk:39

• [ ] Step 5.6: Run all scope-estimator tests

wzxhzdk:40

Expected: all tests pass.

• [ ] Step 5.7: Run tsc --noEmit

wzxhzdk:41

Expected: clean.

• [ ] Step 5.8: Commit

wzxhzdk:42

---

Task 6: Cluster validation: full regression

Files: - (Read-only verification)

• [ ] Step 6.1: Run all factory tests

wzxhzdk:43

Expected: all Spec 01 tests (~23 from Foundations) + all Spec 02 tests (sitemap ~12, path-grouper ~5, scope-estimator ~5) GREEN.

• [ ] Step 6.2: Run full project test suite

wzxhzdk:44

Expected: 407 baseline + 23 Spec 01 + ~22 Spec 02 = ~452 tests, all GREEN.

• [ ] Step 6.3: Run tsc --noEmit

wzxhzdk:45

Expected: clean.

• [ ] Step 6.4: Bundler still produces clean output

wzxhzdk:46

Expected: completes without error; existing api/*.mjs regenerated; fast-xml-parser remains externalized (no bundle bloat from new code).

• [ ] Step 6.5: Mark Spec 02 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

wzxhzdk:47

---

Acceptance criteria: Spec 02 done means:

1. ✅ lib/factory/sitemap.ts exports discoverSitemaps(rootUrl): Promise<string[]> reading robots.txt + falling back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml.
2. ✅ lib/factory/sitemap.ts exports walkSitemap(url): AsyncGenerator<string[]> yielding URL batches of up to 1000.
3. ✅ Walker recurses sitemap-indexes at any depth (verified at depth 3: Microsoft pattern), with cycle prevention via Set<string> (verified with self-referencing index).
4. ✅ Walker decompresses gzipped sitemaps (verified for .xml.gz URL suffix and Content-Encoding: gzip header).
5. ✅ Walker extracts hreflang alternates as additional URLs, deduped against loc.
6. ✅ Walker handles a 50,000-URL urlset (sitemap-protocol max) without OOM, yielding 50 batches of 1000.
7. ✅ Walker fetches sub-sitemaps in parallel batches of 8 (concurrency cap).
8. ✅ lib/factory/path-grouper.ts exports groupByPathPrefix(source): AsyncGenerator<PathGroup> consuming an async iterable of URL batches and emitting validated PathGroup objects.
9. ✅ Path-grouper merges groups with <3 URLs into a single 'other' group, yielded last.
10. ✅ Path-grouper caps sampleUrls at 10 per group.
11. ✅ Path-grouper handles unparseable URLs by routing them to 'other' and logging.
12. ✅ lib/factory/scope-estimator.ts exports estimateScope(rootUrl): Promise<ScopeEstimate> with zod-validated output (ScopeEstimateSchema).
13. ✅ Scope estimator returns urlEstimateRange = [count, count] for flat urlsets and [count×1000, count×50000] for sitemap-indexes.
14. ✅ Scope estimator detects ISO 639-1 language codes (e.g., en, en-us, de-de, fr) from URL patterns.
15. ✅ Scope estimator sets transport: 'playwright' when the top sitemap returns 403 (WAF signal).
16. ✅ All new tests pass (~22 tests across the three modules).
17. ✅ Full project test suite still GREEN.
18. ✅ tsc --noEmit clean.
19. ✅ Per CodingSOPs: every module has a logger, every public function has a docstring, every fallible call has try/catch, every public boundary uses zod validation (PathGroup, ScopeEstimate).

---

Out of scope (handled by other specs)

• Sample fetching (downloading individual page HTML for classification) → Spec 03 (sampler.ts).
• Content classification of pathGroups (CMS fingerprint, JSON-LD, OpenGraph cascade per §3b/§4b) → Spec 04 (classifier.ts).
• Persistence of discovered URLs into url_shard records → Spec 08 (/api/factory/discover endpoint uses Spec 01's SessionStore.appendUrls).
• Resumability (Vercel timeout → resume from last shard) → Spec 08 (discover endpoint orchestrator), relies on the visited-set being persisted between runs; v1 accepts a single-process visited-set scoped to one walk.
• Streaming SAX-style parse for files >5MB: fast-xml-parser does support a streaming chunk handler; v1 of this spec uses the in-memory XMLParser because Node.js fetch buffers the response anyway. The streaming-parse code path is reserved for Spec 03 / 07 if Vercel function memory pressure becomes an empirical issue. The 50K-URL urlset test (Step 3.7) verifies in-memory parse handles the spec ceiling without OOM on test infra.
• Playwright stealth fallback for WAF-blocked sites → Spec 03 (the estimator only flags transport: 'playwright'; actual Playwright execution is in the sampler).
• Pre-flight UI (showing the user the scope estimate and asking proceed/sub-tree/cancel) → Spec 11 (frontend wizard).
• Real network integration tests against Algolia.com / Microsoft Learn / Apple → Spec 13 (smoke test cluster).
• Honoring Crawl-delay at fetch time → Spec 03 (sampler) and Spec 07 (orchestrator). The estimator reads it but doesn't act on it; the walker fetches sitemaps with default 8-concurrent and accepts that some Crawl-delay: 10 Drupal sites will be slower than 8 RPS once samples are pulled.