Dashboard/Crawler-Factory/Core/02-DSS.md
Crawler-Factory

Core/02-DSS.md

Domain Schema Standard (DSS) — Reference

> Extracted from 00-Plan.md §4. The DSS is our internal type system mapping content domains to schema.org types, record schemas, indices, and Algolia config tuning.

Source of truth: 00-Plan.md §4.

4. Data model — Domain Schema Standard (DSS)

The fundamental shift: instead of a single enterprise_ledger record type writing to a single index, we have content domains. Each content domain: - Maps to one or more schema.org types (e.g., marketing → BlogPosting, NewsArticle, OpinionArticle) - Has its own record schema (different fields per domain — a recipe has cookTime, a course has learningResourceType, a support article has lastReviewed) - Writes to its own index with config tuned for that domain (different searchableAttributes, customRanking, attributesForFaceting) - Is the data foundation for a future specialist agent (support-agent on support index, etc.)

4a. The Domain Schema Standard (DSS) — the core taxonomy

DSS is our internal type system. Each row maps a content domain to: schema.org parent types, record schema, index name, and Algolia config tuning. DSS is data, not code — it lives in lib/factory/dss.ts as a typed registry, easy to extend.

Content domain Index name schema.org types in scope Distinctive record fields Algolia config tuning
marketing algoliacentral_marketing BlogPosting, NewsArticle, OpinionArticle, Article, AnnouncementPage, SpecialAnnouncement headline, articleBody, datePublished, author, articleSection, keywords searchable: [headline, articleBody, keywords]; customRanking: [desc(datePublished), desc(view_count)]; facets: [articleSection, author, year]
support algoliacentral_support TechArticle (troubleshooting), FAQPage, Question, Answer, HowTo (fix-it) name, articleBody, lastReviewed, acceptedAnswer, severity, productAffected, supportLevel* searchable: [name, articleBody, acceptedAnswer]; customRanking: [desc(lastReviewed), desc(helpful_count)]; facets: [productAffected, severity, status]
education algoliacentral_education Course, LearningResource, HowTo, Guide, VideoObject (educational) name, description, learningResourceType, educationalLevel, timeRequired, hasCourseInstance, transcript (video) searchable: [name, description, articleBody, transcript]; customRanking: [desc(datePublished), desc(enrollment_count)]; facets: [educationalLevel, learningResourceType, language, duration]
technical algoliacentral_technical TechArticle, APIReference*, SoftwareSourceCode, SoftwareApplication, WebAPI name, articleBody, programmingLanguage, version, applicationCategory, codeSampleType, apiEndpoint* searchable: [name, articleBody, programmingLanguage, codeSampleType]; customRanking: [desc(version), desc(view_count)]; facets: [programmingLanguage, framework, version, apiEndpoint]
customer-stories algoliacentral_customers Article (case study), Review, Person, Organization headline, articleBody, about (Organization name), industry, companySize, region, metrics, quotes searchable: [headline, articleBody, about.name, quotes]; customRanking: [desc(datePublished), desc(prominence)]; facets: [industry, companySize, region, useCase]
product-catalog algoliacentral_products Product, Offer, Service, Brand name, description, sku, offers, brand, category, aggregateRating, image searchable: [name, description, brand]; customRanking: [desc(aggregateRating), desc(popularity)]; facets: [brand, category, priceRange, rating]
events algoliacentral_events Event, EducationEvent, BusinessEvent, Conference name, startDate, endDate, location, organizer, eventType, recordingUrl searchable: [name, description]; customRanking: [asc(startDate)] (upcoming first); facets: [eventType, location, year]
legal algoliacentral_legal DigitalDocument, Legislation, TermsOfService, PrivacyPolicy name, text, version, datePublished, jurisdiction, documentType* searchable: [name, text]; customRanking: [desc(version), desc(datePublished)]; facets: [jurisdiction, documentType]
social algoliacentral_social VideoObject (YouTube), Comment, Conversation, SocialMediaPosting* text, transcript, uploadDate, channel, viewCount, commentCount, platform, sentiment searchable: [text, transcript]; customRanking: [desc(uploadDate), desc(engagement)]; facets: [platform, channel, sentiment]

Asterisked fields are not in schema.org — they're our extensions, defined in DSS as dss: namespace fields.

The DSS is extensible at runtime: a user (or future automation) can add a new content domain by adding a new entry to the registry. No code changes needed if the new domain reuses an existing schema.org type set.

4b. Detection cascade — REVISED based on Web Almanac 2024 + 14-site empirical audit

Cascade order is CMS-first, not schema.org-first. Schema.org is a confirmer/refiner, not the foundation. See §3b for the empirical justification and §14 for the source data.

# Method What it inspects Expected hit rate Confidence when hit
1 CMS fingerprint URL paths (/wp-content/, /content/dam/, /sites/default/files/), HTML class signatures (wp-block-*, region-*, data-cmp-*, data-aura-*), robots.txt patterns (Drupal's Crawl-delay: 10), known JSON endpoints (/wp-json/wp/v2/posts, Shopify product API) 51% of all sites 0.80–0.90
2 URL pattern /blog/, /customers/, /docs/, /help/, /support/, /courses/, /products/, /{YYYY}/{MM}/, etc. (full table in §3b) 100% (always available) 0.50–0.75
3 JSON-LD parser <script type="application/ld+json"> blocks. Walks @type chain. Matches against DSS schema.org types. 41% of pages globally; ~7% with content-classifying types 0.85–0.95
4 OpenGraph + Twitter Card og:type (article/website/video/product), og:article:section, twitter:card 64% of pages 0.55–0.70
5 Microdata + RDFa itemtype="https://schema.org/X"; RDFa typeof 26% / 66% — declining 0.40–0.65
6 Date + author regex "Published: YYYY-MM-DD", "by Author Name", <time datetime=...> 99% of articles 0.40–0.60 (article-family only)
7 Semantic HTML heuristics <article>, <main>, <header>, breadcrumbs, code-block density, table-of-contents shape ~5–8% (low) 0.45–0.60
8 LLM classification Cheerio-extracted text (first 4KB) + DSS table → "which content domain best fits?" Cheap; ~$0.001 per call. 100% (always available) 0.50–0.85

Aggregation rules: - Each layer outputs {contentDomain, confidence} or null. - Final confidence = max-confidence layer + boost when multiple layers AGREE on the same domain (each agreement adds 0.05, capped at 0.95). - LLM (layer 8) only fires when combined confidence from layers 1–7 is < 0.55. - UI shows the contributing layers per pathGroup so the user can see WHY a classification was made and override if needed.

Confidence thresholds: - ≥ 0.85: green — auto-accept, recommend "Configure" - 0.65–0.84: yellow — show confidence, recommend manual review - < 0.65: amber — flag for manual override; default action is LLM tiebreak + manual confirmation

4c. Record schemas — one zod schema per content domain

Each content domain has its own zod schema. Common base + domain-specific extension:

// lib/factory/types.ts

const RecordBaseSchema = z.object({
  objectID: z.string(),                          // hash of url
  url: z.string().url(),
  content_domain: ContentDomainEnum,             // enum from DSS keys
  schema_org_type: z.string(),                   // raw value, e.g. "BlogPosting"
  detected_via: z.enum(['json-ld','microdata','rdfa','opengraph','heuristic','url','llm']),
  detection_confidence: z.number().min(0).max(1),
  language_code: z.string().default('en'),
  source_url_root: z.string(),                   // e.g., "https://www.algolia.com"
  crawled_at: z.number(),
  is_chunk: z.literal(false),
  status: z.literal('indexed'),
});

export const MarketingRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('marketing'),
  headline: z.string(),
  articleBody: z.string().min(200),
  datePublished: z.string().optional(),
  dateModified: z.string().optional(),
  author: z.array(z.string()).optional(),
  articleSection: z.string().optional(),
  keywords: z.array(z.string()).optional(),
  image: z.string().url().optional(),
});

export const SupportRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('support'),
  name: z.string(),
  articleBody: z.string().min(200),
  acceptedAnswer: z.string().optional(),
  lastReviewed: z.string().optional(),
  productAffected: z.array(z.string()).optional(),
  severity: z.enum(['low','medium','high','critical']).optional(),
  supportLevel: z.enum(['self-serve','assisted','enterprise']).optional(),
});

// ... one schema per content domain. Discriminated union for inserts:
export const FactoryRecordSchema = z.discriminatedUnion('content_domain', [
  MarketingRecordSchema,
  SupportRecordSchema,
  EducationRecordSchema,
  TechnicalRecordSchema,
  CustomerStoryRecordSchema,
  ProductRecordSchema,
  EventRecordSchema,
  LegalRecordSchema,
  SocialRecordSchema,
]);

The legacy CrawlerRecord (packages/crawl/src/crawl/models.py) is kept as-is for the existing algolia-central_enterprise_ledger index — it's not deleted. New crawlers from the factory write to per-domain indices using the new schemas. Migration is opt-in.

4d. Sharded session storage (no record-size cap)

A single Algolia record has a 100KB limit. For sites with 100K+ URLs, one giant session record won't fit. Shard:

Index: algoliacentral_factory_sessions  (single index, multiple record_type)
  ├─ record_type: 'session'
  │    objectID: &lt;sessionId&gt;
  │    main session metadata (status, domain, categories[], blueprints[])
  │    NO discoveredUrls inline — only counts and shard pointers
  │
  ├─ record_type: 'url_shard'
  │    objectID: &lt;sessionId&gt;:urls:&lt;shardIndex&gt;
  │    parent_id: &lt;sessionId&gt;
  │    urls: [up to 1000 URLs]
  │    shardIndex: 0..N
  │
  ├─ record_type: 'sample'
  │    objectID: &lt;sessionId&gt;:sample:&lt;urlHash&gt;
  │    parent_id: &lt;sessionId&gt;
  │    pathGroupId: &lt;id&gt;
  │    url, html (truncated to 80KB), structure_analysis, detection_result
  │
  └─ record_type: 'log'
       objectID: &lt;sessionId&gt;:log:&lt;ts&gt;
       parent_id: &lt;sessionId&gt;
       ts, level, message

attributesForFaceting: [filterOnly(record_type), filterOnly(parent_id)]. Reads filter on parent_id:&lt;sessionId&gt; then split by record_type.

The factory streams URLs into shards as discovery proceeds — no waiting for full discovery, no memory pressure.

4e. Factory blueprints — agent-ready metadata

Every successfully created crawler also writes a blueprint to a separate index. This is the metadata a future orchestrator/specialist-agent scaffolder will read.

Index: algoliacentral_factory_blueprints
  objectID: &lt;crawlerId&gt;
  content_domain: 'support'
  index_name: 'algoliacentral_support'
  source_domain: 'algolia.com'
  path_groups: ['/help/*', '/hc/*']
  schema_org_types: ['TechArticle', 'FAQPage']
  recordExtractor_path: 'crawler-configs/algolia.com/support-help.js'
  algolia_config: { searchable: [...], customRanking: [...], facets: [...] }
  agent_slot: null   // populated when a specialist agent is later attached
  created_at: ts
  status: 'live' | 'paused' | 'archived'

This index is the link between the crawl world and the agent world. When we build the orchestrator + specialist agents (future spike), it reads from algoliacentral_factory_blueprints to know what specialists exist and what domains they cover.

4f. Session main record (zod, slimmed for 100KB compliance)

export const FactorySessionSchema = z.object({
  objectID: z.string(),
  record_type: z.literal('session'),
  status: z.enum(['discovering','categorized','configuring','crawling','done','error']),
  source_domain: z.string(),
  source_url_root: z.string(),
  createdAt: z.number(),
  updatedAt: z.number(),

  discovery: z.object({
    sitemapUrls: z.array(z.string()),
    urlCount: z.number(),                       // total discovered (no cap)
    urlShardCount: z.number(),                  // pointer to url_shard records
    schemaOrgCoverage: z.number().min(0).max(1), // % of sample pages with JSON-LD
  }),

  pathGroups: z.array(z.object({
    id: z.string(),                             // slug, e.g., "blog"
    pattern: z.string(),                        // /blog/*
    urlCount: z.number(),
    sampleUrls: z.array(z.string()).max(10),
    detectedDomain: ContentDomainEnum,          // from DSS classification
    detectionConfidence: z.number(),
    detectionMethod: z.enum(['json-ld','microdata','rdfa','opengraph','heuristic','url','llm']),
    selected: z.boolean().default(false),
    blueprint: z.object({                       // populated when configured
      indexName: z.string(),
      recordExtractor: z.string(),
      crawlerConfig: z.object({
        renderJavaScript: z.boolean(),
        rateLimit: z.number(),
        pathsToMatch: z.array(z.string()),
      }),
      sandboxResults: z.array(z.unknown()).optional(),
      realTestResults: z.array(z.unknown()).optional(),
      crawlerId: z.string().optional(),
      reindexTaskId: z.string().optional(),
    }).optional(),
  })),
});

4g. API contract (revised)

GET    /api/factory/session?id=...           → FactorySession (main + denormalized url shard count + log tail)
PUT    /api/factory/session                  → upsert main record only
POST   /api/factory/discover                 → SSE: {type:'log'|'urls'|'pathGroup'|'classified'|'done'}
                                                'urls' streams batches of 1000 → factory writes url_shard records
POST   /api/factory/sample                   → fetches &amp; writes sample records, returns sample IDs
POST   /api/factory/classify                 → runs detection cascade on a pathGroup, returns content domain
POST   /api/factory/generate-extractor       → DSS-aware extractor + config generation
POST   /api/factory/test-sandbox             → runs extractor against sample records
POST   /api/factory/test-real                → real Algolia crawl_urls test
POST   /api/factory/create-crawler           → creates index (if missing) + crawler + blueprint
GET    /api/factory/crawl-progress?id=...    → SSE: per-crawler stats
GET    /api/factory/blueprints               → list all blueprints (for future agent factory)