Dashboard/Crawler-Factory/Research/04-Web-Reality-Catalog.md
Crawler-Factory

Research/04-Web-Reality-Catalog.md

Web Reality Catalog — Empirical Grounding

                        ┌──────────────────────────────────────────┐
                        │            INGESTION TASK                  │
                        │  (the wiring + the trigger/schedule)       │
                        └──────────────────────────────────────────┘
                                          │
          ┌───────────────────────────────┼───────────────────────────────┐
          ▼                                ▼                                ▼
   ┌─────────────┐                 ┌────────────────┐                ┌──────────────┐
   │   SOURCE    │  ──fetch───►    │ TRANSFORMATION │  ──reshape──►  │ DESTINATION  │
   │ (where data │                 │ (optional code/ │                │ (the Algolia │
   │  comes from)│                 │  no-code step)  │                │  index)      │
   └─────────────┘                 └────────────────┘                └──────────────┘
          ▲                                                                  │
          │                                                                  ▼
   ┌──────────────┐                                              indexed records
   │AUTHENTICATION│  holds the credentials (API tokens, OAuth,
   │ (credentials)│  basic auth) that a Source/Destination uses
   └──────────────┘

   Each execution of a Task is a RUN. Each per-record outcome inside a run is an EVENT.
   RUNs + EVENTs are the observability layer (did it work, what failed, why).
Extracted from 00-Plan.md §14. Hard data from Web Almanac 2024 and a 60-site empirical audit conducted during planning. The cascade order, fallback strategies, and risk register all derive from these findings.

Source of truth: 00-Plan.md §14.

14. Web Reality Catalog — empirical grounding

This section anchors the plan in real-world data, not assumptions. All numbers below are sourced from HTTP Archive's Web Almanac 2024 (a census of ~16M websites) and from a 14-site empirical audit conducted during planning. The cascade order, fallback strategies, and risk register all derive from these findings.

14a. Structured-data adoption (Web Almanac 2024)

Format Adoption (mobile pages) Trend
OpenGraph 64% Stable, ubiquitous on social-aware sites
RDFa 66% Declining; mostly legacy WordPress/Drupal
JSON-LD 41% Fastest growing (was 34% in 2022)
Microdata 26% Niche; mostly older WP/Drupal templates
None of the above ~37% Pure HTML, no structured data

Top JSON-LD @type values on mobile pages: - WebSite — 12.7% - Organization — 7.2% - BreadcrumbList — 5.7% - LocalBusiness — 4.0% - ItemList — 2.4% - BlogPosting — 1.4% - Product — 0.8% - Article — 0.18% - FAQPage, HowTo, Course, Recipe, Review — each 

curl -X POST "https://data.us.algolia.com/1/sources" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "json",
    "name": "Product feed (JSON)",
    "input": {
      "url": "https://example.com/products.json"
    },
    "authenticationID": "6c02aeb1-775e-418e-870b-1faccd4b2c0f"
  }'
0.5%

Key implication: Even when JSON-LD is present, it usually only declares structural types (WebSite, Organization, BreadcrumbList) — not content-classifying types (BlogPosting, Article, Product). Content-class JSON-LD is on roughly 7% of pages globally. A factory that depends on JSON-LD for content-domain classification will fail on 

// tool: mcp__algolia__createSource
{
  "type": "json",
  "name": "Product feed (JSON)",
  "input": { "url": "https://example.com/products.json" },
  "authenticationID": "6c02aeb1-775e-418e-870b-1faccd4b2c0f"
}
90% of the web.

Source: 

curl -X POST "https://data.us.algolia.com/1/destinations" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "search",
    "name": "Products index destination",
    "input": { "indexName": "products" }
  }'
https://almanac.httparchive.org/en/2024/structured-data
curl -X POST "https://data.us.algolia.com/2/tasks" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceID": "6c02aeb1-775e-418e-870b-1faccd4b2c0f",
    "destinationID": "c1f8c5 a8-...-...",
    "action": "replace",
    "cron": "0 2 * * *",
    "enabled": true
  }'

14b. Sitemap 
// tool: mcp__algolia__createTask
{
  "sourceID": "6c02aeb1-775e-418e-870b-1faccd4b2c0f",
  "destinationID": "c1f8c5a8-...",
  "action": "replace",
  "cron": "0 2 * * *",
  "enabled": true
}
robots.txt adoption (Web Almanac 2024)

  • robots.txt presence: 83.9% of sites (mobile)
  • Sitemap protocol limits: 50,000 URLs OR 50 MB per file (uncompressed). Gzip permitted.
  • Sitemap-index nesting: Per spec, only ONE level of nesting is allowed — a sitemap-index can point to urlsets, but not to other sitemap-indexes. (Some sites violate this; treat as best-effort with cycle prevention.)
  • Sitemap discoverability: Web Almanac 2024 doesn't publish %, but inferred to be < 60% based on robots.txt-vs-sitemap correlation.

Plan impact: The walker must handle gzipped sitemaps (.xml.gz) natively. The plan's /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml fallback list is correct. &lt;lastmod&gt; is reliable on enterprise/government sites; &lt;changefreq&gt; and &lt;priority&gt; are nearly useless (rarely populated).

Sources: 

curl -X POST "https://data.us.algolia.com/2/tasks/{taskID}/run" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY"
https://almanac.httparchive.org/en/2024/seo
// tool: mcp__algolia__runTask
{ "taskID": "76ab4c2a-ce17-4b92-..." }
, 
curl -X POST "https://data.us.algolia.com/2/tasks/{taskID}/push" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "addObject",
    "records": [
      { "objectID": "prod-123", "name": "Yoga pants", "price": 49 }
    ]
  }'
https://www.sitemaps.org/protocol.html
// tool: mcp__algolia__pushTask
{
  "taskID": "76ab4c2a-ce17-4b92-...",
  "action": "addObject",
  "records": [ { "objectID": "prod-123", "name": "Yoga pants", "price": 49 } ]
}

14c. Semantic HTML adoption (Web Almanac 2024)

Element Adoption
&lt;h1&gt; ~70%
&lt;h2&gt; ~71%
&lt;h3&gt; ~59%
&lt;article&gt; NOT REPORTED (estimated <10%; 14-site sample showed ~5%)
&lt;main&gt; NOT REPORTED (estimated <10%)
&lt;time&gt; with datetime NOT REPORTED (estimated <10%)

Plan impact: Semantic HTML5 elements (&lt;article&gt;, &lt;main&gt;, &lt;time&gt;) are unreliable signals. The plan's earlier reliance on these for heuristic classification is wrong. Strip them to layer 7 (low confidence). Use heading hierarchy (&lt;h1&gt;/&lt;h2&gt;/&lt;h3&gt;) which IS widely adopted.

Source: 

curl -X POST "https://data.us.algolia.com/1/push/products" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "action": "addObject", "records": [ { "objectID": "p1", "name": "Item" } ] }'
https://almanac.httparchive.org/en/2024/markup
curl -X POST "https://data.us.algolia.com/1/transformations/try" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "code": "return { ...record, price: record.price_cents / 100 };",
    "sampleRecord": { "objectID": "p1", "price_cents": 4900 }
  }'

14d. CMS market share (Web Almanac 2024)

  • 51% of all sites use a CMS (up from 46% in 2022)
  • WordPress: 35.6% of CMS sites = 18.2% of all sites
  • Wix: 2.8% / Squarespace: 1.5% / Joomla: 1.5% / Drupal: 1.2% / Shopify: ~1% / others: <1%

Plan impact: Detecting WordPress alone covers ~18% of the entire web. Adding Drupal, AEM, Shopify, Wix, Squarespace pushes coverage to ~30% via CMS fingerprint. This is more reliable than schema.org for these sites.

Per-CMS schema.org defaults: - WordPress: minimal — relies on plugins (Yoast, RankMath) for structured data. Plugin adoption ~35% of WP sites. - Shopify: ships Product + Organization + BreadcrumbList by default. - Wix/Squarespace: proprietary auto-generation, opaque, often non-standard. - AEM (Adobe): no defaults; structured data is bespoke per implementation. - Drupal: minimal defaults; site-specific.

Source: 

// tool: mcp__algolia__tryTransformation
{
  "code": "return { ...record, price: record.price_cents / 100 };",
  "sampleRecord": { "objectID": "p1", "price_cents": 4900 }
}
https://almanac.httparchive.org/en/2024/cms
curl -X POST "https://data.us.algolia.com/1/authentications" \
  -H "x-algolia-application-id: $ALGOLIA_APP_ID" \
  -H "x-algolia-api-key: $ALGOLIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "oauth",
    "name": "Shopify OAuth",
    "input": {
      "url": "https://my-store.myshopify.com/admin/oauth/access_token",
      "client_id": "REPLACE_ME",
      "client_secret": "REPLACE_ME"
    }
  }'

14e. JS framework / SPA adoption (Web Almanac 2024)

  • jQuery: 74% of pages (entrenched, mostly WordPress legacy)
  • React: 10% (slight growth from 8% in 2023)
  • All modern frameworks combined: ~20% of pages

Plan impact: ~80% of pages are server-rendered or static. Cheerio handles them. ~20% need JS execution. SPA detection should err on the side of "static-first, escalate to Playwright on signals" — wasting Playwright cycles on every page is expensive.

Source: <https://almanac.httparchive.org/en/2024/javascript>

14f. Cross-vertical site coverage validated (~59 sites across 13 verticals)

This plan was validated against an extensive cross-section of real sites during the planning session. Each site was probed for robots.txt, sitemap structure, JSON-LD, OpenGraph, CMS signature, WAF posture, and URL conventions.

Vertical Sites validated WAF-block rate JSON-LD coverage §16 row
Enterprise B2B Salesforce, Adobe, Oracle, HP, BMW, Mercedes 6/6 unverifiable (blocked) 16c
CMS / generalist TechCrunch, WordPress.org, Shopify, Drupal.org, Wikipedia, Guardian 0/6 1/6 (Wikipedia partial) 16a, 16b, 16d, 16h
Government USA.gov, GOV.UK 0/2 0/2 16b, 16j
Multi-brand corporates LVMH, Diageo, Unilever, P&G 2/4 rare 16k
Massive scale tech Microsoft, AWS, Apple, Algolia, learn.microsoft.com 0/5 partial 16l
SaaS B2B / API-first commercetools, BigCommerce, OpenText 0/3 rare 16m
Education / online learning Coursera, Khan Academy, MIT OCW, edX 0/4 3/4 (Course schema) (sub-row of 16m)
SaaS dev tools Stripe, Notion, Atlassian, Datadog 0/4 1/4 (Datadog only) (sub-row of 16m)
Healthcare Mayo Clinic, WebMD, Cleveland Clinic 1/3 0/3 16h-extended
News / media Medium, NYT, BBC, Reuters 3/4 1/4 (Medium NewsArticle) 16a-extended
Community / social Stack Overflow, Reddit, Hacker News 2/3 0/3 16h-extended
Non-profit / docs Mozilla, Adobe Helpx 1/2 0/2 16h-extended
Manufacturing 3M, Caterpillar, Siemens 2/3 1/3 (Siemens) 16c-extended
E-commerce retail Walmart, Etsy, eBay, Wayfair, Best Buy 5/5 0/5 (verifiable) 16o
DTC retail commerce Nike, Restoration Hardware, Walgreens, Dell, LL Bean, Oriental Trading, SKIMS, Havertys 5/8 0/8 of accessible 16o
Furniture vertical (subset) Restoration Hardware, Havertys 2/2 (full WAF) unverifiable 16o (furniture sub-schema in v2)

Aggregate WAF rate: ~40% of sites tested fully WAF-block plain fetch from a Vercel-class datacenter IP. Higher concentrations in: enterprise B2B (100%), premium DTC retail (>50%), retail (Walmart/Etsy/Wayfair-class). Lower in: government (0%), open-source CMSes (0%), education with permissive Allow: / (MIT OCW), modern SaaS docs.

Aggregate JSON-LD usefulness for content classification: when content-class @types are needed (Article, Product, Course, MedicalCondition), real-world coverage is <25% across the verticals tested. Education (Course schema) is the single domain where JSON-LD is reliable enough to lead the cascade. Everywhere else: CMS-fingerprint + URL-pattern leads, JSON-LD is a confirmer/enricher.

Universal patterns (high reliability): - robots.txt: present on 95%+ - Sitemap discoverable (via robots.txt or /sitemap.xml fallback): ~70% (lower than expected; many sites WAF-block sitemaps too) - URL path conventions for content domain: ~95% (the universal floor for classification) - CMS signature detectability when CMS is present: ~90%

14g. 14-site initial audit findings (planning session — round 1)

Sites successfully audited (CMS / generalist)

TechCrunch, WordPress.org, Shopify, Drupal.org, USA.gov, GOV.UK, Wikipedia, The Guardian.

Findings: - 0 of 8 had JSON-LD on the listing/index pages tested. - Wikipedia had partial JSON-LD (WebPage, Article, ImageObject) only on actual article pages. - All 8 had robots.txt with at least one Sitemap directive. - All 8 had a discoverable sitemap (variants: top-level urlset, sitemap-index with sub-sitemaps; sub-sitemap counts ranged from 1 to 1,450). - Drupal-based sites (Drupal.org, USA.gov, GOV.UK partially) all enforced Crawl-delay: 10. - WordPress sites (TechCrunch, WordPress.org) had /wp-content/, /wp-json/, wp-block-* classes. Nearly identical signatures. - &lt;article&gt; tags appeared on roughly 1 in 8 sites (low). Date+author was always recoverable from text + visible page elements.

Sites blocked by WAF (enterprise B2B)

Salesforce, Adobe, Oracle, HP (partial), BMW, Mercedes-Benz.

Findings: - Plain fetch from Vercel-class IPs returned 403 / timeout / CAPTCHA on most attempts. - HP's homepage WAS reachable; revealed Adobe AEM signatures (/content/dam/, /digitnav/menu/). - Conclusion: enterprise B2B requires Playwright stealth mode for the discovery+sample phases. The Algolia Crawler itself runs from a different IP range that many enterprise WAFs already allowlist for SEO — so the production crawler usually works once configured.

Plan impact: Playwright fallback moved from v2 to v1. The factory must detect WAF blocks early (repeated 403/timeout patterns) and surface the option.

14g. Detection cascade hit rate (synthesized from above)

For an arbitrary content page in 2026:

Cumulative cascade through layer Expected coverage
1. CMS fingerprint 51%
1+2. CMS + URL pattern ~95% (URL is universal floor)
1+2+3. + JSON-LD when present ~95% (URL already covers; JSON-LD enriches confidence)
1+2+3+4. + OG ~95% with refined confidence
1+2+3+4+5+6. + microdata + date/author regex ~98%
All layers + LLM fallback 100%

Key insight: URL pattern matching is the universal floor. CMS fingerprint is the most-reliable single signal. Schema.org is a confidence booster, not a foundation. The cascade reaches >95% useful coverage WITHOUT relying on JSON-LD.