03 — Record Extraction: recordExtractor wzxhzdk:13 Helpers

What recordExtractor does

recordExtractor is a JavaScript function you write that receives the page content and returns an array of Algolia records. It runs once per matched URL.

"searchParameters": {
  "filters": "is_published:true",
  "attributesToRetrieve": ["title", "url", "section.name", "plain_body", "vote_sum"],
  "attributesToHighlight": ["title", "plain_body"],
  "hitsPerPage": 5
}

---

Built-in helpers

helpers.page() — generic page

wzxhzdk:1

Returns: { objectID, url, hostname, path, depth, fileType, contentLength, title, description, keywords, image, headers, content }

Best for: generic informational pages where you want all text content as a single record.

helpers.article() — structured article

wzxhzdk:2

Returns: { objectID, url, lang, headline, description, keywords, tags, image, authors, datePublished, dateModified, category, content }

Requires: og:type="article" or JSON-LD schema (Article, NewsArticle, BlogPosting, Report).

helpers.product() — product pages

wzxhzdk:3

Returns: { objectID, url, lang, name, sku, description, image, price, priceCurrency, category }

Requires: JSON-LD Product schema on the page.

helpers.docsearch() — documentation

wzxhzdk:4

Returns hierarchical records with lvl0–lvl6 fields. Designed for documentation navigation.

helpers.splitContentIntoRecords() — long pages

wzxhzdk:5

Splits a large page into multiple records, each under maxRecordBytes. Use when a single page would produce a record too large for Algolia (10KB attribute limit) or when you want finer-grained search.

helpers.codeSnippets() — code extraction

wzxhzdk:6

Returns: [{ content, languageClassPrefix, codeUrl }] — extracted from <pre> elements.

---

Actual RC2 Algolia index schema (live — verified 2026-04-30)

Queried from algolia-central_enterprise_ledger (app 0EXRPAXB56). 10,615 records, 9,615 with status: indexed.

Parent doc record — full field schema (crawler sets ALL of these in one pass):

wzxhzdk:7

Chunk record (is_chunk: true) — created by L2 Enrichment (NOT the crawler):

wzxhzdk:8

Key differences from the RC1 n8n/Supabase schema:

Supabase (RC1)	Algolia RC2	Note
markdown	content	renamed
doc_summary	summary	renamed
content_type	record_type + category	restructured
product_tag lowercase	product_tag proper case	"instantsearch" → "InstantSearch"
feature_tag snake_case	feature_tag proper case	"query_rules" → "Rules"
content_hash	not present	dedup via objectID instead
canonical_url	not present	handled pre-crawl
clean_char_count	not present	dropped
is_reject	ingestion_rejections table	separate table in RC1, not in Algolia
not present	solution_tag	new in RC2
not present	customer_tag	new in RC2
not present	industry_tag	new in RC2
not present	is_chunk, parent_id, chunk_index, chunk_total	chunking now in Algolia

source_type values (from live facet count):

doc: 5,475 | support: 1,732 | blog: 1,161 | other: 638 | developer: 256 | customer_story: 149 | academy: 139 | resource: 54 | changelog: 11

product_tag values (from live facet count, proper case):

"AI Search" (5,496) | "InstantSearch" (1,393) | "Analytics" (1,236) | "Recommend" (501) | "Autocomplete" (380) | "DocSearch" (37)

---

Custom extraction pattern (our use case)

For the L1 enterprise knowledge ledger crawl, we write a custom extractor that:

1. Extracts fields matching the current RC2 Algolia schema exactly
2. Sets source_type from URL routing (same logic as n8n Firehose Normalize fields node)
3. Sets status: "pending" — L2 flips to "indexed" after enrichment
4. Sets is_chunk: false — chunking is L2's job
5. Sets explicit objectID for deduplication
6. Returns [] for quality rejects (no separate rejection table in Algolia — just skip)

wzxhzdk:9

---

Tagging strategy: crawl vs enrich

The crawler (recordExtractor) should NOT attempt ML-based classification. The correct separation is:

Layer	What it does
L1 Crawler	Extract clean text, set structural fields (url, title, description, content, domain, path, crawled_at)
L2 Enrichment	Apply ML classification: category, entity_type, industry, vendor, sentiment, etc.

The crawler sets category: null as a placeholder. L2 reads the index, classifies, and updates the record.

---

ObjectID strategy — deduplication anchor

autoGenerateObjectIDs: false — we own the objectID.

Design:

wzxhzdk:10

Why URL-based: - Same URL re-crawled → same objectID → saveObjects overwrites (no duplicate) - URL change → new objectID → new record (correct — it's a different page) - Normalisation: strip UTM/session params via ignoreQueryParams before this

Alternative: use Algolia's distinct: true + attributeForDistinct: "url" for query-time dedup. We use indexing-time dedup (objectID) as primary, query-time distinct as secondary safety.

---

Handling the 750-record-per-page limit

If a page generates wzxhzdk:14 750 records (e.g., a very long article split into paragraphs), the crawl fails. Use:

wzxhzdk:11

Or use helpers.splitContentIntoRecords() with a byte budget that keeps you well under limit.

---

Filtering pages to index

Return [] to skip a page. Use this to exclude: - Navigation-only pages (no real content) - Error pages (check $("title").text().includes("404")) - Pages behind login (check for login form presence) - Pages with too-short content (wzxhzdk:15 200 chars) - Duplicate content detected via canonical tag mismatch

wzxhzdk:12