10xJobs · live
10xJobs: job ingestion pipeline
Collection, preprocessing, type-safe LLM extraction and matching: the pipeline that turns raw job posts into matches for each user's published profile.
Problem
Job posts are written for humans: free text, every board with its own format, the same job reposted on several boards, and the facts an engineer filters on (seniority, remote or not, where, contract, salary, stack) buried in prose or missing. To match posts to profiles automatically, every post first has to become a clean, typed record, and every step has a cost: each scraped post, each LLM call and each comparison is paid for.
Constraints
- Cost per post: scraping and LLM calls are the main costs, so nothing is processed twice and nothing is sent to an LLM that a cheaper step can handle.
- Type safety end to end: the matcher must never see a field it cannot trust, so model output is validated against a schema, not parsed loosely.
- No invented facts: if a post does not state the seniority or salary, the record says so; a guess would create false matches.
- Freshness: new posts should reach users while they are still open.
Architecture
Extraction runs on Claude Sonnet 5.5, and every record is validated against the schema before it is stored.


Key decisions
- Four stages with a stored output each. Collection, preprocessing, processing and matching each write their result, so a failure or a prompt change re-runs one stage, not the whole pipeline, and no post is scraped or sent to the LLM twice.
- Cheap steps before the LLM. Normalisation and deduplication run first; a post reposted on three boards costs one LLM call, not three.
- The LLM fills a schema; the schema decides. The model returns structured output for a typed job record, which is validated before it is stored. Output that fails validation is retried or rejected, never patched by hand in the matcher.
- “Not stated” is a value. Missing facts are explicit, so the matcher can treat them deliberately instead of trusting a hallucinated field.
- Matching is exact on what the user published. A job must satisfy every answer on the profile, and jobs needing tech outside the user’s rated stack are skipped. How that stays fast is the search optimisation.
Results
Numbers to come.
Stack and links
Deep dives
- 10xJobs: search optimisation with segmentation: Matching jobs to profiles like a hash table: both are reduced to segment keys, so each job is only compared with the profiles in its bucket instead of every profile.