Skip to content
Youssef Alouani
← Projects

10xJobs · live

10xJobs: job ingestion pipeline

Collection, preprocessing, type-safe LLM extraction and matching: the pipeline that turns raw job posts into matches for each user's published profile.

Problem

Job posts are written for humans: free text, every board with its own format, the same job reposted on several boards, and the facts an engineer filters on (seniority, remote or not, where, contract, salary, stack) buried in prose or missing. To match posts to profiles automatically, every post first has to become a clean, typed record, and every step has a cost: each scraped post, each LLM call and each comparison is paid for.

Constraints

  • Cost per post: scraping and LLM calls are the main costs, so nothing is processed twice and nothing is sent to an LLM that a cheaper step can handle.
  • Type safety end to end: the matcher must never see a field it cannot trust, so model output is validated against a schema, not parsed loosely.
  • No invented facts: if a post does not state the seniority or salary, the record says so; a guess would create false matches.
  • Freshness: new posts should reach users while they are still open.

Architecture

10xJobs ingestion pipelineCollection: Apify actors pull new posts from job boards on a schedule. Preprocessing: posts are normalised, cleaned and deduplicated by a hash of their normalised content. Processing: an LLM extracts the job's fields into a typed schema, validated before anything is stored, with unknowns kept as Not stated. Matching: each typed job is matched against the published profiles in its segment, and matches are shown to users.1 · CollectionApify actorsjob boards · schedulesRaw postsstored as received2 · PreprocessingNormalise & cleantext · places · titlesDeduplicatecontent hash3 · ProcessingLLM extractionstructured outputSchema validationtype-safe record4 · MatchingSegment lookupsee search optimisationProfile checksevery answer · tech stackMatchesper user

Extraction runs on Claude Sonnet 5.5, and every record is validated against the schema before it is stored.

10xJobs job detail for an AI Engineer post on LinkedIn, with role, seniority, arrangement, places, contract type, engagement and salary fields, several marked Not stated
The output of the pipeline: typed fields extracted from the post, with Not stated instead of a guess.
10xJobs tech-stack ratings from 1 (learning) to 5 (expert) for FastAPI, Claude Code and Data Pipeline
What the matcher compares against: the user's rated tech stack.

Key decisions

  1. Four stages with a stored output each. Collection, preprocessing, processing and matching each write their result, so a failure or a prompt change re-runs one stage, not the whole pipeline, and no post is scraped or sent to the LLM twice.
  2. Cheap steps before the LLM. Normalisation and deduplication run first; a post reposted on three boards costs one LLM call, not three.
  3. The LLM fills a schema; the schema decides. The model returns structured output for a typed job record, which is validated before it is stored. Output that fails validation is retried or rejected, never patched by hand in the matcher.
  4. “Not stated” is a value. Missing facts are explicit, so the matcher can treat them deliberately instead of trusting a hallucinated field.
  5. Matching is exact on what the user published. A job must satisfy every answer on the profile, and jobs needing tech outside the user’s rated stack are skipped. How that stays fast is the search optimisation.

Results

Numbers to come.

Stack and links

Deep dives