Skip to content
Youssef Alouani
← Projects

Qalqul Engine · shipped

Qalqul: classical NLP/NLU engines

Intent classification and entity extraction on language servers and Duckling, tuned per client, before LLMs made it the default.

Diagram: a Darija and French message tagged with its intent, product, date and amount, above the normalisation, classifier and Duckling pipeline

Problem

A banking bot has to know that “I want to block my card” and “my card was stolen” are the same intent, and pull out the amount, the date and the account from a sentence written in a mix of Arabic, French and Darija. In 2021 that meant classical NLP: tokenisation, classifiers and entity extractors, built and tuned for each client.

Constraints

  • Little training data: each client starts with a few examples per intent.
  • Mixed languages and scripts: Arabic, Latin-script Darija and French in the same message.
  • Structured entities: dates, amounts and numbers must come out normalised.
  • Fast: understanding runs on every message, in real time.

Architecture

Qalqul NLU pipelineA message goes through normalisation and language detection, then in parallel to an intent classifier trained per client, to entity extractors, and to Duckling for dates, amounts and numbers. The results are merged into one understanding that bots and routing use.InputMessagewidget · SDK · channelPre-processingNormalisationscripts · spellingLanguage detectionUnderstandingIntent classifierper clientEntity extractionDucklingdates · amounts · numbersConsumersBotpress botsRouting & tickets

Key decisions

  1. Language servers behind one API. Each language model runs as its own server, and bots call one understanding API.
  2. Duckling for structured entities. Dates and amounts are a solved problem; Duckling normalises them instead of custom regexes.
  3. Per-client models. Each client’s intents are trained and versioned separately, so one client’s changes never affect another.

Results

Numbers to come.

Stack and links

Deep dives