Patent · Voice Assistants · Deployed at national scale
Memory-Network NLU for Tmall Genie
Every command a voice assistant hears must be handed to exactly one service. I replaced the brittle rule-versus-model cascade that made this decision with a memory-network architecture that learns the trade-off per utterance — and records it. Production routing accuracy went from about 65% to about 85%.
A smart speaker has dozens of separate services behind it — music, shopping, lights, weather, general questions. Something has to decide which one gets each request. Get that wrong and nothing downstream can save you: the user asked for a song and the speaker went shopping.
The industry did this with hand-written matching rules layered on top of a machine-learning model. As more teams added rules, the rules started contradicting each other, and nobody could tell whether a failure came from a bad rule or a bad model. Accuracy stalled near 65% — roughly one request in three going to the wrong place.
My design stops treating the two as a chain. It keeps the rules as a memory the model can consult, and learns — for each individual request — how much that memory should count. Accuracy reached about 85%, and every decision now carries a record of what drove it.
One detail earns an extra sentence, because the granted claim turns on it. Whenever the system consults that memory, it also consults a permanent entry for every service it can route to — one learned summary of what music requests look like, one for questions, one for lights, and so on. They are always in the room. Without them, a request that happens to trip exactly one rule would have that rule counted as a certainty, not because it fitted well but because there was nothing else to weigh it against. With them, a rule that fires on a request it has misread can be outvoted by the service the meaning actually points to.
The decision that cannot be undone
Domain classification is the traffic controller of a voice assistant. It sits between speech recognition and everything the product can actually do, and it commits to one answer per request.
Before this work, Tmall Genie did what the industry did: rules first, then model. Each product team hand-wrote matching templates for its own domain; a template match won outright, and the learned classifier only ran when nothing matched. At dozens of domains and hundreds of templates owned by different teams, every fix broke somebody else’s cases, and when a request went wrong there was no way to tell whether a rule had misfired or the model had. Accuracy sat at roughly 65%.
The uncomfortable part is that a cascade gives you nothing to tune. A template either matches or it does not, so the architecture offers exactly one decision — which component outranks the other — and it is taken once, in the wiring, for every request the system will ever handle. Try both orderings.
Interactive — try both orderings
There is no dial here. Only which component wins.
The industry standard was rules first, then model: hand-written templates are checked, and a template that matches routes the request outright — the model never runs. There is nothing to tune. A template either matches or it does not, so the only decision the architecture offers is which component outranks the other, taken once and applied to every request the system will ever see. Try both orderings on these seven real request types.
This interactive needs JavaScript. In short: a cascade offers only two wirings — templates outrank the model, or the model outranks the templates — and each routes five of these seven correctly while failing on a different pair. Some requests need the template overruled, others need it obeyed, and a fixed ordering cannot be both. The patented memory network weighs the two per utterance instead.
Illustrative model of the mechanism, not production data — seven cases chosen so the trade-off is visible at a glance instead of across millions. The production result of replacing the cascade with learned fusion is on the chart further down: ~65% → ~85% domain-classification accuracy, and 35% fewer routing errors from the fusion layer alone.
Neither ordering wins, and the reason is worth stating plainly: a match tells you a pattern fired, not that it understood. 周杰伦的父亲 and 周杰伦的青花瓷 are the same template on two entities that both genuinely exist — 父亲 really is a song title — and one is a request for music while the other is a question about a person. Put the rules first and the question is answered with a song. Put the model first and you lose the cases where a template is the only thing that survives a transcription slip. The requests need opposite decisions, and a fixed ordering can only make one of them.
What the patent does differently
The invention is to stop chaining the two sources of evidence and start weighing them. Rule and template knowledge is kept as an addressable memory of template embeddings. The neural model’s semantic vector becomes a query against that memory: each matched template’s similarity to the query is normalised into a weight, the templates are summed in those proportions into a single memory vector, and that vector is concatenated with the semantic vector and classified.
Two consequences follow, and both matter:
Accuracy
The rule-versus-model weighting is learned from context rather than fixed by a human, so utterances that need the rules overruled and utterances that need the rules obeyed can both be handled by one system.
Diagnosability
The attention weights are an output, not an internal detail. An operator seeing a bad route can read which evidence drove it — a rule conflict or a model error — instead of guessing.
The detail claim 1 turns on
Normalising the similarities is what makes the weights readable — they sum to one, so each is the share of the decision that template accounts for. It also creates a failure that is easy to miss. A weight is a relative quantity: similarity divided by the sum of all similarities in the read. How many templates a request pulls into that read is not fixed — some utterances match a dozen, some match one. And when one is all that matched, the sum is that template, so its weight is 1.00 no matter how badly it fits. Normalised attention has no way to express “nothing here really matches”; the arithmetic will not produce it.
This is not a corner case, and it does not need bad data to happen. Jay Chou’s father is a question about a person; Jay Chou’s Blue and White Porcelain is a request for a song. Both are @{artist}'s @{song}, both fire the same music template on two entities that genuinely exist in the catalogue — 父亲 really is a song title. The template match is identical. Nothing at that layer can separate them.
So the independent claim requires that the matched set is never just what happened to match. It must “include at least a preset basic template.” In production these were one per domain class, each a learned representation of what that class of request looks like, read for every utterance regardless of what else matched. They do two jobs at once. They occupy the denominator, so a template’s weight has to be earned against standing competition rather than awarded by default. And because each one stands for a class, the weight that flows to them is not merely withheld from the template — it goes to the service the semantics actually point at. On Jay Chou’s father the music template scores 0.10 and the question-answering class scores 0.70, and the read comes out as a question. The patent states the purpose in its own terms: including them means “the influence of dirty data can be reduced, and more accurate interaction results can be obtained even if the dirty data is hit by the search match.”
Interactive — the mechanism in claim 1
Why one weak match becomes total certainty
Attention weights are normalised: each template's weight is its similarity divided by the sum of all of them. So a weight does not measure how well a template fits — it measures how well it fits compared with everything else in the read. Take the everything else away and a poor match is indistinguishable from a perfect one. The basic templates are what stays in the read: one per domain, each a learned representation of that class. Switch them off below and watch an unchanged similarity score turn into total confidence.
This interactive needs JavaScript. In short: if one template matches an utterance, normalised attention must assign it all of the weight however badly it fits, because there is nothing else in the denominator. CN112002313B requires the matched set to always include preset basic templates — in production, one learned representation per domain class — so the read is never a single template, and a class the semantics actually point at can outvote a template that merely fired. On "Jay Chou's father" the music template scores 0.10 against a question-answering class at 0.70, and takes 10% of the read instead of 100%.
Illustrative model of the mechanism. The weights on the first tab are the patent's own worked example — CN112002313B, Figure 2, where three basic templates and the matched template score 0.10, 0.70, 0.10 and 0.10 against the semantic vector — applied here to a request that shows the failure in production terms; the patent's own illustration uses garbled catalogue entities instead. A real read also contains every template that matched, not just one. The patent states the purpose directly: including basic templates means "the influence of dirty data can be reduced, and more accurate interaction results can be obtained even if the dirty data is hit by the search match."
Two further mechanisms shipped alongside it: open-set handling, so unsupported requests are declined instead of forced into the nearest domain, and an unclear-expression model that reads raw acoustic features rather than transcribed text, catching cases where meaningless speech produces plausible-looking text.
What changed in production
Commercial consequence
A 65% router is not a shippable consumer product. One request in three failing is the difference between a device people use daily and a device that ends up in a drawer. The accuracy work was the precondition for scale, not a refinement after it.
The architecture stayed in production as Tmall Genie’s routing layer from 2018 through 2020. In 2019, midway through that period, Tmall Genie shipped 15.61 million units in China and took the number-one position in the market, growing 87.9% year on year (IDC). Globally, Canalys recorded 25.7 million cumulative units across 2018–2019. By 2023 — two years after I left the company — Alibaba Cloud was publicly describing the platform as serving 40 million households and 350 million connectable IoT devices, across 1,000+ device manufacturers and 1,600+ brands.
The platform’s corporate entity received the 9th Wu Wenjun AI Science and Technology Progress Award in 2019, during the period this architecture was in production.
Timeline
- 2017Placed on the founding team of Tmall Genie, Alibaba's smart-speaker platform.
- Mid-2018 → mid-2019Open-set handling, the memory-network classifier, and the unclear-expression model deployed in sequence to live traffic.
- 9 May 2019Patent CN112002313B filed, first inventor.
- 2019Tmall Genie becomes China's number-one smart speaker: 15.61 million units, up 87.9% year on year.
- 2020Still the production routing layer. This is as far as my direct knowledge extends; I cannot speak to what replaced it, if anything.
- 7 Apr 2023Patent granted after examination for novelty and non-obviousness.
- 2023–2024Cited as prior art by Ping An Life Insurance and Tianjin University in unrelated fields.
The underlying evidence
- PatentCN112002313B — Interaction Method, Apparatus, Speaker, Electronic Device and Storage Medium
- FamilyWO2020224570A1 — international application in the same family
- Cited byCN112559687B — Ping An Life Insurance, financial-services question identification
- Cited byCN114036823B — Tianjin University, power-transformer load forecasting
- RelatedIndicator Loss for context-aware ASR — the other foundational layer every Tmall Genie command passed through