Patent · Speech Recognition · Deployed at national scale

Indicator Loss for Context-Aware ASR

A voice assistant has to hear names it was never trained on — your contacts, your devices, the song you played last night. I invented an explicit per-step supervision signal that makes that personalisation both instantly updatable and, for the first time, auditable when it goes wrong. Negative user feedback fell 32.9%.

Patent
CN113808593B
Role
First inventor
Filed
16 Jun 2020
Granted
3 Jun 2025
Assignee
Alibaba Group
In one minute, without the jargon

Speech recognisers learn from enormous amounts of general speech. They have never heard your friend's name, your lamp's name, or an obscure song title — and those are exactly the words people say to a smart speaker. So systems keep a per-user list of important phrases and try to steer recognition toward it.

The problem is how you steer. The industry standard, Google's CLAS, leaves that decision implicit: the decoder attends to the list and works it out for itself, so what you can read back afterwards is a soft attention weight rather than a decision. When a phrase is missed, that is not enough to tell you whether the steering was applied and wrong, or never applied at all. And the older generation of systems needed a full language-model rebuild — weeks of work — every time the vocabulary changed.

My design supervises the decision directly: at every step of decoding, for every phrase on the list, the model is trained on an explicit yes/no target saying whether that phrase applies right now. The list can then be swapped at runtime with no retraining, and when something goes wrong an operator can see precisely where the steering did or did not fire.

−32.9%
Negative user feedback versus the traditional stack
22.38% → 15.01% on live traffic
78 → 87%
Multi-turn task completion
+11.17% relative
weeks → live
Time to add a new phrase to the vocabulary
Language-model retrain cycle eliminated
−9.86 → −1.24
Accuracy gap between single-turn and multi-turn, in points
Sep 2019 to May 2020
2023
Alibaba Cloud still describing Tmall Genie's production speech stack in these terms
Three years after deployment, two after I left

A failure nobody could see

I was not assigned this problem. I found it by reading daily user-session logs, where the same pattern kept appearing: a request would fail, the user would repeat it, and it would fail again. On 1 January 2020 one user asked for the song 山谷里的思念 — “Longing in the Valley” — six times before the system got it.

These failures were invisible to ordinary monitoring because the system was not producing garbage. It was producing fluent, grammatical, entirely valid text that happened to be the wrong text. Measured across production, recognition accuracy in multi-turn dialogue ran 7.64 to 12.07 percentage points below single-turn, and task completion 2.44 to 8.89 points lower.

The widget below lets you drive the mechanism directly. Turn an entry off and watch a correct transcript decay into a plausible wrong one.

Interactive — edit the list, watch the transcript

Personalisation you can switch on and off, one step at a time

Speech recognition decodes one character at a time. Indicator Loss adds a supervised yes/no target at every step, for every phrase on the user's personal list — contacts, device names, things they have played before. Tick an entry off below and watch the transcript degrade into a valid-looking phrase that means something else entirely. Tick it back on and it recovers. Nothing is retrained in between.

This interactive needs JavaScript. In short: each entry on a user's personalisation list carries a per-step binary target telling the decoder exactly when that phrase is in play. Removing an entry changes the transcript immediately, with no model retraining; and because the targets are explicit, an operator can see whether biasing fired rather than inferring it from the output.

The three scenarios are drawn from documented Tmall Genie failure modes. The syllables are identical in each pair — only the characters differ, which is why the failures are invisible to ordinary monitoring: the system returns fluent, grammatical text that happens to be the wrong text.

What the patent does differently

The invention is called Indicator Loss. Alongside the ordinary training objective — produce the right transcript — the model is trained on a second, explicit target: at each decoding step, for each entry on the personalisation list, a binary label for whether that entry is in play.

That single change produces three properties the prior art did not have together:

Updatable

The bias list is an input, not a trained parameter. Adding a contact or a newly launched device brand takes effect on the next utterance instead of requiring a language-model rebuild measured in weeks.

Auditable

Because biasing is supervised explicitly, whether it fired is a logged value. A failed request can be traced to a cause rather than reproduced by guesswork.

Accurate

Head to head against Google CLAS, character error rate improved on IoT device names, contact names, and a combined general set — while keeping a language-model module that CLAS does not have.

Three generations of personalised speech recognition compared First-generation hybrid ASR combines an acoustic model, an n-gram language model and a context language model through shallow fusion, which requires hand-written grammars. Google CLAS uses an encoder, decoder and bias encoder joined by multi-head attention and concatenation, trained only on decoder loss, so its use of the bias list stays implicit in the attention. The Indicator Loss design keeps an encoder, a decoder with a language model, and a bias encoder, joined by multi-head attention and ColdFusion, and adds explicit binary supervision. Only the third design keeps a language-model module, combines real-time vocabulary updates with sub-50-millisecond response and grammar-free operation, and exposes at every step whether biasing fired. How the field solved this, and what changed The same task — bias recognition toward a user’s own contacts, devices and content — attempted three ways. First-generation ASR hybrid pipeline, pre-2019 Acoustic model n-gram language model Context language model Shallow fusion Transcript Not end-to-end. Every new phrase needs a hand-written grammar and a language-model retrain measured in weeks. Google CLAS the industry benchmark Encoder Decoder Bias encoder key phrases Multi-head attention Concat → transcript End-to-end, but the only training signal is the final transcript. Attention shows where the model looked, never whether it decided. No LM module. Indicator Loss deployed on Tmall Genie, 2019–2020 Encoder Decoder + language model Bias encoder key phrases Multi-head attention ColdFusion → transcript Plus a supervised binary target at every step, for every phrase on the list — so biasing is an explicit, inspectable decision. WHAT ONLY THE THIRD DESIGN DOES Keeps a language-model module. CLAS has none, so it cannot draw on linguistic context while decoding. Was the only design in the head-to-head to combine real-time vocabulary updates, sub-50 ms response and grammar-free operation. Exposes, at every step, whether biasing fired — so a bad transcript can be traced to a cause instead of guessed at.
How the field approached the same task, and what the third design adds.

A complementary mechanism shipped alongside it: a self-correction data loop that mines production logs for fail-then-succeed retry pairs — exactly the six-retry pattern above — and feeds them back as training signal. That loop alone cut negative feedback by 20.16%.

What changed in production

Production results across three generations of the Tmall Genie speech stack Negative user feedback fell from 22.38 percent with traditional ASR to 15.90 percent with an end-to-end model without context, and to 15.01 percent with the context-aware architecture — a 32.9 percent reduction against the traditional baseline. Multi-turn turns that failed to complete fell from 21.72 percent to 14.50 percent to 12.98 percent, meaning completion rose from 78.28 to 87.02 percent. In a head-to-head evaluation against Google CLAS, character error rate was 20.00 versus 20.90 on IoT device names, 21.38 versus 21.88 on contact names, and 16.54 versus 16.85 on a combined general and contact set. Production impact across three generations of the speech stack Measured on live Tmall Genie traffic — the 53.8% of sessions that involve multi-turn dialogue. Lower is better in both charts. Negative user feedback rate 25% 15% 5% 0 22.38% 15.90% 15.01% Traditional ASR v1.0 End-to-end, no context (v2.0) Context-aware this work −32.9% against the traditional baseline Multi-turn requests that failed to complete 25% 15% 5% 0 21.72% 14.50% 12.98% Traditional ASR v1.0 End-to-end, no context (v2.0) Context-aware this work Task completion 78.28% → 87.02% HEAD-TO-HEAD AGAINST GOOGLE CLAS — CHARACTER ERROR RATE, LOWER IS BETTER IOT DEVICE NAMES Indicator Loss 20.00 Google CLAS 20.90 CONTACT NAMES Indicator Loss 21.38 Google CLAS 21.88 GENERAL + CONTACT Indicator Loss 16.54 Google CLAS 16.85
Live Tmall Genie metrics across three generations of the speech stack, plus the head-to-head evaluation against Google CLAS.

The character-error-rate margins against CLAS are real but modest. The decisive difference is operational: this was the only design in the evaluation that combined real-time vocabulary updates, sub-50-millisecond response, and grammar-free operation — and the only one where an engineer could tell, after the fact, whether personalisation had fired.

Commercial consequence

Why this is a business result, not only a technical one

Recognition failures are not a quality metric on a dashboard — they are the mechanism by which a household stops using a device. Internal analysis across 12.34 million monthly active users found negative-feedback rate inversely correlated with engagement (r = −0.26): users in the low-failure band averaged 16.3 active days per month, users in the high-failure band 11.5. Roughly a fifth of the base sat in that high-failure band.

Removing the retrain cycle also removed a recurring operating cost and a competitive constraint. Under the old architecture, a newly launched smart-device brand or a trending song title could not be recognised until a language-model rebuild shipped. Under this one, the vocabulary is data, changed in seconds.

The architecture long outlasted my tenure. In May 2023 — three years after the deployment period closed, and two years after I left the company — an Alibaba Cloud article describing Tmall Genie’s production speech stack still set it out in the same terms: fusion recognition, and personalised, scenario-aware recognition working along the user dimension and the dialogue-context dimension. Those are the two subsystems this patent covers. A stack still described that way years later, on a platform of this size and through successive model generations, is load-bearing infrastructure rather than an experiment that happened to ship.

Timeline

The underlying evidence

Google Patents record for CN113808593B, Voice interaction system, related methods, devices and equipment
Granted patent CN113808593B — the primary record.
Internal Alibaba slide comparing first-generation ASR, Google CLAS, and the personalised decoder with Indicator Loss
The internal industry-comparison slide from the period (Chinese). The diagram above is its English rendering; the worked example in the right-hand column is the contact-name case reproduced in the interactive.
Internal slide showing Indicator Loss results compared with Google CLAS across IoT and contact test sets
Head-to-head character error rate against CLAS on the IoT and contact personalisation sets.

← Back to home · All projects