Speech recognition and Indian English

Vendor accuracy numbers are measured on clean, read speech in one language. Your callers are on a mobile, in a market, switching between Hindi and English mid-sentence, and saying a street name no training set has ever seen.

Four things break, and none of them are fixed by choosing a "better" model.

1. Code-switching mid-sentence

The most common speech pattern in urban India, and the one most systems handle worst:

Haan, main kal aa sakta hoon, but morning slot chahiye.

Systems configured for a single language pick one and mangle the other half. Systems set to auto-detect often flip languages between turns, so the agent replies in Hindi to an English question and vice versa.

Test for it explicitly. Ask a vendor to demo one sentence that switches language in the middle — not two sentences in two languages, one sentence. It is the single most revealing test you can run, and it takes ten seconds.

2. Proper nouns

Place names, area names, building names, surnames. "Kondapur", "Andheri East", "Vaishali Nagar" — these appear rarely in general training data and get transcribed as something phonetically close and semantically useless.

This matters most in exactly the fields you care about: the address, the branch they want, their name. Mitigations that actually work:

• Supply a vocabulary hint list if the platform supports one — your branch names, your product names, the areas you serve. Big accuracy gain for ten minutes of work.
• Do not ask for spellings you can look up. If you know their number, you may already know their area.
• Read it back. "That's Kondapur, correct?" costs two seconds and prevents a wasted visit.

3. Numbers

Three separate problems that all land in the same field:

• Lakhs and crores. "Two point five lakh" has to become 250,000. Many pipelines return the literal words.
• Phone numbers spoken in pairs. "Ninety-eight, double-three, one..." Double and triple are extremely common in Indian speech and frequently mishandled.
• Digits versus words. Whether "one two three" arrives as 123 or "one two three" decides whether your validation passes.

Test each of these with your own data before you commit. They are cheap to check and expensive to discover.

4. The channel itself

A mobile call is 8kHz narrowband audio with compression artefacts and, frequently, a two-wheeler in the background. That is a materially harder input than the microphone your demo used.

Always evaluate over a real phone call. A browser-based demo tells you very little about how the same system performs on the network your customers are actually on.

Design around it, not just for it

Even with everything tuned, some turns will come back wrong. The agent's design should assume it:

• Confirm anything consequential — dates, amounts, addresses — by reading it back.
• Ask one thing at a time. Compound questions produce compound transcription errors and you cannot tell which half failed.
• Offer a fallback. "I'll send you a link on WhatsApp to confirm" is a perfectly good escape from a name it cannot get.
• Hand over after two failed attempts at the same field, rather than a third. Name it as a handover condition.

A one-hour evaluation

1. Record twenty real calls from your own line, chosen to include your worst audio.
2. Run them through each vendor you are considering.
3. Score only the fields you actually need — name, number, date, area — not overall word accuracy.
4. Include at least five code-switched sentences and five spoken phone numbers.

Word-error-rate across a whole call is close to meaningless for your purposes. Getting the appointment date right is the thing. Voice is set per language on the bot in Convarza, so an agent can answer properly in more than one without being two agents — how that is configured.