
AI Call Center Automation with Speech Recognition and Diarization
Challenge
A European insurance provider needed a more efficient way to evaluate call center performance. Quality analysts manually listened to recordings, scored conversations and gave recommendations, which was slow and resource-heavy, and the company had no in-house AI expertise to run a pilot quickly.
WebbyLab was asked to design and validate an AI quality assurance system that converts audio to text with accurate support for Spanish dialects, separates agent and client speech (speaker diarization), runs entirely on the client's own infrastructure for data confidentiality, and exposes APIs for integration into existing call center workflows. The system had to cope with background noise, overlapping voices, mixed languages and long recordings, matching the accuracy of human analysts while scaling far beyond manual review.
A European insurance provider needed a more efficient way to evaluate call center performance. Quality analysts manually listened to recordings, scored conversations and gave recommendations, which was slow and resource-heavy, and the company had no in-house AI expertise to run a pilot quickly.
WebbyLab was asked to design and validate an AI quality assurance system that converts audio to text with accurate support for Spanish dialects, separates agent and client speech (speaker diarization), runs entirely on the client's own infrastructure for data confidentiality, and exposes APIs for integration into existing call center workflows. The system had to cope with background noise, overlapping voices, mixed languages and long recordings, matching the accuracy of human analysts while scaling far beyond manual review.
Solution
We evaluated leading speech recognition models (OpenAI Whisper, Mozilla DeepSpeech, Wav2Vec 2.0) on real call recordings against dialect support, accuracy, local deployment, diarization support and performance. Whisper extended with WhisperX proved the most accurate, handling accents, noise and mixed languages.
For diarization we combined WhisperX with PyAnnote: audio is segmented with voice activity detection, each segment is transcribed, and every text segment gets a speaker ID.
To embed the system into the client's workflows we built a Node.js / Express.js backend with an API to submit audio for transcription and another to check task status and retrieve transcripts. Jobs are queued in Redis for horizontal scaling and parallel processing, results are stored in MySQL, and the solution was deployed on-premise, with a React dashboard and storage ready for migration to the cloud.
We evaluated leading speech recognition models (OpenAI Whisper, Mozilla DeepSpeech, Wav2Vec 2.0) on real call recordings against dialect support, accuracy, local deployment, diarization support and performance. Whisper extended with WhisperX proved the most accurate, handling accents, noise and mixed languages.
For diarization we combined WhisperX with PyAnnote: audio is segmented with voice activity detection, each segment is transcribed, and every text segment gets a speaker ID.
To embed the system into the client's workflows we built a Node.js / Express.js backend with an API to submit audio for transcription and another to check task status and retrieve transcripts. Jobs are queued in Redis for horizontal scaling and parallel processing, results are stored in MySQL, and the solution was deployed on-premise, with a React dashboard and storage ready for migration to the cloud.
Results
The R&D prototype was tested on 50+ real call center recordings and delivered about 92% transcription accuracy (word error rate around 8%) even with noise, accents and mixed languages; about 87% diarization accuracy; and around 90% accuracy in automatic Spanish dialect detection. One minute of audio is processed in 45 to 80 seconds on CPU.
Business impact: manual listening time dropped by 70 to 80%, so analysts focus on edge cases instead of every call; reviewing a call now takes under a minute instead of 5 to 10 minutes; and the volume of analyzed calls grew dramatically, giving broader insight into customer satisfaction. The solution is a foundation for further NLP and sentiment analysis.
The R&D prototype was tested on 50+ real call center recordings and delivered about 92% transcription accuracy (word error rate around 8%) even with noise, accents and mixed languages; about 87% diarization accuracy; and around 90% accuracy in automatic Spanish dialect detection. One minute of audio is processed in 45 to 80 seconds on CPU.
Business impact: manual listening time dropped by 70 to 80%, so analysts focus on edge cases instead of every call; reviewing a call now takes under a minute instead of 5 to 10 minutes; and the volume of analyzed calls grew dramatically, giving broader insight into customer satisfaction. The solution is a foundation for further NLP and sentiment analysis.