Nov 16, 2026
No image
AI Call Center Automation with Speech Recognition and Diarization
Completed

AI Call Center Automation with Speech Recognition and Diarization

$10,000+
2-3 months
Spain
2-5
view project
Service categories
Service Lines
Artificial Intelligence
Machine Learning
Domain focus
Banking & Financial Services
Programming language
JavaScript
Python
Frameworks
Express
Node.js
React.js
Subcategories
Artificial Intelligence
AI Integration
AI Proof of Concept
Machine Learning
Natural Language Processing (NLP)

Challenge

A European insurance provider needed a more efficient way to evaluate call center performance. Quality analysts manually listened to recordings, scored conversations and gave recommendations, which was slow and resource-heavy, and the company had no in-house AI expertise to run a pilot quickly.

WebbyLab was asked to design and validate an AI quality assurance system that converts audio to text with accurate support for Spanish dialects, separates agent and client speech (speaker diarization), runs entirely on the client's own infrastructure for data confidentiality, and exposes APIs for integration into existing call center workflows. The system had to cope with background noise, overlapping voices, mixed languages and long recordings, matching the accuracy of human analysts while scaling far beyond manual review.

Solution

We evaluated leading speech recognition models (OpenAI Whisper, Mozilla DeepSpeech, Wav2Vec 2.0) on real call recordings against dialect support, accuracy, local deployment, diarization support and performance. Whisper extended with WhisperX proved the most accurate, handling accents, noise and mixed languages.

For diarization we combined WhisperX with PyAnnote: audio is segmented with voice activity detection, each segment is transcribed, and every text segment gets a speaker ID.

To embed the system into the client's workflows we built a Node.js / Express.js backend with an API to submit audio for transcription and another to check task status and retrieve transcripts. Jobs are queued in Redis for horizontal scaling and parallel processing, results are stored in MySQL, and the solution was deployed on-premise, with a React dashboard and storage ready for migration to the cloud.

Results

The R&D prototype was tested on 50+ real call center recordings and delivered about 92% transcription accuracy (word error rate around 8%) even with noise, accents and mixed languages; about 87% diarization accuracy; and around 90% accuracy in automatic Spanish dialect detection. One minute of audio is processed in 45 to 80 seconds on CPU.

Business impact: manual listening time dropped by 70 to 80%, so analysts focus on edge cases instead of every call; reviewing a call now takes under a minute instead of 5 to 10 minutes; and the volume of analyzed calls grew dramatically, giving broader insight into customer satisfaction. The solution is a foundation for further NLP and sentiment analysis.