CV enabled American Sign Language recognition

Abto Software took part in an ambitious competition to design a model to recognize and classify ASL signs, aimed at education gamification. Having covered initial discovery, approach determination and evaluation, our team created a TensorFlow Lite language model, trained on data extracted using the MediaPipe Solution. To allow the algorithm to run across devices and limit the latency, the videos aren’t stored on a public cloud. That means the inference must be smoothly conducted on the user’s device.

The project's main goals: 1. To develop an ASL recognition model prioritizing accuracy 2. To ensure the machine learning model can run on Android and iOS The stages our team has covered: 1. Preliminary discovery – we explored different approaches to gain more insight into the best practices 2. Architecture design – we designed a model to associate data with the corresponding ASL signs 3. Model training – feeding the built model with the training dataset to help it learn different patterns 4. Model evaluation – providing the created model with the testing dataset to compare the predicted output with the actual labels 5. Model fine-tuning and optimization (iterative tweaking and retraining to achieve better accuracy) 6. Model conversion into the requested format (TensorFlow Lite)

The designed ASL model is optimized for deficient internet connection, which makes it suitable for use even in developing countries. The described ASL model doesn’t use original input (user video’s), which preserves patient security and privacy and ensures regulatory compliance. With integrated ASL recognition into products and services, people with hearing loss can enjoy: - Improved communication - Educational support and gamification - Social inclusion - Practical implementations (convenient control over technology, including phones, wearable devices, and home automation systems)