Finger Recognition Detection System Using Mediapipe as Communication Solution for People with Disabilities

Abstract

Sign language is a language used by people with disabilities, especially the deaf and mute to communicate. The problem is, not everyone can understand sign language. This study aims to create a system that can translate sign language. The system can also produce sound with the text to voice method. The system is built using Mediapipe to detect fingers that form sign language. The system performs classification by combining 2 machine learning models with the Artificial Neural Network (ANN) method. The first model is used to classify letters A-Z and the second model is used to classify movement patterns in letters J and Z. The accuracy of the first model is 94% and the accuracy of the second model is 95%. The model will also be evaluated using the confusion matrix technique to find recall, f1-score, and precision of 25 letters or classes.