Multimodal Human–Computer Interaction Using Hand Gestures

Abstract

Human-Computer Interaction (HCI) has evolved significantly with the integration of natural and intuitive communication methods. This project, Multimodal Human-Computer Interaction Using Hand Gestures, presents a system that enables users to interact with computers through hand gestures combined with other input modalities such as voice or facial expressions. The system uses computer vision and machine-learning techniques to detect, track, and recognise hand gestures in real time using a standard camera; by interpreting these gestures as commands, users can perform tasks such as navigation, media control, and application management without physical contact. The multimodal approach enhances accuracy, flexibility, and user experience by combining multiple input sources, and the system is particularly useful in environments where touch-based interaction is limited or impractical. The prototype is implemented in Python with OpenCV for image processing, MediaPipe for hand tracking, and a Streamlit interface; gestures map to system actions including volume up/down, cursor movement, left and right click, and brightness increase/decrease, with keyboard and PowerShell fallbacks for robustness. Hand landmarks (42 features) are extracted from the webcam feed and classified to predict the gesture, and a confidence threshold prevents actions from being triggered by uncertain predictions. The system was validated through thirteen functional test cases covering application launch, webcam initialisation, hand detection, landmark extraction, gesture recognition, confidence validation, and each control action, all of which passed. Overall, the project creates a natural, efficient, and user-friendly interface that improves accessibility and paves the way for advanced interactive systems.