Multimodal Human–Computer Interaction Using Hand Gestures

Abstract

—Human-Computer Interaction (HCI) has evolved significantly with the integration of natural 
and intuitive communication methods. This project, Multimodal Human-Computer Interaction Using Hand 
Gestures, presents a system that enables users to interact with computers through hand gestures combined 
with other input modalities such as voice or facial expressions. The system uses computer vision and 
machine-learning techniques to detect, track, and recognise hand gestures in real time using a standard 
camera; by interpreting these gestures as commands, users can perform tasks such as navigation, media 
control, and application management without physical contact. The multimodal approach enhances 
accuracy, flexibility, and user experience by combining multiple input sources, and the system is 
particularly useful in environments where touch-based interaction is limited or impractical. The prototype 
is implemented in Python with OpenCV for image processing, MediaPipe for hand tracking, and a Streamlit 
interface; gestures map to system actions including volume up/down, cursor movement, left and right click, 
and brightness increase/decrease, with keyboard and PowerShell fallbacks for robustness. Hand landmarks 
(42 features) are extracted from the webcam feed and classified to predict the gesture, and a confidence 
threshold prevents actions from being triggered by uncertain predictions. The system was validated through 
thirteen functional test cases covering application launch, webcam initialisation, hand detection, landmark 
extraction, gesture recognition, confidence validation, and each control action, all of which passed. 
Overall, the project creates a natural, efficient, and user-friendly interface that improves accessibility and 
paves the way for advanced interactive systems.