Real-Time 3D Virtual Object Manipulation Using MediaPipe Hands for Web Based Human Computer Interaction

Abstract

Real-time hand tracking is one of the challenges in the field of computer vision, particularly in supporting the development of intuitive and natural Human Computer Interaction (HCI) systems. This research aims to design and implement a system for manipulating three-dimensional (3D) virtual objects based on MediaPipe Hands and WebGL in a web-based environment. The system utilises an RGB webcam as an input device to detect 21 hand landmark points, which are subsequently processed into three main types of gestures: ‘grabbing’ for object translation, ‘rotating’ for object rotation, and the ‘pinch’ gesture for changing the object’s scale. The system was evaluated by testing variations in the user’s distance from the camera (0.5 m, 1.0 m and
1.5 m), lighting conditions (bright, dim and dark), and measuring system performance based on frame rate (FPS), Detection Confidence and Tracking Confidence. Test results showed that the system was capable of operating in real time at an average of 25–30 FPS, with Detection Confidence of 80–95 per cent and Tracking Confidence of 80–94 per cent under adequate lighting conditions. The best performance was achieved at distances of 0.5–1.0 m, whilst a decrease in accuracy occurred at greater distances, under low-light conditions, and in the presence of occlusion. The contribution of this research is the development of a browser-based 3D object manipulation system that integrates MediaPipe Hands, JavaScript and WebGL, enabling the entire process of hand detection, gesture recognition and object visualisation to be carried out markerlessly without the need for specialised hardware or a GPU accelerator. The research results indicate that the proposed system has the potential to be applied to various Human Computer Interaction applications, virtual simulations, virtual reality, and gesture-based interfaces.