Moving Robot Arm Controlled by Hand Gestures
Touchless Human-Robot Interaction Using Computer Vision & Wireless Communication
Python
OpenCV
Arduino
C++Project Gallery
Image Placeholder 1
Add your project image here
Image Placeholder 2
Add your project image here
Image Placeholder 3
Add your project image here
Image Placeholder 4
Add your project image here
Project Overview
A human-computer interaction (HCI) project that enables touchless, intuitive control of a robotic manipulator. By leveraging computer vision and wireless communication, this system translates natural hand movements into precise mechanical actions in real-time, bridging the gap between digital gesture recognition and physical actuation.
The system eliminates physical controllers (joysticks/keyboards) by using a standard webcam to track hand movements, achieving 95%+ recognition accuracy while providing immediate visual feedback through skeletal overlay visualization.
Technical Architecture
Gesture Recognition Node
Computer vision pipeline running on host machine that captures video, tracks hand landmarks, and classifies gestures using MediaPipe
Communication Bridge
Wireless link using Bluetooth (HC-05) transmitting encoded command signals from host computer to robot controller
Robotic Control Node
Arduino-based embedded system that parses received commands and drives servo motors to replicate intended motion
Hardware Components
- â–¸Arduino Uno: Central MCU for parsing commands and generating PWM signals
- â–¸HC-05 Bluetooth: Wireless serial communication (~10m range)
- â–¸Servo Motors (SG90/MG995): Precise angular positioning for joints
- â–¸3-4 DOF Robotic Arm: Multi-joint chassis with base, shoulder, elbow, gripper
- â–¸Mobile Wheeled Base: Adds mobility to manipulation capabilities
Software Stack
- â–¸Python (OpenCV): Image processing and vision pipeline core
- â–¸MediaPipe: Robust hand landmark detection (21 3D keypoints)
- â–¸Arduino C/C++: Firmware for serial data interpretation
- â–¸PySerial: Handles serial communication to Bluetooth COM port
- â–¸Gesture Classification Logic: Analyzes finger landmark positions
Core Features & Capabilities
Real-Time Hand Gesture Recognition
95%+ recognition accuracy using MediaPipe's ML model to detect 21 3D hand keypoints with visual skeletal overlay feedback
Wireless Bluetooth Control
HC-05 module enables untethered operation with low-latency command transmission (~10m range)
Multi-DOF Control
Maps specific hand gestures to robotic axes: Index up (lift), Fist (grip), Open palm (release), Hand tilt (drive base)
Integrated Mobility
Wheeled chassis enables Pick and Place tasks with unified gesture interface for manipulation and navigation
Performance Specifications
Implementation Highlights
🎯 Computer Vision Pipeline
Captures frames, converts to RGB, processes through MediaPipe Hands solution extracting 21 landmark coordinates. Custom logic calculates Euclidean distances between landmarks (e.g., thumb tip to index tip) for dynamic gesture recognition.
🤖 Inverse Kinematics (Simplified)
Translates abstract gestures into specific servo angles. For example, "V-sign" maps to "set servo 2 to 90 degrees" ensuring predictable movement patterns for intuitive control.
📡 Robust Serial Communication
Implements concise protocol sending single-character command bytes rather than complex strings. Arduino processes instructions within millisecond loop cycle required for smooth servo PWM generation.
Technical Challenges & Solutions
âš¡ Jitter & False Positives
✓ Implemented moving average smoothing filter and confidence threshold (min_detection_confidence=0.7) to ignore low-quality frames
âš¡ Power Supply Isolation
✓ Separate high-current power source for motors while keeping Arduino logic on stable regulated supply with shared common ground
âš¡ Latency Management
✓ Optimized Python loop with lower resolution (640x480) and increased Bluetooth baud rate (38400+) to maximize throughput
Real-World Applications
Prosthetics
Controlling artificial limbs using muscle signals or visual gestures
Industrial Safety
Operators control heavy machinery from safe distance without contact
Medical Robotics
Touchless control of surgical tools in sterile environments
Smart Home Automation
Control appliances using simple hand signs for elderly/disabled
Education & Research
Platform for HCI and robotics learning and experimentation
Entertainment
Interactive gaming and immersive gesture-based experiences
Future Enhancements
Project Impact
The Gesture-Controlled Robot Arm successfully demonstrates the power of integrating modern AI-based computer vision with classical embedded robotics. It creates a seamless, intuitive interface that makes controlling complex hardware as simple as waving a hand, highlighting the future of natural Human-Computer Interaction.
Interested in This Project?
This project demonstrates advanced human-robot interaction, computer vision, and wireless embedded systems. Feel free to reach out for collaboration or technical discussions.