Computer vision-enhanced human-robot interaction using a 7 DOF manipulator with application to rehabilitation

Abstract

Human-robot Interaction (HRI) plays a crucial role in enabling robots to operate safely alongside a human's upper extremity in a rehabilitation environment. This thesis focuses on the design and implementation of a robust HRI system using a 7 degree-of-freedom (DoF) robot for potential use in upper extremity rehabilitation and motor recovery. The system is designed to support multiple interaction modes, including passive, resistive, and assistive modes, tailored to the potential participant's physical capabilities and rehabilitation needs. A real-time vision system enables the 3D visualization of the human body movement and simulation of the robot in a shared simulation environment. Utilizing a camera and pre-trained models, the human body keypoints and the robot's base frame are detected. This visualization enables precise, real-time monitoring of human body movements, primarily in the robot's proximity. To boost participants' engagement and therapeutic outcomes, a custom 2D interactive game was created where participants control a paddle to catch a moving ball, emphasizing shoulder joint movement. The paddle is connected to the robot's end-effector, enabling control through physical movement. The robot provides dynamic assistance or resistance based on the selected mode, making the system adaptable for patients with different levels of motor ability. The game-based interface is designed to motivate participation and support rehabilitation through repeated, goal-focused motor exercises for the shoulder joint. The safe and real-time integration of vision-based human motion capture, multimodal robotic interaction, and gamified therapy creates a versatile platform for potential neurological rehabilitation. This work contributes to the growing field of assistive robotics by providing a scalable, interactive solution that can support physical therapy while maintaining user engagement.May 202