IEEE GEM 2024

Responsive Teleoperation of a Robotic Arm
via Wearable Inertial Sensors

Alessandro Fornaro, Daniela D'Auria, Hamza Amrani, Paolo Napoletano
University of Milano-Bicocca
Paper
93.8%
Users: Intuitive
System usability
16
Participants
User study
3
IMU Sensors
Wearable setup

TL;DR — We present a responsive teleoperation system using three wearable IMU sensors to control a robotic arm, validated with 16 users who rated it 94% intuitive — demonstrating that low-cost inertial sensors can enable natural, accessible human-robot interaction.

System architecture: IMU units on human arm communicate via MQTT to ROS2 controlling a simulated UR10e in Gazebo

Figure 1. System architecture overview. Three IMU-equipped wearable units (upper arm, forearm, hand) transmit orientation quaternions via MQTT to a ROS2 motion package, which orchestrates Cartesian motion control and gripper actuation for a simulated Universal Robot UR10e arm in Gazebo.

Abstract

Human-in-the-loop teleoperation stands as a primary method for controlling robotic arms, offering significant practical and research applications across various fields such as industry, medicine, and home automation. In this paper, we propose a responsive system for the remote control of a robotic arm utilizing wearable inertial sensors. The system relies on three inertial units connected to low-cost wireless microcontrollers, which are placed on the user's arm. A server-side application, built on the Robot Operating System (ROS), orchestrates the interaction between these units and the robotic arm. The robotic arm's functionality is demonstrated within a simulated environment using Gazebo. To validate the efficacy of our proposed system, we conducted an experiment involving 16 users and evaluated their interaction using both objective measures (task execution time) and subjective metrics (user satisfaction). Our results reveal that users can gradually reduce teleoperation times with experience, and 93.75% found the system intuitive and usable.

Our key contributions are:

  1. A low-cost wearable teleoperation system using three IMU sensors (Bosch BNO055) on Arduino MKR1000 boards that captures full arm kinematics as quaternion orientation data at 50 Hz.
  2. A modular ROS2-based software architecture with MQTT-bridged communication, Cartesian motion control, and real-time Gazebo simulation of a Universal Robot UR10e with Robotiq 2F-85 gripper.
  3. A comprehensive user study with 16 participants evaluating the system through both objective task-completion metrics and subjective satisfaction surveys, demonstrating rapid skill acquisition and high usability.

Method

The system consists of three wearable IMU units placed on the operator's upper arm, forearm, and hand. Each unit houses a Bosch BNO055 inertial measurement unit connected to an Arduino MKR1000 wireless microcontroller, acquiring orientation data as quaternions at 50 Hz. The quaternion data is transmitted via MQTT to a ROS2 server application running on a desktop computer.

The ROS2 architecture is organized into three core nodes: (1) an MQTT Listener that receives sensor data from the three wearable units, (2) a Cartesian Motion Controller that computes the desired end-effector pose through quaternion multiplications and generates joint trajectories, and (3) a Gripper Controller that handles open/close commands for the Robotiq 2F-85 gripper. Quaternion multiplications across the three IMU readings compute the desired end-effector orientation, mapping the user's natural arm movements to the simulated UR10e robotic arm in Gazebo.

Users perform pick-and-place tasks of increasing complexity within the Gazebo simulator. Three task levels are defined: a simple single-object transfer, a medium two-step manipulation, and a complex multi-object rearrangement. Each task is repeated three times per user to assess the learning effect.

User wearing IMU sensors while teleoperating the robotic arm in Gazebo

Figure 2. A user wearing the three IMU-equipped wearable units teleoperates the simulated UR10e robotic arm. The Gazebo environment displays the pick-and-place task with designated source and target zones.

Results

We evaluated the system with 16 users performing three pick-and-place tasks of increasing complexity. Each task was repeated three times to assess skill acquisition. The experiment measured both objective performance (task completion time) and subjective user experience through satisfaction questionnaires.

Task Completion Times

The table below compares average teleoperation times against real (physical) manipulation baselines. While initial teleoperation is significantly slower, users demonstrate consistent improvement across repetitions, with total completion time decreasing from 3:13 to 2:12 across three repetitions.

Metric Task 1 (Simple) Task 2 (Medium) Task 3 (Complex)
Real Time (avg) 4.78s 7.82s 7.87s
Teleop Time (avg) 19.18s 40.71s 54.61s
% Increase +319% +429% +608%

Table 1. Comparison of average task completion times between physical manipulation and teleoperation. The overhead increases with task complexity, reflecting the additional cognitive and motor demands of remote control.

User Satisfaction

Subjective evaluation through post-experiment questionnaires reveals strong positive reception across all measured dimensions. The majority of participants found the system easy to learn and use despite the inherent challenges of teleoperation.

Satisfaction Metric Positive Rate
Intuitive & Usable 93.75% (15/16)
Responsive 81.25% (13/16)
Realistic 75.00% (12/16)

Table 2. User satisfaction survey results. Participants rated the system on three dimensions after completing all tasks.

Learning Effect

The figures below demonstrate a clear learning effect: total average completion time across all three tasks decreases consistently from the first to the third repetition, confirming that users adapt and improve their teleoperation skills with practice. Inter-user variability reflects different levels of prior experience with robotic systems.

Stacked bar chart showing decreasing completion times across repetitions

Figure 3. Average task completion times across three repetitions. Total time decreases consistently, showing a clear learning effect.

Per-user percentage increase in execution time compared to real manipulation

Figure 4. Per-user percentage increase in execution time relative to physical manipulation, broken down by task. Inter-user variability reflects different levels of prior experience with robotic systems.

BibTeX

@inproceedings{fornaro2024responsive,
  title     = {Responsive Teleoperation of a Robotic Arm via
               Wearable Inertial Sensors},
  author    = {Fornaro, Alessandro and D'Auria, Daniela and
               Amrani, Hamza and Napoletano, Paolo},
  booktitle = {IEEE GEM 2024},
  year      = {2024},
  organization = {IEEE}
}