Dual-arm safe robot manipulation with second arm assistance

Abstract

In this thesis, the challenge of achieving optimal and safe manipulation policies is addressed for dual-arm robots in complex environments using deep reinforcement learning. Representing both the goal and safety constraints in a single reward function often leads to policies that are either unsafe or overly conservative. To address this issue, a dual-arm cooperation method is proposed, which involves decomposing the overall task into subtasks assigned to each robot arm. This approach ensures that the goal is achieved while adhering to safety constraints, thereby preventing potential unsafe situations. The experimental setup involves the Baxter humanoid robot performing a stir task in the CoppeliaSim simulation environment, controlled via PyRep. The environment includes a bowl containing balls of various sizes, and the robot must use a spoon to stir the bowl. The objective of this thesis is to prevent unsafe situations such as sliding, spilling and hence overturn failures during the stir task. There are six experimental scenarios to evaluate the proposed approach: Single-Arm Stir, Single-Arm Hold, Right-Hand Assisted Stir, Single-Agent Dual-Arm, and Multi-Agent Dual-Arm. In the Single-Arm Stir scenario, the robot uses one arm to stir the bowl without any assistance, focusing solely on the stirring task which may result in safety issues such as slide or spill. The Single-Arm Hold scenario involves the robot using one arm to hold the bowl in place without performing the stirring task. In the Right-Hand Assisted Stir scenario, the robot's right arm stabilizes the bowl using a pre-trained model while the left arm performs the stirring action. The Single-Agent Dual-Arm scenario treats both arms as part of a single-body problem with a 6-dimensional array, controlling both arms together to perform the task. The Multi-Agent Dual-Arm scenario employs a multi-body approach with two separate 3-dimensional arrays, where each arm is controlled independently but works towards a coordinated task. These scenarios provide a comprehensive evaluation of the dual-arm cooperation method, demonstrating its effectiveness in achieving the task goal while adhering to safety constraints. The state space includes the positions of the bowl and the end effectors of both arms, as well as their relative positions and the desired position of the bowl. The action space involves the movements of the arms in three dimensions. The Deep Deterministic Policy Gradient (DDPG) method is used to learn the policies, optimizing the action-value function based on the Bellman equation. The results show that the Right-Arm Assisted Stir scenario achieves the highest stir reward and the lowest error rates. This scenario effectively minimizes slide and spill, demonstrating the advantage of using a secondary stabilizing mechanism. The Dual-Arm Stir scenario, while achieving higher stir rewards than Single-Arm Stir, still faces challenges due to the complexity of simultaneous learning, resulting in higher spilling rates. To address the complexities observed in the bimanual scenarios, a refined approach was adopted. Initially, a single model was trained to handle both arms with a 6-dimensional array, but the results indicate that the complexity of the environment and the state-action space made it difficult for the robot to learn the task efficiently. Consequently, a dual-model system was implemented where each arm learned its respective task (holding and stirring) separately in distinct environments. Pre-trained models from these separate learning phases were then combined in the Right-Hand Assisted Stir case, leading to optimal performance as this setup allowed each arm to focus on its specific goal without interference. In conclusion, our dual-arm cooperation method significantly enhances the safety and effectiveness of dual-arm robots performing intricate tasks. This approach not only improves task performance but also reduces the risk of accidents, making it applicable in various fields such as manufacturing, healthcare, and service robotics. Future work will focus on sim-to-real transfer to validate these methods in real-world applications.