Non-verbal communication with a social robot peer: Towards robot assisted interactive sign language tutoring

Abstract

This paper presents a humanoid robot assisted imitation based interactive game for Sign Language (SL) tutoring. The game is specially designed for children with communication impairments. The work presented is a part of Robot Sign Language Tutor∗ project. In this study, a Robovie R3 humanoid robot is used to express a set of chosen words in SL using hand movements, body and face gestures. The robot platform is specially modified with LEDs in face (for non-manual facial gestures), additional DoFs in wrist and 5 independent fingers at hands for robust SL generation. The robot is able to communicate with the participant by recognizing signs and colored flashcards through a RGB-D camera based system and generating a selected subset of signs, including motivating facial gestures, in return. The game also aims to improve children's imitation and turn-taking skills and to teach the words semantically. Current paper presents results from the preliminary study with a group of hearing impaired children, where children had almost 100% score in recognizing the robot's signs from a subset of 12 words.