Social behavior learning for an assistive companion robot

Abstract

Designing robots having the ability to build and maintain interpersonal relationship with humans by exchanging social and emotional cues has attracted much attention recently because robots are vastly in use in a wide variety of places with a diverse range of tasks from healthcare to edutainment industry. In order to interact with humans in a natural way, emotion recognition and expressive behavior generation are crucial for these robots to act in a socially aware manner. This requires the ability to recognize the affective states, actions and intentions of their human companions. The robot should be able to interpret the social cues and its human companion's behavioral patterns to be able to generate assistive social behaviors in return. There are several popularly known robotic platforms such as Kismet, Nao and Pepper which are able to recognize its human partner's emotional state based on facial expressions, vocal cues and body postures and express simple human emotions. However these robots do not have an emotional character or an affective state, they are just capable to mirror the human's expressions independently from the social context of the interaction. For the robots to be accepted as social entities, they should be endowed with the capability to interpret the human's mood as well as his/her emotions and the social context of the interaction. In order to achieve this purpose, it is not enough to treat expressive behaviors of humans as only a mirror of their internal state. Therefore, it is crucial to incorporate a generative account of expressive behavior into an emotional architecture. This requires perceiving and understanding others' affective states and behaving accordingly, it corresponds to the most generic definition of empathy. Empathy is a focal point in the study of interpersonal relationships therefore it should also be considered as one of the major elements in the social interaction between the robots and humans. Humans have the ability to feel empathy not only for other humans but also for fictional characters and robots. But the robots are not able yet to display any empathic behavior in return. The motivation of this thesis study is to design and implement a computational emotion and behavior model for a companion robot. The proposed model is designed to process multi-channel stimuli in order to interpret the affective state of its human companion and to generate in-situ affective social behaviors based on the processed information coming from the human companion and the environment, that is the social context of the interaction. This dissertation attempts to explore the following research objectives: - Would the robot be able to display basic emotions? Could the human companion identify correctly the emotions displayed by the robot? - How could a social robot infer and interpret its companion's emotional states? - How can we computationally model an artificial emotional mechanism and implement it on a social robot to provide a natural social interaction? - Which learning techniques should we use for the affective robot assistant to learn which emotional behavior to express during the interaction? - Could a social robot designed with emotional understanding and expression foster the interaction gain in a HRI scenario based on assistance? In order to explore the answers of these research questions, user studies with children having different developmental profiles and two affective robots, Pepper and Kaspar, were conducted in coordination with two research projects. The first project, titled as RoboRehab, we aimed to use an affective social robot to support the auditory tests of children with hearing disabilities in clinical settings. During their hospital visits, children take several tests to determine the level of their hearing and to adjust their hearing aid or cochlear implant, if necessary. The audiologists mention that the children usually get stressed and tend to be in a negative mood, when they are in the hospital during their consultation. This affects children's' performance in the auditory perception tests negatively, and their cooperation decrease significantly. In Roborehab, we used machine learning-based emotion recognition approaches to detect the children's emotional state and adjust the robot's actions accordingly. We designed a feedback mechanism to reduce the stress of children and help the children to improve their mood during the audiometry tests. We used a socially assistive humanoid robot Pepper, enhanced with emotion recognition, and a tablet interface, to support children in these tests. We investigated the quantitative and qualitative effects of the different test setups involving a robot, a tablet app and the conventional method. We employed traditional machine learning techniques and deep learning approaches to analyze and classify the physiological data (blood volume pulse, skin temperature, and skin conductance) of children collected by E4 smart wristband. The second project, entitled as "Affective loop in Socially Assistive Robotics as an intervention tool for children with autism (EMBOA)", was a research project with the aim of combining affective computing technologies with the social robot intervention in children with autism spectrum disorder (ASD). Children with ASD are known to display limited social and emotional skills in their routine interactions. Inspired by the promising results presented in the social robotics field, we aimed to investigate affective technologies with a social robot, Kaspar, to assist children with ASD in the development of their social and emotional skills, help them to overcome social obstacles and make the children more involved in their interactions. Interaction studies with Kaspar were conducted in 4 collaborating country; Poland, North Macedonia, United Kingdom and Turkey; with more than 65 children with ASD and interaction data collection by different sensor modalities (visual, audio, physiological and interaction-specific data) was performed within a longitudinal study design. In this dissertation, a computational model for emotional understanding and emotional behavior generation and modulation was designed and implemented for a companion robot based on the collected data and findings through The RoboRehab and EMBOA projects. The presented models were designed: (1) to process multi-channel stimuli, i.e. vision-based facial landmarks, physiological data-based signals, in order to detect the affective states of the human companion; (2) to generate in-situ affective social behaviors for the robot based on the interaction context; (3) to adapt the intensity of the robot's emotional expressions based on the preferences of the human companion. Despite challenging situations, with Covid-19 outbreak at the top of the list, and computational limitations, the previously mentioned research objectives were investigated in detail and the results were presented in this dissertation. We were able to answer the five research questions on the affective social behavior of a companion robot. The user studies conducted with hearing-impaired children in RoboRehab showed that Pepper robot was able to display emotional behaviors and the children could correctly identify and interpret them. Moreover, the RoboRehab findings revealed that the affective robot was able to trigger some emotional changes in children and cause difference in their physiological signals. These signals were used to distinguish the emotions of children with machine learning and deep learning approaches in different setups. On the other hand, the results from the EMBOA studies with children with ASD showed that a multimodal approach based on behavioral metrics, i.e. facial action units, physiological signals, interaction-specific metrics, can be used to understand the emotional state and infer the emotional model of the human companion. Different frameworks were presented in order to model an artificial emotional mechanism for a social robot. The first one was a simple linear affect model based on the two-dimensional valence and arousal representation. The second one, a vision-based model where the robot detects the affective state of its companion and develops a behavioral strategy to adapt its emotional behaviors and improve the negative mood of its companion. And finally, a behavioral model were presented for the robot to predict the preferences of its companion and regulate the intensity of its emotional behaviors accordingly. The results of the user studies and the findings revealed that an emotional model can be computationally modeled and implemented for a social robot to adapt its emotional behaviors to the preferences and needs of its companion, consequently, to make the interaction between the human companion-robot pair more natural. The RoboRehab and EMBOA user studies with different groups of children (with typical development, with hearing impairment, with autism) showed that independently from their profile the children were more involved and paid attention to the social robot. Even though the objective evaluation metrics did not point a significant difference, the subjective evaluation of the audiologists, therapists, pedagogues complied with the presented hypothesis. Furthermore, the self-report surveys with children and their parents showed that the children accepted the affective robot as an intelligent and funny social being. These results demonstrate that an affective social robot with personalized emotional behaviors based on the preferences of its companion can foster the interaction gain in an assistive human-robot interaction scenario.