Scene analysis through auditory event monitoring

Abstract

The ability to categorize objects and outcomes of events using auditory signals is rather advanced in humans. When it comes to robots, limitations in sensing pose many challenges for this type of categorization specifically required in many robotic applications. In this paper, we propose auditory scene analysis methods for robots in order to monitor events to detect failures and learn from their experiences. Audio data are convenient for these purposes to detect environmental changes surrounding a robot and especially complement visual data. In our study, we investigate supervised learning methods using informative features from sound data for efficient categorization in manipulation scenarios. Furthermore, we use these data for robots to detect execution failures in runtime to prevent potential damages to their environment, objects of interest and even themselves. Firstly, the most distinguishing features for categorization of object materials from a set including glass, metal, porcelain, cardboard and plastic are determined, and then the performances of two supervised learning methods on these features for material categorization are evaluated. In our experimental framework, the performances of the learning methods for categorization of failed action outcomes are evaluated with a mobile robot and a robotic arm. Particularly, drop and hit events are selected for this analysis since these are the most likely failure outcomes that occur during the manipulation of objects. Using the proposed techniques, material categories as well as the interaction events can be determined with high success rates.