Design and development of audio-emotional serious games for audiology therapy

Abstract

In recent years, with the increase in the use of technological tools, the act of playing games in people's daily lives has also started to occupy a large place. With this demand and the rapid development of the digital game industry, the digital game industry has become a major shareholder in the entertainment industry. In the digital game world, which includes a wide range of genres, the concept of serious game has emerged, which aims to provide information, education and therapy on any subject rather than just entertaining the players. The cornerstone of how people interact with their environment is communication. Communication is an indispensable phenomenon of people's daily lives. Emotions are one of the most important elements that enable effective communication. Although there are many differences between people in the world, the common concept shared by all people's emotions. Emotions are a very important building block that affects every thought or behavior in people's daily lives. It is very important for people to be able to express their emotions correctly and to understand the emotions of the others while communicating with their environment, in terms of establishing a smooth and efficient communication. At this point, especially hearing-impaired students at elementary school level, have difficulty in expressing their emotions while communicating with their peers and understanding the emotions conveyed in their dialogues with their peers. Within the scope of this thesis, two serious game projects were designed to help children who have difficulties in communicating effectively with their environment, including the mechanics of emotion analysis over voice. During the literature review, scientific studies on subjects such as emotions, expression of emotions, and serious games were examined, and many scientific research papers containing serious game projects in the field of therapy were explored. By combining the information obtained from all these scientific articles and similar studies with the principles of game design, serious games that are easy to play, enjoyable, and entertaining were developed, as indicated by the ratings provided by the participants in the experiment. The first game developed within the scope of the project is a 2D endless runner game. The first game project is called EmotiRUN. The game is designed to be as simple and easy to play as possible for the target audience of children. The main aim of the players in the game is to earn the highest score they can in a game round. The game focuses on three main emotions. These emotions are happiness, sadness and anger. These emotions are shown with animal symbols in the game. Respectively; the monkey represents happiness, the bird sadness and the snake anger. In the game, the players try to jump between the platforms and not fall into the gaps by jumping at the right time with the characters they control. At the same time, they earn points by collecting animal coins on the platforms as they progress through the game. When they collect 20 pieces of the same animal coin during the game, the players are faced with the emotion task about the related animal. Players who successfully complete this task earn extra points in the game. In the game, if the players cannot jump with their characters at the right time or if they hit into a platform and fall into the gap, the game is over. The second game developed for research is a 2.5 dimensional drag and shoot game. The name of the second game is EmotiFARM. There are many different game objects and mechanics in the game. The game consists of 10 different levels in total. There are different animals in the game levels and these animals have 3 different needs. These needs are water, food and attention. Players must select the suitable objects for these needs in the game and throw them to the right animal. If players can throw the right object to the right animal, they earn points. Players have the right to make 3 moves and choose 3 game objects in each level. If the players run out of these rights, they are faced with the emotion mission panel. On this screen, if the players can successfully complete the task, they gain in game moves or in game objects. The level ends when all the needs and tasks in a level are completed. Players who successfully complete 10 levels in total are finished the game. Both projects were developed using the Unity game engine. While designing the games, cartoonish graphics, cute objects and sounds were used to attract children's attention. During the design process, the design of the levels and the user interface design were designed to be simple, aesthetic, easy to understand and comfortable to use. Confusing and complex design is avoided as children can be easily distracted. A product called entertAIn play was used to analyze emotions from the voice in games. entertAIn play is a plugin powered by devAIce developed by the company named audEERING GmbH that can analyze emotions from voice with artificial intelligence. Within the scope of the research, the company was contacted and they agreed to support this project through their free academic license. Using this tech, the emotions of happiness, sadness and anger from the voices of the players in the games were detected in real time on the device without sending the data to any cloud server. Within the scope of the research, game experiments were conducted with 11 participants. While the target audience is especially hearing impaired children, the games in the research were tested on hearing impaired, autistic, dyslexic and normally developing children. 5 of the participants are hearing impaired, 1 has autism, 1 is dyslexia and 4 is typically developed children. 4 of the participants are female and 7 are male and their ages are between 10 and 16. Within the scope of the study, the first game was tested 5 times, and the second game was tested for a maximum of 30 minutes over 10 levels. At the end of the experimental sessions, the opinions of the participants about the features of the games were recorded using a 5-point Likert-type scale consisting of 14 items under 7 main headings. No additional explanation was given to any participant during the experiment, and they were expected to solve and play the games on their own. It was observed that the participants participating in the experiment played both games with great interest, fun and enthusiasm during the game testing process. When the test results were examined, it was seen that normally developed children, regardless of the type and complexity of the game, were able to express their emotions better with voice than other participants. Although hearing impaired participants achieved better results than dyslexic participant, it was observed that the success rates in expressing emotion over voice were very close to each other. The autistic participant was the participant with the lowest percentage of success in expressing emotion with voice in both games. Among the hearing impaired participants, the participants who had previously received speech therapy were more successful in both games than those who did not receive speech therapy. When the attitudes of the participants towards the game features were examined, it was determined that the first game with simpler mechanics was found to be easier to understand, fun and easy to play by the hearing impaired, dyslexic and autistic participants than the other game. In contrast, typically developed participants found the second game with complex mechanics and different levels more enjoyable. Apart from these, the second game, which has 2.5 dimensional graphics and contains many various game objects, was generally found more attractive among the participants in terms of gameplay and visually. Within the scope of this thesis study, when the feedback of the participants participating in the experiment were examined, it was seen that they had a pleasant time while playing the games and they had fun while playing the games, even though they were serious games. The mechanics of emotion analysis over sound have been successfully integrated into the games and can be used without disrupting the flow of gameplay. It has been seen that serious games can be used in the field of audiological therapy and it has been determined in general terms what features should be included in a therapy oriented serious game project whose target audience is children. In terms of areas that can be improved within the scope of the project, new levels, scenarios, mechanics, objects and emotion types can be added for future studies to increase the replayability of the games developed in this study and for a deeper therapy game experience. By increasing the number of children with hearing impairment, autism and other communication difficulties participating in the experiments, the data group can be expanded and more and reliable feedback can be collected in terms of game designs. In addition, the long term effectiveness of the use of serious games in therapy can be determined by extending the experimental periods of the games over a longer period. It is believed that the serious game projects presented within the scope of this thesis study, along with future developments and research, will provide inspiring information on how to design and develop a serious game project in the most effective way for audiological audio-emotion therapy, how serious games can help people to overcome their deficiencies in certain points and how serious game projects can contribute to keeping individuals motivated during the rehabilitation process and play an important role in having an enjoyable and effective therapy experience.