Xpressionist: facial expressions as affect-aware game mechanics

Abstract

Empathy is an essential skill for social communication, enabling individuals to understand others and interpret social situations. It can be experienced not only through direct social interaction but also through media such as films, books, music, and games. The interactive nature of games makes them particularly effective tools for fostering emotional experiences and developing empathy compared to traditional media. Although games are often criticized for promoting negative behaviours, many studies show that games can enhance social skills, increase empathy, and support mental well-being. People play games to experience different emotions. Advances in technology have enabled new methods for creating richer emotional experiences in digital games. These approaches focus on players' emotional states and allow emotional data to influence gameplay. They can be used as game input, allowing the game to adapt to players' emotional states, or they can enable players to directly control the game through their expressions. These methods not only enhance player engagement but also influence players' emotional experiences during gameplay. Another significant area of emotion-related research involves serious games. This study examines games designed specifically for children with autism spectrum disorder (ASD). These games provide opportunities for children to practice facial and emotion recognition while developing empathy and social skills. By creating safe, non-judgmental environments, the games enable learning through enjoyable gameplay. Various methods have been employed to teach emotions and facial expressions, with most applications relying on observational learning approaches. This study bridges serious games and affective games by creating a proof of concept for children with ASD that facilitates active participation while providing opportunities for emotional experiences. To build a foundation for this vision, a pilot study was conducted with neurotypical adults. This initial phase focused on validating the facial expression mechanics and ensuring the game environment responds accurately to user input before adapting the system for a younger, more sensitive audience. In this exploratory study, emotions serve as an interaction tool within the game. The goal is to highlight emotions' role in nonverbal communication and help children understand cause-effect relationships. This prototype prioritizes happy and sad expressions for their recognition reliability, general familiarity, and ability to simply convey positive and negative emotions before moving to complex emotions. The game comprises two levels, each emphasizing a different emotion. To reinforce each level's emotional theme, the game includes a brief narrative and the objective in both levels is to navigate obstacles and reach their friend. At the start, the game asks players to choose their control method. Players choosing keyboard input receive instruction on the keys mapped to expressions before beginning the main game. Players selecting facial expression input proceed to a calibration phase that establishes personalized detection thresholds. Following calibration, players complete a tutorial section that introduces how facial expressions function in the game and teaches how to manipulate objects with this mechanic. The first section emphasizes sadness, with the objective of reaching and comforting a sad friend. Players use sad expressions to calm environmental elements and remove obstacles. The second section has a brighter and more playful atmosphere with a playground-like course. Players progress through this level by smiling, which makes the world colourful and moving environment and allows them to create new paths. The study was first tested with neurotypical adults to validate the expression mechanics, and the experiment was conducted with 20 participants. Participants evaluated the mechanics' usability and effectiveness by playing with both facial expression and keyboard controls. Condition order was counterbalanced across participants. The experiment explored how players interacted with the game's controls and how the experience influenced their emotional awareness and overall engagement. At the end of the game, players found both control methods provided comparable game experience, but facial expressions were more challenging. Keyboard control offered faster, more accurate responses and felt more natural due to familiarity. However, facial expression control was rated as more engaging and enjoyable despite its difficulty. Although happy expressions were easily detected in the game, sad expressions were difficult to detect for most players. This affected the experience and enjoyment between the two levels. Players struggled in the first level due to detection difficulties, but in the second level, most players completed it as quickly as with keyboard control because they naturally smiled while playing. Expression data were analysed separately for each level. In the first level, both expressions were used at similar average rates, but happy expressions (51,97%) were used more than sad expressions (48,03%). In the second level, happy expressions (79,08%) were used predominantly over sad expressions (20,92%). In addition, results from the study showed that positive emotions increased while negative emotions significantly decreased after playing the game. These results match the facial expression data, which recorded a high frequency of happy expressions during the game. The players' feedback indicated that the game was engaging overall and fostered an emotional connection with the virtual environment. The participants suggested that improvements to sad expression recognition and inclusion of additional emotions could increase enjoyment. They also noted that the design and mechanics in both sections demonstrated logical connections to their respective emotional themes. This study demonstrates how a particular game genre can be transformed into emotionally meaningful experiences through affect-aware mechanics. Future work will prioritize refining recognition accuracy through models tailored to children's facial movements, and shifting toward low-pressure, exploration-based design. The environment will be designed to react dynamically to the player's expression state, offering a responsive and gentle digital playground rather than fixed objectives. This creates a safe, self-paced space where children can experiment with their expressions without the fear of failure. This ensures that the technology serves as a bridge for practicing social and non-verbal communication.