Dynamic difficulty adjustment by changing enemy behavior using reinforcement learning

Abstract

Dynamic difficulty adjustment (DDA) systems are essential in modern gaming to provide to the diverse skill levels of players. These systems ensure that games remain challenging yet enjoyable by automatically adjusting the difficulty based on the player's performance. Traditional fixed difficulty settings often fail to provide an optimal experience for all players, leading to frustration for less skilled players and boredom for more skilled ones. Implementing DDA systems aims to enhance player engagement and satisfaction by maintaining an appropriate level of challenge throughout the game. Various techniques have been explored to implement DDA systems. One common approach is dynamic scripting, which involves adjusting the game's rules and parameters in real-time based on the player's actions. This technique allows for a more responsive and adaptable gaming experience. Other methods include player modeling, which uses data from the player's performance to predict their future behavior and adjust the difficulty accordingly, and machine learning algorithms that continuously learn and adapt to the player's skill level over time. Reinforcement learning (RL) has emerged as a powerful tool for developing DDA systems. In this approach, Artificial Intelligence (AI) agents are trained to play the game and learn optimal strategies to maximize their rewards. These agents can then dynamically adjust the game's difficulty by modifying the behavior of non-player characters (NPCs) or the game's mechanics based on the player's performance. This method allows for a more nuanced and effective DDA system that can adapt to the player's skill level in real-time. In this thesis, a DDA framework was created to be used in various gaming environments. Three different game scenarios were developed to demonstrate its effectiveness: a basic shooter, a basic action game, and a complex action game. Each of these scenarios provided a unique set of challenges and complexities, allowing for a thorough evaluation of the framework's adaptability and performance. The developed framework is capable of analyzing the performance of AI agents against human players and suggesting new difficulty levels accordingly. All parameters for these difficulty adjustments can be modified in the editor, providing game developers and designers with the flexibility to tweak the system to suit their specific needs. This capability ensures that the DDA system remains effective and relevant across different games and player demographics. The BrainBox plugin, developed for Unreal Engine, is a versatile tool designed to facilitate the creation of environments for DDA systems. It seamlessly communicates with a Python backend, managing the complex interplay between game environments and AI training processes. The plugin efficiently handles the creation and management of game environments, executes player and agent actions, calculates rewards, and xxii implements difficulty change procedures. This integration ensures that game developers can easily implement and tweak DDA systems, enhancing the gaming experience by maintaining an optimal level of challenge. A Python backend was created for training and evaluating the RL models. This backend communicates with the game environments created in Unreal Engine using Transmission Control Protocol (TCP), facilitating seamless integration between the training process and the game. The backend is responsible for managing the training data, running simulations, and updating the models based on the results, ensuring a robust and efficient training process. In the complex action game scenario, models were trained and evaluated to determine their effectiveness. The models were ordered by their median rewards across 20 episodes and mapped into difficulty levels. This process allowed for a detailed analysis of the models' performance and provided insights into their learning capabilities and adaptability to different levels of game complexity. A test case was conducted with 20 participants of varying game experience and skill levels. In this test case, the DDA was benchmarked with multiple testers. All sessions were logged, and a comprehensive analysis was performed on the collected data. This analysis provided valuable feedback on the system's performance and effectiveness in real-world scenarios, highlighting areas for improvement and potential future developments. In conclusion, the DDA demonstrated a robust capability in tailoring game difficulty to individual player needs. Its ability to adapt in real-time, guided by both player performance and feedback, highlights its potential to enhance gaming experiences significantly. The findings suggest that the DDA system not only improves player engagement and satisfaction but also offers a scalable solution for balancing difficulty in a wide array of games. Future implementations could benefit from refining this system to further optimize player retention and enjoyment, ensuring the game remains accessible and rewarding for all players regardless of their initial skill level.