Scalable planning and learning framework development for swarm-to-swarm engagement problems with reinforcement learning

Abstract

Development of guidance, navigation and control frameworks/algorithms for swarms attracted significant attention in recent years. Since existing conventional aerial defense systems are optimized for a small number of heavy-hitting adversaries such as cruise missiles or fighter aircraft, these systems are often in a critical disadvantage against large-scale aerial swarm attacks that cover a wide area. Thus, defending against the aerial swarm attacks is one of That being said, algorithms for planning swarm allocations/trajectories for engaging with enemy swarms is largely an understudied problem. Although small-scale scenarios can be addressed with tools from differential game theory, existing approaches fail to scale for large-scale multi-agent pursuit evasion (PE) scenarios. To solve this problem, two main approaches are presented in this study. First, a reinforcement learning (RL) framework that controls the density of a large-scale swarm for engaging with adversarial swarm attacks is proposed. Although there is a significant amount of existing work in applying artificial intelligence methods to swarm control, analysis of interactions between two adversarial swarms is a rather understudied area. Most of the existing work in this subject develop strategies by making hard assumptions regarding the strategy and dynamics of the adversarial swarm. The main contribution is the formulation of the swarm to swarm engagement problem as a Markov Decision Process and development of RL algorithms that can compute engagement strategies without the knowledge of strategy/dynamics of the adversarial swarm. Simulation results show that the developed framework can handle a wide array of large-scale engagement scenarios in an efficient manner. Secondly, a reinforcement learning (RL) based framework to decompose to large-scale swarm engagement problems into a number of independent multi-agent pursuit-evasion games is proposed. Variety of multi-agent PE scenarios are simulated, where finite time capture is guaranteed under certain conditions. The calculated PE statistics are provided as a reward signal to the high level allocation layer, which uses an RL algorithm to allocate controlled swarm units to eliminate enemy swarm units with maximum efficiency. This approach is verified in large-scale swarm-to-swarm engagement simulations.