Resource allocation mechanisms for end-to-end delay optimization of 5G URLLC services

Abstract

5G and beyond networks aim to satisfy the challenging requirements of a variety of vertical services and domains such as automatic driving, health services, augmented/virtual reality and gaming, and streaming in addition to traditional mobile communication services and applications. Each service has its own specific QoS requirements such that enhanced mobile broadband (eMBB) services aim to provide higher data transmission rates and higher spectral efficiency for bandwidth-hungry applications and high-volume data processing services while ultra-reliable low-latency communications (URLLC) services are optimized for critical applications with stringent delay and reliability requirements. Allocating resources to higher priority URLLC services for its stringent delay and reliability requirements needs to be done carefully without jeopardizing the throughput performance of eMBB services. Simultaneously meeting the requirements of distinct services is a challenging task and requires innovative and intelligent resource allocation solutions. Network Slicing (NS) and Multi-Access Edge Computing (MEC) have emerged as promising enablers that can be utilized to manage network resources to satisfy application-specific requirements. NS enables the formation of end-to-end logical networks over an underlying infrastructure, spanning multiple network segments. This allows for cooperative allocation of resources across access, transport, and core components. NS simplifies the management of network resources, making it easier to provide tailored services for various applications, such as eMBB and URLLC. MEC serves as a crucial building block in mobile networks by facilitating the execution of computation tasks offloaded through wireless links. MEC enhances efficiency by bringing computational capabilities to the network's edge, resulting in faster response times and enhanced user experiences. Resource management for RAN slicing is a highly challenging task due to limited radio resources including frequency and power, stringent service requirements, dynamic wireless channel conditions, and random traffic arrivals. Effective management of communication and computation resources in RAN is needed to optimize resource utilization and satisfy service requirements. Innovative and intelligent resource allocation solutions utilizing artificial intelligence (AI) and machine learning (ML) will be key enablers to jointly optimize the performance of multiple services by providing real-time adaptation of the edge with respect to time-varying network conditions. Reinforcement Learning (RL) has become a popular tool for intelligent resource management in RAN. RL is an interdisciplinary area of machine learning in which an agent is trained to make a sequence of decisions by interacting with the environment whereby the agent chooses an action from a set of possible actions after observing the current system state and then receives a reward. Upon executing the action, the environment transitions to a new state. The primary goal of RL algorithms is to optimize action selection to maximize cumulative long-term rewards. In this study, we employ deep Q-learning (DQL) as our RL method. DQL employs a deep neural network (DNN) to achieve an action selection policy which maximize the expected cumulative reward. This sub-discipline of RL is referred to as deep reinforcement learning (DRL). In this thesis, we propose DRL-based resource management mechanisms for RAN slicing where URLLC and eMBB slices co-exist. The proposed resource distribution mechanisms aim to maximize the throughput for eMBB traffic while simultaneously satisfying the delay requirement of URLLC traffic. DRL-based resource allocation design includes hierarchically placed layers. The main DRL agent located at the upper layer performs inter-slice resource distribution while the URLLC and eMBB sub-agents are responsible for intra-slice resource allocation. In addition, we presented methods to reduce state and action spaces for computationally efficient DRL training and scalable design. Furthermore, we proposed methods to make the agents independent from each other in the hierarchical resource allocation design. In the first study, DRL-based resource allocation design is utilized for downlink transmission in RAN. In this study, the communication resource under consideration is the Resource Block (RB). In the second study, the resource allocation system is designed to jointly allocate communication (resource block) and computation (CPU cycle frequency) resources available in the base station and MEC server for task offloading operations. In both studies, packet delay analysis is presented by including queuing, transmission and computation delays. Moreover, resource allocation problems are formulated by including the QoS requirements of the URLLC and eMBB slices. The experiments are performed using various traffic scenarios and numerical results are compared with different baseline algorithms.