Reinforcement learning based battery thermal management controller for electric vehicle charge time optimization using horizon data

Abstract

Electric vehicles (EVs) are gaining popularity today due to the shift towards sustainable transportation. They offer significant benefits in terms of reduced emissions when compared to internal combustion engine (ICE) vehicles. Battery is a fundamental component of EVs, playing a critical role in effecting vehicle performance and range, in other words, convenience of the vehicle for users. Today most battery packs in the EVs use lithium-ion based battery cells, because they have higher energy and power density, longer life cycles and higher efficiency when compared to other battery types like lead-acid or nickel-based batteries. Batteries provide the electrical energy by electrochemical reactions that is happening inside them. These reactions are affected by the temperature conditions thus influencing the performance of the battery. Batteries generate heat during charging and discharging due to their internal resistances, extreme temperatures may lead to battery degradation and reduction in life cycle. If the temperature exceeds certain limit, it can lead to thermal runaway, where the battery becomes unstable and may catch fire or explode. That is why EV batteries have a battery management system (BMS), responsible for constantly monitoring the battery pack in order to ensure safe and robust operation. The BMS monitors voltage, current, temperature and state of charge (SOC) of the battery while controlling discharge and charge power limits and also it is responsible for battery pack's temperature management. Thermal management system (TMS) provides the necessary cooling or heating for the battery pack and rest of the vehicle systems. With the algorithm inside the BMS it controls the TMS to ensure the battery temperature remains within the optimal range for every operating condition. Charge performance is also important when it comes to users convenient. Battery packs charge performance usually depends on the cell types that are used, battery pack configuration, SOC and temperature conditions. In order to have advertised maximum charge power of the EV, battery pack temperature has to be in optimum range. That is not always the case where too hot or cold ambient operation conditions thus charge power is usually limited during these conditions. That is why also TMS control is important in order to keep battery pack in optimum operating temperature range to get improved charging performance. The traditional approach for battery thermal management control is the rule-based control method. The rule-based control method uses battery temperature and ambient temperature as input for look-up tables to create cooling or heating requests. Rule-based control methods are relatively easy to implement which makes them suitable for many applications. However, since this method uses predefined rules, it is not dynamic and limited by the set of rules, so it may not always give the optimum performance in terms of energy consumption or operating temperature. In order to address these issues of rule-based controller, optimum control methods are also widely used for TMS. Model predictive control (MPC) is the most used method in this area. For using MPC, system has to be modeled in high fidelity in order to predict system behaviour based on operation conditions. After that a cost function is created depending on the application and the MPC algorithm solves the optimum control problem by minimizing the cost function and it controls the heating or cooling actions accordingly. This method requires more computational resources compared to rule-based controllers but results in more efficient control of TMS. Artificial neural networks (ANNs) also widely used to model system behaviour for TMS. When working with real systems, these approach does not require to create a high fidelity system model since ANNs learns from the inputs and outputs of the system to predict its behaviour. ANNs are used in combination with some optimum control methods in order to increase their real time calculation performance and reduce the modeling dependency. Also as a subset of ANNs, reinforcement learning (RL) method is an optimum control method that ANNs are trained by using a reward function that represents goals to achieve in the system. RL aims to learn to react to inputs of the system that maximizing the reward. Reward function must be created in a way that rewards the good actions that aligned with the goal and punishes the actions that are undesired so that RL can learn to optimize its outputs to achieve that goal. As the main goal usually determined as optimized energy consumption for thermal management. This study focuses on minimizing the charging duration of an EV by using predictive battery thermal management control. A RL based control method is proposed for battery TMS for different environment conditions using horizon data. A drive cycle is used for the use case of driving until charge station and starting to charge in different environment conditions. Vehicle dynamics, propulsion, energy storage, and comprehensive thermal systems are modeled in this study to create a training and testing environment for RL based controller. A cost function is created based on the optimization problem of charge duration and energy consumption for thermal management of the battery. The RL controller is trained with the simulation environment to minimize the cost using the drive cycle as horizon data. Relation between ambient temperatures and charge performances are investigated with RL based TMS method and traditional rule based TMS method. RL controller and the rule-based controller is investigated for the same initial conditions like temperature and SOC for the given test scenario. Based on the results, 14% decrease in charging duration observed using RL controller compared to the rule-based controller.