Multi-objective optimization of regenerative active suspension systems using data-driven reinforcement learning

Abstract

Multi-Objective Optimization of Regenerative Active Suspension Systems (RASS) of Electric and Hybrid Electric Vehicles is examined, within the context of ride comfort and energy regeneration, by means of data-driven reinforcement learning. The study uses a 2-DOF quarter car model that includes an electromagnetic actuator, so it can operate in four quadrants; therefore, it can regenerate energy as well as consume it. An important contribution to methodology was to extend Lopez's off-policy Q-learning algorithm to include cross-coupling terms in the LQR cost function, such that the controller could recognize between energy consuming and energy regenerating actions. The cross-coupling matrix N relates the mechanical power (actuator force times relative velocity) to the control objectives, so that the controller can optimize comfort and energy recovery simultaneously. The data-driven approach is based on Willems' Fundamental Lemma that states that persistently exciting input-output data trajectories are sufficient to describe all system behavior without having to perform explicit system identification. This model-free approach is especially useful for automotive applications because the vehicle parameters (e.g., sprung mass) are subject to varying loading conditions. Three multi-objective optimization approaches were used and compared: Grid Search for exhaustive deterministic mapping of the weight space (225 evaluations), Indirect NSGA-II for evolutionary optimization of LQR weights, and Direct NSGA-II for direct optimization of feedback gains as a theoretical roofline to ensure that the LQR structure did not artificially limit the solution space. Simulation results, based on both deterministic bump-and-dip profiles and stochastic road profiles, demonstrated that the proposed framework is effective. Ride-comfort oriented controllers had about 75% of the reduction in RMS sprung mass acceleration compared to passive suspension; however, these ride-comfort oriented controllers required net energy consumption. Energy-regeneration oriented controllers regenerated as much as 257 Joules; however, they provided ride comfort at nearly the same level as passive suspension. Therefore, balanced controllers are considered the most practical solutions to this problem since they regenerated between 135 and 156 Joules of net energy, while improving ride comfort by 50% compared to passive suspension. Results from the validation show that the RL gains learned by the RL agent matched the analytical DARE-derived LQR solutions, with virtually no difference; therefore, the RL agent converged to the global optimal solution(s). The fact that the Pareto fronts obtained from all three optimization methods match indicates that the proposed data-driven RL agent identified optimal linear control policies.