Fuel cell air supply system control using deep q-network

Abstract

The increasing global population and economic growth are causing a continuous rise in energy consumption. Energy production remains an indispensable component of industrial development for every country. However, traditional fossil fuel sources such as coal, oil, and natural gas are causing serious environmental problems due to their rapid depletion as well as greenhouse gas emissions that accelerate global warming. Considering the deteriorating environmental conditions, rising global temperatures, and greenhouse gas emissions, the transition from fossil fuels to renewable energy sources has become imperative. In this context, clean and efficient energy sources such as solar, wind, geothermal, and hydrogen emerge as the most viable alternatives to replace conventional sources. Particularly hydrogen exhibits great potential as both a fuel and an energy carrier and storage medium due to the intermittent nature of solar and wind energy. Through the carbon-free alternatives it offers, hydrogen is increasingly being recognized as a globally viable energy source and fuel alternative. Hydrogen is positioned as the primary fuel for a new generation power generation device that produces electricity, heat, water, and zero pollution using oxidant and hydrogen. The fuel cell (FC) operates through an electrochemical redox reaction: the reaction of hydrogen and oxygen produces energy, water, and heat. At the anode, hydrogen undergoes catalytic decomposition to form protons and electrons; while protons pass through the proton exchange membrane to the cathode, electrons are transferred to the external circuit to provide energy to the power load. At the cathode, oxygen reacts with protons and electrons to produce water and heat. When evaluating the environmental impacts of mobility vehicles, three main propulsion systems emerge: internal combustion engines, battery electric vehicles, and FC electric vehicles. Internal combustion engines produce significant amounts of carbon dioxide, nitrogen oxides, and particulate matter emissions as they are based on the direct combustion of fossil fuels. Battery electric vehicles, while offering zero tailpipe emissions, have disadvantages such as limited range and long charging times. Proton exchange membrane fuel cells (PEMFC) have become a prominent technology, particularly in the automotive sector, due to advantages such as zero emissions, low noise, fast response time, long range, and short refueling time. PEMFC systems are electrochemical devices that directly convert chemical energy into electrical energy and find a wide range of applications from portable power applications to hybrid electric vehicles and FC electric vehicles, with characteristics of high efficiency, high power density, low operating temperature, and minimal pollution. PEMFC systems generally consist of 5 subsystems: thermal management subsystem, water management system, hydrogen supply subsystem, air supply subsystem, and power electronics management subsystem. While the hydrogen supply subsystem uses fast electrical valves to minimize the pressure difference between the anode and cathode, the air supply subsystem provides the necessary oxygen to the cathode through a compressor. Since water and temperature dynamics change much more slowly compared to air flow dynamics, many studies assume that the FC operates under constant temperature conditions. The power electronics subsystem controls load connection and includes application-specific converters. The most critical component of the air supply system is the air compressor. For stable operation of the FC, the presence of sufficient oxygen at the cathode side is mandatory. The oxygen excess ratio (OER, denoted with λ) is defined as the ratio of the amount of oxygen supplied to the cathode to the amount of oxygen consumed in the electrochemical reaction and is one of the fundamental indicators of system efficiency. Due to the slow air flow dynamics of the compressor and air supply manifolds, when the load current changes suddenly, oxygen starvation can occur at the cathode if sufficient oxygen cannot be rapidly supplied. This situation can lead to the formation of hot spots on the membrane surface and irreversible damage to polymeric membranes. Additionally, oxygen starvation causes serious problems such as low power generation, membrane degradation, and even shortening of stack operating life. The manifold system is modeled as a collective volume representing all pipes that supply gas to the stack and drain gas from the stack. The supply manifold pressure depends on downstream cathode dynamics, and the dynamics of cathode variables depend on both the air from the supply manifold and the amount of external current load. Based on the ideal gas assumption, the partial pressures of oxygen and nitrogen can be derived through differential equations. OER control is of great importance for both the economic efficiency and safety of PEMFC systems. The system faces challenges such as nonlinearity, parametric uncertainty, and load disturbances. During rapid load current increase, the chemical reaction within the FC accelerates substantially, and this situation can lead to excessive oxygen consumption and even oxygen starvation that can cause permanent cell damage. The OER at the cathode must be kept greater than 1; however, this should not be interpreted to mean that higher OER always implies better performance. Conversely, excessively high OER causes the air compressor to consume more power and consequently reduces net power output. Net power is defined as the difference between stack power generation and parasitic power consumed by the compressor, and at high loads, these parasitic losses can reach up to thirty percent of the FC's power consumption. Research shows that maximum net power for different stack currents and OER values is obtained when OER is between 2 and 2.5. Therefore, the primary objective of OER control is to prevent oxygen starvation while minimizing compressor power consumption and thus optimizing system efficiency. The optimal OER reference varies depending on load current and is generally found in a value range between 1.5 and 3. One of the greatest challenges encountered in control design of PEMFC systems is the complex and highly nonlinear dynamic structure of the system. Traditional model-based control approaches require an accurate system model; however, PEMFCs exhibit nonlinear and time-varying uncertainties due to aging and degradation. This situation creates significant limitations as the performance of methods such as model predictive control (MPC) is largely dependent on an accurate system model. In this context, reinforcement learning-based approaches offer significant advantages. The Q-learning algorithm is a model-free approach that determines an optimal policy without requiring estimation of the environment's dynamics. This method works in a data-driven manner to address challenges such as system nonlinearity, parametric uncertainty, and load disturbances, and does not require characterization of system dynamics. The findings demonstrate that PI and reinforcement learning collaboration markedly improves transient OER control by increasing response speed and diminishing both safety-critical OER violations and efficiency-reducing deviations, especially when the baseline PI controller is inadequately calibrated. The decline in steady-state accuracy in specific configurations underscores the necessity for precise reward shaping and action constraint design to prevent ongoing corrective behavior approaching equilibrium. The findings indicate that hybrid control performance is fundamentally dependent on configuration and support that reinforcement learning is most effective as a supplementary corrective mechanism rather than a standalone alternative for classical PI control in PEMFC air management systems, where durability and efficiency must be concurrently maintained.