Fractional-order derivative based adaptive methods for control and optimization

Abstract

Fractional calculus generalizes classical differentiation and integration from integer orders to non-integer orders, thereby offering enhanced flexibility and effectiveness in many areas of science and engineering. By adapting these orders dynamically, one can achieve a smooth transition between different levels of differentiation and integration. This thesis investigates such adaptive fractional‑order derivatives, in which the order varies in response to external parameters. Making the differentiation order adaptive unlocks new possibilities in fields where differentiation is fundamental, such as control and optimization. We first develop adaptive fractional-order derivative based methodologies to tackle long-standing challenges in fractional-order controllers, achieving improved performance under both nominal and uncertain operating conditions. Secondly, we introduce methods leveraging fractional-order derivatives to obtain robust and adaptive optimization of deep neural networks, to provide the necessary flexibility in adjusting robustness against noise. Our experiments include both simulated and real‑world systems and datasets, advancing both the theoretical understanding and practical applications. The unifying theme of these three approaches presented in this thesis is leveraging the advantages of adaptively applying fractional‑order derivatives to advance performance in control systems and optimization. In the realm of control systems, traditional controllers often employ integer-order integrals and derivatives, which constrain their performance and flexibility. Fractional operators have been shown to significantly enhance the modeling and control of dynamic systems, providing improved performance and adaptability. However, many existing fractional controllers in the literature rely on fixed‑order fractional integral and derivative operators, limiting their effectiveness in complex systems and rapidly changing environments. By contrast, variable‑order (VO) fractional controllers offer a dual advantage: superior overall performance compared to fixed‑order controllers and effective disturbance rejection through dynamic adjustment of the fractional orders. Although studies on VO fractional controllers demonstrate potential in specific domains, developing online tuning methods for VO fractional operators that preserve system stability, particularly for real‑world applications such as quadcopter control, remains a significant challenge. Currently, no definitive method exists for establishing the stability of VO fractional controllers, and their practical applications have been limited. To address this, this thesis introduces an online tuning method for the VO fractional derivative based on a metric known as normalized acceleration, which quantifies the convergence rate toward the desired reference. By adaptively adjusting the fractional-order derivative, the controller can strategically accelerate or decelerate the system response, thereby enhancing performance. Extensive simulation studies and real‑time experiments on quadcopter position control, conducted under wind gust disturbances and measurement noise, demonstrate that this adaptive tuning not only outperforms conventional fixed‑order controllers but also preserves closed‑loop stability, as verified by D‑decomposition‑based stability analysis. A critical challenge in optimization is managing noisy data and outliers, since optimization parameters must be chosen according to the dataset's noise characteristics. In modern optimization methods, especially for deep neural networks, the choice of loss (objective) function is crucial, as it directly influences model convergence speed, robustness, and generalization performance from noisy data. Conventional loss functions such as L1, L2, log‑cosh, and Cauchy each offer unique benefits; for example, L1 loss is often preferred for noisy datasets over L2 loss for its lower sensitivity to large errors. However, no single loss function is universally optimal. Recognizing this limitation, this thesis introduces a novel family of adaptive robust loss functions based on adaptive fractional-order derivatives, for both areas of supervised learning: regression and classification. For regression, we propose Fractional Loss Functions (FLFs), which leverage a fractional‑order derivative \(\mu\) and adaptively adjust it to control the shape of any given regression loss. Increasing \(\mu\) reshapes the loss landscape to reduce outlier influence and enhance robustness, while decreasing \(\mu\) accelerates convergence. To address the challenge of selecting an optimal \(\mu\) a priori, we transform it into a parameter that is dynamically adjusted during optimization. This adaptive mechanism ensures the loss function self‑tunes to balance robustness and convergence speed, mitigating the adverse effects of noisy data. Comparative experiments on battery cycle‑life prediction, system identification, image synthesis, and denoising demonstrate that models trained with adaptive FLFs significantly outperform those using conventional loss functions. Next, the thesis extends the adaptive fractional-order derivative framework to the optimization for classification tasks. The current paradigm of machine learning is driven by large models trained on vast datasets. However, this reliance on large-scale data acquisition often prioritizes quantity over quality, leading to datasets that contain significant label noise, which can degrade model performance and generalization. In supervised classification tasks, data labeling—whether performed automatically, by humans, or through web scraping—produces large datasets that frequently contain mislabeling errors. These errors can result from human mistakes, limited expertise, or the error-prone nature of automated methods. Such mislabels introduce various forms of label noise that negatively affect model training. While loss functions like the mean absolute error offer theoretical robustness, they often suffer from slow convergence. To overcome this challenge, this thesis introduces Fractional Classification Loss (FCL), which is derived by applying the adaptive fractional-order derivative to the Cross-Entropy loss and uses it together with the mean absolute error loss. This loss formulation uses a single interpretable robustness hyperparameter, fractional derivative order $\mu$, which is adaptively tuned during optimization. By dynamically balancing the trade-off between enhanced robustness and convergence speed, FCL ensures that the loss function adapts to the noise characteristics of the dataset without the need for manual hyperparameter tuning. To assess the effectiveness of FCL, we conduct ablation studies and first demonstrate that mistuned hyperparameters degrade performance in existing robust losses, an issue avoided by FCL's adaptive learning of \(\mu\). Next, we evaluate performance across different initializations of \(\mu\) and compare models with fixed versus adaptive \(\mu\), in addition to assessing the time complexities of existing losses and FCL. Finally, experiments on MNIST, CIFAR-10, and CIFAR-100 with both symmetric and asymmetric label noise confirm that FCL consistently achieves state-of-the-art performance while automatically tuning \(\mu\) for robustness.