Learning general type-2 fuzzy logic systems for uncertainty quantification

Abstract

Deep learning has been widely used in various domains such as computer vision, natural language processing, large language models, autonomous driving, and robotics because it provides us with the flexibility to design complex architectures and achieve high performance. Consequently, we no longer hesitate to apply these models in high-risk areas like medical treatment and finance. However, these ambitions will fall short if our models yield unreliable outcomes under diverse conditions. In this context, uncertainty estimation becomes crucial: it tells us when to trust our predictions and helps us handle anomalies, outliers, and out-of-distribution examples. In recent studies, different deep learning models such as bayesian neural networks, deep ensembles, monte carlo dropout, gaussian processes, and quantile regression have been used for uncertainty estimation. For example, bayesian neural networks model the weights of a neural network as probability distributions, providing uncertainty by capturing posterior distributions over weights. However, this approach comes with a high computational cost and stability issues on large-scale datasets. On the other hand, quantile regression is easy to implement with a single model and simple loss functions, and it also scales well with large datasets. Type-2 fuzzy logic systems can be great candidates for estimating uncertainty. It has been shown that type-2 fuzzy logic systems are capable of handling uncertainties through their inherent structural model, which provides a degree of freedom, referred to as the footprint of uncertainty, for modeling these uncertainties. In recent studies, interval type-2 fuzzy logic systems, which are simplified versions of general type-2 fuzzy logic systems, have been used for modeling uncertainty while simultaneously generating highly accurate predictions. To achieve this, the type-reduced set of interval type-2 fuzzy logic systems is employed to estimate uncertainty through a pinball loss. On the other hand, the output of interval type-2 fuzzy logic systems is used for point-wise estimation with an appropriate empirical loss definition, resulting in a composite loss function. Furthermore, general type-2 fuzzy logic systems are also utilized to generate reliable prediction intervals and estimate highly accurate predictions by exploiting the shape and size of the secondary membership functions. It has been shown that using the secondary membership functions for point-wise predictions offers an efficient way to handle both uncertainty and accuracy. In most studies, general type-2 fuzzy sets, based on Mendel and John's definition, are widely used, although Zadeh first defined the concept of general type-2 fuzzy sets. This is due to the 𝛼-plane representation of general type-2 fuzzy sets, which facilitates the parameterization of the secondary membership function and demonstrates the equivalence between a general type-2 fuzzy logic system and a set of 𝛼-plane associated interval type-2 fuzzy logic systems. However, we identify some drawbacks in this definition, particularly regarding the direct dependency of the secondary membership functions on the primary membership functions. To define the secondary membership functions, the primary membership functions must first be defined. We believe this dependency could potentially reduce the learning performance of general type-2 fuzzy logic systems and also affect the design flexibility of general type-2 fuzzy sets negatively. In this master's thesis, we revisit the definition of general type-2 fuzzy sets as originally defined by Zadeh. We first present Zadeh's definition of general type-2 fuzzy sets. This structure offers the flexibility to design the secondary membership functions of general type-2 fuzzy sets without depending on the primary membership functions. In this context, we propose the mathematical foundations of both the secondary and primary membership functions, each of which is a type-1 fuzzy set. Afterwards, to define the output of Zadeh's general type-2 fuzzy logic systems, we integrate the 𝛼-plane representation into Zadeh's general type-2 fuzzy sets. Subsequently, we define the 𝛼-cuts of the secondary membership function and extract the equivalent lower and upper membership functions corresponding to the 𝛼-planes of Zadeh's general type-2 fuzzy set. These membership grades are then directly used to calculate the output of the general type-2 fuzzy logic system, which is formulated based on the 𝛼-plane approach. This approach enhances modeling flexibility and learning efficiency. Furthermore, we develop a method to address the curse of dimensionality problem that arises in fuzzy logic systems due to the rule firing strengths. This method adjusts the primary membership grades based on the input dimensions, effectively overcoming the challenges associated with high-dimensional datasets. Additionally, we propose parameterization tricks to ensure that the definitions of general type-2 fuzzy sets are not violated. These tricks allow us to formulate an unconstrained optimization problem, which can be efficiently handled using deep learning optimizers and automatic differentiation methods. We propose a deep learning framework to learn dual-focused Zadeh's general type-2 fuzzy logic systems. In this context, we first assign distinct roles to the interval type-2 fuzzy logic systems associated with each 𝛼𝑘 -plane within a composite loss function. This loss function consists of two components, simultaneously focusing on uncertainty and accuracy. To address both aspects, we present two loss definitions, leveraging the shape and size of the secondary membership function. For both loss definitions, we use only the type-reduced set of the 𝛼0-interval type-2 fuzzy logic system to learn the prediction interval by estimating the upper and lower quantile levels for a given confidence level in the uncertainty component of the composite loss function. On the other hand, for the accuracy component, we define two loss functions. For the first, we utilize the output of the general type-2 fuzzy logic system, and for the second, we use the output of the 𝛼𝑘 -plane interval type-2 fuzzy logic system as a point-wise estimator. Then, we present the comparative performance analysis of Zadeh's general type-2 fuzzy logic systems on high-dimensional datasets by comparing them to their Mendel and John's general type-2 fuzzy logic systems and interval type-2 fuzzy logic systems counterparts. The statistical results show that Zadeh's general type-2 fuzzy logic systems can serve as an effective approach for achieving highly accurate point-wise estimations and generating high-quality prediction intervals, meaning narrow bands that capture uncertainty at a given coverage level. We also present a deep learning framework based on Zadeh's general type-2 fuzzy logic systems to learn the inverse cumulative distribution function by estimating all quantile levels. This approach helps prevent the need for multiple training sessions for different desired coverage levels with given quantile pairs. Instead, any quantile pair can be selected to generate a prediction interval that provides the desired confidence level after one training section. In this context, we reformulate the output of the general type-2 fuzzy logic system by enforcing it to learn a specific quantile level, 𝜏, through the assignment 𝛼 = 𝜏. In this way, each output of the 𝛼-plane associated interval type-2 fuzzy logic system is set to learn a quantile level function. To learn the inverse cumulative distribution with a general type-2 fuzzy logic system, we reformulate the simultaneous quantile regression by sampling random quantile levels. To enhance learning, we develop an approach called adaptive simultaneous quantile regression, which incorporates a miscalibration measure during training. This approach allows us to generate additional quantile levels from miscalibration areas, ensuring they are trained effectively using the general type-2 fuzzy logic system. Afterwards, we compare our method with state-of-the-art deep learning methods to show the superiority of our method.