A new fuzzy cluster-aware regularization of neural networks

Abstract

A frequently encountered case in developing a classification model is the presence of embedded clusters, formed by data used for training. A good example for this case may be the differences in purchasing styles of e-commerce customers in a purchase propensity modelling problem. While some customers prefer a detailed research about prices, functionalities and comments, some others may need a shorter examination to make a purchase decision. Although feeding such cluster information into the classification model has been shown by recent studies to improve the prediction performance, this valuable information has been largely ignored in classical modeling techniques in general and neural networks in particular. This paper proposes a feedforward neural network regularization method which incorporates cluster information into networks’hidden nodes. Within the forward propagation and backpropagation calculations of the network, a non-randomized matrix is used to assign hidden nodes to different observation clusters. This matrix manipulates the activation value of a hidden node for each observation in line with the observation’s membership degree to the cluster that the node is assigned to. Also, through the alternating use of randomized binary and non-randomized matrices within iterations, the proposed method successfully fulfills the regularization task. Experiments were performed for different settings and network architectures. Empirical results demonstrate that the proposed method works well in practice and performs statistically significantly better than existing alternatives.