Predictive error compensated wavelet neural networks framework for time series prediction

Abstract

Machine learning algorithms have gotten considerable attention and recognition in the context of addressing time series prediction problems; however, the task of constructing an accurate machine learning model with optimal architecture and hyperparameters becomes highly challenging if the data is non-linear, encompasses multi variable characteristics with chaotic or stochastic properties, and has sensitivity to environmental factors. In this context, common issues encountered in time series prediction models and frameworks include overfitting, a machine learning problem arising in situations with limited labeled data but a large variety of input data; generalization issues due to insufficient or inadequate input data; the need for extensive feature engineering to properly set internal weights in artificial neural networks; dependency on network parameters in developed solutions and limited adaptability to different problems; and being computationally expensive and time-consuming. Predictive Error Compensated Wavelet Neural Networks (PECNET) is an innovative artificial neural network architecture for time series predictions. It avoids overfitting by training the data separately in cascaded networks based on different frequency bands and using the remaining errors of each subsequent network as target data. In the PECNET architecture, data is fed into the networks with a low-frequency band in a wide time window, and the subsequent network is trained with narrower time windows and high-frequency data while using the error from the previous low-frequency network as the target data. This method improves the orthogonality of data features across time windows and lets you choose orthogonal features in data fusion applications. It also makes predictions more accurate as more networks are added, which lowers the risk of overfitting. Additionally, by applying wavelet transform as a feature extraction method to the various frequency components of each network , it is possible to distinguish and extract the variety of patterns present in the data. PECNET also overcomes the traditional normalization problems for non-stationary time series data by using adaptive normalization techniques. In conclusion, PECNET is a very good alternative for solving time series prediction problems due to its high prediction accuracy without overfitting, unique structure that allows adaptation to different problems independently of network parameters, and being computationally cheap and low time-consuming just with two-layer MLP structure. The PECNET model, due to its composition of cascade networks, modular feature extractions, and fusion networks, presents challenges in its implementation at the coding level. PECNET contains many sequential cascaded neural networks that are trained by each other's errors. When training the sequential networks with input data sequences, it is necessary to shift the input values to the previous time window. Otherwise, it would result in using data from the future, which is not feasible in practice. The continuous use of error data sequences as both label and input data in training and the use of the final error in the fusion network for predicting the remaining error value of the time series as both input and target data increase computational complexity for solutions involving numerous data sources and networks. This can lead to execution errors in time synchronization management. To overcome these challenges, PECNET framework software has been developed within the scope of this thesis, ensuring optimal use of memory and processor resources along with the modular design. In the development of the PECNET framework software, the C compiler and Python interpreter were utilized. NumPy, Pandas, PyWavelets, and Matplotlib libraries were used for data processing tasks. The PyTorch library was chosen for constructing the artificial neural network model due to its extensive modification features and options for interacting with the graphics processing units (GPUs). The design adhered strictly to object-oriented programming principles, and a syntax similar to Keras was used. Additionally, the machine learning application cycle followed the sklearn flow (fit-predict-eval). The PECNET framework is made up of various modules that work together. In the "models" module, the BasicNN class forms the core of the neural network architecture. This class manages key tasks like initializing the network, fitting data, computing loss, and adjusting the model during training. In the "network" module, specialized classes like ErrorNetwork, FinalNetwork, and VariableNetwork handle specific stages of the prediction process. ErrorNetwork focuses on correcting prediction errors; FinalNetwork integrates predictions from previous networks for a final output; and VariableNetwork manages training input data across different frequencies and integrates data fusion mechanism for multivariate data. In the same module, PECNET's functionality is encapsulated in the Pecnet class, which coordinates the workflow among different networks. It manages the data flow between cascaded networks, error compensation, and final prediction generation. The PecnetBuilder class provides a fluent interface to construct the Pecnet object. It sequentially adds various network components, ensuring a streamlined building process for the PECNET model. In the "preprocessing" module, the DataPreprocessor class plays a crucial role in data preparation. It lets users change how data is processed into different frequency bands, sampling periods, and sequence sizes that are appropriate for each of cascaded networks in the model. It also does scaling, adaptive normalization, denormalization, and wavelet transform with the help of other classes in the same module. This makes sure that the input data is in the best possible shape for the prediction pipeline. In the "utils" module, the Utility class facilitates hyperparameter optimization, offers tools for loading datasets and plotting results. Overall, PECNET's code-level functionality revolves around these classes, each contributing to the framework's ability to process and predict time series data efficiently. PECNET has already demonstrated successful outcomes on various datasets as a discrete code, showing promising results against existing machine learning models such as ARIMA, MLP, CNN, LSTM. In this study, as a framework form, it is initially tested on a historical time series dataset of daily adjusted close prices of Apple stocks and then compared with LSTM, which is known for its strong memory and sequence comprehension capabilities. As a result, in terms of the RMSE metric, the LSTM model has had an error of \$2.55, while PECNET has had an error of \$1.24. In terms of the R2 metric, the LSTM model has achieved a value of 0.94, whereas PECNET has reached 0.98. Following that, it is comparatively tested with LSTM for seismic energy estimation on real-time chaotic Electric Field Data (EFD) which is collected within the scope of earthquake prediction research project conducted at Istanbul Technical University (ITU). In this experiment, in terms of the RMSE metric, the LSTM model has showed errors ranging between 300J-400J, while PECNET has showed errors ranging between 130J-150J. In terms of the R2 metric, the LSTM results has fluctuated between 0.2-0.3, while PECNET has achieved values between 0.5-0.6. In both scenarios, PECNET outperforms LSTM, and the developed framework is being integrated into the portal software for real-time earthquake prediction. In conclusion, the developed modular and customizable framework facilitates the use of PECNET, which is highly performant and robust against overfitting, for various types of time series predictions by other developers in real-time machine learning systems without requiring specific coding knowledge of PECNET.