Fog computing-based real-time emotion recognition using physiological signals

Abstract

Emotion recognition plays a pivotal role in affective computing and human-computer interaction, with physiological signals such as ElectroEncephaloGram (EEG), ElectroCardioGram (ECG), ElectroMyoGram (EMG), respiration, and Galvanic Skin Response (GSR) are more reliable indicators of emotions. Unlike facial expressions, gestures, or speech signals, these physiological signal measures offer greater consistency in detecting emotions. This reliability arises from their reduced susceptibility to subjectivity and external factors, such as environmental noise and language barriers. There are numerous uses for emotion recognition to monitor multi-user emotional states simultaneously in real-time such as smart homes, workplaces, education, healthcare, and entertainment. For smart homes, emotion recognition may be integrated to provide automation for family members by modifying lighting, music, and other environmental elements according to the overall emotional states of the households. For workplace wellness, employers may have obtained the ability to track the health of many employees, recognizing signs of stress and tiredness; thus, they can take timely actions for a healthier work environment. In educational settings, emotion recognition can provide educators with insights into student engagement and understanding for groups of students. That helps educators adjust their teaching strategies to suit the demands of each class and improves the quality of the learning process overall. In a clinical setting, emotion recognition can help healthcare professionals to monitor the emotional states of multiple patients simultaneously, receiving immediate alerts and providing timely interventions. In the entertainment industry, especially in multi-player gaming and virtual reality environments, emotion recognition may enhance the experience by adapting environmental settings to the emotional states of individual users in real time while ensuring low latency and high performance. This study explores the implementation of fog computing for real-time emotion recognition using physiological signals. Implementing an emotion recognition system in the Internet of Things (IoT) requires powerful computational resources. Therefore, existing studies highlight the potential of cloud and edge computing architectures for emotion recognition but reveal a gap in leveraging fog computing for scalable, real-time applications. Building on these insights, this study integrates fog computing to improve latency, response time, and scalability efficiency. Hence, sensor-to-cloud architectures face challenges like latency, high bandwidth requirements, and security concerns. Fog computing provides low latency, enhanced security, scalability, and efficient resource utilization by processing data closer to its source, ensuring reliable real-time performance for multi-user emotion recognition systems. The research adopts a comprehensive methodology, starting with the use of the DREAMER dataset, which contains EEG and ECG signals recorded from 23 participants under various emotional stimuli. Signals in the dataset were segmented into 3-second windows with a 2-second overlap. Then, the data in the windows were preprocessed, and 36 time-based statistical features were extracted from the signals. By merging the features obtained from 2-channel ECG and 14-channel EEG signals, a data vector of 576 features was obtained for each sample. The dataset was divided into training and testing sets to train and evaluate the machine learning models. Eight machine learning models are employed to predict emotions. Based on accuracy, recall, precision, and F1 score metrics, the Light Gradient Boosting Machine (LGBM) model demonstrated the best performance. Since the models are designed for real-time use, the inference time for a single sample was measured. The LGBM model provided the highest accuracy with an acceptable prediction time, making it the preferred choice for the proposed real-time system. The LGBM model's inference times on the worker and cloud devices were 7.26 ms and 2.85 ms with an accuracy of 85.27 %. Accuracy variations were analyzed as the number of features changed, and accuracy plateaued at around 85 % after 44 features were used. The maximum accuracy of 86.25 % was achieved using 136 features, resulting in an average response time of 61.01 ms. However, considering resource utilization and time performance requirements for real-time systems, the system was configured to use 48 features, yielding 84.85 % accuracy with a 33 % reduction in processing time. In the proposed system, the emotion recognition process runs every second and takes about 40 ms. This pre-trained machine learning model, based on 48 physiological signal features, was integrated into the fog computing architecture, allowing for real-time emotion recognition. A fog computing architecture is designed, comprising cloud, broker, and worker nodes, to manage data processing tasks efficiently. All computation unit components of the architecture were tested in real-time scenarios using the pre-trained model. Unit performances were evaluated based on metrics such as latency, queuing delay, jitter, total response time, and resource usage, with experimental results showing that a worker node can efficiently handle computational tasks. Overall, the emotion recognition procedure begins every second and takes approximately 40 ms, including 4 ms of latency and 33 ms of execution time. Therefore, the results demonstrate fog computing's superiority over edge and cloud computing. The proposed fog computing system also outperforms existing studies in response time for real-time feature extraction from physiological signals, confirming that the fog architecture is well-suited for a real-time emotion recognition system. The system's scalability and usability in a multi-user environment were also assessed. Cloud, broker, and worker devices supported up to 11, 5, and 6 users, respectively. Although the single worker device could serve fewer users than the cloud, the fog architecture addressed this by issue incorporating multiple, cost-effective worker devices. Furthermore, using multiple processes for data processing on the worker device enhanced multi-user capacity with an optimal configuration of 6 processes allowing a worker device to serve up to 11 users. The fog architecture, utilizing multiple workers, was also evaluated. Stress tests revealed that the system can scale to accommodate 30 users with 3 worker devices. The study's findings highlight the effectiveness of fog computing in real-time emotion recognition. There are three key results including that the LGBM model achieved the highest accuracy of 85.27 % and mean single inference time of 7.26 ms for worker device, outperforming other machine learning models. The second key result is that fog computing significantly reduced latency and response time compared to cloud-based architectures, ensuring faster processing of physiological signals. Lastly, the system with 3 worker nodes and a 6-process configuration demonstrated scalability, handling up to 30 users with stable response times of approximately 40 ms. Resource utilization was optimized, with fog nodes distributing computational workloads effectively to avoid bottlenecks. Several research areas remain unexplored in this study such as optimizing computational resources, expanding the dataset and emotion classes, conducting real-world experiments to assess the practical usability of the proposed system, and implementing deep learning models within the fog computing framework using larger datasets.