LEE- Bilgisayar Mühendisliği-Yüksek Lisans

Bu koleksiyon için kalıcı URI

http://hdl.handle.net/11527/19207

Gözat

Yazar "Azmoudeh, Ali" ile LEE- Bilgisayar Mühendisliği-Yüksek Lisans'a göz atma
  • Öge
    Facial expression analysis foran online usability evaluation platform
    (Graduate School, 2025-01-07) Azmoudeh, Ali ; Ekenel, Hazim Kemal ; 504211546 ; Computer Engineering
    Facial Expression Analysis (FEA) is crucial in human-computer interaction (HCI), because recognizing and responding to user behavior can greatly improve engagement and satisfaction. It helps evaluate user experiences in customer service and interactive online usability platforms. However, applying FEA in online environments faces many difficulties, such as different lighting setups, diverse facial structures, and spontaneous expressions. These factors can reduce the accuracy and reliability of current expression recognition models. To address these challenges, researchers have developed POSTERv1, an advanced deep-learning model. POSTERv1 uses a feature extraction block, a transformer block, and a multilayer perceptron (MLP) classifier to classify facial expressions. Its cross-fusion transformer encoder (CFTE) layer supports the interaction of appearance-based and structural features, helping the model capture emotional cues. By including cross-fusion multi-head self-attention (CFMSA), POSTERv1 focuses on the most important areas of the face, reducing irrelevant features. A simplified version of POSTERv1 was also introduced to handle practical and computational constraints. This version removes the pyramid structure of the original model and uses only one CFTE block, making it faster and still highly accurate in most tests. In this study, we built a custom dataset of videos showing customers performing tasks in different online shopping environments. We aim to observe user behaviors on shopping platforms to evaluate the usability of the designed platform and designed two scenarios: one with the help of a moderator and one without it. In the moderator-assisted setting, participants received guidance and support when faced with difficulties. These conditions allowed us to compare emotional responses in moderator-assisted versus independent scenarios. We also evaluated and compared the latest FEA models on popular public datasets such as AffectNet, RAF-DB, FER2013, and CK+. We specifically tested how well models trained on AffectNet would perform when cross-evaluated on FER2013 and CK+ datasets. Notably, the simplified POSTERv1 model delivered better and faster performance while preserving accuracy in most public datasets than the original POSTERv1. Besides offering faster performance, the simplified model showed slightly greater robustness, making it a possible candidate for real-time emotion analysis. Based on these promising results, we used the simplified POSTERv1 to predict emotions in our custom dataset. The model showed reliable speed and efficiency in expression recognition, although factors like lighting and individual facial features had impacts on performance. In addition, an additional subset of our custom dataset included text-based sentiment annotations, derived from participants' speech, labeled as positive, negative, or neutral. We compared predicted facial expressionswith sentiment labels to assess the performance of simplified POSTERv1 model. Outputs from the custom dataset utilizing simplified POSTERv1 model showed that having a moderator encouraged more positive expressions and fewer negative reactions, demonstrating the model's ability to capture changes in emotional responses. However, the absence of annotations limits the ability to thoroughly assess the model's performance. Factors such as poses, like looking downward, and variations in environmental conditions, including lighting, can impact the reliability of the model. These considerations underscore the need to address aspects like illumination, facial diversity, and contextual user interactions when developing and deploying FEA tools. In conclusion, this thesis demonstrates how advanced deep-learning models like POSTERv1 and its simplified variant can effectively handle real-world challenges in FEA. These models achieve reliable real-time FEA by focusing on the most important facial features and adapting to different environmental and social conditions. Tests on both public datasets and a custom online shopping dataset show that the simplified POSTERv1 is faster and more efficient for real-time predictions, making it suitable for practical HCI applications. However, issues such as lighting conditions and certain facial poses point to areas that need further study. These challenges point to areas for further improvement, but the approach presented in this thesis can help designers develop online platforms that better understand and respond to user behavior, ultimately improving engagement and satisfaction across various interactive digital services and online usability evaluation platforms.