Multimodal vision-based driver monitoring system in autonomous vehicles

Abstract

Driver fatigue and distractions are major causes of accidents and fatalities on the roads, and there is a pressing need for effective technologies to detect and mitigate these risks. In this thesis, we presented a vision-based driver monitoring system (DMS) that aims to improve road safety by detecting and alerting drivers to potential dangers or distractions, and by providing a more comprehensive and robust representation of the driver's actions and behaviors. The DMS is based on a multimodal data source, comprising a camera, vehicle CAN data, and other sensors, to provide a wide range of information about the driver and the driving environment. To improve the accuracy and reliability of the DMS, we developed a unique dataset containing synchronized output from multiple sensors in both RGB and IR formats, which can be used for training, testing, and validation of the DMS. This dataset is unique in that it contains synchronized output from multiple sensors in both RGB and IR formats, which allows for the development and evaluation of DMS modules that can operate on both types of data. To detect and classify different types of distractions and impairments, we developed a hybrid visual distraction module that combines head pose and gaze estimation. We also developed an adaptive gaze estimation model that works on both RGB and IR data, and we adapted the HourGlass CNN to work on IR data. To detect drowsiness, we used the Mediapipe framework and Empatica e4 wristband, and to detect phone usage, driver presence, and eating/drinking, we used a combination of computer vision and detection algorithms. To evaluate the performance of the DMS, we used a variety of metrics and benchmarks, including accuracy, precision, false positive rate, etc. The results showed that the DMS achieved high accuracy and reliability in detecting and classifying different types of distractions and impairments. Overall, this work makes a significant contribution to the field of driver monitoring and road safety by providing a novel and effective approach for detecting and mitigating driver fatigue and distractions using multimodal data and a hybrid visual distraction module. The unique dataset and the proposed DMS can be used as a benchmark for future research and development in this area. In addition, the results of this study have the potential to inform policy and practice related to driver monitoring and road safety, and to improve the safety and efficiency of transportation systems.