Fusion of RGB and thermal image pairs for object and face liveness detection

Abstract

Reliable visual perception is essential for intelligent systems operating in real-world environments. This is particularly important in applications such as surveillance, autonomous systems, and biometric analysis. In practice, these systems often face challenging conditions, including low illumination, nighttime operation, adverse weather, and cluttered backgrounds. Under such conditions, system performance strongly depends on the sensing modality. This dependency becomes even more critical in real deployments. Visible-spectrum cameras remain the most commonly used sensors in computer vision. Their widespread use is mainly due to their high spatial resolution and rich texture information. However, RGB-based detection methods are sensitive to illumination changes and often experience performance drops in low-light or nighttime scenarios. Thermal cameras, on the other hand, capture infrared radiation emitted by objects and operate largely independently of ambient lighting. This allows them to function reliably in darkness and harsh environments. However, thermal images usually have lower spatial resolution and less structural detail than RGB images. As a result, they may struggle in tasks that require sharp object boundaries or fine geometric features. Because RGB and thermal modalities have complementary strengths, combining them is a natural choice for multimodal perception systems. RGB images provide structural detail, while thermal images offer robustness to lighting conditions. When used together, they can produce more reliable visual representations. However, effective performance depends not only on having both modalities, but also on how well they are aligned and fused. In this thesis, robust multimodal perception is addressed through the joint use of RGB and thermal imaging for object detection and face liveness detection. To prevent synchronization and temporal alignment problems that commonly occur in multi-sensor systems, data are collected using a hybrid RGB-T camera that captures paired RGB and thermal images simultaneously. Although the acquisition is synchronized, geometric differences between the sensors still exist. Therefore, stereo camera calibration and image rectification are performed to achieve pixel-level alignment before fusion. The main contribution of this thesis is a simple and effective edge-enhanced RGB-T fusion representation. In this approach, structural edge information extracted from RGB images is combined with geometrically aligned thermal images using a weighted linear fusion method. Instead of using complex fusion networks or separate modality-specific branches, the method integrates structural and thermal information into a single hybrid image. This fused image can be directly used by standard YOLO-based detection models, allowing efficient training, lower computational cost, and improved interpretability. To enable comprehensive evaluation, two RGB-T datasets are developed. The ARISTO dataset is a real-world RGB-T object detection dataset that includes paired RGB and thermal images collected in both indoor and outdoor environments, covering a diverse set of object categories. To the best of our knowledge, ARISTO is one of the few publicly available indoor RGB-T datasets and also among the RGB-T datasets that contain a relatively large number of outdoor object classes. In addition, a dedicated RGB-T face liveness detection dataset, named ARISTOF, is collected under controlled conditions and includes both live and spoof face samples. Due to privacy considerations, ARISTOF is used only for experimental evaluation within this study. Because real RGB-T datasets are limited in size, this thesis also explores synthetic thermal image generation using large-scale RGB data. Two approaches are considered: depth-based thermal generation using shift-and-scale invariant depth estimation, and GAN-based RGB-to-thermal image translation. The generated synthetic thermal images are fused with RGB-derived edge information using the same fusion strategy. This enables large-scale training and consistent evaluation across both real and synthetic data. Object detection experiments are conducted using the YOLOv8 model on real and synthetic hybrid datasets. The results show that thermal-only detection is limited by insufficient structural detail, while RGB-only detection remains sensitive to difficult lighting conditions. The proposed edge-enhanced RGB-T fusion improves detection performance compared to thermal-only inputs and achieves results close to RGB-only models, while maintaining the robustness of thermal information. In addition, fine-tuning models pretrained on synthetic data with real RGB-T samples further improves performance, especially in outdoor environments. Cross-dataset evaluations on the LLVIP and FLIR benchmarks also demonstrate the generalization ability of the proposed method. The fusion framework is further evaluated for face liveness detection using the ARISTOF dataset. The same edge-enhanced RGB-T representation is adapted for close-range facial analysis. The results show that the fused representation improves face detectability compared to thermal-only inputs and enables reliable discrimination between live and spoof faces. High precision, recall, and mAP values on validation and test sets confirm the effectiveness and generalization capability of the method. In conclusion, this thesis demonstrates that integrating RGB-derived structural edge information with thermal imagery provides a practical and efficient fusion strategy for multimodal vision tasks. The proposed framework improves both object detection and face liveness detection under challenging conditions. In addition, the developed datasets and experimental results contribute useful resources and insights for future research in RGB-T multimodal perception.