Benchmarking generative ai in structured domains: A critical evaluation of LLM–RAG architectures for flight recommendations

Abstract

The accelerated adoption of Large Language Models (LLMs) across domains has often outpaced rigorous domain-specific validation, particularly in structured, transactional environments. This thesis critically examines the effectiveness of Retrieval-Augmented Generation (RAG) architectures within the domain of flight ticket recommendation, a complex setting characterized by behavior-driven decisions, structured fare data, and time-sensitive preferences. Using a proprietary, large-scale booking dataset from BiletBank, we benchmark a RAG pipeline and a behaviorally enhanced RAG pipeline against a traditional item-based collaborative filtering (CF) baseline, evaluating both top-k recommendation accuracy and computational efficiency. The results reveal that while CF achieves superior accuracy across all thresholds, the behaviorally enhanced RAG system offers competitive performance with substantially lower runtime costs. This study provides one of the first empirical comparisons between classical and retrieval architectures in a real-world, structured enterprise setting. By foregrounding the role of behavior-informed retrieval and segmentation in improving retrieval performance, the thesis contributes to the ongoing discourse on responsible AI deployment. It underscores the necessity of empirical benchmarking and domain-specific architectural adaptation before adopting generative models in operational, structured domains such as travel recommendation systems. While the results are promising, the study is limited to a single proprietary dataset and domain, and future work is needed to generalize findings across industries and model configurations.