Bot detection through behavioural signals

Abstract

Automated traffic continued to surpass human-generated traffic on the web, and a rising proportion of this automation was explicitly malicious: in 2024, automated requests represented 51% of all web traffic, while bad bots made up about 37% of total requests, with attacks increasingly enabled by AI. Attackers no longer operate just noisy, low‑skill crawlers; they increasingly rely on AI augmented evasive bots that pretend to be real users, even solve Captchas and use headless browser tools to mimic human interaction patterns at scale. The result is a rapidly escalating arms race in which defenders harden signals and raise the cost of detection, and attackers borrow the same automation and obfuscation techniques to evade those defenses. All of the previous reasons underscore the importance of developing robust detection methods. In this thesis the term behavioural signals is used broadly to include not only human interaction patterns (mouse, keystroke dynamics) but also client-level behavioral traces observable at the protocol and device layer. Protocol-level choices made during automated interactions such as TLS handshake parameter ordering, extension usage, and cipher preferences, reflect the implementation behavior of client software and libraries. JA4 TLS fingerprints capture these implementation behaviors and therefore function as a form of behavioral signal that is harder for bots to spoof at scale. We examine how device fingerprinting, specifically TLS fingerprinting with the JA4 method, can help distinguish bots from real users in real-world web traffic. Because traditional methods such as IP blocking and CAPTCHA are often bypassed by advanced bots, this research examines how passive, protocol-level, and device-level fingerprinting can offer a stronger, less disruptive way to manage bots. Two gradient-boosted machine learning classifiers (XGBoost and CatBoost) were trained and evaluated on a dataset of real TLS fingerprints (JA4DB) after feature extraction, which derived informative signals from JA4 fingerprints that describe TLS handshake parameters, yielding a feature set that captures characteristics of client TLS connections. To make sure the approach is technically feasible, the models were evaluated based on metrics like accuracy, precision-recall, and F1-score. The CatBoost model did better and got a very high score. It got an AUC of 0.998 and an F1 score of 0.9734. It was accurate 0.9863 of the time on the test set. The XGBoost model showed almost similar results. Feature importance analyses identified JA4 components, especially ja4_b, cipher_count, and ext_count, as the most significant contributors to model performance. This work contributes to machine learning for cybersecurity by correlating protocol-level fingerprinting and web bot detection. It provides an explicit methodology and practical strategy for researchers and professionals. To conclude, this thesis demonstrates that device fingerprinting is capable of working well, especially TLS fingerprinting techniques like JA4, and offers promise to be a scalable solution to protect against bots. The proposed approach achieved strong results when tested on a real dataset. Looking ahead future research direction will extend this method to new protocols such as HTTP/3 and add additional device-fingerprinting features to test how well the system resists advanced bot evasion tactics. Overall, the findings emphasize the importance of protocol-level fingerprinting in advancing bot detection in modern cybersecurity.