Neutrino particle direction estimation using transformer-based deep learning

Abstract

The neutrino sector represents a crucial area for advancing our comprehension of the cosmos, extending beyond the confines of the Standard Model of particle physics. Nevertheless, the weak interaction cross-sections, coupled with the intricate data structures derived from large-scale detectors such as the IceCube Neutrino Observatory, present substantial obstacles in the detection and reconstruction of these elusive particles. Conventional algorithms and standard Convolutional Neural Networks (CNNs) encounter considerable limitations when processing the irregular, non-grid geometry of the IceCube detector, especially in the context of "cascade" events, where directional resolution is intrinsically constrained. Consequently, this thesis introduces a Deep Learning methodology, grounded in the Transformer architecture, to estimate neutrino arrival directions, specifically azimuth and zenith angles. The main goal is to use the self-attention mechanism in Transformers to capture global dependencies and long-range spatiotemporal correlations in pulse series of varying lengths. This strategy allows for the direct processing of the sequential raw data, independent of fixed geometric constraints. This approach uses the complete simulation dataset from the "IceCube - Neutrinos in Deep Ice" competition, which contains over 138 million events. The research followed a structured experimental process, from Exp1 to Exp7, to improve the model's architecture and training methods. The input data, which consisted of pulse series, went through thorough preprocessing. This included adding Spherical coordinates (radial distance and angular orientation) to standard Cartesian coordinates, along with normalization, to better represent the detector's geometry. The sequences were analyzed using a Transformer Encoder architecture, which included eight attention heads. The training process changed from using the usual Mean Squared Error (MSE) loss functions to Cosine Similarity Loss, a method better suited for angular data. Learning rate schedulers, including warm-up periods, were also used. Some of the experimental work was done on the National Center for High Performance Computing (UHeM) infrastructure at Istanbul Technical University. In addition, the training workflows were improved using distributed computing methods, such as PyTorch Distributed Data Parallel (DDP). Although early experiments showed difficulties in geometric generalization, the final configurations (Exp6 and Exp7) demonstrated both stability and the ability to generalize spatially. In the Exp7 phase, the fine-tuned model attained a Mean Angular Error of 63.80 degrees on the test set, indicating an approximate 11-degree enhancement relative to the baseline model. Visual assessments corroborated the model's capacity to resolve the head-tail ambiguity present in particle track reconstruction, enabling it to differentiate between individual event topologies instead of merely predicting the dataset mean. Moreover, the model exhibited its applicability to real-time data streams, achieving a high inference speed that allowed for the processing of roughly 375 events per second. Based on the experimental results, it can be stated that the Transformer architecture shows promise for this field. Furthermore, it is concluded that further success could be achieved by combining the Transformer with Graph Neural Networks (GNNs), which have demonstrated general success in the literature, within a hybrid architecture.