Analyzing and predicting e-commerce customer behaviors using process mining techniques

Abstract

Advancements in technology and commerce have been the drivers for improving consumer reach since the beginning of the new millennium. Ever-growing product ranges and the new channels to access the markets facilitate trade. From this perspective, e-commerce is regarded as one of the most significant innovations of today. Despite all the improvements they bring to people's lives, innovations come with specific problems. For example, conversion rates in e-commerce are significantly lower than in traditional brick-and-mortar channel. In general for retail and specifically in e-commerce, conversion rate (CR) is one of the most important performance indicators. On average, the e-commerce CR is around 1.5%, whereas in the traditional brick-and-mortar retail, immediate action is required when it falls below 15%. The fact that e-commerce has a conversion rate ten times lower than traditional channel indicates that increasing CR is among the fundamental challenges in e-commerce that require a solution. In addition to its advantages, e-commerce provides tools to address its challenges. For example, in e-commerce, vast amount of visitor data can be collected and analyzed to find solutions. Unlike traditional channels, e-commerce provides quick, easy, and unobtrusive records on visitors' activities, which are known as clickstream data. Using clickstream data it is possible to detect and track the source of traffic, which products a specific visitor browsed or compared, which products were added to cart and purchased. The evolution of trade is also shaping new consumer behaviors. It can be argued that e-commerce channel visitors are acting more freely compared with the traditional channel customers which is related with the advantages that e-commerce provides For instance, in traditional retail, it is rare for a customer to add numerous items to their cart and finalize the purchase days later. On the other hand, this behavior is common among e-commerce customers. In the scope of this study, it was argued that so as to improve CR, first of all it is a prerequisite to understand emerging visitor behaviors and map an end-to-end visitor journey map in e-commerce. Following that, it was aimed at improving the results of the studies such as prediction and product recommendation which are mainly done to improve CR by using the outcomes of the e-commerce visitor journey analysis. In summary, unlike most studies in the e-commerce domain, a holistic and phased was adopted approach. In this thesis, process mining was utilized for analyzing and mapping the e-commerce visitor journeys. Until the late 2000s, process analysis and improvement initiatives were conducted manually by process analysts. The resulting process diagrams obtained by using the information collected during interviews and workshops often did not align with actual process instances. On the contrary to this, the process diagrams mapped with process mining algorithms are produced by using system logs and they consist of more objective flows. As a result, process mining was adopted by many organizations in a very short time period and has become an outstanding approach for managing and improving the processes. From this perpsective, process mining is a groundbreaking inovation in data science. As mentioned earlier, innovations come with both advantages and challenges. Process mining is not exempted from that. One of the biggest challenges in process mining is dealing with unstructured processes. A key characteristic of unstructured processes is the hard to understand process diagrams resulting from creating more realistic process flows. In the early phases of this research, the focus was on unstructured processes, particularly due to the lack of studies on the structuredness dimension in process mining using e-commerce clickstream data. However, at this point the existing measures for the outcomes of process mining structuredness were mostly not applicable for DFG notation which was needed to be used due to the dataset size. Additionally, measures developed for process diagrams before process mining emerged as a discipline were not suitable for the purpose. Given these constraints, initial research was conducted to develop a structuredness measure for process mining. In line with this objective, first of all a synthetic process diagram dataset was created algorithmically. After that, subject matter experts classified elements of this visual dataset in terms of their structuredness. This process resulted in a dataset suitable for a classification problem. Then this problem was modelled with logistic regression and developed a structuredness measure for process mining. In the second phase, using a real-life big dataset which has over twenty-million rows, activity-, behavior-, and process-level e-commerce visitor journeys was defined. Exploitation and exploration were the most common journeys, and it was revealed that journeys with exploration behavior had significantly lower conversion rates. At the process level, the backbones of eight journeys were mapped and their qualities were tested with the empirical structuredness measure. By using cart statuses at the beginning and end of these journeys, a high-level end-to-end e-commerce journey that can be used to improve recommendation performance was obtained. Additionally, new metrics were proposed to evaluate online user journeys and to benchmark e-commerce journey design success. By the end of the second phase, a milestone in understanding e-commerce visitor behaviors based on the dataset was reached. However, the procedure that was applied to discover process mining diagrams in which the structuredness measure was used was iterative and labor-intensive. At that point, an additional research was initiated to automatize this task. By utilizing the process mining structuredness measure, an algorithm was developed. The proposed algorithm was tested by manipulating the algorithm parameters using several scenarios created from eight different datasets. Experiments demonstrated that the proposed approach was capable of helping process mining researchers and practitioners obtain better process discovery outcomes. The results indicated that with high-quality user perception models, it was possible to satisfy users' expectations while saving considerable time. The algorithm's outputs were also tested with Petri nets. ANOVA results showed that the clusters obtained using proposed algorithm had significantly better fitness and simplicity values. In the fourth and fifth phases, the focus was on leveraging e-commerce customer behavior analysis for product recommendation and prediction. From the literature, it was confirmed that Session-based Recommender Systems (SBRS) had significant potential for improving conversion rates in e-commerce by assisting visitors to find products and services that suit their preferences. Many SBRS models have progressively improved recommendation performance for the same datasets. Instead of developing even more complex algorithms than the existing ones, in the fourth phase,the focus was on harnessing the context of visitor behaviors and decided to use exploration and exploitation behaviors as a contextual pre-filtering. In addition, a new strategy for reorganizing session logs to have better contextual settings. The proposed methodology was tested using five existing SBRS models, which were run with variants of two publicly available session logs. Experiments on the exploration and exploitation of visitor behaviors showed that with more homogeneous datasets, improvements exceeding 50% in Recall@20 and 60% in MRR@20 could be achieved compared to the baseline implementation of existing SBRS models. In the last phase, next activity prediction was applied which is an emerging process mining technique. It was identified that the commonly used GNN-based algorithms in the SBRS domain had not been considered in process mining studies with e-commerce clickstream data, so a GNN-based next activity prediction algorithm was developed. It was then tested with both clickstream and business process data. Next, the results were benchmarked against two prominent next activity prediction algorithms from the literature. Experiments showed that GNN-based algorithms outperformed LSTM-based approaches in e-commerce datasets. A comparison of runtimes, especially for larger datasets, revealed that proposed approach was clearly superior to the benchmarked methodologies. In conclusion, in this study a comprehensive approach was presented to improving e-commerce conversion rates by deeply analyzing visitor behaviors through process mining. A new structuredness measure was developed and validated for enabling more accurate and automated process discovery. Product recommendation systems were also enhanced by incorporating behavioral context and a GNN-based next activity prediction algorithm was introduced that outperformed traditional methods. Overall, the findings highlighted the value of combining process mining with behavioral insights to optimize the online customer journey.