Data Linkage Methods for Big Data Management in Industry 4.0

Abstract

In recent years, the use of various digital devices that continuously generate massive amounts of heterogeneous, structured or unstructured data has increased. In parallel to generation, data collection, storage, and analysis technologies have developed. Big data sources have a variety of data quality. Preparing and clearing data is one of the first step of mining big data. It is often important to address the full data set found in different data sources to achieve the right result. Various techniques have been used to increase the accuracy of the data comparison. Deterministic and probabilistic linkage algorithms are the two main techniques used in literature. They have different steps to reach qualified and integrated results. To easily interpret the results of the linkage algorithm, a confusion matrix can be used. Measurements such as sensitivity, specificity, positive predictive value, negative predictive value, false positive rate, and false negative rate, are considered to evaluate output quality.