Semantic segmentation of historic dome systems considering part-whole relations

Abstract

This research investigates the usage of different deep learning methods of semantic segmentation in the context of classifying the parts of the historic domes of Anatolian monumental architecture regarding part-whole relations. Documentation of cultural heritage is a detailed process that focuses on recording the state of the items with high accuracy and all the required information. However, manual documentation techniques are error-prone that lead the human-caused inconsistencies in the collected data. Even though photogrammetry and laser scanning-based tools supply three-dimensional data, the workflows that use the collected data just as a reference for human-involved, re-drawing projects such-as ortho-photo drawings are both time-consuming and error-prone. Particularly, in the Heritage-Building Information Modeling context scan-to-HBIM tasks that focus on modeling the architectural elements with different layers of information by preserving unique shapes and deformations, transferring detailed and unique elements requires precise solutions. The presented study provides a semi-automatic, deep learning-based, as-is modeling workflow that interprets the semantic relations of the structural parts of the domes of historical Anatolian buildings. Deep neural network-based semantic segmentation algorithms can examine the given data and learn the generalized relation of the parts in the scenes. Effective usage of the networks relies on the existence of variants and numerous instances of models in the data set to train. However, the collected real-world data is limited due to the impossibility of capturing the multiple variations of different architectural elements. A synthetic data set, namely, Historic Dome Data Set(HDD) including 1537 models, is created to address this constraint by using the parametric modeling tools referring to the simplified definitions of the four parts of the historical Anatolian domes. The models include only position and normal values without color definition. Four types of deep neural networks are trained using the created data set to examine the results of different data processing approaches. The part-whole relations are usually well defined in the design, construction, and documentation tasks for humans. There are usually sequential but interrelation connections of the architectural units on different scales. However, the definition of the relations is based on the identified instances of the architectural elements. On the other hand, deep neural networks investigate the contextual, part-whole relations of the different classes to extract semantic information using relatively ambiguous patches and interpret the link between them in various ways. The presented study examines the outcomes of the direct implementation of multi-layer perceptron, graph, sequential, transformer-attention-based semantic segmentation algorithms into the historic dome system documentation process. To evaluate the semantic segmentation outcomes both visual and numerical results are provided and discussed. In addition to HDD, eleven real-world dome system models from the Anatolian Seljuk period are collected using photogrammetry tools to assess the proposed methods. To sum up, the presented research examines different deep learning-based semantic segmentation methods to acquire meaningful and accurate results based on the relations of the architectural elements. Also, a synthetic data set and an adaptive parametric modeling method for dome systems are provided that can be adapted into similar research. The results show that even if the trained networks perform well in the synthetic data set, and the real-world data includes all the parts of the dome systems, the exceptions in the real-world examples, such as sparsity of the points, noise in the collected data, unseen parts, and unlabeled elements confuse the segmentation process. Further studies might implement unlabeled objects, train the deep learning models with partial models, and focus on a unit-based investigation that focuses on creating the different sizes of patches on the point clouds to examine the relations of the part in the point clouds in an architecturally meaningful way.