Remerciez-le!

Remerciez @Admin pour avoir partagé cet document gratuitement, de la manière la plus simple, en partageant sur les réseaux sociaux.

21.09.2021 16:01 Semantic-based automated reassembly of heritage fragments http

21.09.2021 16:01 Semantic-based automated reassembly of heritage fragments https://journals.openedition.org/ceroart/8053 1/9 CeROArt Conservation, exposition, Restauration d’Objets d’Art 12 | 2020 (Open issue) Flux 2020-2021 Articles Semantic-based automated reassembly of heritage fragments Marie-Morgane Paumard https://doi.org/10.4000/ceroart.8053 Abstracts Français English L'intelligence artificielle apporte des solutions à de nombreux problèmes de conservation du patrimoine. Parmi ceux-ci, la reconstruction d'artefacts pourrait être facilitée grâce à l’automatisation. Après la diffusion des techniques de réassemblage basées sur la continuité des motifs ou des fractures, la dernière décennie a vu l'essor des algorithmes basés sur la sémantique. Ceux-ci permettent d'effectuer des tâches visuelles en utilisant le sens de ce qui est représenté sur les images. Nous présumons que ces méthodes aideront à résoudre les problèmes patrimoniaux ne pouvant être solutionnés par les méthodes classiques, par exemple lorsque les fragments sont trop érodés pour utiliser les continuités. Artificial intelligence provides archeologists and conservators with solutions to many problems. Among them, artifact reconstruction would surely benefit from smart automation. After the spread of the vision-based reassembly techniques, that use either fractures or pattern continuity, the last decade has seen the rise of semantic-based algorithms that allow performing computer vision tasks using the meaning of what is represented. We expect that such methods would help to solve some of the most difficult heritage problems that cannot be addressed by traditional methods, often because the fragments are too eroded to make continuity-based deductions. Index terms Keywords : apprentissage profond, vision par ordinateur, optimisation, remontage automatique Keywords: deep learning, computer vision, optimization, automatic reassembly Full text 21.09.2021 16:01 Semantic-based automated reassembly of heritage fragments https://journals.openedition.org/ceroart/8053 2/9 Introduction Fig.1 Vaux-de-la-Celle The blocks from Vaux-de-la-Celle (a) are stored for future reassembly of the temple (b, c). ©La Gazette Val d’Oise, ©Le Parisien, ©Association APSAGE. The archaeological site of Vaux-de-la-Celle (Fig.1), in the Val d'Oise, is an exceptional Gallo-Roman sanctuary made up of the ruins of a temple with two cells surrounded by a circulation gallery, pools, and a theater that can accommodate up to 8,000 people (Barrière 2019). 1 The first fanum is dated to the middle of the first century AD and the architectural ensemble was built a few decades later. During the 3rd century, the site was gradually abandoned after the 3rd century and its monuments served as a stone quarry until modern times. The first excavations were undertaken in 1935 by Pierre Orième. From 1960, the team of volunteers led by Pierre-Henri Mitard discovered most of the known remains: theater, main temple and annex temples, settlements, sacred area. The existence of an agglomeration was brought to light in the early 1990s. The 5,000 ritual and everyday objects discovered, as well as the sculpted blocks, are kept at the Archaeological Museum of Val d'Oise. Archaeologists now want to reassemble the building from the hundreds of sculpted blocks. The task was too complex for people to undertake, and it is thanks to advances in artificial intelligence that computer reassembly is now possible. This reassembly problem gave rise to the Archepuz-3D research project, funded by the Heritage Science Foundation, which aims to develop a 3D assembly algorithm of archaeological blocks. Vaux-de-la-Celle is one of the two sites selected for the project. Its sculpted blocks are being digitized by photogrammetry and 3D acquisition, and will ultimately constitute a dataset. In December 2019, 55 blocks were digitized. The other site selected is the Roc-aux-Sorciers, a rock shelter dating from the Upper Paleolithic, whose sculpted ceiling has collapsed (Pinçon 2009). 2 Sets of heritage fragments are characterized by the erosion of the pieces, the absence of certain pieces, and the mixture of pieces from several objects, which increases the difficulty of the reassembly. The automatic methods seek to match the contours (Papaioannou 2017, Sizikova 2017) and visual continuities of the fragments (Son 2014, Paikin 2015). When they are damaged, it is no longer possible and experts use semantics to make sense of the whole. Having an algorithm capable of understanding semantics would overcome the limitations of current models. In recent years, deep learning algorithms have learned to use semantic features to perform different tasks, such as detecting cancer cells and translating documents. Our goal is to design a similar 3 21.09.2021 16:01 Semantic-based automated reassembly of heritage fragments https://journals.openedition.org/ceroart/8053 3/9 Theoretical notions of deep learning Method method to solve all types of archeological jigsaw puzzles. The method presented here is as effective as the experts on a simple semantic task, solving the 9-square 2D fragment jigsaw puzzle. Algorithms that use visual semantic reasoning are recent and are improving little by little. Ultimately, the semantic methods should be versatile and allow the reassembly of many archaeological sites. Our contributions include an extension of the 3×3 problem introduced by Doersch et al. (Doersch 2015) that allows solving jigsaw puzzles from pairwise relations: we proposed three reassembly solving methods (greedy, exact, and heuristic). We extended the original setup to deal with missing fragments and outsider fragments, which are frequent in archaeology (Paumard 2020). We also propose a new dataset of 14,000 heritage images, a refined neural network architecture, few metrics to assess the reassembly quality, and merging functions (Paumard 2018). In this article, we present the basics of deep learning and the main ideas of our methods to focus on the results and the potential applications for archaeologists and conservators. 4 Given data, a deep learning algorithm is trained to answer a question, such as: where are the faces in each photography? To solve a jigsaw puzzle, a relevant question is: what is the relative position of two fragments? The answer can then be: the first fragment is above and to the right of the second. An algorithm based on this question learns to organize the pairs of fragments into nine categories (that we call “classes”). The first eight correspond to the main position relations (cardinal and intercardinal directions). The ninth class indicates that the two fragments are not adjacent. 5 To predict the class, we use a neural network, which applies a succession of operations to the data, to obtain an answer to the question. A neural network is organized in several layers, each containing thousands of nodes; each node contains an operation. The nine nodes in the last layer correspond to the classes and return the probability that it is the correct answer. In the beginning, the operations of the node are randomly initialized and the prediction errors are high. To minimize them, we must optimize each node operation. Theoretically, if the number of nodes is sufficient and the operations are well-chosen, the network can answer any question. 6 Deep learning consists of automatically associating each node with a relevant operation to answer the question. The network is asked to predict answers for a large number of images and compare its predictions to the solutions. Then, it adjusts slightly the nodes operations to decrease the last prediction errors. The network is thus optimized by trial and error until the prediction error is acceptable. Therefore, when the dataset is too limited (either in quantity or diversity of pictures), the neural network is not able to generalize well and will provide weak answers to new problems. 7 At the end of the learning phase, the neural network can predict responses for unknown data. The accuracy of the answer depends on the difficulty of the question, the number of layers and nodes in the neural network, and the data on which the network has been trained. 8 Solving a jigsaw puzzle amounts to predicting the relative position of all pairs of fragments and finding the best arrangement (Fig.2). For this purpose, we use a dual- head convolutional network that extracts the features of a pair of fragments made of the central fragment and one of the lateral fragments. The features are then concatenated and abstracted so that the last layer predicts the spatial relation between them. Once the predictions have been made for all pairs, we can solve the puzzle: we only have to find the best arrangement among all available assemblies. If we assume that the neural network is perfect (i.e., it always assigns the highest probability to the correct class), the correct reassembly is the one that maximizes the sum of the probabilities at the output 9 21.09.2021 16:01 Semantic-based automated reassembly of heritage fragments https://journals.openedition.org/ceroart/8053 4/9 Fig.2 Resolution method The green lateral fragment is compared to the red central fragment: the features of each fragment are extracted (b) and then the neural network predicts the relative position (c). Here, the most likely position is to the left of the red fragment. The predictions are organized in a graph (d): each line contains all the remaining positions. Thus, the position of the blue fragment depends on the location of the green fragment. The final resolution (e) is obtained after traversing the graph. ©Marie-Morgane Paumard. Fig.3 Dataset preparation of the neural network. This calculation is performed using the shortest path algorithm in the pruned graph that links fragments to positions. The graph is weighted by the neural network predictions. We proposed two uploads/Management/ semantic-based-automated-reassembly-of-heritage-fragments.pdf