Towards Automatically Interpreting Dutch Tombstone Inscriptions

Johan Bos, Cristian Marocico, Yasmin Mzayek and Emin Tatar

Digital preservation of tombstones is important in the context of cultural heritage but a costly process. We propose a way of automatically reading tombstone inscriptions with the aim of assisting human annotators and data curators. Our method comprises a pipeline of dedicated components where the input is an image of a tombstone, and the output is an interpretation (represented as a directed acyclic graph) comprising the name of the deceased, date of birth, date of death, place of birth, place of death, and biblical references. The three main components in the pipeline are (1) Label Detection, (2) Optical Character Recognition (OCR) and (3) Interpretation. The Label Detection component uses the YOLOv5 Deep Learning algorithm trained on tombstone images to detect the bounded boxes and labels for the entities mentioned above (names, locations, and dates). The OCR component then takes in each of the detected labels and recognizes the text contained therein. Finally, the interpretation component performs post-processing, normalizes dates and places, and puts all the information together into a meaning representation coded as a directed acyclic graph. There are several challenges that need to be addressed, such as correcting OCR errors, interpreting unusual segmentation of words, recognising abbreviations, dealing with multiple languages, toponym grounding, and accommodating different notational variants of dates. The system is developed and evaluated with the help of an annotated corpus of 1,100 tombstone inscriptions. Evaluation is carried out by calculating graph overlap of the system output and gold standard. The first results are encouraging.