Semantics-driven annotation of patient-specific 3D data: a step to assist diagnosis and treatment of rheumatoid arthritis

In the digital era, patient-specific 3D models (3D-PSMs) are becoming increasingly relevant in computer-assisted diagnosis, surgery training on digital models, or implant design. While advanced imaging and reconstruction techniques can create accurate and detailed 3D models of patients’ anatomy, software tools that are able to fully exploit the potential of 3D-PSMs are still far from being satisfactory. In particular, there is still a lack of integrated approaches for extracting, coding, sharing and retrieving medically relevant information from 3D-PSMs and use it concretely as a support to diagnosis and treatment. In this article, we propose the SemAnatomy3D framework, which demonstrates how the ontology-driven annotation of 3D-PSMs and of their anatomically relevant features (parts of relevance) can assist clinicians to document more effectively pathologies and their evolution. We exemplify the idea in the context of the diagnosis of rheumatoid arthritis of the hand district, and show how feature extraction tools and semantic 3D annotation can provide a rich characterization of anatomical landmarks (e.g., articular facets, prominent features, ligament attachments) and pathological markers (erosions, bone loss). The core contributions are an ontology-driven part-based annotation method for the 3D-PSMs and a novel automatic localization of erosion and quantification of the OMERACT RAMRIS erosion score. Finally, our results have been compared against a medical ground truth.