Using unsupervised topic modeling to uncover document hierarchy and latent topics in prostate cancer clinical texts

Abstract

AbstractObjectiveMedical records reflect patients’ health status and journey through healthcare services and medical specialties. As a result, document collections contain various text types, even about one patient, which makes medical texts unique. Our aim was to take advantage of the implicit document hierarchy of medical records to evaluate contextualized neural topic modeling as a tool to gain insight into a mixed set of notes related to prostate cancer treatment, i.e., to uncover document types, their contents, and relations.Materials and MethodsWe collected clinical text documents from 21,872 prostate cancer patients and organized the documents into a hierarchy using the document metadata. We trained neural topic models without the metadata to index the document collections, performed rigorous numerical evaluations of topic and clustering quality to optimize the topic count, visualized the latent representation of the models, and evaluated the topic clusters with respect to document metadata.ResultsTopic clusters reflected the structure of the document hierarchy and provided information about the contents of different text types. The determination of the optimal number of topics required complementary information by topic and clustering quality metrics.DiscussionThe topic modeling was found useful in visualizing and indexing large document collections, in providing an understanding of document contents, and in revealing document organization comparable to metadata-based hierarchy.ConclusionHospital databases contain masses of text documents, and topic modeling can provide means for analysts and researchers to group documents into discernable and explainable classes.