Meta-research: How many diagnostic or prognostic models published in radiological journals are evaluated externally?

Abstract

Abstract Objectives Prognostic and diagnostic models must work in their intended clinical setting, proven via “external evaluation”, preferably by authors uninvolved with model development. By systematic review, we determined the proportion of models published in high-impact radiological journals that are evaluated subsequently. Methods We hand-searched three radiological journals for multivariable diagnostic/prognostic models 2013–2015 inclusive, developed using regression. We assessed completeness of data presentation to allow subsequent external evaluation. We then searched literature to August 2022 to identify external evaluations of these index models. Results We identified 98 index studies (73 prognostic; 25 diagnostic) describing 145 models. Only 15 (15%) index studies presented an evaluation (two external). No model was updated. Only 20 (20%) studies presented a model equation. Just 7 (15%) studies developing Cox models presented a risk table, and just 4 (9%) presented the baseline hazard. Two (4%) studies developing non-Cox models presented the intercept. Just 20 (20%) articles presented a Kaplan–Meier curve of the final model. The 98 index studies attracted 4224 citations (including 559 self-citations), median 28 per study. We identified just six (6%) subsequent external evaluations of an index model, five of which were external evaluations by researchers uninvolved with model development, and from a different institution. Conclusions Very few prognostic or diagnostic models published in radiological literature are evaluated externally, suggesting wasted research effort and resources. Authors’ published models should present data sufficient to allow external evaluation by others. To achieve clinical utility, researchers should concentrate on model evaluation and updating rather than continual redevelopment. Clinical relevance statement The large majority of prognostic and diagnostic models published in high-impact radiological journals are never evaluated. It would be more efficient for researchers to evaluate existing models rather than practice continual redevelopment. Key Points • Systematic review of highly cited radiological literature identified few diagnostic or prognostic models that were evaluated subsequently by researchers uninvolved with the original model. • Published radiological models frequently omit important information necessary for others to perform an external evaluation: Only 20% of studies presented a model equation or nomogram. • A large proportion of research citing published models focuses on redevelopment and ignores evaluation and updating, which would be a more efficient use of research resources. Graphical abstract