GPT for RCTs?: Using AI to measure adherence to reporting guidelines

Abstract

AbstractBackgroundAdherence to established reporting guidelines can improve clinical trial reporting standards, but attempts to improve adherence have produced mixed results. This exploratory study aimed to determine how accurately a Large Language Model generative AI system (AI-LLM) could measure reporting guideline compliance in a sample of sports medicine clinical trial reports.MethodsThe OpenAI GPT-3.5 AI-LLM was evaluated for its ability to determine reporting guideline adherence in a sample of 113 published sports medicine and exercise science clinical trial reports. For each paper, the model was prompted to answer a series of nine reporting guideline questions. The dataset was randomly split (80/20) into a TRAIN and TEST dataset. Hyperparameter and model fine-tuning were performed using the TRAIN dataset. Model performance (F1-score, classification accuracy) was assessed using the TEST dataset.ResultsAcross all questions, the AI-LLM demonstrated acceptable performance (F1-score = 86%). However, there was significant variation in performance between different reporting guideline questions (accuracy between 70-100%). The model was most accurate when asked to identify a defined primary objective or endpoint and least accurate when asked to identify an effect size and related confidence interval.DiscussionThe AI-LLM showed promise as a tool for assessing reporting guideline compliance. Next steps should include developing a cost-effective, open-source AI-LLM and exploring methods to improve model accuracy.