Beyond benchmarking: towards predictive models of dataset-specific single-cell RNA-seq pipeline performance

Abstract

AbstractThe advent of single-cell RNA-sequencing (scRNA-seq) has driven significant computational methods development for all steps in the scRNA-seq data analysis pipeline, including filtering, normalization, and clustering. The large number of methods and their resulting parameter combinations has created a combinatorial set of possible pipelines to analyze scRNA-seq data, which leads to the obvious question: which is best? Several benchmarking studies have sought to compare methods to answer this, but frequently find variable performance depending on dataset and pipeline characteristics. Alternatively, the large number of publicly available scRNA-seq datasets along with advances in supervised machine learning raise a tantalizing possibility: could the optimal pipeline be predicted for a given dataset? Here we begin to answer this question by applying 288 scRNA-seq analysis pipelines to 86 datasets and quantifying pipeline success via a range of measures evaluating cluster purity and biological plausibility. We build supervised machine learning models to predict pipeline success given a range of dataset and pipeline characteristics. We find both that prediction performance is significantly better than random and that in many cases pipelines predicted to perform well provide clustering outputs similar to expert-annotated cell type labels. Finally, we identify characteristics of scRNA-seq datasets that correlate with strong prediction performance that could guide when such prediction models may be useful.