Metrics for Evaluating the Accuracy of Diagnostic Fault Detection Systems

Abstract

This paper presents a method for providing metrics to evaluate the accuracy and cost effectiveness of diagnostic decision support systems. One intention of engine health management (EHM) fault detection systems is to have engines identified for removal and refurbishment as soon as there is evidence of an adverse gas generator trend shift. The benefits of EHM diagnostics and prognostics tests are derived from the resulting improved safety, the reduced operating costs, and most importantly, the good will and trust of the customer. The method presented in this paper is a generalized way of evaluating the performance of some of the tests that are used to make inspection, removal, and maintenance decisions [Ref 1,2]. The detection of faults from shifts in classification data is the first step in EHM systems that use diagnostics and prognostics [Ref 3,4,5]. The minimum parameter shift required to trigger a fault indication is called the threshold. Typically, it is a predetermined multiple of the standard deviation of the parameter measurements. Root cause isolation is usually invoked following these detection tests for the gas path parameter shifts. This paper shows how the achievable accuracy of diagnostic and prognostic system tests can be determined from the signal to noise ratio (SNR), and the system’s design (sensitivity and specificity). From these tests we extract two features, true positives (TP) and false positives (FP) that can be used to compare the accuracy of any simple or complex decision support system. This method is conducive to efficiently handling large amounts of data from multiple sensor tests because it avoids explicit correlation among individual diagnostic tests, and focuses instead on the net results. Each piece of classification information is used to reduce ambiguity. In this approach, the individual diagnostic tests and any data fusion weighting factors can be parametrically varied to optimize the accuracy of the decisions. The resulting plot of TP versus FP is then directly compared to the results of simple idealized classifier systems having known SNRs. This paper applies the receiver operating characteristics (ROC) process to evaluate the potential accuracy of EHM decisions. The paper also shows that the actual accuracy depends on how thresholds are set, and on the local shape of the ROC in the regions where it is used. The method presented can be applied to test the relative accuracy of each phase of the EHM decision-making process. The effects of test accuracies, event probabilities, and consequential event costs on the value of the decision support system are also presented.