|
Sign In to gain access to subscriptions and/or personal tools.
|
Affirmative Actions
Can the Discriminant Accuracy of a Test Be Determined in the Face of Selection Bias?
George A. Diamond, MD
Clinical estimates of test efficacy can be distorted by the differential referral of positive and negative test responders for outcome verification. Accordingly, a series of computer simu lations was performed to quantify the effects of various degrees of this selection bias on the observed true-positive rate, false-positive rate, and discriminant accuracy of a hypothetical test. The error in observed true- and false-positive rates was positive with respect to diag nosis, and negative with respect to prognosis. The magnitude of error was highly correlated with the magnitude of bias associated with the test response (primary selection bias), but not with the magnitude of bias associated with additional independent factors (secondary selection bias). Mathematical correction for preferential referral based on the test response using a previously published algorithm completely removed the correlation with primary selection bias for both diagnosis and prognosis. Although a significant correlation with sec ondary selection bias persisted at intermediate base rates, its magnitude was small. Dis criminant accuracy was assessed in terms of area under a receiver operating characteristic (ROC) curve. Biased values of true- and false-positive rates were distributed along the curve defined by the actual true- and false-positive rates of the test for both diagnosis and prognosis. As a result, the areas under ROC curves calculated from biased true- and false-positive rates were within 2% of the areas calculated from the actual rates. Only when the primary and secondary observations were independent with respect to one outcome and dependent with respect to the other outcome did a systematic error appear in ROC area. These data indicate that: 1) selection bias significantly distorts the determination of diagnostic and prog nostic test accuracy in directionally opposite ways; 2) the distortion can be partially offset by a previously published mathematical algorithm; and 3) the area under an ROC curve is insensitive both to the primary bias associated with the test response itself and to the secondary bias associated with concomitant clinical information under a variety of circum stances. Key words: diagnosis; prognosis; referral bias; ROC curves; sensitivity; specificity; verification bias; workup bias. (Med Decis Making 1991;11:48-56)
Medical Decision Making, Vol. 11, No. 1,
48-56 (1991)
DOI: 10.1177/0272989X9101100109
 CiteULike  Complore  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter What's this?
This article has been cited by other articles:
|
|
|
|
 
P. Whiting, A. W.S. Rutjes, J. B. Reitsma, A. S. Glas, P. M.M. Bossuyt, and J. Kleijnen
Sources of Variation and Bias in Studies of Diagnostic Accuracy: A Systematic Review
Ann Intern Med,
February 3, 2004;
140(3):
189 - 202.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. P. Morise, G. A. Diamond, R. Detrano, M. Bobbio, and E. Gunel
The Effect of Disease-prevalence Adjustments on the Accuracy of a Logistic Prediction Model
Med Decis Making,
June 1, 1996;
16(2):
133 - 142.
[Abstract]
[PDF]
|
|
|
|
|
|
|
Y. T. Van Der Schouw, H. Straatman, and A. L.M. Verbeek
ROC Curves and the Areas under Them for Dichotomized Tests: Empirical Findings for Logistically and Normally Distributed Diagnostic Test Results
Med Decis Making,
October 1, 1994;
14(4):
374 - 381.
[Abstract]
[PDF]
|
|
|
|
|
|
|
G. A. Diamond
Off Bayes: Effect of Verification Bias on Posterior Probabilities Calculated Using Bayes' Theorem
Med Decis Making,
February 1, 1992;
12(1):
22 - 30.
[Abstract]
[PDF]
|
|
|
|
|
|
|
G. A. Diamond
Scotched on the ROCs
Med Decis Making,
August 1, 1991;
11(3):
198 - 200.
[Abstract]
[PDF]
|
|
|
|
|
