Medical Decision Making current issue

Using the Principles of Randomized Controlled Trial Design to Guide Test Evaluation

Lord, S. J., Irwig, L., Bossuyt, P. M. M. — Thu, 12 Nov 2009 10:52:04 PST

The decision to use a new test should be based on evidence that it will improve patient outcomes or produce other benefits without adversely affecting patients. In principle, long-term randomized controlled trials (RCTs) of test-plus-treatment strategies offer ideal evidence of the benefits of introducing a new test relative to current best practice. However, long-term RCTs may not always be necessary. The authors advocate using the hypothetical RCT as a conceptual framework to identify what types of comparative evidence are needed for test evaluation. Evaluation begins by stating the major claims for the new test and determining whether it will be used as a replacement, add-on, or triage test to achieve these claims. A flow diagram of this hypothetical RCT is constructed to show the essential design elements, including population, prior tests, new test and existing test strategies, and primary and secondary outcomes. Critical steps in the pathway between testing and patient outcomes, such as differences in test accuracy, changes in treatment, or avoidance of other tests, are displayed for each test strategy. All differences between the tests at these critical steps are identified and prioritized to determine the most important questions for evaluation. Long-term RCTs will not be necessary if it is valid to use other sources of evidence to address these questions. Validity will depend on issues such as the spectrum of patients identified by the old and new test strategies.

Proposals for a Phased Evaluation of Medical Tests

Lijmer, J. G., Leeflang, M., Bossuyt, P. M. M. — Thu, 12 Nov 2009 10:52:04 PST

Background. In drug development, a 4-phase hierarchical model for the clinical evaluation of new pharmaceuticals is well known. Several comparable phased evaluation schemes have been proposed for medical tests. Purpose. To perform a systematic search of the literature, a synthesis, and a critical review of phased evaluation schemes for medical tests. Data Sources. Literature databases of Medline, Web of Science, and Embase. Study Selection and Data Extraction. Two authors separately evaluated potentially eligible papers and independently extracted data. Results. We identified 19 schemes, published between 1978 and 2007. Despite their variability, these models show substantial similarity. Common phases are evaluations of technical efficacy, diagnostic accuracy, diagnostic thinking efficacy, therapeutic efficacy, patient outcome, and societal aspects. Conclusions. The evaluation frameworks can be useful to distinguish between study types, but they cannot be seen as a necessary sequence of evaluations. The evaluation of tests is most likely not a linear but a cyclic and repetitive process.

Decision-Analytic Modeling to Evaluate Benefits and Harms of Medical Tests: Uses and Limitations

Trikalinos, T. A., Siebert, U., Lau, J. — Thu, 12 Nov 2009 10:52:04 PST

The clinical utility of medical tests is measured by whether the information they provide affects patient-relevant outcomes. To a large extent, effects of medical tests are indirect in nature. In principle, a test result affects patient outcomes mainly by influencing treatment choices. This indirectness in the link between testing and its downstream effects poses practical challenges to comparing alternate test-and-treat strategies in clinical trials. Keeping in mind the broader audience of researchers who perform comparative effectiveness reviews and technology assessments, the authors summarize the rationale for and pitfalls of decision modeling in the comparative evaluation of medical tests by virtue of specific examples. Modeling facilitates the interpretation of test performance measures by connecting the link between testing and patient outcomes, accounting for uncertainties and explicating assumptions, and allowing the systematic study of tradeoffs and uncertainty. The authors discuss challenges encountered when modeling test-and-treat strategies, including but not limited to scarcity of data on important parameters, transferring estimates of test performance across studies, choosing modeling outcomes, and obtaining summary estimates for test performance data.

Additional Patient Outcomes and Pathways in Evaluations of Testing

Bossuyt, P. M. M., McCaffery, K. — Thu, 12 Nov 2009 10:52:04 PST

Before medical tests are introduced into practice, they should be properly evaluated. Randomized trials and other comprehensive evaluations of tests and test strategies can best be designed based on an understanding of how tests can benefit or harm patients. Tests primarily affect patients’ health by guiding clinical decision making and downstream management, such as the decision to order more tests or to start, stop, or modify treatment. In this article, the authors demonstrate that tests can have additional effects on patient outcome, which may be cognitive, emotional, social, or behavioral. They present a framework to help researchers and policy makers consider the cognitive, emotional, social, and behavioral effects of testing. These additional effects may be important themselves and may also influence the clinical outcomes of testing through different pathways. The authors provide examples from test evaluations in the literature to illustrate how these additional effects can be important in the evaluation of testing or indeed any health intervention.

Effects of Categorizing Continuous Variables in Decision-Analytic Models

Bentley, T. G. K., Weinstein, M. C., Kuntz, K. M. — Thu, 12 Nov 2009 10:52:04 PST

Purpose. When using continuous predictor variables in discrete-state Markov modeling, it is necessary to create categories of risk and assume homogeneous disease risk within categories, which may bias model outcomes. This analysis assessed the tradeoffs between model bias and complexity and/or data limitations when categorizing continuous risk factors in Markov models. Methods. The authors developed a generic Markov cohort model of disease, defining bias as the percentage change in life expectancy gain from a hypothetical intervention when using 2 to 15 risk factor categories as compared with modeling the risk factor as a continuous variable. They evaluated the magnitude and sign of bias as a function of disease incidence, disease-specific mortality, and relative difference in risk among categories. Results. Bias was positive in the base case, indicating that categorization overestimated life expectancy gains. The bias approached zero as the number of risk factor categories increased and did not exceed 4% for any parameter combinations or numbers of categories considered. For any given disease-specific mortality and disease incidence, bias increased with relative risk of disease. For any given relative risk, the relationship between bias and parameters such as disease-specific mortality or disease incidence was not always monotonic. Conclusions. Under the assumption of a normally distributed risk factor and reasonable assumption regarding disease risk and moderate values for the relative risk of disease given risk factor category, categorizing continuously valued risk factors in Markov models is associated with less than 4% absolute bias when at least 2 categories are used.

Incorporating Herd Immunity Effects into Cohort Models of Vaccine Cost-Effectiveness

Bauch, C. T., Anonychuk, A. M., Van Effelterre, T., Pham, B. Z., Merid, M. F. — Thu, 12 Nov 2009 10:52:04 PST

Background. Cohort models are often used in cost-effectiveness analysis (CEA) of vaccination. However, because they cannot capture herd immunity effects, cohort models underestimate the reduction in incidence caused by vaccination. Dynamic models capture herd immunity effects but are often not adopted in vaccine CEA. Objective. The objective was to develop a pseudo-dynamic approximation that can be incorporated into an existing cohort model to capture herd immunity effects. Methods. The authors approximated changing force of infection due to universal vaccination for a pediatric infectious disease. The projected lifetime cases in a cohort were compared under 1) a cohort model, 2) a cohort model with pseudo-dynamic approximation, and 3) an age-structured susceptible-exposed-infectious-recovered compartmental (dynamic) model. The authors extended the methodology to sexually transmitted infections. Results. For average to high values of vaccine coverage (P > 60%) and small to average values of the basic reproduction number (R₀ < 10), which describes school-based vaccination programs for many common infections, the pseudo-dynamic approximation significantly improved projected lifetime cases and was close to projections of the full dynamic model. For large values of R₀ (R₀ > 15), projected lifetime cases were similar under the dynamic model and the cohort model, both with and without pseudo-dynamic approximation. The approximation captures changes in the mean age at infection in the 1st vaccinated cohort. Conclusions. This methodology allows for preliminary assessment of herd immunity effects on CEA of universal vaccination for pediatric infectious diseases. The method requires simple adjustments to an existing cohort model and less data than a full dynamic model.

Planning Posttherapeutic Oncology Surveillance Visits Based on Individual Risk

Filleron, T., Barrett, A., Ataman, O., Kramar, A. — Thu, 12 Nov 2009 10:52:04 PST

The main objective of posttherapeutic surveillance in oncology is to detect recurrent disease associated with treatment failure. Current follow-up schedules are easy to apply because they are planned on a regular basis (for instance, every 3 months) but do not take into account prognostic factors associated with time to failure. We propose a 2-stage strategy to individualize surveillance by first identifying prognostic factors for time to failure, then modeling cumulative risk or cumulative incidence to plan visits according to equal quantiles of risk or probability of failure, respectively. Using data from a clinical trial of radiotherapy in non—small cell lung cancer patients, we demonstrate how this method could improve the early detection of relapse.

Incorporating Extrinsic Goals Into Decision and Cost-Effectiveness Analyses

Hazen, G. B., Schwartz, A. — Thu, 12 Nov 2009 10:52:04 PST

It has not been widely recognized that medical patients as individuals may have goals that are not easily expressed in terms of quality-adjusted life years (QALYs). The QALY model deals with ongoing goals such as reducing pain or maintaining mobility, but goals such as completing an important project or seeing a child graduate from college occur at unique points in time and do not lend themselves to easy expression in terms of QALYs. Such extrinsic goals have been posited as explanations for preferences inconsistent with the QALY model, such as unwillingness to trade away time or accept gambles. In this article, the authors examine methods for including extrinsic goals in medical decision and cost-effectiveness analyses. As illustrations, they revisit 2 previously published analyses, the management of unruptured intracranial arteriovenous malformations (AVMs) and the evaluation of preventive strategies for BRCA + women.

Development of Goal-Sensitive Health-Related Utility Assessment Procedures

Schwartz, A., Hazen, G. B., Leifer, A., Heckerling, P. S. — Thu, 12 Nov 2009 10:52:04 PST

Purpose. To measure the degree to which people express willingness to trade life or health for nonmedical goals. Method. In 3 studies, outpatients provided important life goals. In study 1, patients performed time-tradeoff between life-years and goal achievement and chose between states that varied in goal achievement, life expectancy, and disability; in study 2, patients made choices that traded off health state and goal achievement; in study 3, patients performed time-tradeoff assessments in 3 different goal achievement contexts. Results. In study 1 (n = 58), participants were eager to trade life-years for goal achievement, trading, on average, 71% of their remaining life for certain achievement v. certain nonachievement or 54% of their remaining life for their expected likelihood of achievement v. nonachievement. Life expectancy, disability status, and goal achievement each had a significant main effect on utility. In study 2 (n = 54), participants equally preferred a moderately impaired health state with goal achievement to perfect health without goal achievement and more strongly preferred the moderately impaired state with goal achievement than other less impaired states without goal achievement. Study 3 (n = 62) demonstrated that the mere discussion of goals and goal achievement or nonachievement in the context of a standard time-tradeoff assessment (without trading off goals) did not impact the assessment. Conclusions. Nonmedical life goals are important determinants of quality of life. People express willingness to trade off life and health in pursuit of these goals, which are extrinsic to the standard quality-adjusted life-year model.

The Influence of Treatment Effect Size on Willingness to Adopt a Therapy

Aberegg, S. K., O'Brien, J. M., Khoury, P., Patel, R., Arkes, H. R. — Thu, 12 Nov 2009 10:52:04 PST

Background. Physicians are slow to adopt novel therapies, and the reasons for this are poorly understood. The authors sought to determine if the size of the treatment effect of a novel therapy influences willingness to adopt it. Methods. We developed 2 experimental vignette pairs describing a trial of a therapy for a hypothetical disease that showed a statistically significant mortality benefit. The size of the mortality effect was varied in vignettes of a pair (3% v. 10%). The 2 experimental vignette pairs differed in whether study enrollment was reported. Vignettes were mailed to a random sample of physicians using an intersubject design. The main study outcome was respondents’ willingness to adopt the hypothetical therapy, based on the results of the hypothetical trial. Results. There were 124 and 89 respondents to vignette pairs 1 and 2, respectively. In vignette pair 1, 91% versus 71% of respondents adopted the therapy when it reduced mortality by 10% and 3%, respectively (P = 0.0058). For vignette pair 2, 88% versus 51% of respondents adopted the therapy when it reduced mortality by 10% and 3%, respectively (P = 0.0002). In both vignette pairs, nonadopters were more likely than adopters to report side effects of the therapy as a principal reason for their decision. Conclusions. In this study, respondents were less likely to adopt a lifesaving therapy if its associated mortality reduction was 3% compared to 10%. Because most therapies for major medical conditions reduce mortality within or below this range, and because there were no opportunity costs associated with the adoption of the therapy, we believe that this effect represents a bias. Further investigation will be required to determine its prevalence and mechanism.

Diagnostic Certainty as a Source of Medical Practice Variation in Coronary Heart Disease: Results from a Cross-National Experiment of Clinical Decision Making

Thu, 12 Nov 2009 10:52:04 PST

The authors examined physician diagnostic certainty as one reason for cross-national medical practice variation. Data are from a factorial experiment conducted in the United States, the United Kingdom, and Germany, estimating 384 generalist physicians’ diagnostic and treatment decisions for videotaped vignettes of actor patients depicting a presentation consistent with coronary heart disease (CHD). Despite identical vignette presentations, the authors observed significant differences across health care systems, with US physicians being the most certain and German physicians the least certain (P < 0.0001). Physicians were least certain of a CHD diagnoses when patients were younger and female (P < 0.0086), and there was additional variation by health care system (as represented by country) depending on patient age (P < 0.0100) and race (P < 0.0021). Certainty was positively correlated with several clinical actions, including test ordering, prescriptions, referrals to specialists, and time to follow-up.

The German Coronary Artery Disease Risk Screening Model: Development, Validation, and Application of a Decision-Analytic Model for Coronary Artery Disease Prevention with Statins

Stollenwerk, B., Gerber, A., Lauterbach, K. W., Siebert, U. — Thu, 12 Nov 2009 10:52:04 PST

Background. Coronary artery disease (CAD) is a major cause of death in industrial countries, leading to high health-related costs and decreased quality of life. Objective. To develop and validate a decision-analytic model for CAD risk screening in Germany (German Coronary Artery Disease Screening Model). Design. Markov model. Target Population. Age- and gender-specific cohorts of the German population. Data Sources. Mortality rates posted by the German Federal Statistical Office, the German Health Survey, social health insurance institutions, the MONICA Augsburg study, and the literature. Time Horizon. Lifetime. Interventions. CAD risk screening for high-risk individuals using Framingham risk equation and use of statins as the primary preventive measure, compared with a setting without screening. Outcome Measures. Life-years (LY) gained, quality-adjusted life-years (QALYs) gained. Results. The model-based CAD incidence corresponds well with empirical data from the MONICA Augsburg study. Health outcomes depend on the screening threshold (cutoff value of Framingham 10-year risk) and on the age and gender of the cohort screened (0.03 to 0.26 LYs and 0.06 to 0.42 QALYs gained per person screened in cohorts of 50- and 60-year-old men and women, respectively). Conclusions. The model provides a valid tool for evaluating the long-term effectiveness of CAD risk screening in Germany. Using statins as a primary prevention intervention for CAD in high-risk individuals identified by screening could improve the long-term health of the German population.

Web Exclusive White Paper Series on Diagnostic Test Evaluation

Helfand, M. — Thu, 12 Nov 2009 10:52:04 PST