Evaluation of Process and Effects of Change

Differences Between Data Sources

An understanding of the differences between quality improvement data and clinical research will help users to structure systems changes and their evaluation so as to effectively close the gap between knowledge and practice.

• Collection of Data - Clinical Research v. Quality Improvement:
  Since the aim in clinical research is the development of new knowledge, rigorous methods are used to prevent the effects of confounding variables. The aim is to see the effect of the variable of interest. To eliminate sources of bias, tests are blinded in studying a single hypothesis. Usually a large study collects large amounts of data to thoroughly study the process.
  
  By comparison, clinical improvement studies aim to improve outcomes through the application of known information . The tests are observable to the investigator w(e.g., self-management education to enable patients to meet their treatment goals)ho has a stable bias in seeing a positive outcome (Will this intervention result in more patients participating in self-management education?). Just enough data is collected to test whether the change results in an improvement (Did patient participation increase as a result of using a reminder/recall system?). Multiple hypotheses are tested in a sequential fashion (Cycle 1: letter of invitation to participate. Cycle 2: telephone call inviting participation).
   
• Enumerative vs. Analytic Statistics
  Enumerative statistics are used in clinical research to evaluate the outcome of testing a hypothesis. The analysis assumes a stable system - one in which all variables are held constant except the one under study. The goal is to estimate whether the outcomes between the control and study group are different. The statistics ascribe a degree of confidence to the accuracy of the estimate.
  
  Analytic statistics are used to evaluate clinical improvement such as Plan-Do-Study-Act (PDSA) cycles. The goal of the analysis is to determine the stability of the process producing the data. For example, will the patient recall system that increased the rate of eye exams from 36 percent to 70 percent consistently result in the higher percentage of patients having annual exams? In this example the accuracy of the measure is not the issue -- was the improvement in the rate of eye exams 70 percent or 68 percent or 72 percent? Rather, if the process is statistically stable, one can assess its current performance and take action either to predict future performance or to measure the effects of an improvement intervention. For example, now that eye exam rates have improved to 70 percent, how can we further improve the system to increase the rate to >90 percent?
   
• Translation of Clinical Trials
  While much clinical knowledge of diseases and their treatment is generated through clinical trials, the results of those trials will best be applied to patient populations through the application of clinical improvement methods. Unlike the trials that generated such knowledge, patients live in a world with many sources of variation that cannot be controlled. Practice environments vary from primary care providers in office settings to sub-specialists in tertiary care centers. Payment schedules vary from public health clinics to managed care organizations to fee-for- service practices.
  
  Individual physicians will interpret the results of studies and determine whether or how to incorporate the findings in their clinical practice. Finally, unlike clinical studies, the patients receiving treatment are not a selected population - results of clinical trials are applied in an environment that differs from a research setting. Therefore, to make better predictions and decisions, analytic statistics must be used to assess deviations from expected results and pinpoint sources of variation.⁶

-Return to the Top-
-Return Home-

Continue to Evaluation: Examples of Data Use