QUERI Implementation Guide
Section I Part 4: Formative Evaluation
Planning a Formative Evaluation
If a purely developmental formative evaluation is being planned (e.g. to identify determinants of gaps or to generically identify barriers and facilitators) that would be developed as a typical descriptive/observational study. On the other hand, if the formative evaluation is part of an implementation project all three types can be used. Formative evaluations of a full implementation project are the topic of this Section.
As in any evaluation or research endeavor, choices must be made about what to study, and the same is true for the formative evaluation of an intervention project. More than likely, it will not be feasible to assess and evaluate every component of the project, so choices about the most critical aspects must be made. Identifying the aims for the formative evaluation is the first step, . The aims depend on the overall aim of the depending upon the intervention project and its conceptual framework, as well as the planned activities and as well as what is already known about the planned interventions. Then, as in other research endeavors, investigators must:
- Identify the primary questions that derive from the aims,
- Develop instruments and methods to collect data,
- Conduct systematic data collection, and
- Analyze and report data.
The unique character of formative evaluation is that it occurs during the research project, thus the results can be used to describe and inform the process. One use of formative evaluations is to identify parts of the process that need changing refinement so as to maximize the effect of the project. While formative evaluations can be used during the research project, the data may be analyzed in relation to summative findings (outcomes) as well, in order to better interpret findings, particularly in light of a conceptual model. For example – What influenced the degree of success or failure? What was required to "make the change happen?" How did the stakeholders feel about the process? Such information is critical to the expected roll-out of VA implementation projects to the broader system.
The goal of the SCI QUERI Vaccine Initiative project was to increase vaccination rates; the project incorporated several intervention components. One intervention was directed at patients; several were aimed at modifying behaviors of practitioners; and several were designed to cause changes in policy and in information technology interactions. Therefore, each intervention required different formative evaluation plans. For example, one formative evaluation was conducted to learn about any problems being encountered when personnel used computerized clinical reminders for influenza vaccine so that these problems could be corrected. The formative evaluation for an intervention to encourage the use of standing orders for vaccines by nurses consisted of contact with both the hospital policy offices and the staff at the clinics where the patients were seen. For this intervention to improve vaccine rates, the policy had to be in place, it had to be known to the practitioners who saw patients, and it had to be put into practice.
Both qualitative and quantitative methods are commonly used for formative evaluations in translation projects. Qualitative observations of participants, or discussions with participants, may uncover things that are working well and not working well, and whether program elements are implemented as intended. Quantitative data on certain activities may be collected on an ongoing basis and used to determine whether changes are being made. For example, in a project that intends to have its providers use computerized clinical reminders, whether tthe extent to which remindershose reminders are used as prescribed or planned could be tracked to see if change occurs after certain educational activities. References and Internet sources of information are provided at the end of this chapter for those desiring further information on design and planning issues.