|
|
|
|||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||
A Guide to Developing a TB Program Evaluation PlanAppendix A: A Sample Evaluation Plan
TB Support Program - Evaluation Plan for January – March 2004 Prepared by: Ana Garcia TB Nurse Department of Health and Human Service November 15, 2003
Tuberculosis was not a major concern in Lull County until in the recent years. According to the county report, an average of 3 cases was reported per year from 1995 to 2001. In 2002, six new cases of TB were identified within three month period. Unlike previous years, the infected suspects were new immigrant Salvadoran men rather than the predominant white population. In response to this epidemic, “TB Support Program” was implemented by the county health department director, Henry Evans, to control and prevent further transmission of TB in Lull County. Evaluation Goal The goal of this evaluation is to determine the effectiveness of “TB Support Program” in preventing transmission of TB among Salvadoran immigrants. This evaluation will investigate components of the TB Support Program that are performing optimally and should be expanded and replicated in future initiatives. In addition, this evaluation will help determine the funding needs of TB Support Program for the following year. Evaluation Team Our team consists of TB Support program staff and a nurse from the TB clinic who works within the system and can access clinic/program data easily. A member of the county commission was asked to participate on the team to review plans and ensure needs for information are met. Carlos, a leader from the local community who has a background in the social sciences, was also asked to participate to ensure the evaluation is culturally appropriate to the target community. Table 1. Roles and Responsibilities of the Evaluation Team Members
At the monthly TB program staff meeting, the evaluation of TB Support Program was discussed. The following stakeholders were identified, along with their interests and perspectives and how each stakeholder should be involved in the process. After the meeting, Carlos was also contacted to give perspective on how patients and the community should be engaged. The following table summarizes the plan for stakeholder engagement. Table 2. Stakeholder Assessment and Engagement Plan a. Persons involved in program operations
b. Persons served or affected by the program
c. Intended users of evaluation findings
II. BACKGROUND AND DESCRIPTION OF THE TB PROGRAM AND PROGRAM LOGIC MODEL Need
A culturally competent TB prevention program such as TB Support Program is needed to combat the rise of TB incidence among the Salvadoran immigrant community of Lull County. Newly arrived persons from a high TB incidence country to Lull County are a major concern. Persons with TB disease and infection are not being reached for timely treatment. Consequently, TB cases are found on hospital admission after extensive health damage has occurred and community transmission demonstrated. Some newly discharged TB patients are lost to follow-up. Treatment cost increases along with the rising morbidity, ill-health, and suffering of the community members. The need for immediate attention to TB is great. If the problem is left unattended, backlash against foreign born population is possible. Context Historically, Lull County has been a low TB incidence area. Recent surge in the number of TB cases has raised concerns in the community. TB prevalence among the large number of newly arrived immigrants was fairly high. Although the need for TB elimination and prevention in this community is great, the capacity and resources needed to address the magnitude of these surging TB issues in Lull County is limited. TB Support program, an outreach program using “lay health advisors” (LHAs) to target new immigrants in the community, is implemented in response to this need. Under TB Support program, Spanish-speaking lay health advisors are to provide directly observed therapy (DOT) and social support in the immigrant community to increase targeted testing and treatment adherence and promote early case finding in the community. Target Population The target population for TB Support Program is newly arrived Salvadoran immigrants who cannot speak English, have limited resources, a small pool of family and friends if any at all, and may possibly have legal problems with their immigration status. Objectives The goal of TB Support Program is to eliminate TB transmission in the Salvadoran community, improve quality of life, and ultimately, to eliminate TB in Lull County. In order to achieve this overarching goal, program objectives were identified for the first year of the program.
Because the program and problem are new, specific baselines, i.e., how much of an increase can be expected, is unknown. Stage of Development The program is relatively new; in operation for less than a year. Resources/Inputs Lay heath advisors (LHAs), TB staff, funding, community based organizations and community health centers serving the Salvadoran community are key inputs of TB Support program. Activities Lay health advisors (LHAs) training, community outreach/education, TB screening/testing, prescribing treatment, and DOT visits make up the major activity categories. Outputs As a result of the efforts done in TB Support Program, LHAs were hired and trained, counseling provided, TB outreach/education conducted, test conducted, referrals made, a treatment plan developed, and DOT was administered. Outcomes A list of short-term outcomes include: patient’s knowledge increased, trust is built, patients accept treatment for TB or LTBI, patients identify contacts and possible source of cases, and patients use TB and other services. Intermediate outcomes: patients adhere to treatment, complete treatment, reduce hospital admissions for TB among Salvadorans, and reduce transmission of TB. Long-term outcomes: Salvadoran patients are healthier; eliminate TB in Salvadoran community, and healthier community.
Table 3. Project Description of TB Support Program Stakeholder Needs
From the beginning, four groups of people were identified as the core users of the evaluation findings. They will be using the findings in different ways and for different purposes. The table below summarizes the users of this evaluation, what information they need or would like to get from the evaluation, and how they intend to use that information to achieve what they need or set out to do.
Evaluation Questions
Although the evaluation team generated many possible questions in a brainstorming session, the evaluation team prioritized the following as representing the most important aspects of the program that could be examined at this time. To determine if the program has been implemented as planned:
To determine if the program is meeting its objectives:
Evaluation Design The evaluation team decided to use multiple data sources because the program was small (few number of people served), the need to obtain information to improve the program was high, and in-depth information was needed. No control or comparison group seemed appropriate. The team discussed reviewing records for Salvadoran patients in the past and comparing these data to current program information, but realized that it would be difficult since data in the files were incomplete. The decision was made to assess the data against the benchmarks the program has set. Further, it was decided that the data collected for this evaluation would serve as a baseline for later evaluations. Resource Consideration
Resource available for evaluation is limited. Manpower consists of 1 full time staff member who is only able to devote a few hours a week to the evaluation. The other evaluation team members are able to devote only a limited amount of time to the evaluation. However, several of the evaluation team members work together and most stakeholder groups meet regularly. Thus, some evaluation work can be done as part of existing routines. Personnel records, training curriculum and clinical records are available as existing data sources. Additional data collection measuring patients’ perception will be needed to understand whether patients feel trust is built with their health care providers and whether their TB knowledge has increased. Evaluation Standards The evaluation team agreed that the evaluation data will be useful, and should be feasible to collect. Propriety will be addressed through consent form and all data will be kept confidential. While the group agreed that there may be some issues regarding record keeping or patient willingness to answer honestly, the accuracy of the strategy was judged to be acceptable. IV. GATHERING CREDIBLE EVIDENCE: DATA COLLECTION Indicators The evaluation team recognized that the terms used the evaluation questions, e.g., “appropriately,” needed to be operationalized, and the following list of indicators show the team’s decisions. Since several program objectives were vague, the evaluation team had to determine benchmarks that seemed reasonable, given the limited information available. Each member of the evaluation team informally talked with other stakeholders to ensure consensus. Table 4. Indicators and Program Standards for Evaluation Questions
Data Collection The evaluation utilizes several methods of data collection in various ways: interview, survey, observation, and record review. Staff interview will be conducted to assess knowledge and behavior in accessing translators and LHAs. Written survey (in Spanish) will be administered to the patients in the clinic waiting room to assess patients’ perception of the clinic (access to care, hours, acceptable environment/staff) and their comfort level (attitudes and beliefs) about the staff (a compromise due to limited funding). Observation will be conducted to assess staff behaviors and language used in TB awareness signs. Lastly, review of records, charts and training materials will be conducted. Table 5 summarizes the data source and data collection methodology that will be used to gather evidence for evaluation. Table 5. Data Collection Plan
Plan Timeline Table 6. Illustrative Timeline for Evaluation Activities
V. JUSTIFY CONCLUSIONS: ANALYSIS AND INTERPRETATION Analysis Both quantitative and qualitative methods will be used to analyze the data. Simple counts of frequency will be use for quantitative data analysis. Qualitative methods such as content analysis will be used to review training curriculum, patient charts and records for themes and patterns. Table 7. Analysis Plan
Interpretation Stakeholders including the county administrator, program manager/staff, LHAs, medical staff and CBO representatives will be included in a scheduled meeting to interpret the findings. The data from the evaluation will be compared to the established program benchmarks. Stakeholders and those involved in program operations will be given an opportunity to justify the findings and make recommendations accordingly. VI. ENSURE USE AND SHARE LESSONS LEARNED: REPORT & DISSEMINATION Dissemination Evaluation finding will be disseminated via various channels. Presentations will be given at the program staff meeting and to the health care providers at regular staff meetings. A short report will be drafted and a presentation offered for the county health commissioners. An article will also be added to the newsletter the health department periodically compiles. Carlos will also write an article to go in a local Spanish newspaper and offer to present at community meetings. Table 8. Dissemination Plan
Use Libby, the program manager, and staff will use the findings to refine program strategies for TB Support Program. The findings will help guide the program to focus on areas that are most crucial for effective service delivery. Health care professional will use the finding to improve their medical practices in serving Salvadoran community. County health department administrators, led by Don, will use the findings to determine the future funding distribution for the program. The community liaison, Carlos, will use the findings in advocacy efforts. Finally, findings from this evaluation will be used for future evaluations.
Last Reviewed: 05/18/2008 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
If you would like to order any of the DTBE publications please visit the online order form. You will need Adobe Acrobat™ Reader v5.0 or higher to read pages that are in PDF format. Download the Adobe Acrobat™ Reader. If you have difficulty accessing any material on the DTBE Web site because of a disability, please contact us in writing or via telephone and we will work with you to make the information available. Division of Tuberculosis Elimination
Centers for Disease Control & Prevention |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||