TB Notes Newsletter

No. 4, 2011

TB PROGRAM EVALUATION NETWORK UPDATE

Highlights from the 2011 TB PEN Conference and Program Evaluation Focal Point Meeting

For a third year, the TB Program Evaluation Network (TB PEN) joined forces with the TB Education and Training Network (TB ETN) to conduct a joint TB ETN/TB PEN Conference. The TB PEN Training Team collaborated with the TB PEN Conference Planning Committee to develop the first TB PEN New Member Orientation.  The New Member Orientation preceded the conference opening session and provided attendees with an introduction to TB PEN, defined the role of the focal point, presented resources for program evaluation, and provided an overview of the TB PEN Conference sessions. Conference activities also included joint plenary sessions, skill-building sessions, expert consultation, and a poster session. The conference social at the end of the first day brought TB PEN and TB ETN members together for an excellent evening of networking.

TB PEN participants were able to attend TB PEN skill-building sessions which combined theory with practical application and provided a variety of speakers representing both CDC and state and local TB programs. Evaluation session topics included evaluation 101, basic data analysis / statistics 101, moving forward from evaluation findings, program evaluation plan assessment, and a panel with selected programs on four program evaluation focus areas.  TB PEN plenary sessions focused on cohort review and CDC guidance on developing an evaluation plan and interim and annual reports. 

As a direct result of previous conference evaluations, the expert consultation sessions were redesigned this year to provide participants with an opportunity to interact one-on-one or in small groups with their respective program consultant or program evaluation representative.  This was structured to assist focal points with gaining a better understanding of program evaluation expectations and effectively addressing specific requirements pertaining to their cooperative agreement.

Another change this year saw the TB program evaluation (PE) focal point meeting incorporated into the conference as a breakout session. Approximately 46 individuals who serve as the assigned points of contact for TB program evaluation within their respective agencies were in attendance. The focal point meeting provided updates on TB PEN activities, incorporated a skill building session on survey design / designing questionnaires, and included a focal point feedback session. The focal point feedback session was an opportunity for PE focal points to brainstorm challenges and solutions to program evaluation in their respective jurisdictions and to provide the TB PEN Steering Committee and DTBE feedback on their needs for program evaluation guidance and technical assistance.

Attendees also participated in the poster session featuring 34 posters, 14 of which represented TB program evaluation. Two of the posters representing TB PEN were selected for oral presentation. The posters, titled: Challenges Associated with Decentralizing the Entry of Tuberculosis; and Evaluation of a New Health Code Requirement for Discharging Infectious Tuberculosis (TB) Patients from New York City Hospitals were presented respectively by Sandra Matus from Maryland and Michelle Macaraig from New York City during the local presentations plenary session.

Overall attendance at the TB PEN breakout sessions averaged 65 participants. TB PEN focal point meeting evaluations indicated that the majority felt it was useful to attend the TB PEN focal point meeting (88%), thought the focal point feedback discussion was engaging (83%), and agreed that the survey design session was informative (95%). Focal points also indicated that they prefer to attend the TB PEN Conference on an annual basis (70%) in the month of September (61%).
Suggestions for the 2012 TB PEN Conference include:

• Designing sessions to be more participatory and how-to;
• Providing more subject matter sessions, such as data analysis and evaluation plans; and
• Offering more practical examples and sharing from program areas.

The TB PEN Conference Planning Committee convened again in November 2011 and will begin planning for the 2012 TB ETN & TB PEN Conference in January 2012.  Please contact the TB PEN Conference Planning Committee at tbpen@cdc.gov if you have further suggestions for next year’s conference or would like to participate on the planning committee.

—Reported by Brandy L. Peterson, MPH, MCHES
Div of Elimination

Top of Page

TB PEN Focal Point Needs Assessment: Preliminary Results

Introduction
With the increasing emphasis on program evaluation activities within TB prevention and control programs and the introduction of the National TB Indicators Project (NTIP), the leadership at CDC’s Division of Tuberculosis Elimination (DTBE) determined the value in having a designated individual responsible for evaluation activities. Initiated by DTBE in 2009, a designated individual at each of the 68 federally funded TB prevention and control programs was to serve as the evaluation focal point for their program. Together, these individual focal points formed a TB Program Evaluation Network (TB PEN). The role of the TB PEN and the evaluation focal point was designed to parallel the TB Education and Training Network (TB ETN), in existence since 2001, along with the designated focal point for training and education.

In addition, DTBE convened a TB PEN steering committee in recognition that the newly formed network would benefit from the guidance of a diverse group of TB and evaluation professionals representative of the TB PEN membership. The Steering Committee includes representatives from state and local/city TB programs, the Regional Training and Medical Consultation Centers, National TB Controller’s Association, TB ETN, TB Epidemiologic Studies Consortium, TB Trials Consortium, and DTBE. The Steering Committee guides TB PEN by reviewing and assessing applicability of program findings; advising DTBE on national evaluation focus; and identifying needs for evaluation training, technical assistance, and tools.

The TB PEN steering committee decided to assess the training needs of the evaluation focal points to gain greater clarity about the individuals serving in the role, as well as the additional training and support that might be required for the focal points, and the programs, to be successful in the evaluation activities expected of them. This preliminary report focuses on the results obtained in several areas of greatest interest to the TB PEN steering committee: 1) the characteristics of those individuals assigned the role of TB PEN focal point, including their previous program evaluation experience; 2) current program evaluation activities conducted in the state TB programs; and 3) additional training identified by the respondents that would increase their success in evaluation activities.

Methods
Members of the TB PEN steering committee developed the assessment which targeted several areas: 1) the role of TB PEN focal point; 2) program evaluation activities of the program, including the NTIP indicators selected by the programs for initial emphasis; 3) previous evaluation experience; 4) training needed in general, and 5) additional training specifically focused on cohort review and evaluation tools.

The on-line assessment tool was launched on April 18, 2011, using Qualtrics Survey Software. The assessment was distributed to the individuals listed as the program TB PEN focal point.  The assessment remained open for 4 weeks, and two follow-up reminders were sent to the study universe after the initial invitation to participate in the assessment was distributed.

Results
A total of 88 individuals were invited to participate in the survey. Although 66 surveys were initiated, there are complete data for 50 respondents (57% response rate). Response rates vary from question to question; the number of respondents (N) is reported for each question.

Description of the individuals who serve as TB PEN Focal Points
The TB PEN Focal Points who completed the needs assessment represent diverse roles within their respective state TB Prevention and Control Programs (Table 1).  Almost one quarter of the respondents (24%) were Program Managers; however, Epidemiologists (22%), TB Controllers (20%), and Nurse Consultants (19%) were also heavily represented among the respondents.  Those indicating “other” roles included public health advisors (PHAs), IT/Data Analysts, and other TB coordinating roles.

Table 1. Respondents’ Role within the State TB Program.

Role	Number	Percent
TB Controller	12	20
Program Manager	14	24
Health Educator	2	3
Epidemiologist	13	22
Nurse Consultant	12	21
Other	10	10

The majority of the respondents reported working in TB prevention and control for more than 5 years (62%).

The majority (69%) did not have a dual role as both the designated TB ETN and TB PEN focal point, and had been the designated TB-PEN focal point for more than 1 year (61%). Three quarters (74%) indicated that they had less than 50% for their time allocated to TB program evaluation activities; however, almost half (48%) indicated that they had less than 25% of their time allocated to evaluation. Nevertheless, 52% of all respondents indicated that the time allocated was sufficient to carry out TB program evaluation activities.

Seven (12%) respondents stated that they were exclusively responsible for evaluation; however, 6 of the 7 (86%) stated that they did not feel supported in this role. Self-identified barriers included competing priorities both for themselves and their staff members, as well as a need for evaluation skills training for themselves or adding staff members trained in evaluation.

Evaluation experience
When asked about their evaluation experience, 39 (75%) of the respondents reported having received formal training in program evaluation. The most common forms of training described were from an academic setting, usually as part of a graduate program. Trainings provided by CDC or TB PEN were also often cited. In addition, respondents mentioned evaluation training provided via webinars and other conferences.  Over half of the respondents (52%) reported having been involved in program evaluation activities prior to the DTBE requirement for program evaluation activities; data collection (96%) and data analysis (92%) activities were reported by the greatest percentage of respondents who were engaged in evaluation activities prior to the establishment of the requirement. 

Several evaluation documents have been developed by CDC, and respondents were asked to identify which of these resources they had read or consulted in their role as the evaluation focal point.  The majority of respondents reported that they have used/read the different documents available to them (Graph 1).

Graph 1. Percentage of respondents who have accessed existing evaluation documents

Program evaluation activities
When asked about program evaluation activities in their state, 77% indicated that the TB program had developed clear and achievable goals for program evaluation, and 90% of those programs had developed specific and measurable objectives to achieve these goals. Almost three quarters (70%) had developed an action plan for evaluation; 83% of those with an action plan reported having a timeline for completion of activities for each objective.

Respondents were asked to indicate the National TB Objectives/Performance targets identified as a focus for their state’s evaluation plan (Graph 2). The top five most frequently cited targets were contact investigation (60%), completion of treatment (48%), known HIV status (40%), sputum culture conversion (35%), and evaluation of immigrants and refugees (29%).  Additional targets cited by more than one quarter of the respondents included data reporting (27%), program evaluation (27%), and universal genotyping (25%).

Graph 2. Reported National TB Objectives/Performance Targets

Training Needed by the TB PEN Focal Points
The assessment was designed to determine the additional training required for focal points to feel confident in their role and to be successful in guiding the evaluation activities of their TB program. A list of potential training topics was provided for ease; however, the respondents were also able to identify topics not included in the list. The percentages of respondents identifying the additional training topics are reported on Graph 3. Although all of the topics for additional training were identified as important to their success by 25% of the respondents, the most frequently cited additional training was the development of various types of evaluation tools (survey design, chart abstraction) and data analysis (both cited by 51% of the respondents).  Approximately half of the respondents (46%) also indicated that additional training in the overall design of the evaluation would be beneficial to their success.

The “other” training responses referenced the need for additional time or funding to perform the evaluation activities rather than having more training or training in time management to balance the competing claims on their time.

Graph 3. Additional training topics identified for evaluation focal points

Respondents were asked to identify additional training or information needs related to specific program evaluation tools (Graph 4). The most frequently requested training/information were assistance related to locating tools, models, or plans for evaluation (52%); logic models (42%); NTIP (34%); and Cohort Review (16%). One quarter (26%) of the respondents indicated they did not need any additional training or information at this time.

A question about the type of training format which would best fit the needs of the focal points revealed that 56% of the respondents preferred live web-based seminars, or webinars.  Interactive on-line courses were identified by 50% of the respondents, and in-person training and archived webinars were both identified as formats meeting their needs by 48% of the respondents. Self-study module formats were also identified by 46% of the respondents as a format best meeting their training needs.

Graph 4. Tools identified by respondents as those that require more training or information

Conclusions
These preliminary results indicate that the current cohort of evaluation focal points have numerous years of TB experience and represent diverse roles within a TB program.  The majority of focal points are assigned exclusively to this role and do not also have responsibilities as the education and training focal point.  Three quarters of the focal points have had formal evaluation training, most commonly at a university.  The amount of time allocated to evaluation responsibilities varied, although over half of the respondents believed that the time allocated for evaluation activities was appropriate and sufficient.

Three quarters of the respondents indicated the need for additional training to adequately and successfully fulfill their evaluation role. The additional areas of training identified included assistance related to locating the appropriate tools, models, or evaluation plans; and an understanding of how to use logic models, NTIP, and Cohort Review.  The preferred formats for these trainings included webinars, interactive online courses, and stand-up trainings. Slightly fewer respondents indicated that archived webinars and self-study modules were desirable formats for learning.

These preliminary results will be used by the TB PEN steering committee as the foundation for future activities and trainings aimed at supporting these individuals in their roles as evaluation focal point.

—Reported by Paula Hamsho-Diaz, MD, MA, Ellen Murray, RN, BSN, Donna Hope Wegener, MA,
Southeastern National TB Center;
Brandy L. Peterson, MPH, CDC/DTBE; and
Stephen E. Hughes, PhD, NY State TB Control Program

Top of Page