Part 2: Program monitoring

Monitoring refers to the simple description, counting, and tracking of processes or events, without in-depth analysis or comparisons. Monitoring will answer the questions What? Where? When? and How much or how many? but not Is it effective? or Why is it effective? Thus, monitoring will be important for assessing if something is being done and if it is being done as intended, if it is being done where and when intended, and if it is being done as much as intended. In particular, monitoring will be useful for assessing adherence to and changes in policies, procedures, and protocols and progress toward achieving objectives. Domains that should be addressed by monitoring include 1) program implementation and management, 2) processes, 3) program performance, 4) achievement of goals and objectives, and 5) resource requirements (i.e., staffing and cost).

Monitoring program implementation and management

Monitoring implementation and management of the program (e.g., hiring staff, purchasing equipment, developing policies and procedures) will ensure that critical operational issues are being addressed and that program implementation is on schedule, and will provide a better understanding of what is needed to develop a social networks program.

Protocols should include plans for implementing the program, as well as implementing objectives (e.g., “By [date], all program staff will be hired and in place.”) and time lines.

Monitoring processes

The program monitoring plan should identify key processes and describe how the agency initially plans to accomplish them. The plan also should include barriers encountered, how the barriers were addressed, and lessons learned (e.g., What was their initial plan for enlisting recruiters to participate in the program? Were there any problems or barriers? Were the methods for enlisting recruiters changed to address problems or barriers? If so, in what way were they changed?). Monitoring will be done using qualitative data, which may come from the contractor’s original site protocols, interviews with staff, debriefing notes of outreach and program coordinator staff, weekly reports of activities incorporated in quarterly reports, and interviews with staff.

Lessons Learned

“Have stop-gap measures to monitor productivity, don’t get caught up in it if it’s not producing.”

Monitoring program performance

The program monitoring plan should quantitatively measure the performance of each key step or procedure, including, at a minimum, the following:

• Number of HIV-infected persons invited to serve as a recruiter
• Number of HIV-infected persons who agree to serve as a recruiter
• Demographics of recruiters
• Number of network associates identified by each recruiter
• Number of network associates located and offered CTR services
• Demographics of located network associates
• HIV risk characteristics and HIV testing histories of network associates
• Number of network associates who receive CTR services
• Number of network associates who test HIV-positive and were previously undiagnosed
• Number of network associates who test HIV-positive and were previously diagnosed
• Number of network associates testing positive who receive test results
• Number of network associates who test negative
• Number of network associates testing negative who receive test results
• Number of network associates testing positive who are successfully referred for medical evaluation, treatment, and prevention services
• Number of high-risk network associates testing negative who are successfully referred for prevention services

Lessons Learned

“Streamline indicators such that program managers will know when they are going in the right or wrong direction.”

A useful way to illustrate the networks being targeted by your program, and to monitor program success, is to periodically create a network diagram that visually represents all the individuals who have been identified and/or tested to date. Network diagrams can help program managers determine which networks are more productive than others (i.e., which networks lead to a greater number of undiagnosed HIV-positive individuals but require fewer people be tested). These diagrams are also useful tools to show outreach workers and other staff in the field how well the testing strategy is working within various networks. On the next page is an example of a network diagram for a site that enlisted 15 recruiters in its program (Figure 3). Which networks are worth pursuing in the future (e.g., network of recruiter #2-1) and which networks may not be worth continued efforts (e.g., network of recruiter #2-277) can quickly be seen.

Network diagrams also help identify indirect network associates (people who want to get tested because they learned of the program through a network associate). Indirect network associates may also be invited to be recruiters if they meet eligibility criteria.

		“Network diagrams are very useful and help illustrate what’s going on in a community. They can show how the virus flows within the networks.”
Lessons Learned		“Diagrams can be an incentive to encourage people to work harder and determine which network needs more focus. Overlaying information is visually impressive and can be a great communication tool to program workers and recruiters.”

Figure3. Example network diagram for a site (15 total recruiters)

Recruiters   Network associates   Indirect network associates Black node   HIV-positive White node   HIV-negative Gray node   HIV status unknown

Click to enlarge

Monitoring achievement of goals and objectives
Monitoring the achievement of program goals and objectives is critical for knowing which elements of the program need to be modified or improved. Agency goals and process and outcome objectives should serve as the framework if progress reports are submitted to stakeholders.

1. Goals

The program monitoring and evaluation plans should include the agency’s broad goals for its program (e.g., Increase the number of HIV-infected African American men who have sex with men, 18-30 years of age, in the Tenderloin district of San Francisco, who are aware of their infection).

2. Process objectives

The monitoring and evaluation plans should include process objectives for the program (e.g., By [date], [number] recruiters will have been enlisted to participate in the program). These process objectives should be based on the key steps or procedures identified in the implementation plan.

3. Outcome objectives

The primary outcomes for this program should be: a) HIV-positive persons not previously aware of their infection will be identified and linked to medical, prevention, and other services; and b) HIV-negative persons at high risk for becoming infected with HIV will be identified and linked to prevention and other services.

Monitoring and evaluation plans should include at a minimum the following outcome objectives:

• Number and percentage of persons tested who were HIV-positive and previously unaware of their infection
• Number and percentage of newly identified HIV-positive persons who were successfully linked to appropriate services
• Number of high-risk HIV-negative persons who were identified
• Number and percentage of high-risk HIV-negative persons identified who were successfully linked to appropriate services

Monitoring resource requirements (costs and personnel)
Agencies should track and document costs and staffing requirements (e.g., number of personnel, skills and expertise) needed to implement their program. This will provide information about resource requirements to other organizations interested in implementing similar programs.

Progress reports
Progress reports should include both quantitative data and qualitative information regarding implementation of the program and lessons learned, such as facilitators and barriers to the program and how barriers were addressed. Progress report procedures will include the following:

• Progress report format, to be structured around the stated goals and implementation, process, and outcome objectives
• Standard format data reports for assessing program performance

Go to Section Four; Part 3