| The President's Emergency Plan for AIDS Relief: Second Annual Report to Congress Released by the Office of the U.S. Global AIDS Coordinator February 8, 2006 Chapter 10: Improving Accountability and Programming  PDF version The collection and analysis of high-quality data in underresourced settings is thus a critical challenge. Given the range of challenges faced by health care personnel, it is difficult for them to focus on patient tracking and records management. The weakness of health management information system (HMIS) infrastructures compounds the challenge. In order to support host nations in addressing these issues, U.S. Government (USG) in-country teams, in coordination with host governments and partners, determine strategic information (SI) activities and priorities for the upcoming fiscal year as the part of the Country Operational Plan (COP) process. PEPFAR-supported activities at the country level focus on both the availability and the quality of data – both must be assured. Efforts to meet these challenges include:
Best Practices Sporadic military hostilities gave way to a tense, unpredictable standoff when the AIS was launched in September 2004. Implementation of the survey required overcoming serious obstacles presented by the political-military crisis, which had divided the country into three zones, only one of which was fully controlled by government forces. The challenge of coordinating the implementation of a nationally representative high quality survey in such an environment was assured by an AIS Steering Committee, comprised of host country entities and international partners, with technical leadership provided by a PEPFAR implementing partner. Mapping for sampling and training in data and blood sample collection were completed in the first quarter of 2005, despite large-scale political demonstrations and violence. PEPFAR provided support for in all technical aspects of the survey, including preparation of questionnaires and manuals, training, data processing, and tabulation and analysis plans, even in the northern region where security agents had to be deployed with the technical experts to ensure safe passage. Data and dry blood spot collection were successfully carried out nationwide between August 4 and October 20, 2005. Analysis of data from the field implementation of the survey is continuing. Despite ongoing tensions and the lack of unified leadership in the country, the response rate of the population in all three zones (government-controlled, rebel-controlled and buffer) was very high. All data were compiled and entered in the database in Abidjan, completing part of the AIS, for which the crisis conditions increased both the price tag and payoff in experience and knowledge gained. Little is known about the dynamics of HIV/AIDS in crisis conditions, and yet such conditions are fairly common among those countries and populations most affected. The Côte d’Ivoire AIS, which could not have been implemented without Emergency Plan support, will provide high-quality information from such a setting. These data will be invaluable, informing USG HIV/AIDS programming and ensuring cost-effective use of Emergency Plan funding to mitigate the impact of the disease on highly vulnerable populations. International Coordination The USG has come together with WHO, UNAIDS, the UK Department for International Development (DfID), the World Bank, and the Global Fund to Fight AIDS, Tuberculosis and Malaria for a series of meetings on data harmonization. Among the results of these productive consultations were the first-ever joint analysis and release of data on treatment results in January 2005, improved estimates of treatment results by international partners (including PEPFAR), and initial steps toward improving estimates of prevention and care results. The organizations have also collaborated to produce coordinated guidance for reporting of future results. USG strategic information staff have come together with staff from international organizations for joint training in such areas as monitoring and evaluation and management information systems. PEPFAR, along with UNAIDS, WHO, World Bank and Global Fund staff, offered two regional strategic information training meetings in Africa in fiscal year 2005, with the option of further meetings as needed. During these meetings, coordinated country monitoring and evaluation plans were discussed and outlined. Working to ensure data quality, PEPFAR has worked with WHO and others to issue standards for the transfer of data among systems and data storage and reporting, and has worked with the Global Fund and the Health Metrics Network to develop a data quality diagnostic tool. Standardized data abstraction, storage, and access procedures will enable stakeholders in USG agencies and partner organizations to participate in antiretroviral treatment (ART) program evaluation and improvement. Consultations were held by international agencies and donors, including the USG, on use of HL7 standards in data transmission and common data storage and confidentiality procedures. These data collection standards will allow programs to efficiently transfer and better monitor progress and identify problems in order to refine and adapt implementation strategies. Analyses of these data will inform clinical decision-making, encourage improvement of ART program implementation, and assess impact of the USG’s investment in treatment programs on individuals, families, and communities. Ensuring accountability is predicated on obtaining high quality information on program results. In recognition of this joint objective, PEPFAR has partnered with the Global Fund and the Health Metrics Network to develop a framework for data quality. Three tools are linked together within this framework that target different stakeholders involved in data collection and results reporting: host country governments, grantees and prime partners, auditors, and USG or Global Fund managers. Initial pilots of the data quality harmonization framework and toolkit in 2005 have received favorable reviews from host country partners and grantees, which benefit greatly from standardized and coordinated approaches to results reporting and strategic information systems development. Development of common indicators, to the extent possible, is another priority area for Emergency Plan strategic information efforts. PEPFAR is working with other international donors to create or improve indicator guides in such areas as treatment, counseling and testing, stigma, most at-risk populations, and orphans and vulnerable children (a joint effort with UNICEF). The Emergency Plan and other key partners have collaborated within countries to share common data on program results and surveillance. At the international level, international partners have worked with host nations and each other to compare sub-recipients to help eliminate unnecessary duplication. With the Global Fund and UNAIDS, PEPFAR has established a Joint Monitoring and Evaluation Technical Assistance Facility. This facility, housed at UNAIDS, links countries requesting technical assistance with consultants who can assist them in the areas of need. The Global Fund and the Emergency Plan are also jointly supporting work by the U.S. Census Bureau to estimate infections averted in the 15 focus countries and other key countries. The Emergency Plan has also provided support for the WHO in laboratory surveillance, development of ART patient monitoring and reporting guidance, management information systems standards, and monitoring and evaluation. For example, PEPFAR has supported WHO’s Service Availability Mapping project – a census of health facilities that identifies which facility offers which HIV/AIDS services, critical information for international partner coordination.
An important activity in fiscal year 2005 was the development of a number of SI technical working groups focused on developing standards and supporting country programs. Working groups address the following technical areas:
The Scientific Steering Committee, through its subcommittee on targeted evaluations, also guides headquarters participation in targeted evaluations. Twenty-five targeted evaluations, including some that had previously been initiated, were conducted in fiscal year 2005, and the number is expected to grow in the coming year. In addition, fiscal year 2005 saw headquarters SI advisors help country teams to address challenges in Emergency Plan implementation, including target setting, COP submission, and results reporting. SI country liaisons helped to ensure submission of high quality data from the field. A text-only version of following table is available.
The reporting burden task force formed in fiscal year 2005, which is discussed in the chapter on Implementation and Management, addressed and resolved many of the SI reporting issues raised by country teams. As noted in the chapter on Prevention, PEPFAR created a Scientific Advisory Board to review data on the possible protective effect of male circumcision and to provide evidence-based guidance to Emergency Plan programs. Surveillance Information to Track HIV Incidence, Prevalence, and Mortality In Kenya, the USG’s investments in surveillance systems and surveys have been critical in producing data to monitor trends in HIV prevalence. Sentinel surveillance in pregnant women has been conducted annually since 1990, and the past year’s analysis of the 2004 surveillance data demonstrated a continued drop in HIV prevalence among pregnant women from a high of 13.5 percent in 2000 to a national estimate of 7.4 percent in 2004. Additional USG support was provided in 2005 for the Kenya Demographic and Health Survey, the first national survey of HIV prevalence. Analysis of this survey’s data provided further evidence of a decline in HIV prevalence in Kenya from 10 percent to 7 percent among adults 15 – 49 years of age from 1998 to 2003. Sentinel surveillance data were collected again in 2005, and will provide important measures of the potential impact of the USG’s contributions to the prevention of HIV transmission in Kenya. Behavioral surveillance is becoming more closely linked to HIV testing, with those surveyed offered testing, yielding additional data points on HIV prevalence. PEPFAR is also planning sentinel site surveillance to monitor treatment outcomes, as well as HIV-related morbidity and mortality. During 2005, the Emergency Plan drafted a protocol for ARV resistance surveys. SAVVY (Sample Vital Registration with Verbal Autopsy) is an established best practice to collect sample vital registration information on national populations. In places where vital records systems are weak, SAVVY can improve the ability to estimate the levels and causes of morbidity and mortality through "verbal autopsies" conducted by interviewing surviving household members. PEPFAR supports development of comprehensive curriculum and training materials to improve tuberculosis (TB)/HIV surveillance in TB clinics. This complements the successful electronic TB registers the Emergency Plan supports, which aid in tracking people living with HIV/AIDS (PLWHA) receiving TB care and treatment. The USG supported testing of the BED-capture enzyme immunoassay (VED-CIA) to assess overall incidence trends in HIV epidemics. Testing efforts included working to make the assays available, training laboratory staff, and developing protocols to bring the assays to surveillance sites. In recognition of the need to adapt surveillance strategies to the realities of particular countries, PEPFAR, in conjunction with international agencies, has developed tools and plans for surveillance in countries with concentrated epidemics. Results of PEPFAR-supported surveillance efforts to date include improvements in HIV baseline estimates, methods for estimating and projecting incidence and prevalence, definition for AIDS case reporting, and methods for ART resistance surveillance. Program Reporting and Monitoring Mechanisms PEPFAR’s move to electronic reporting via the COPRS database is streamlining the process significantly, as country teams directly upload information that headquarters personnel can then download and review. The Emergency Plan conducted training and workshops on monitoring and evaluation for strategic information personnel in the focus countries – each of which now has an SI team. PEPFAR is also working to integrate monitoring into staff training curricula on an ongoing basis. In fiscal year 2005, common indicator guidance was developed for USG agencies to use in their results reporting. All USG partners provide semi-annual program results reports to USG activity managers on progress and challenges. PEPFAR staff also conduct regular site visits to review partner activities and assess the quality of data reported by partners. In South Africa, many new local partners were engaged to rapidly scale up HIV/AIDS programming in 2005. The USG SI team prioritized strengthening the M&E capacity of these partners as an essential component of effective program implementation. Implementing partners all attended one or more M&E capacity building workshops in 2005 and benefited from individualized site visits from M&E experts. As a long term strategy, the South Africa team is leveraging other USG investments in the University of Pretoria, which implements an M&E track in the Masters of Public Health program. Through an innovative internship program, recent graduates of the Masters program are matched with PEPFAR partners that are in particular need of M&E support. PEPFAR invested in the development of a data quality diagnostic tool as a practical guide for program managers to improve the quality of data collected according to Emergency Plan guidance. Initial piloting of the tool has proven its effectiveness in aiding program managers to generate and manage data for advocacy and program improvement decisions. The USG has leveraged WHO and Global Fund in support of the development of a broad set of data quality tools, with the intent of training host country partners in their use in fiscal year 2006. The Emergency Plan M&E capacity-building technical working group has being working collaboratively with UNAIDS, the Global Fund, and WHO on the development of guidelines for the evaluation of M&E capacitybuilding in recipient countries. Application of these guidelines is planned for fiscal year 2006, with the identification of best practices for wide dissemination.
Health Management Information Systems
HMIS developments in fiscal year 2005 include the application of USG and other health reporting technologies in developing country reporting and patient medical records monitoring systems. Technologies being piloted include smart cards, cellular phone reporting systems, and integrated patient management and prescription electronic records. Software tools for laboratory information systems have been inventoried, and PEPFAR has increased its use of geographical information systems. PEPFAR support for strategic information tools to help host nations improve clinical care management is described in the chapter on Building Capacity for Sustainability. Examples of Emergency Plan-supported HMIS enhancements in 2005 include the following:
Internal Reviews In addition to its data and financial auditing efforts, the Office of the U.S. Global AIDS Coordinator (OGAC) has engaged an independent contractor to conduct a special Program Audit to assess implementation of ABC prevention activities by Emergency Plan grantees and subgrantees. The Emergency Plan has also funded the MEASURE Evaluation Project to provide:
These products will help PEPFAR develop systems and processes that contribute to long-term, sustainable, high-
External Reviews The Government Accountability Office (GAO) is reviewing HIV/AIDS prevention under PEPFAR. This review is being done at the request of the U.S. House of Representatives’ Committees on International Relations and Government Reform. The review addresses PEPFAR’s strategy for HIV/AIDS prevention, its interpretation and implementation; obligation, expenditure, and allocation of prevention funding; and the award process for implementing partners. During fiscal year 2005, GAO representatives traveled to four focus countries and regularly discussed prevention efforts with senior staff members of OGAC. Congress has also mandated the U.S. Institute of Medicine (IOM), a non-governmental entity, to evaluate Emergency Plan implementation. The task of the IOM Committee for the Evaluation of PEPFAR Implementation is to examine a variety of measures of program success, in order to provide constructive information to the U.S. Congress. Evaluation measures are being derived from a variety of sources, including baseline information and information on activity design at the point of funding decisions, at the point of implementation, and at intervals thereafter. During fiscal year 2006, members of the Committee will conduct site visits to focus countries. In fiscal year 2005, the IOM Committee produced a report, entitled Healers Abroad, that provided recommendations relating to potential involvement of American volunteers in PEPFAR activities. Identifying Best Practices Because of the Emergency Plan’s focus on producing results through sustainable programs of high quality, identifying evidence-based best practices is a core principle. Determining key evaluation questions, monitoring methods and results, and evaluating programs and activities on the ground are critical elements in bringing about continuous program improvement. Many promising proposals for targeted evaluations have been considered by the subcommittee on targeted evaluations of the PEPFAR Scientific Steering Subcommittee. This body of USG program implementers, evaluators, and research scientists conducted technical review of proposed targeted evaluations for fiscal year 2006, and its input was reflected in funding decisions. As noted above, PEPFAR is also considering the possibility of joining with other donors to establish a centralized mechanism for technical assistance to countries implementing monitoring and evaluation. The Second Annual Field Meeting on the Emergency Plan in Addis Ababa, Ethiopia, included many presentations on information collection and serialization by country teams, affording another important opportunity for dissemination of best practices to in-country teams as well as to host governments and other partners represented at the meeting. Key Challenges and Future Directions Disruptions to national health systems include major setbacks to developing nations’ efforts to monitor and evaluate programs, as these activities are often the first things to be abandoned during emergencies. During fiscal year Attribution of results of upstream Emergency Plan program support (as described at the end of the Prevention, Treatment, and Care chapters) for national programs is an ongoing challenge. There is typically a lack of USG presence at the sites where the upstream support is being put to use, making PEPFAR dependent on host government reports. In addition, the fact that Emergency Plan support for national efforts is provided in close collaboration with other donors makes attribution of results especially difficult. PEPFAR is working with host governments to refine attribution of results in these contexts. Another key challenge is posed by the new joint USG/WHO ARV guidelines, which provide medical records standards. The Emergency Plan is working to ensure technical assistance and training in medical records systems as this change takes place. As noted above, ensuring the data quality in resourceconstrained settings remains difficult, and PEPFAR is addressing this issue by providing training for country teams and partners on data quality tools. PEPFAR has continued to work to build systems incountry for partner accountability and reporting. Some host nations have embraced database approaches analogous to PEPFAR’s COPRS, and the USG is working with additional countries to facilitate their adoption of similar tools. Another challenge in fiscal year 2006 will be the implementation of results reporting by a greatly expanded group of country teams – those in nations beyond the focus countries that receive more than $1 million in PEPFAR support. |
 | |||
|
" Overcoming extreme poverty will require humanitarian aid that focuses on results, not merely on inputs and other flawed measures of compassion. True compassion is measured by real improvements in the lives of men, women and children. And that is the goal and that is the focus of American policy." 
In fiscal year 2005, the USG agencies implementing the Emergency Plan made further advances in integrating their surveillance, HMIS, targeted evaluation, and M&E activities. Building on success in 2005, both headquarters and country USG offices developed updated annual activity plans and joint budgets for strategic information for fiscal year 2006. 
In fiscal year 2005, the Country Operational Plan and Reporting System (COPRS) became fully operational. This web-based information system allows for the annual entry and updating of Emergency Plan COPs, annual and semiannual program results, and budget information by the focus country teams. Information can now be rapidly searched and reported by key factors and areas of interest to the program, including partner and partner type, program area, geographic coverage, gender issues, indicator targets and results, and budget. 