Detailed Information on the
Afghanistan Health Initiative Assessment

Establishing baselines and ambitious targets for annual performance goals.

Establishing regular procedures to achieve efficiencies and cost-effectiveness.

Demonstrating through individual performance plans how federal managers are held accountable for program performance.

Developing a program evaluation plan and schedule for an independent evaluation that is high quality, sufficient in scope, and unbiased.

Measure: The time to hire and deploy essential staff trainers.

Explanation:Given the security challenges of working and recruiting for Kabul, HHS took a multi-pronged approach for clinical training of RBH staff: (1) working closely with Awardees on the ground and (2) direct engagement of HHS clinical experts within RBH for the provision of didactic, on-site clinical training, with a focus on quality assurance for ceasarean deliveries (C-sections). Beginning in 2008, HHS clinical teams have spent three rotations in Kabul (January/February, June/July and September/October), for a total of 4 months. Because of the security situation in Afghanistan and budget restrictions, it is difficult to recruit and retain qualified staff trainers. Additionally, U.S. government personnel are required to stay at the U.S. Embassy in Kabul which has severe housing limitations. To help recruit trainers, HHS is reaching out to the Indian Health Service, to provide additional personnel. Also, HHS is working with academic institutions throughout the United States to encourage participation in the project.

Measure: The percentage of staff trainers who fulfill the agreed-upon in-country contract.

Explanation:The retention of clinically-competent educators remains challenging given that: (a) Kabul and Afghanistan more broadly remains a security risk/conflict situation; (b) RBH is a women's hospital, gender, culture, and linguistic competencies need to be taken into account for any training situation; and (c) RBH pay-scales are low and negatively affect retention. For these reasons, enabling HHS clinical experts to educate/training through on-site didactic clinical rotations at RBH (see above) was implemented in 2008.

Measure: The in-hospital maternal mortality rate per 100,000 caesarean sections at Rabia Balkhi Women's Hospital in Kabul, Afghanistan.

Explanation:The Afghanistan Health Initiative works to reduce the intrapartum and postpartum maternal mortality rate associated with caesarean sections at this major publicly-funded hospital by providing training to hospital staff aimed at reducing the risk of death to mothers during labor, delivery, and the post-partum period.

Measure: The rate of fetal deaths occurring during labor or delivery among newborns who weigh at least 2500 grams at birth at Rabia Balkhi Women's Hospital in Kabul, Afghanistan per 1,000 such births.

Explanation:The predischarge perinatal intrapartum mortality rate is the number of fetal deaths occurring during labor and delivery among newborns who weighed 2500+ grams at birth, divided by the total number of births of babies weighing at least 2500 grams, per 1,000 such births.

Measure: The percentage of women who have a caesarean section delivery who subsequently develop a post-operative infection at Rabia Balkhi Women's Hospital in Kabul, Afghanistan.

Explanation:This measure tracks the number of women who develop post-operative infections as a result of a caesarean section divided by the total number of women who have had caesarean sections. This may be difficult to measure because (1) women do not return to RBH for infection care after hospital discharge, (2) at the current time, HHS does not have an accurate follow-up program for tracking in the community after hospital discharge. The establishment of this surveillance system is a 2009 and beyond effort.

Measure: The newborn predischarge mortality rate for babies weighing at least 2500 grams at birth at Rabia Balkhi Women's Hospital in Kabul, Afghanistan per 1,000 births.

Explanation:This measure tracks the number of perinatal (intrapartum and newborn pre-discharge) deaths occurring from delivery to the time of discharge (ranging from a few hours to a week or more after birth) among newborns who weigh 2500+ grams at birth at RBH, divided by the total number of live born 2500+ gram newborns, and multiplied by 1,000.

Measure: The percentage of nurse midwives at Rabia Balkhi Women's Hospital who meet selected competency measures on the 37 Afghanistan Standards of Practice.

Explanation:Competency measures are indicators of the training results.

Measure: The percentage of trainees enrolled in courses.

Explanation:Hospital staff, including physicians, nurses, and midwives will take part in a variety of training activities to improve the quality of care provided to their patients.

Is the program purpose clear?

Explanation: The purpose of the Afghanistan Health Initiative is " to improve maternal and child health and reduce maternal and child mortality" in Afghanistan, as authorized by the Afghanistan Freedom Support Act of 2002. The program primarily focuses on providing continuing medical education and refresher training in maternal and infant healthcare for physicians, residents, midwives and other health professionals at Rabia Balkhi Women's Hospital (RBH) in Kabul, Afghanistan. It also provides RBH administrators with leadership and management training and has a patient education component as well. An Afghan Family Health Book, which is an electronic interactive health education tool using sound and pictures, was developed and distributed to women and families across Afghanistan to improve knowledge of disease prevention and early childhood development.

Evidence: (1) Afghanistan Freedom Support Act of 2002 (P.L. 107-327 Sec. 103(a) (4) (H)). (2) FR 04-19462 and FR 05-17596. (3) Afghan Ministry of Public Health, Progressive Report on the Distribution of Afghan Family Health Book (July 2006). (4) International Medical Corp, Afghan Family Health Book Final Report (September 2005). (5) FR E6-15541.

Does the program address a specific and existing problem, interest, or need?

Explanation: Afghanistan has one of the highest maternal mortality rates (MMR) in the world, with a rate of 1,600 maternal deaths per 100,000 live births. In Kabul, the MMR is considerably less at 400 maternal deaths per 100,000 live births, but is still significant considering the MMR in the United States is estimated at 18 maternal deaths per 100,000 live births. Preventable complications related to childbirth cause more than 85 percent of deaths among women of childbearing age in Afghanistan, and an estimated one in five infants dies before reaching his or her first birthday. These indicators are high in comparison to other developing countries and remain the highest in Asia.

Evidence: (1) World Health Organization (WHO), Reproductive Health Indicator Database available at: http://www.who.int/reproductive indicators/countrydata.asp.; (2) WHO Eastern Mediterranean Regional Office (EMRO), Division of Health System and Services Development (DHS), Health Policy and Planning Unit, Health Systems Profile Country: Afghanistan (May 14, 2005); (3) Islamic Republic of Afghanistan, Ministry of Public Health, A Basic Package of Health Services for Afghanistan (2005/1384). (4) Dott, Orakail, Ebadi, Hernandez, MacFarlane, Riley, Prepas and McCarthy, Implementing a Facility Based Maternal and Perinatal Health Care Surveillance System in Afghanistan - 2003, Journal of Midwifery & Women's Health, Vol. 50, No.4 (July/August 2005) (5) Islamic Government of Afghanistan, Ministry of Public Health National Policy on Reproductive Health (March 2006).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Over the course of three separate missions to Afghanistan, HHS, in consultation with U.S. Government (USG) partners, such as the U.S. Agency for International Development (USAID) and the U.S. Department of Defense (DOD), as well as international partners such as UNICEF and the Afghan Ministry of Public Health (MOPH), developed this program in a manner complementary to other ongoing programs, so as not to duplicate investments or provide excess resources to a community that is already receiving benefits. As a result of these consultations, HHS' program focuses on maternal and infant health needs and targets specialty physicians such as obstetricians and gynecologists. These efforts contribute to the future development of a network of highly trained physicians who work in referral hospitals in Kabul and surrounding urban centers. This complements other existing programs supported by USAID and others that focus on rural areas and on community-level health clinics. To ensure HHS' efforts were efficient, in 2005 the Office of Global Health Affairs (HHS/OGHA) contracted out an independent evaluation of all active HHS Interagency Agreements and Cooperative Agreements to identify potential redundant or duplicative tasks or funding commitments in Afghanistan. These findings resulted in the discontinuation of one Agreement and a reallocation of that funding stream.

Evidence: For assessments of Maternal Child Health (MCH) needs and services at RBH see (1) Dott, Hernandez, McCarthy, Assessment of Obstetrical Services, Rabia Balkhi Hospital, Kabul Afghanistan (2004) and (2) Hernandez, Afghanistan Reproductive Health Initiative Summary Report (July 1, 2005). For assessments of USAID activities in Afghanistan see (1) Afghanistan Reconstruction: Despite Some Progress, Deteriorating Security and Other Obstacles Continue to Threaten Achievement of U.S. Goals, GAO 05-742 (July 2005) and (2) Rural Expansion of Afghanistan's Community-based Healthcare (REACH) Program Semi-Annual Report (December 2004 - May 2005). For assessments of HHS activities in Afghanistan see (1) DB Consulting Group, Department of Health and Human Services Afghanistan Mid-project Outcome Evaluation, Part II, Technical Assistance and Support to Afghanistan Rabia Balkhi Women's Hospital 2002-2006 (May 31, 2006).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Before initiating the program, HHS conducted three assessment missions (December 2002, March 2003, and September 2003), and contracted with an experienced program evaluator in February 2003 to assess the needs at RBH and to design a program that met those needs. These assessments included an evaluation of medical staff capacity, operations, quality of healthcare delivery at RBH, equipment and supplies needs, and staff training programs interest surveys. Similar assessments were conducted in 2005 and 2006 as well. These assessments governed the design and direction of the Afghanistan Health Initiative. In the Fall of 2003, HHS/OGHA formed an Afghanistan Advisory Committee composed of senior officials from USAID, DOD, the U.S. Department of Veterans Affairs and various HHS agencies (the CDC, the Indian Health Service [IHS], the HRSA, and the National Institutes of Health [NIH]) to ensure the Afghanistan Health Initiative stays aligned with the original mission. This advisory committee met periodically over the course of several months to provide feedback on the design of the program and its cooperative agreements and eventually evolved into the Afghanistan Interagency Group. Presently, the Afghanistan Interagency Group meets quarterly to assess the program's performance and direction.

Evidence: (1) Report of Fact Finding Trip to Kabul, Afghanistan for DHHS by Peter van Dyck, Associate Administrator for MCH/HRSA (Dec. 6-16, 2002). (2) Report on Fact Finding trip to Kabul, Afghanistan for DHHS by Peter van Dyck, Associate Administrator for MCH/HRSA (March 3-12, 2003). (3) Survey and Evaluation of Rabia-e-Balkhi Hospital Kabul, Afghanistan by M. Linda Brown (March/April 2003) (4) Meeting Agendas for Afghanistan Interagency Group (January 5 and April 17, 2007). (5) Center for International Health (CIH) Initial Assessment Report (February 13-27, 2005). (6) CIH Second Trip Report (June 22-July 9, 2005). (7) Tulane University Resident Assessment Rabia Balkhi Hospital (September 2006).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: As stated in the report from the December 2002 assessment mission, HHS and the Afghan Ministry of Public Health (MOPH) chose RBH because it is the largest women's hospital in Kabul and in great need of improvement. After enduring more than two decades of civil war, the staff had not been retrained in over 20 years, equipment and supplies were scarce, and the physical infrastructure was in disrepair. Nonetheless, over 15,000 babies were delivered at RBH in 2004 and to this day, the hospital continues to have a large patient base. In designing the program, HHS determined it needed partners that were experienced in the delivery of healthcare services and training of healthcare workers in the difficult environment of Afghanistan. In 2004, HHS/OGHA solicited applications from groups experienced in working in Afghanistan and skilled at managing accredited ob/gyn residency training programs with the goal of ultimately establishing an ob/gyn residency training program at RBH. In 2005, as it became increasingly clear that ongoing significant investments in Kabul hospitals were unsustainable without improved hospital management, HHS/OGHA solicited applications from groups to provide effective leadership and management training to RBH hospital administrators. Both International Medical Corps (IMC), the recipient of the Cooperative Agreement in 2004, and CURE International, the recipient of the Cooperative Agreement in 2005, have extensive experience working in Afghanistan providing comprehensive healthcare services and training programs to high-risk populations in resource-poor settings.

Evidence: (1) Report on Fact Finding trip to Kabul, Afghanistan for DHHS by Peter van Dyck, Associate Administrator for MCH/HRSA (Dec. 6-16, 2002). (2) FR 04-19462. (3) Applications submitted by IMC in response to FR 04-19462. (4) FR 05-17596. (5) CURE International's Application to the US Department of Health and Human Services - Management Improvement Rabia Balkhi Women's Hospital in 2005 in response to FR 05-17596.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has a limited number of long-term performance measures, which are focused on outcomes and reflective of the purpose of the program. The purpose of the program is to improve the quality of maternal and infant health care at RBH by improving the skills and training of the hospital staff. The desired long-term outcomes are a reduction in maternal mortality, predischarge neonatal mortality, intrapartum mortality, and post-operative infection rate. These four measures represent basic indicators of the quality of patient care at the facility and are included in the HHS/OGHA FY 2007 and FY 2008 budget submissions.

Evidence: The long-term measures are to achieve the following targets from 2004 (baseline) to 2012: (1) to decrease the maternal mortality rate from 189 per 100,000 live births to 95 per 100,000 live births, (2) to decrease the predischarge neonatal mortality rate from 2.7 per 1,000 live births to 1.5 per 1,000 live births, (3) to decrease the intrapartum mortality rate from 7 per 1,000 live births to 4.8 per 1,000 live births, and (4) to decrease the post-operative infection rate from 3.7 to 1.8.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Afghanistan Health Initiative has quantifiable baselines and targets that are especially ambitious due to the current conditions in Afghanistan. Afghanistan has been ravaged by more than twenty years of civil war and strife. As a result, a large number of trained professionals fled the country over the past two decades. Those who remained did not have the benefit of any type of continuing medical education. Furthermore, during the years of the Taliban, hospitals in particular suffered from a lack of equipment, supplies and decent hospital administration. Women and girls had very limited access to services, and most female health-care workers were not allowed to work. Presently, conditions in Afghanistan remain such that it is difficult to find skilled personnel to live and train in Afghanistan, due to concerns of personal safety and political instability. Insufficient resources, inadequate infrastructure, and a poorly functioning healthcare system continue to be major constraints to improving the quality of healthcare service and delivery throughout Afghanistan. Therefore, given the numerous obstacles in the current working environment the achievement of the desired long term-outcomes through the training of hospital staff is ambitious.

Evidence: In the REACH Semi-Annual Report, (December 2004 - May 2005), USAID lists civil unrest, attacks and threats of attack, high staff turn over and delays in furniture, equipment and operating budgets as constraints on its programs. In this report, USAID also cited human resources development as a major long-term challenge in Afghanistan. In its 2005 Health Systems Profile of Afghanistan, World Health Organization (WHO) Eastern Mediterranean Regional Office (EMRO) described the overall situation in Afghanistan as "grim" and reported that "most of the Afghan population does not have access to the basic services that could make a large difference to their health". Furthermore, WHO declared, "Afghanistan is today where most developing countries were 40 years ago".

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: HHS/OGHA has implemented a limited number of annual goals to measure the performance of the training programs for physicians and other staff at RBH. Due mainly to security concerns, it is very difficult to recruit and retain qualified staff trainers, especially female ob/gyn practitioners with clinical training expertise. Therefore, two of the annual measures are to reduce the time to recruit and deploy essential staff trainers and to increase the staff trainer retention rate. The purpose of the training program itself is to improve the quality of maternal and infant healthcare offered at RBH by providing continuing education and refresher training to physicians, residents, midwives and other health professionals to equip them with specific skills in how to maintain infection control, manage difficult obstetrical cases, conduct neonatal assessment and triage, and conduct continuous quality improvement. The remaining two annual goals relate to the number of trainees enrolled and the improved competencies of the trainees at the conclusion of the training. These annual goals are linked to the long-term goals of reducing maternal mortality, predischarge neonatal mortality, intrapartum mortality, and post-operative infection rate, and they are included in the HHS/OGHA FY 2007 and FY 2008 budget submissions.

Evidence: The annual measures are to achieve the following targets from 2005 (baseline) to 2012 unless specified: (1) to increase the percent of nurse midwives that successfully complete competency assessments from 40 percent to 97 percent; (2) to increase the percent of trainees enrolled in courses from 50 percent to 98 percent; (3) to reduce the time needed to hire and deploy essential staff trainers from 3 months to 1 month; (4) to increase the percent of staff trainers who fulfill the agreed upon in-country contract time from 80 percent to 96 percent.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Afghanistan Health Initiative has quantifiable baselines and targets that are especially ambitious due to the current conditions in Afghanistan. Afghanistan has been ravaged by more than twenty years of civil war and strife. As a result, a large number of trained professionals have fled the country over the past two decades. Those who remained did not have the benefit of any type of continuing medical education. Furthermore, during the years of the Taliban, hospitals in particular suffered from a lack of equipment, supplies and decent hospital administration. Additionally, conditions in Afghanistan are such that it is difficult to find skilled personnel to live and train in Afghanistan, due to concerns of personal safety and political instability.

Evidence: In a series of assessments in 2005, the Center for International Health (CIH) noted that RBH operated in an environment that lacked adequate accountability, discipline, and organizational collaboration. CIH also noted a patient care delivery model that did not promote collaboration or teamwork. An assessment by Lieutenant Ruiz-Beltran in 2006 identified an ineffective administration system for equipment and supplies, poor storage conditions and inadequate inventories as primary areas of concern. Therefore, the achievement of the desired annual goals is ambitious given the numerous obstacles in the current working environment and the progressive targets that are projected through 2012.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: HHS worked both externally, internally, and with grantees, contractors, etc., to ensure that all parties are working toward the same goals and that HHS' goals were in line with U.S. foreign policies. This policy has been laid out in numerous planning documents, and the Government has asked that all partners work together in accomplishing these common long-term goals. Internally, HHS/OGHA has worked with each grantee to develop a set of annual performance goals that match the long-term goals of HHS. Currently, both HHS/OGHA grantees operate under Letters of Understanding with HHS, which defined the key commitments each grantee is making to RBH to improve the quality of services at RBH, and the means by which they will coordinate activities and share relevant information.

Evidence: For example, in the "National Health Policy: Policy Statement Document Final Draft," by MOPH (undated), Afghanistan reproductive health is listed as one of the eight health sector priorities. Within this section it states that the MOPH will work to ensure access to reproductive health services to reduce maternal and infant deaths. One of the ways in which this will be achieved is through the training of midwives. Other evidence includes the Letter of Understanding (LOU) between CURE International and International Medical Corps (IMC) for the benefit of Rabia Balkhi Hospital (November 2006) which delineates grantee responsibilities, sets a joint performance goal (to improve the quality of care at RBH so that maternal and neonatal mortality rates decrease) and mandates increased coordination of activities and information sharing to include joint weekly meetings and joint monthly performance reports. The Memorandum of Understanding (MOU) between the Ministry of Public Health (MOPH) and CURE International (May 9, 2006) designates CURE as the party responsible for establishing a high quality system of hospital leadership and facilities management at RBH, which will be sustainable beyond the scope of this project.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: HHS/OGHA is planning on enlisting the expertise and technical assistance of evaluation professionals within HHS to assist in the planning and design of methodologies, which will evaluate the impact of the training programs at RBH. The evaluation will be contracted out, either through the Assistant Secretary for Planning and Evaluation (ASPE) or directly through OGHA. A preliminary statement of work has been drafted and a 2008 implementation of the evaulation is planned. A July site-visit is scheduled as a precursor to the more comprehensive evaluation. Attendees will include the Afghanistan Health Initiative projoect officer and a team of experts from HHS Operating Divisions outside of OGHA. Previously, in 2005, HHS/OGHA contracted with IMC to expand its scope of work beyond the provision of training at RBH to include the implementation of a series of comprehensive evaluations on the effectiveness of the Afghan Family Health Book in rural communities throughout Afghanistan. The final evaluation report completed in September 2005 concluded that overall, health knowledge improved among Afghans who used the Afghan Family Health Book with the most significant improvements being demonstrated among women on nearly all health topics. HHS and the Afghan MOPH are currently planning to distribute more books in regions not previously included. An evaluative component is also being incorporated into this plan.

Evidence: (1) OGHA/ASPE Evaluation Plan Draft Statement of Work (July 2007) (2) RBH Site Visit Agenda & Attendees (July 2007). (3) IMC Afghan Family Health Book Deliverables "Plan for Follow-up Survey" (April 11, 2005). (4) IMC Afghan Family Health Book Deliverables "Recruitment and Training of Surveyors, Survey Supervisor and Community Health" (April 5, 2005). (5) Initial survey results of Afghan Family Health Book from several districts. (6) IMC Afghan Family Health Book Interim Report (June 30, 2005). (7) IMC Afghan Family Health Book Final Report (September 16, 2005).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Afghanistan Health Initiative has not conducted a marginal cost analysis and as such cannot demonstrate what outcomes an increase in funding would deliver. HHS/OGHA's FY 2008 Congressional Justification provides a detailed program description of the Afghanistan Health Initiative which describes the activities to date and identifies the performance measures and goals for upcoming years. HHS/OGHA's FY 2007 and FY 2008 budget requests include program performance goals and targets along with a detailed program description to align program performance with budget requests.

Evidence: (1) FY 2007 Congressional Budget Justification, (2) FY 2008 Congressional Budget Justification

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: HHS/OGHA, in response to recommendations from the Afghanistan Advisory Committee, has made numerous adjustments to the program's strategic planning. These include more emphasis on administrative needs at the hospital to sustain a long-term training program. For example, the Committee members reviewed early reports from the field that the hospital's administrative deficiencies were severe enough to put the entire training program in jeopardy. This necessitated a reconsideration of the program objectives and the inclusion of hospital administration training to ensure that trainees had access to drugs, equipment and other supplies, as well as had the benefit of a proper laboratory and pharmacy at the hospital. In 2005, in response to this need, HHS/OGHA awarded a Cooperative Agreement to CURE International to provide leadership and management training to RBH staff and developed annual measures to assess the performance of the program. Furthermore, in 2006, HHS/OGHA awarded a Cooperative Agreement to the MOPH to address the critical need for prenatal care.

Evidence: Both Federal Register Announcements (FR 05-17596 and FR E6-15541) evidence the programmatic expansions in response to identified needs such as inadequate administrative practices among RBH leadership and management (i.e., RFA to CURE) and a lack of adequate prenatal care and a nonfunctional referral system (i.e., RFA to MOPH). The IMC Quarterly Reports (October 19,2004 & September 14, 2005) identify those needs while the Islamic Government of Afghanistan, Ministry of Public Health National Policy on Reproductive Health (March 2006) not only identifies those needs as well, but also establishes them as priorities and sets the protocol for addressing the needs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Both IMC and CURE provide monthly, quarterly and annual performance reports on their activities. Monthly reports are a collaborative effort while quarterly and annual reports focus on activities specific to each grant. For example, IMC reports provide information on the types of training held and the number and type of participants; whereas, CURE reports provide information on the status of ongoing management and leadership activities to include training conducted, operational changes implemented; supplies procured; and administrative policies, systems and processes developed. Furthermore, both grantees regularly report on challenges encountered. In addition, HHS/OGHA routinely makes quarterly site visits to the project each year, which provide additional information for use in managing the program. These reports, along with weekly and monthly calls, quarterly site visits and ad hoc reporting, enable all program partners to discuss improvements and areas of concern. As a result, all parties have made adjustments according to needs on the ground.

Evidence: In its September 14, 2005 Quarterly Report, IMC noted an overall lack of accountability and a general reluctance to assume responsibility for patient care among the RBH staff, which was further exacerbated by low staff wages that promoted the prioritization of private practice and discouraged fulfilling work commitments at RBH. By the end of 2006, with the assistance of CURE International, a higher paying wage system was introduced at RBH, personnel policies to govern attendance, over-time, leave and absenteeism were approved and staffing policies to establish 24-hour coverage and shift scheduling were implemented. Since this policy change went into effect, there have been reports of a gradual increase in the number of trainees attending classes and a decrease in the rate of absenteeism among RBH staff members.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: OGHA is unable to demonstrate how individual Federal managers are held responsible for program performance through individual performance plans. Beginning in 2004 with the awarding of the first Cooperative Agreement and continuing through 2005, HHS/OGHA staff members have made biannual site visits to RBH. In 2006, the frequency increased to quarterly visits. Furthermore, numerous telephone consultations and in-person meetings have transpired to ensure that both IMC's and CURE's activities are in line with agreed plans and expectations. Based on recommendations informed by the site visits and feedback from IMC and CURE and consultations with U.S.-based experts, HHS/OGHA develops action plans to address concerns. In addition, the program manager ensures HHS/OGHA expends funds according to approved budgets, and that steps are in place to provide continuation funding based on acceptable performance.

Evidence: (1) See grantee progress reports on the RBH Support Project, RBH Project, Afghan Family Health Book Project and related articles, conferences and presentations: http://www.globalhealth.gov/Afghan_index.shtml. (2) Key Decisions and Action Items from Afghanistan Interagency Work Group Meeting - June 30, 2005. (3) Rabia Balkhi Hospital - Meeting Minutes (November 4, 2006). (4) Notes from OGHA Meeting with EBH Partners (July 10, 2006). (5) Notes from Interagency Health Work Group - Afghanistan (January 22, 2007)

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: To track actual expenditures, the program used the CORE accounting system until FY 2007 when the UFMS accounting system was implemented. Actual spending for FY 2006 matched the appropriated funds closely. For example, Congress appropriated $5,888,000 to the Afghanistan Health Initiative and UFMS records indicate that 99 percent of the funds were obligated accordingly. Further financial oversight is practiced through the compilation of monthly status of funds reports, which are compiled using UFMS and supplied to the Program Director for review and approval. The HHS/OGHA Operations Office also keeps track of funds transferred to IMC and CURE. In addition, OGHA works daily with the Program Support Center (accounting office for the Office of the Secretary) to reconcile all expenditure accounts, when planned obligations differ significantly from reported actuals.

Evidence: (1) FY 2007 Congressional Budget Justification. (2) FY 07 Undelivered Orders Report, UFMS Accounting System (April 6, 2007). (3) PMS Disbursement Transactions Report (December 6, 2006). (4) UFMS FY 2006 Obligations Report.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The efficiency measures OGHA has in place for the Afghanistan Initiative does not relate directly to the cost of the program and fails to demonstrate cost effectiveness. HHS/OGHA has established an efficiency measure for the Afghanistan Health Initiative, time to hire and deploy essential staff trainers, which has quantifiable baselines and targets. HHS/OGHA uses the services of the HHS Office of Public Health and Science (OPHS) and the HHS Program Support Center to provide grants and contracts management expertise and additional financial oversight thus eliminating the need to have the necessary expertise within OGHA. HHS/OGHA also uses contract staff to provide technical oversight and logistical support to supplement the roles of the Federal program managers.

Evidence: Current efficiency measures: (1) The time to hire and deploy essential staff trainers. (2) The percent of staff trainers who fulfill the agreed upon in-country contract.

Does the program collaborate and coordinate effectively with related programs?

Explanation: HHS, IMC and CURE work with multiple partners both in Afghanistan and in the United States, to offer both programmatic advice and technical assistance. Since the Afghan MOPH owns the hospital facility and all hospital staff are employees of the MOPH, all parties must keep the MOPH informed of its progress and ensure coordination with the MOPH. Senior HHS officials have regular contact with international agencies, Afghan MOPH staff and other USG partners to ensure there is no duplication of investments and to assist with setting program direction. Lastly, HHS/OGHA staff and leadership meet on an ad hoc basis with other partners to ensure that the overall direction of the program is appropriate.

Evidence: (1) Rabia Balkhi Hospital - Meeting Minutes (November 4, 2006). (2) Notes from OGHA Meeting with EBH Partners (July 10, 2006). (3) Meeting Agendas for Afghanistan Interagency Group (January 5 and April 17, 2007).

Does the program use strong financial management practices?

Explanation: For the eighth year in a row, independent auditors have issued the Department an unqualified or "clean" opinion in HHS' FY 2006 Performance and Accountability Report. HHS/OGHA uses the UFMS accounting system, newly implemented in FY 2007, to record all expenditures and assist in the proper management and the exercise of stewardship of the program funds. The HHS/OGHA Operations Office prepares bi-weekly status of funds reports and reconciles these with the UFMS system to capture all expenditures properly. Monthly status of funds reports are then complied and supplied to the Program Director for review and approval. The HHS/OGHA Operations Officer must approve all expenditures, and the HHS/OGHA Deputy Director for Operations signs off on any expenditure over $5,000. Also, HHS/OGHA Operations and the Program Officer monitor grantee performance and expenditures through quarterly and annual financial status reports submitted by the grantee as well as reports retrieved from the HHS Division of Payment Management (DPM) Payment Management System (PMS).

Evidence: OGHA does the majority of its fiscal monitoring of grantees via submitted financial reports, retrieved reports from the UFMS or PMS system or informal communications with its grantees. (1) FY 07 Undelivered Orders Report, UFMS Accounting System. (2) PMS Disbursement Transactions Report (December 6, 2006); (3) HHS, Office of Finance, FY 2006 Performance and Accountability Report accessible at: http://www.hhs.gov/of/reports/account/; (4) CURE Annual Progress Status Report/Financial Status Report Year 1 (December 11, 2006); (5) Quarterly Financial Status Report for IMC (January - March 2007).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: HHS/OGHA has regular opportunities to meet or speak with partner agencies to get feedback and discuss future directions. During these discussions, HHS/OGHA makes decisions to address identified concerns or deficiencies. One identified management concern was the difficulty both IMC and CURE were having in recruiting and retaining fully qualified staff to live and work in Afghanistan, as called for in the Cooperative Agreements. Political instability, general insecurity and a lack of available housing in Kabul has been a barrier to attracting good candidates.

Evidence: To address the recruitment challenges, HHS granted IMC and CURE verbal approval to expand the pool of recruits to beyond the United States. In addition, HHS/OGHA has assisted in directing qualified applicants to both grantees. As a result, both grantees reported improved recruitment capabilities overall, however, the recruitment of fully qualified and Board certified female obstetricians and gynecologists continues to be challenging.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: HHS/OGHA advertised the Request for Applications for the Cooperative Agreements in the Federal Register, and encouraged several potential grantees to send in applications. Criteria for assessing applications were included in the Request for Applications. A multi-agency panel reviewed the proposals based on specified objective review criteria and developed a project summary highlighting the strengths and weakness of the application. Three-year Cooperative Agreements were awarded to both IMC and CURE with funding for the second and third year of each contingent upon the availability of funds. In the second and third year, both grantees must submit non-competing continuation applications to receive additional funding. The Cooperative Agreements will be open for competition again after the third year.

Evidence: (1) FR 04-19462. (2) Guidance for Objective Review of the Applications for the Training Program for Rabia Balkhi Women's Hospital (September 2004). (3) Objective Review Project Summary. (4) FR 05-17596

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: In the early critical stages of this program, HHS maintained a daily presence at the hospital in 2004 and 2005 to supplement and validate the work of the grantee. Through 2006, a CDC representative provided oversight and feedback on IMC and CURE operations to HHS officials as well. In addition, HHS had a representative in Kabul who regularly visited the hospital and attended meetings of the principal staff. HHS/OGHA staff currently conducts quarterly site visits of at least two-week duration to see first-hand the conditions and to provide technical guidance, as warranted. Furthermore, the Federal program manager within HHS/OGHA consults weekly with the representatives of IMC and CURE in the United States to go over major programmatic and budgetary issues. Finally, OGHA/HHS monitors grantee performance and expenditures through quarterly and annual progress and financial status reports submitted by the grantees as well as reports retrieved from PMS.

Evidence: (1) Grantee progress reports on the RBH Support Project, RBH Project, Afghan Family Health Book Project and related articles, conferences and presentations are posted on the Afghanistan webpage on the OGHA website available at: http://www.globalhealth.gov/Afghan_index.shtml (2) Rabia Balkhi Hospital - Meeting Minutes (November 4, 2006). (3) CURE and IMC Combined Monthly Report (January 2007). (4) PMS Disbursement Transactions Report (December 6, 2006). (5)IMC Rabia Balkhi Training Project Annual Report (2006).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: HHS/OGHA collects grantee performance data on a monthly, quarterly and annual basis, and makes this information accessible to the public through press releases highlighting major developments in HHS/OGHA activities in Afghanistan and through links on the Afghanistan webpage of the OGHA website. Additionally, HHS/CDC staff, working with partners in Afghanistan, publishes articles in scientific journals highlighting progress made in implementing programs in Afghanistan.

Evidence: (1) Grantee progress reports on the RBH Support Project, RBH Project, Afghan Family Health Book Project and related articles, conferences and presentations are posted on the Afghanistan webpage on the OGHA website available at: http://www.globalhealth.gov/Afghan_index.shtml; (2) Kitt, Khalid, Rahimi, McCarthy, An Occupational Health Services Initiative at a Women's Hospital in Kabul, Afghanistan (Public Health Reports, November-December 2006).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Afghanistan Health Initiative has demonstrated some progress toward long-term performance goals. Preliminary data suggests there was a significant change at RBH in two of the four long-term measures. Between 2004 and 2006, RBH experienced a drop in both the maternal mortality rate and the predischarge neonatal rate. The other two long-term measures have not yet demonstrated a statistically significant positive trend. The post-operative infection rate and the intrapartum neonatal mortality rate have showed varied results since 2004. The program suggests elements outside of RBH are contributing for the mixed results surrounding long-term performance, such as poor access to prenatal care.

Evidence: The maternal mortality rate at RBH decreased from 189 in 2004 to 136.5 in 2006. In addition, the data indicate that during this same period there was a decrease in predischarge neonatal mortality rate for infants weighing at least 2500 grams from 2.7 per 1,000 live births in 2004 to 2.54 per 1,000 live births in 2006. The post-operative infection rate increased from 3.7% in 2004 to 6.3% in 2006, and the intrapartum mortality rate for infants weighing at least 2500 grams increased from 7 per 1,000 live births in 2004 to 8.7 per 1,000 live births in 2006.

Does the program (including program partners) achieve its annual performance goals?

Explanation: While the Afghanistan Health Initiative is making progress in achieving annual outcome goals, there is minimal if any progress being made toward the annual efficiency goals. In 2006, HHS/OGHA exceeded its annual performance goal for the percent of nurse midwifery's meeting competency measures on the 37 Afghanistan Standards of Practice. The program has also demonstrated achievements toward reaching its other annual goal, the percent of trainees enrolled in courses. Preliminary data for the annual efficiency measures, "time it takes to hire/deploy staff trainers," has not yet demonstrated a positive trend and "the percentage of staff trainers who fulfill the agreed upon in-country contract," has fallen short of annual targets. One the efficiency measures begin to show improvement the rating will increase.

Evidence: The annual measure, to increase the percent of nurse midwives that successfully complete competency assessments had a 2006 target of 50% and a 2005 baseline of 40%. In CY 2006, the goal achieved for this measure was 75% which exceeds the 2006 target by 25% and shows an increase of 35% from the 2005 baseline. In 2006, the percent of trainees enrolled in courses reached 70%, nearing the 2006 target of 75% and showing an increase of 20% from the 2005 baseline of 50%. In 2006, the percent of staff trainers who fulfill the agreed upon in-country contract achieved 85%, falling short of the 89% target. The time it took to hire and deploy essential staff trainers took 4.2 months in 2006, 1.7 months longer than the target and 1.2 months longer than the baseline.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The established efficiency measures have not showed significant progress towards achieving the program goals nor has a measure been established to gauge cost effectiveness. In 2005, HHS/OGHA adopted two efficiency measures, the time to hire and deploy essential staff trainers and the percent of staff trainers who fulfill the agreed upon in-country contract. Conditions in Afghanistan however, have not contributed to the attainment of the hire/deploy goal in 2006 as it continues to be difficult to find skilled personnel to live and train in Afghanistan, due to concerns of personal safety and political instability. Likewise, the percent of trainers who fulfill the agreed upon in-country contract has improved only modestly and is falling short of the established targets.

Evidence: In 2005, the efficiency measure, to reduce the time needed to hire and deploy essential staff trainers achieved a baseline of 3 months. In 2006, however the time increased slightly to 4.2 months. Therefore, the target for 2007 remains at 3 months. In 2006, the percent of staff trainers who fulfill the agreed upon in-country contract achieved 85%, missing the target of 89%.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The comparisons of the Afghanistan Health Initiative to other programs have ignored the core component - training of medical staff. As part of the Afghan Government's efforts to improve maternal and infant health care in Afghanistan, the MOPH has commissioned several assessments of hospital conditions over the past four years. These assessments have included evaluations of RBH, as well as other maternity hospitals, and give HHS opportunities to assess independently how RBH is doing compared to other hospitals and programs. However, training of medical staff was not assessed. These comparisons focus on widely-used standards of quality such as infection prevention, hospital administration and management, and clinical management. Past assessments indicated that conditions at RBH were comparable to hospital conditions at Malalai Maternity Hospital in terms of patient population and better than conditions at most hospitals outside of Kabul. These assessments will continue to be made on an adhoc basis by the MOPH and shared with HHS allowing HHS to continue to assess improvement over time.

Evidence: (1) "Hospital Management Improvement Initiative of REACH in 4 Provincial Hospitals", USAID/Kabul, slide presentation. (2) "Improving Maternal Health Services in Afghanistan: Standards-Based Management", USAID/Kabul, slide presentation, June 10, 2005. (3) Need Assessment Results of Kabul Province Maternal Health Services, RH Project of MOPH/JICA, slide presentation (2005). (4) Report on Visits to Five Ministry of Public Health Community Health Clinics (CHCs) in Kabul City (March 15, 2005)

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: An independent evaluation is in the planning stages, but unlike Question 2.6, credit cannot be given for future evaluations.

Evidence: HHS/OGHA is in the initial process of planning a comprehensive methodology for conducting an independent evaluation which will be of sufficient quality and rigor to provide information on the effectiveness of the Afghanistan Health Initiative. Once designed, this evaluation will assess the impact of the training programs on the quality of health-care provided at RBH.