Detailed Information on the
Office on Women's Health Assessment

The Office on Women's Health (OWH) is implementing a Performance Management System (PERMS), which is a web-based data collection system that OWH contractors and grantees will use to electronically submit progress reports to a centralized database. Training will be provided to all OWH staff, contractors and grantees.

The Office on Women's Health is developing and implementing a comprehensive OWH Strategic Plan for 2010-2015.

The Office on Women's Health will convene a strategic planning and budget meeting to identify program opportunities, challenges and priorities for FY 2010.

The Office on Women's Health is developing a biannual budget review/justifiction and program performance review to evaluate program progress, effectiveness challenges and opportunities.

The Office on Women's Health undertook a strategic planning process to define its two major goals: Develop and evaluate Model Programs on Women's Health, and Lead Education/Collaboration efforts to improve Women's Health. OWH seeks to identify gaps and influence changes in healthcare for women and girls.

The Office on Women's Health developed a set of long-term and annual performance measures that link to OWH's mission and priority focus areas. OMB approved OWH's efficiency measure August 3, 2006.

The Office on Women's Health developed baselines and targets for all of its revised performance measures.

Obtain OMB approval for the revised Office on Women's Health long-term and annual measures.

The Office on Women's Health focused program resources on initiatives that target the women's health priority focus areas - cardiovascular disease, diabetes, HIV, mental health, and lupus.

Measure: Annual number of visitors to OWH websites (www.4woman.gov; www.womenshealth.gov; www.girlshealth.gov).

Explanation:In the 2005 PART Assessment, OWH was noted as having a strong health information dissemination role, most notably through its National Women's Health Information Center, an award-winning website and gateway to customized women's health information. The womenshealth.gov website provides original health information on special topics like pregnancy, breastfeeding, body image, HIV/AIDS, girls health, heart health, menopause and hormone therapy, mental health, and violence against women. The girlshealth.gov website, targeting adolescent girls ages 9-17, provides health information on issues such as nutrition, fitness, body image, drugs, alcohol and smoking, safety and bullying. OWH has two major program goals, and educating and influencing health organizations, health care professionals, and the public (women) is one of the two program goals.

Measure: Increase the percentage of women who are aware of the eight early warning symptoms and signs of a heart attack and the importance of accessing rapid emergency care by calling 911.

Explanation:The eight early warning signs of a heart attack are: chest pain; neck pain; shoulder pain; arm pain; chest tightness; shortness of breath; nausea; and fatigue. Heart disease is the leading cause of death for all people in the United States. Disparities exist in treatment outcomes following a heart attack: Females, in general, have poorer outcomes following a heart attack than do males: 44% of females die within a year, compared with 27% of males. At older ages, females who have a heart attack are twice as likely as males to die within a few weeks. OWH funds a series of community outreach interventions in each of the ten US Public Health Service Regions on an annual basis. OWH also periodically funds interventions with faith-based organizations and Women's Health Centers to increase awareness of heart attack symptoms. The data source for this measure is the 2006 National Study of Women, funded by the American Heart Association (which reflects 2005 data). This data is collected every three years.

Measure: Increase the percentage of women-specific Healthy People 2010 objectives and sub-objectives that have met their target or are moving in the right direction.

Explanation:The 311 health outcome measures of Healthy People 2010 that focus on women's health are encompassed in this single measure. The data source for this measure is DATA2010, which is a compilation of data from multiple sources. The baseline measurement at 2005 is based on the Midcourse Review, an assessment the Office of Disease Prevention and Health Promotion (ODPHP) made of the progress in meeting these objectives, in collaboration with the lead agencies and CDC's National Center for Health Statistics. The 2010 target for the final review is based on historical experience, i.e., the percent of total HP 2000 objectives that had met the target or were moving in the right direction. OWH has a representative on each HP2010 focus area working group, to influence discussion around objectives and progress. The Deputy Assistant Secretary for Women's Health participates actively in all HP2010 progress reviews to ensure high-level discussion of gender issues. As a result, every second-round progress review (conducted since 2006) has included gender analysis and detailed examination of causes for differences with program staff and agency and OPHS leadership.

Measure: Increase the percentage of women-specific Healthy People 2020 objectives and sub-objectives that have met their target or are moving in the right direction.

Explanation:The hundreds of health outcome measures of Healthy People 2020 that will focus on women's health are encompassed in this single measure. The data source for this measure is DATA2010, which is a compilation of data from multiple sources. The baseline measurement at 2005 is based on the Midcourse Review, an assessment the Office of Disease Prevention and Health Promotion (ODPHP) made of the progress in meeting these objectives, in collaboration with the lead agencies and CDC's National Center for Health Statistics. The 2010 target for the final review is based on historical experience, i.e., the percent of total HP 2000 objectives that had met the target or were moving in the right direction. OWH has a representative on each HP2010 focus area working group, to influence discussion around objectives and progress. The Deputy Assistant Secretary for Women's Health participates actively in all HP2010 progress reviews to ensure high-level discussion of gender issues. As a result, every second-round progress review (conducted since 2006) has included gender analysis and detailed examination of causes for differences with program staff and agency and OPHS leadership.

Measure: Number of girls ages 9-17 and women ages 18-85+ that participate in OWH-funded programs (e.g. information sessions, website user sessions, outreach) per million dollars spent annually.

Explanation:OWH participants are defined as the number of website user sessions, participants attending information sessions, and recipients of outreach - collected through OWH's new PERMS grantee database. This measure is calculated by dividing the total number of participants by the program's total budget (future years' targets are estimated using the program's enacted 2008 funding level) which is then multiplied by 1 million. PERMS is a web-based data collection system that OWH contractors and grantees use to submit progress reports concerning the number of participants in OWH-funded programs.

Is the program purpose clear?

Explanation: The program purpose is to improve the health and well-being of women by coordinating women's health efforts, supporting health programs and disseminating health information. The program focuses on prevention of health conditions that are unique to, disproportionately affect, or have different impact on women.

Evidence: In 1991, Secretary Louis Sullivan created the Office on Women's Health (OWH) to support the Public Health Service Coordinating Committee on Women's Health Issues. (Announced in a July 24, 1991 Federal Register Notice (Vol.56. No.142)) Since its creation, the program's purpose evolved as evidenced by the current OWH mission statement and the FY 2005 Congressional Justification. The coordinating committee now supports the OWH in its work as the coordinator for women's health efforts and in its role of promoting health education and disease prevention.

Does the program address a specific and existing problem, interest or need?

Explanation: The program addresses all health conditions that affect women, but focuses on a smaller set of health issues, which tends to vary from year to year. Extensive research supports the need for intervention in the following set of health issues that the program currently focuses on. 1) Comprehensive health centers for women: Lack of fragmented health care services and gaps in services; 2) Heart disease:The first leading cause of death among American women, claiming the lives of more than 500,000 women each year or 41.3% of all female deaths; 3) Diabetes:The sixth leading cause of death listed on U.S. death certificates in 2000; 4) HIV/AIDS: Women with HIV/AIDS now account for an estimated 30% of new HIV infections. In fact, African American and Hispanic women represent less than 25% of all women in the U.S., but account for more than 78% of AIDS cases reported among women; 5) Violence againast women: Approximately 1.5 million women are raped or physically assaulted by an intimate partner each year; 6) Depression: Depressive disorders affect nearly twice as many women as men each year in the U.S.

Evidence: 1) October 1992 article in JAMA (Vo. 268. No. 14) by Carolyn Clancy M.D. and Charlea T. Massion M.D. "American Women's Health Care: A Patchwork Quilt with Gaps." 2) December 2003 AHA article, "Tracking Women's Awareness of Heart Disease: An American Heart Association National Study." 3) Various CDC publications on Diabetes. 4) October 2003 Kaiser Family Foundation publication, "Women and HIV/AIDS in the U.S." 5) February 2004 United Nations AIDS Initiative: The Global Coalition on Women and AIDS press release, "HIV Prevention and Protection Efforts are Failing Women and Girls." 6) 1999 CDC report (Vol.11, No.2) "HIV/AIDS Surveillance Report: Year-End Edition." 7) Urban Institute estimates of the March 2000 Current Population Survey, U.S. Bureau of the Census, for the Kaiser Family Foundation. 8) 2000 CDC report, "Extent, Nature, and Consequences of Intimate Partner Violence." by P. Tjaden and N. Thoennes. 9) 1993 article in the Archives of General Psychiatry by D. Regier, W. Narrow, and et al. "The De Facto Mental and Addictive Disorders Services System."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: OWH provides Department wide collaboration on women's health and ensures that women's health issues are represented and supported throughout HHS. Although the coordinating role is unique to the program, public health education and outreach activities are duplicative of other efforts. For example, national organizations such as the American Heart Association, American Cancer Society and the American Diabetes Association target women through their websites, national health campaigns, and health promotion events. State and local health departments also target women's health issues through similar means. OWH partners with sub-agencies at HHS to address key women's health issues. Despite this effort, there appears to be some overlap as evidenced by CDC's REACH 2010 program, which aims to eliminate health disparities for breast and cervical cancer, cardiovascular diseases, diabetes and HIV/AIDS. The Black Women's Health Imperative is a REACH 2010 grantee that targets cardiovascular disease in black women.

Evidence: Women's health education resources are shown on the following websites:1) www.americanheart.org (American Heart Association)2) www.cancer.org (American Cancer Society)3) www.diabetes.org (American Diabetes Association)4) www.healthywoman.org (National Women's Health Resources Center)OWH awards contracts for activities that are similar to the grants made by the Robert Wood Johnson Foundation (www.rwjf.org.) RWJF Grants Include: a) The National Council of Negro Women, Inc. for The African-American Women's Health Information Project. b) The Women's Project for Faith in Action. c) Brigham & Young Hospital Inc. for The Harold Amos Medical Faculty Development Program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program design is free of major flaws and enables the program to be effective. The program coordinates with women's health offices across HHS sub-agencies. For example, the program organized a nationwide listening session with nearly 1,000 constituents from health professionals, administrators, advocates, consumers, and state and local organizations to identify women's health needs and gaps. The program used this information to lead the coordinating committee to review the women's health needs and priorities and to develop a framework for women's health for the FY 2001 budget in a document entitled, "Women Living Long, Living Well." The program also competitively awards contracts to community and faith based organizations, that women know of and trust, to distribute public health messages.

Evidence: In the August 1992 GAO report entitled, "Women's Health Information: HHS Lacks an Overall Strategy", GAO concludes that "while HHS puts out much information and its component units do their own planning, there is no overall strategy to direct the various agency activities." The program addresses GAO concern with coordination as a major program design element.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Program resources target women to provide key information on women's health issues. Most notably, the National Women's Health Information Center (NWHIC), a national website and toll-free hotline, is a national gateway on women's health with customized sections for all women, including women of color, Spanish-speaking women and women with disabilities. NWHIC is designed to help women make informed decisions by providing reliable health information. Next, the National Centers of Excellence in Women's Health (CoE) are new models of comprehensive health care for women through academic health centers. Funding provides administrative support and guidance for developing linkages within a university and its schools, clinics, departments, and centers to provide optimal health care to women. Similarly, the National Community Centers of Excellence in Women's Health (CCOE) provides comprehensive, integrated, interdisciplinary services to underserved women by employing case managers, eligibility specialists, and patient advocates. Finally, there are collaborative initiatives and partnerships with extensive networks, which all help the program to reach women.

Evidence: The FY 2005 Congressional Justification cites the following statistics. In 2003, NWHIC had 6.7 million visitors to the website, 42,858 calls to the call center, and 2,878 e-mails. Also, the quarterly program reports show that the program is targeting and reaching the intended beneficiaries. For example, CCOEs provided health services to 13,989 individuals and provided education and outreach to 8,446 individuals in the second quarter of FY 2004. In the first quarter of FY 2004, CoEs had 124,685 billable encounters with a health professional, 16,370 educational visits, and 10,115 resource center visits.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has output based measures, which do not clearly tie to the program's mission of supporting health programs and disseminating health information and there are no measures that address the program's mission of coordinating women's health efforts. The program also cites the Healthy People 2010 (HP2010) objectives as long-term performance measures. Of the 400+ Healthy People measures, the program cites that 236 measures are relevant to OWH and OWH contributes directly to HP2010's overarching goals to increase quality and years of healthy life and to eliminate health disparities. Additionally, all HP2010 measures fall within ten "Leading Health Indicators" (LHI). Although the overarching HP2010 goals and the LHI are outcome oriented, they do not quantify the percent increase in quality years of life or percent decrease in health disparities.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH) identifies the following as long-term measures: 1) Number of research, demonstration, or evaluation studies completed and findings disseminated; 2) Number of communities, NGOs, state and local agencies, or federal entities, that adopt policies and recommendations targeting health disparities that are generated or promoted by OWH through reports, etc; 3) Number of peer-reviewed texts published by government or externally.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program does not have a timeframe or targets beyond FY 2006 for its long-term output measures. Also, targets are not ambitious for several of the long-term output measures because the targets remain constant or only increase slightly from one fiscal year to the next.

Evidence: The OPHS FY 2004 GPRA Plan for OWH. The target for number of research, demonstration, or evaluation studies completed and findings disseminated is 3 in FY 2004 and is 4 in FY 2005 and FY 2006. Next, the target for the number of communities, NGOs, state and local agencies, or federal entities, that adopt policies and recommendations targeting health disparities that are generated or promoted by OWH through reports and etc, is 16 in FY 2004 and 17 in FY 2005 and FY 2006. Finally, the target for the number of peer-reviewed texts published by government or externally is 5 in FY 2004 and FY 2005 and 6 in FY 2006.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program lists 15 annual performance measures, which do not contribute to long-term outcomes and program purpose. The program also lacks efficiency measures.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH) identifies annual performance measures, which include number of visitors to websites, number of prevention oriented initiatives and number of workshops or conferences with professional associations.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baseline and annual targets exist for FY 2003, FY 2004, FY 2005, and FY 2006. However, the program's annual performance measures do not tie to the long-term outcomes and thus, the baselines and targets are ineffective.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH).

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: For the most part, partners do not commit to and work towards program-wide annual and long-term outcome goals since the program only has output goals and measures. Some partners, specifically the CoEs have performance measures and collect data, but they do not clearly link to the program-wide output oriented long-term and annual measures.

Evidence: CoEs collect data on the following: maintain and expand a preexisting comprehensive, integrated clinical care center for women; develop a comprehensive women's health research agenda; develop and implement a comprehensive community outreach strategy; and develop culturally competent health care professional training in women's health.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Based on funding levels, CoE, CCOE, NWHIC, AIDS, osteoporosis, and the CVD prevention campaign are OWH's largest programs. There are no evaluations for the osteoporosis program, CVD prevention campaign, and NWHIC. NWHIC collects output data such as number of website users and conducts surveys to assess the website's functionality, look and feel, ease of navigation, and site performance. Without an outcome based evaluation, the data merely show the high website traffic and high customer satisfaction, and does not show improved health status or increase in health knowledge. Similarly, the AIDS evaluation is process based and not outcome based. For CoE evaluations, some of the directors of CoEs are on the evaluation staff, which compromises the independence of the evaluation. Finally, the CCoE program evaluation established a baseline and did not assess health status changes in women who participated in the program. However, this evaluation recognized the need for future evaluation efforts to measure progress towards program goals and the program expects to evaluate CCoEs for outcomes in FY 2006.

Evidence: 1) NWHIC January 2004 Status Report. 2) September 2003 ForeSee Results report entitled, "American Customer Satisfaction Index: E-Government Satisfaction Index." 3) December 2003 Research Evaluation Development Analysis (REDA) International, Inc. report entitled, "Evaluation of Women and HIV/AIDS/STD Programs." 4) November 2002 OWH report entitled, "An Evaluation of the National Centers of Excellence in Women's Health." 5) November 2003 Matthews Media Group report entitled, "Office on Women's Health: National CoE/CCOE Joint Project Process Evaluation." 6) 2001 Booz, Allen and Hamilton report entitled, "National Community Centers of Excellence in Women's Health: Draft Executive Summary."

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Resource needs and performance are not clearly link to the budget requests. Although it is not clearly evident in most cases, the budget request ties to annual performance measures such as number of website visits, number of workshops, number of public health education campaigns, and number of contracts that illustrate partnerships. In contrast, budget requests do not tie to the long-term measures of number of peer reviewed texts published; number of research, demonstration or evaluation studies completed and findings disseminated; and organizations adopting recommendations in OWH's reports. The long-term measures suggest that resources be allocated to scientists and researchers, but the program, in reality, funds public health education and outreach through competitive contracts.

Evidence: 1) FY 2005 Congressional Justification for General Departmental Management. Line items are organized by health conditions such as osteoporosis, diabetes, Lupus, HIV/AIDS, and mental health, which are ambiguous and do not illustrate how the resources will be used. In practice, funding for various health conditions are used for public education and outreach. 2) Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Last year, the program identified a weak strategic planning process as its main deficiency and began to engage in a year long process to address this weakness. A program staff retreat was held in September 2003 to begin developing organization-wide priorities by identifying accomplishments, community needs, and staff needs. In addition to the retreat, the planning task force was formed to establish program priorities and a structure task force was also formed to design the organizational structure. Until recently, the strategic planning process has not focused strongly on the development of new or implementation of existing long-term and annual performance goals. Instead, the program has focused on assessing accomplishments, identifying community needs, and enhancing communication and coordination within OWH. In March of 2004, two additional workgroups were formed, the strategic planning workgroup and the diversity workgroup. One of the key responsibilities of the strategic planning workgroup is to review and assess long-term and annual performance measures and link these goals to the program's strategic planning process.

Evidence: Strategic planning meeting agendas demonstrate the strategic planning process currently underway.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Program managers collect performance information from contractors through quarterly progress reports and verify this information through annual site visits, which helps OWH monitor its goal of supporting health programs. Another key program goal is to disseminate health information and to this end, OWH collects the following performance information: 1) quarterly reports on the total number of media impressions for health campaigns, such as the National Bone Health Campaign; 2) monthly data on number of NWHIC website visits, and randomly select website users to complete a customer satisfaction surveys on the web, which in turn, is used to redesign the website and to make it more user-friendly; 3) findings from a focus group, recently held with local DC community organizations to obtain feedback on the 2004 Women's Health Daybook, which will shape the design and content of the 2005 Women's Health Daybook.

Evidence: 1) Site visit reports. 2) Quarterly progress reports. 3) Monthly NWHIC user data. 4) September 2003 ForeSee Results customer satisfaction summary of NWHIC website users. 5) Quarterly media impression memo for national health campaigns. 6) OWH 2004 Daybook Focus Group summary.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Director of OWH is held accountable for program operation, including performance results, through the performance contract. The Employee Performance Management System (EPMS) was used to rate the project officers at OWH on their ability to monitor their contracts until this year, when a standard performance review document was implemented. Although contractor officers monitor cost and schedule of the contracts, the project officers monitor performance results so that the project officer has the ability to approve or disapprove the contractor's invoice for payment when the schedule of deliverables is not met. Next, program partners are held accountable for cost, schedule and performance results through performance based contracts. For example, the program has discontinued funding for poorly performing CCoEs when the CCoEs were unable to meet the deliverables of the contracts. One CCoE was suspended and was given a list of deliverables that it had to meet within a specified time frame. The suspension resulted mainly from failing to established a CCoE comprehensive clinical care component for women that was clearly recognizable to all staff. Corrective action included in-service training to educate all staff and recruitment of at least 500 active participants in the CCoE program. Similarly, OWH has not exercised the option years for poorly performing CoEs.

Evidence: 1) Performance based contract for the Director of OWH. 2) Employee Performance Management System (EPMS). 3) Standardized performance review document entitled, "Performance Management Plan and Rating" form. 4) Invoices where project officer disapproved payment.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: To track actual expenditures, the program uses an accounting system called the CORE. Actual spending, as documented in the CORE match the appropriated funds closely. For example, OWH was appropriated $28,658,000 in FY 2003 and CORE showed that 99% of the funds were obligated accordingly. Also, the administrative office at OPHS keeps a "MOA/MOU/IAG/IPA log", which records transfer of money to different contractors. Next, project officers review the invoices submitted by contractors and use their project knowledge to ensures that funds are spent for the intended purpose. For example, each CoE submits quarterly invoices which can only bill for the activities covered by the contract period of performance.

Evidence: 1) FY 2005 Congressional Justification. 2) Table comparing appropriations against GovNet and CORE obligation reports. 3) MOA/MOU/IAG/IPA Log for FY 2002.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program lacks actual cost efficiency measures and targets. However, the CoE contracts are performance based and performance incentives are built into the contract so that each CoE can earn an annual incentive in the amount of $1,000 for meeting the "acceptable quality level" for CoE performance measures. For example, for the CoE performance measure of mainintaining and expanding a preexisting comprehensive, integrated clinical care center for women, the CoE would receive a bonus if it dedicated a minimum of 20% of exam rooms as CoE-designated rooms and provides 20 hours of women's health care services per week. The program will award its first round of bonus payments in September of 2004. The CoE contracts also have a cost sharing component to move the contractee towards sustainability with the goal of the contractee continuing the work after funding ends.

Evidence: 1) CoE annual performance incentives. 2) CoE cost sharing contract.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program collaborates with related programs, most often Federal programs, on women's health activities. For example, the program worked with HRSA's community health centers on its breastfeeding initiative where HRSA agreed to identify the barriers to breastfeeding and to develop a model to assist hospitals and other birthing facilities in building a sustainable breastfeeding support program. The effectiveness of the public health message on breastfeeding was tested and recently implemented nationwide. The program also collaborated with the Indian Health Service's mobile women's care facility at Aberdeen, SD and played a significant role in shaping the types of health services offered through the mobile women's care facility. Initially, IHS has a vision to focus on mammography, but OWH successfully advocated for expanding the health screenings to include blood pressure and blood sugar. The mobile women's care facility now includes immunizations for children and provides community health care during the facility's after hours. Finally, to address cancer as a women's health issue, OWH knew that there were related programs whose funding and mission focused on cancer. OWH determined that the most effective way of allocating resources to target cancer in women was not to develop its own cancer materials, but to tap into the expertise of existing programs. This led to OWH's collaboration with NIH's National Cancer Institute in which OWH contributes a small amount of funding to NCI.

Evidence: 1) Inter-agency agreement between OWH and HRSA. 2) E-mail traffic on IHS collaboration. 3) "Women, Tobacco and Cancer" conference agenda. (OWH collaboration with the NCI at NIH.)

Does the program use strong financial management practices?

Explanation: HHS received a clean audit opinion and there are no material weaknesses for the program. Also, the program has routine practices to ensure strong financial management practices. For example, prior to awarding each CoE contract, these academic health centers must document and demonstrate sound financial management practices by submitting specific documents to the contracts officer. Once the contracts are awarded, the finances of most contracts are monitored by project officers, although some are monitored by GovWorks. The program also works closely with Administrative Resource Center at OPHS to run monthly financial reports, which leads to staff notices alerting staff to obligate funds on a timely basis and to allocate resources by the procurement deadlines.

Evidence: OPHS is audited annually as part of the HHS overall annual audit and the FY 2004 HHS performance and accountability report shows that there are no material weaknesses or other deficiences reported relative to the OWH program.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: At the program level, there was a concerted effort to review and reform the office structure and function. A workgroup was formed to assess the current structure and recommendations were made to to the OWH management team. Recommendations included using a "team project" format for meetings, projects, and initiatives; developing an office action plan each fiscal year; and creating an "office of orientation" for new hires. The OWH management team is expected to make final comments on the workgroup's recommendations and determine an implementation plan by September 30, 2004. At the contract level, project officers play a key role in identifying and correcting management deficiencies. Corrective action, most often results in discontinued funding. For example, site visit to a CoE over a period of 3 years showed that the site moved each year and saw a total of 8 patients. Subsequently, this contract was cancelled and the CoE solication was revised in FY 2003 so that it was clear that the funding was not to be used to build actual health centers.

Evidence: 1) CoE letters cancelling the contract. 2) CoE solicitation prior to FY 2003. 3) CoE solitation after FY 2003.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: CoE contracts and CCOE cooperative agreements are announced in the Federal Register and awarded through a competitive process. A 10 person federal government technical panel of experts convenes to evaluate each CoE proposal by the published evaluation criteria. This panel are Federal employees and non-OWH program staff. OWH staff also make site visits to the top scoring CoEs prior to making the award to ensure the accuracy of the information presented in the proposal. The program uses the grant authority of HRSA and Office of Minority Health to award cooperative agreements to CCOEs. (Note that cooperative agreement is a grant mechanism in which the Government plays a substantive role, along with the grantees, in the development and implementation of the project. Awards made through this mechanism are referred to as grant awards.) Similar to the CoE, a grants review panel reviews the CCOE proposals by the evaluation critiera and there is a pre-award site visit.

Evidence: 1) FY 2002 CoE and CCOE Requst for Proposal. 2) FY 2001 OWH CoE Site Visit Guidance Manual.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program provides oversight for its contracts through annual site visits, quarterly progress reports, and quarterly invoice review. At the end of each site visit, the OWH team debriefs the CCoE staff on program strengths and areas in need of further improvement. The site visit and the debriefing sessions serve as the primary means of communicating programmatic concerns. For example, a site visit report for one CoE cites the improvements made from the previous site visits, mainly addressing the lack of space needs by securing space in a new hospital owned building, and records expectations of a greater level of details in future quarterly progress reports that reflect all of the CoE's programs and activities.

Evidence: 1) Quarterly report for CoE & CCoE. 2) Site visit report for CoEs. 3) Quarterly and site visit report for HIV/AIDS program.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program placed the CoE evaluation on the web as a link from the NWHIC site. However, performance data are not collected in aggregate program wide level or disaggregated at the contractee level and made available to the public.

Evidence: NWHIC website link to CoEs evaluations.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Adequate outcome measures are unavilable and thus, it is not possible to measure the program's progress in achieving its long-term performance goals.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH).

Does the program (including program partners) achieve its annual performance goals?

Explanation: The annual performance goals do not link to the long-term performance goals. Therefore, the annual goals are not adequate and do not meet the standards of appropriate annual goals.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does not have efficiency or cost effectiveness measures and it is not possible to measure the program's progress in this area.

Evidence: Office of Public Health and Science (OPHS) FY 2004 GPRA Plan for the Office on Women's Health (OWH).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Without a clearly defined set of long-term and annual performance measures, it is difficult to compare the program's performance to other programs with similar purpose and goals. However, the NWHIC site has shown to score the same rate in customer satisfaction as the search engine Google and thus, compares favorably to other health sites. As for the other key programs at OWH, mainly CoEs and CCoEs, there are plans to compare program performance to other similar programs. For example, OWH will convene a comprehensive, integrated model meeting and will compare CoEs to five other Federal programs that utilize the comprehensive health care model. CoE and CCoEs was one of ten semifinalists in Innovations in American Government Award, sponsored by the Institute for Government Innovation at the John F. Kennedy School of Government at Harvard University. While this speaks to the solid program design, program information that compares its performance to other similar programs is lacking.

Evidence: 1) December 2003 Washington Post article by Anne Hull, "Measuring Public Satisfaction with Government Agencies." 2) September 2003 Washington Post article by Stephen Barr, "Handful of Customer-Savvy Federal Web Sites Score Big in New Survey." 3) Semifinalist certificate for the Innovations in American Government Award from the John F. Kennedy School of Government at Harvard University.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program lacks independent, outcome based evaluations. There are questions of conflict of interest with program directors acting as evaluators and where independent evaluations are available, they are focused on outputs and are process evaluations. However, there is some evidence of progress. For example, the process evaluation for the HIV/AIDS program cites the use of gender and culturally appropriate materials, successfully forging community links, and having credibility with the target population. Similarly, the CoE evaluation cites CoEs as a catalyst for change in widening the scope of women's health, and enhancing collaborations among researchers and practitioners. However, independent, outcome-based evaluations are needed to assess the actual impact on the health of the individuals served and to demonstrate results achieved.

Evidence: 1) NWHIC January 2004 Status Report. 2) September 2003 ForeSee Results report entitled, "American Customer Satisfaction Index: E-Government Satisfaction Index." 3) December 2003 Research Evaluation Development Analysis (REDA) International, Inc. report entitled, "Evaluation of Women and HIV/AIDS/STD Programs." 4) November 2002 OWH report entitled, "An Evaluation of the National Centers of Excellence in Women's Health." 5) November 2003 Matthews Media Group report entitled, "Office on Women's Health: National CoE/CCOE Joint Project Process Evaluation." 6) 2001 Booz, Allen and Hamilton report entitled, "National Community Centers of Excellence in Women's Health: Draft Executive Summary."