Detailed Information on the
National Health Service Corps Assessment

Develop an Information System to replace BHCDANEtT(legacy system) to better manage data on scholars, loan repayers and service sites.

The Administration will serve areas of greatest need by better targeting NHSC placements and taking into account foreign physicians who serve in areas with a shortage of health providers through visa waivers.

The Administration will support more underrepresented minorities and other students and health professionals from disadvantaged backgrounds through the program by enhancing recruitment efforts.

Provide data for newly adopted performance measures.

Improve efficiency through greater flexibility in the allocation of funds between scholarships and loans.

Develop recommendations and respond to legislative inquiries in preparation for the reauthorization of the NHSC program to find more meaningful ways of improving access to care in health profession shortage areas.

Measure: Patients served through the placement and retention of NHSC clinicians.

Explanation:The NHSC Loan Repayment program offers fully trained primary care clinicians the opportunity to receive assistance to pay off qualifying educational loans in exchange for service in a federally designated Health Professional Shortage Area. This service commitment is a minimum of two years. Many of these clinicians extend their service by applying for amendments to their contracts in their 3rd, 4th and 5th year. As the program awards more amendments to existing contracts, there is less funding for new contracts. Additionally, most of the NHSC measures are dependent on the size of the Field Strength. The larger the Field Strength, the greater number of people served, and the pool of clinicians who fulfill the service commitment to be potentially retained increases. HRSA attempts to balance the number of new NHSC clinicians with the retention of current members through loan amendments in order to preserves access to care for the underserved beyond the period of service commitment. The recent decreases in field strength reflect the ending service obligations for a large number of loan repayments recipients from previous years, an increase in the debt load of graduating clinicians, and a focus by the program on loan amendments, which can be more expensive than bringing on additional NHSC members.

Measure: Patients served through NHSC placements and retention, as well as other sources (Communities with a compelling need for providers that do not receive a NHSC clinician may more easily recruit a provider from another source as a result of increased exposure from the program.

Explanation:The NHSC Loan Repayment program offers fully trained primary care clinicians the opportunity to receive assistance to pay off qualifying educational loans in exchange for service in a federally designated Health Professional Shortage Area. This service commitment is a minimum of two years. Many of these clinicians extend their service by applying for amendments to their contracts in their 3rd, 4th and 5th year. As the program awards more amendments to existing contracts, there is less funding for new contracts. Additionally, most of the NHSC measures are dependent on the size of the Field Strength. The larger the Field Strength, the greater number of people served, and the pool of clinicians who fulfill the service commitment to be potentially retained increases. HRSA attempts to balance the number of new NHSC clinicians with the retention of current members through loan amendments in order to preserves access to care for the underserved beyond the period of service commitment. The recent decreases in field strength reflect the ending service obligations for a large number of loan repayments recipients from previous years, an increase in the debt load of graduating clinicians, and a focus by the program on loan amendments, which can be more expensive than bringing on additional NHSC members.

Measure: Average Health Professional Shortage Area (HPSA) score of areas receiving NHSC clinicians (HPSA scores gauge provider shortages and whether the program targets communities well.

Explanation:NHSC members can only serve in a Federally designated Health Profession Shortage Areas (HPSA). Each HPSA is scored on a scale of 0 to 25, with higher scores indicating greater relative need for primary care providers. The HPSA score is based on four elements: the ratio of population to primary care physicians, poverty rate, infant mortality rate or low birth weight rate, and travel time or distance to nearest available source of primary care. This measure indicates whether the NHSC is targeting areas of greatest need effectively, as a higher HPSA score reflects greater need. With the exception of 2005 the average score for all placements has trended upwards. The average HPSA score is based on the number of new NHSC scholar and loan repayment placements in a given year.

Measure: Increase the number of NHSC-llist vacancies filled through all sources.

Explanation:The NHSC Loan Repayment program offers fully trained primary care clinicians the opportunity to receive assistance to pay off qualifying educational loans in exchange for service in a federally designated Health Professional Shortage Area. This service commitment is a minimum of two years. Many of these clinicians extend their service by applying for amendments to their contracts in their 3rd, 4th and 5th year. As the program awards more amendments to existing contracts, there is less funding for new contracts. Additionally, most of the NHSC measures are dependent on the size of the Field Strength. The larger the Field Strength, the greater number of people served, and the pool of clinicians who fulfill the service commitment to be potentially retained increases. HRSA attempts to balance the number of new NHSC clinicians with the retention of current members through loan amendments in order to preserves access to care for the underserved beyond the period of service commitment. The recent decreases in field strength reflect the ending service obligations for a large number of loan repayments recipients from previous years, an increase in the debt load of graduating clinicians, and a focus by the program on loan amendments, which can be more expensive than bringing on additional NHSC members.

Measure: Average cost to the NHSC program of a patient encounter.

Explanation:The cost per encounter is based on the total estimated cost to the program for one year of service of all the clinicians in the Field Strength. This needs to be an estimate as there is considerable variation in the amount of the award by program (scholarship vs. loan repayment) and among the disciplines. This total is then divided by the estimated number of patient encounters generated by NHSC clinicians in a year. Patient encounters are drawn from the NHSC Uniform Data System (UDS) and health center UDS data sets. It is anticipated that the cost per encounter will continue to decline as the number of scholars (scholarships in exchange for in service) decreases, which is a function of the NHSC allocating a greater percentage of its funding to loan repayment (in exchange for service) beginning in 2003. This is due to the fact that loan repayers are less expensive than scholars.

Is the program purpose clear?

Explanation: There is a consensus of program purpose among interested parties on the National Health Service Corps and the program has a clear and relatively straightforward mission. The overarching goal of the program is to improve care in underserved communities by placing health professionals in selected areas. The program's immediate purpose is to place health care practitioners in underserved areas through a combination of scholarships and loan repayments. In exchange for this support, practitioners agree to serve for a minimum of two years. The program places primary care, oral and mental and behavioral health clinicians in underserved areas. The agency also determines the health professions shortage area (HPSA) definitions and designations. HPSA designation is used for its own purposes and as a funding guide for other Federal programs. The exact purpose of the NHSC field program, which focuses on recruitment, outreach and technical assistance to communities, is less clear.

Evidence: The National Health Service Corps was first authorized in 1971 (section 331-338 of the Public Health Service Act). Agency and Congressional reports related to the program are consistent with the program purpose as outlined in the authorizing legislation. The program is run by the Health Resources and Services Administration (HRSA).

Does the program address a specific interest, problem or need?

Explanation: The program addresses the problem of communities that have too few primary care, dental, mental and behavioral health care providers. National shortages are relative and subjective, but there is ample evidence that having limited access to a healthcare provider in a community is a barrier to care in and of itself. There are regions and pockets of the country that face shortages of physicians and other healthcare providers known as health professions shortage areas (HPSA). The HPSA designation criteria includes primary medical care, mental health, and dental care. These shortages limit access to healthcare in these areas regardless of the availability of health insurance. By definition, the places where the NHSC clinicians must serve are areas of need.

Evidence: The February 2002 update of shortage areas prepared by the agency includes 2,781 primary medical care, 798 mental health, and 1,580 dental HPSAs, and 56 million people living in a primary medical care HPSA. The agency estimates that as of August 2000, 26,657 clinicians would be needed to meet desired ratios in these underserved areas, assuming a perfect distribution of those clinicians.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The program is designed to have a significant impact in the context of all other factors that is reasonably known and can be measured. The program is designed to target areas of greatest need for primary medical care, mental health and dental clinicians. As a condition of scholarship or loan repayment, the program places clinicians in shortage areas. The program also maintains a list of communities that are eligible to receive a NHSC provider. This list is available to non-NHSC physicians and visiting physicians on J-1 visas who may also seek to work in the designated community.

Evidence: Over 30 years, the NHSC has placed over 22,000 clinicians in shortage areas. Currently, 2,366 clinicians serve in every State, the District of Columbia, and territories. The Office of the Inspector General found in 1994 that 90% of facility directors believe their facility could not adequately serve patients without NHSC providers. As of August 15, 2002, there were 2,434 sites listed as eligible to receive a NHSC clinician.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: Under a strict interpretation, the NHSC is the only Federal program that provides a financial incentive directly to providers as a means of improving access to health care in specific communities. The mechanism and point in the process at which they engage with the provider varies from other Federal programs that share the goal of improving the distribution of health care providers. A separate but related HRSA program, the Health Professions, includes as one of its principal aims to improve access to care in medically underserved communities by improving the distribution of health care providers.

Evidence: The GAO noted in 1995 the NHSC is the Federal government's main program for placing physicians and other providers in health professions shortage areas. The Council on Graduate Medical Education also notes the program is "specifically designed to address geographic maldistribution." The Nursing Education Loan Repayment and Scholarship Program offers similar support, but only for registered nurses. Title VII health professions programs aim to improve the distribution of health care providers by providing training grants and other support to students and institutions.

Is the program optimally designed to address the interest, problem or need?

Explanation: The program is administered through scholarships and loan repayments paid directly to the provider. Given a cost differential, greater flexibility in the allocation of funds between loans and scholarships and by discipline can improve program efficiency. NHSC providers were Federal employees until 1980. The majority are now employed by the facility in which they practice.

Evidence: There is no evidence that a block grant to states, tax incentive, regulation or other mechanism would be more efficient or effective in addressing the problem. With respect to the more narrow issue of scholarships versus loan repayments, a 1995 GAO report recommended loan repayments above scholarships as a more cost effective means of placing providers.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program adopted new long-term goals during the assessment process. The long-term goals focus on increasing access to the nation's neediest populations through the placement and retention of NHSC clinicians and the placement of independent physicians through other program efforts.

Evidence: The program has two long-term goals with targets: 1) Increase by 20% by 2010 the number of individuals served among the Nation's neediest populations through the placement and retention of NHSC clinicians; 2) Increase by 20% the number of individuals served in all communities seeking NHSC assistance. (This measure also captures placement through other sources resulting from NHSC involvement.)

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: During the assessment process, the program has adopted new annual performance goals that would demonstrate progress toward desired long-term outcomes.

Evidence: The first goal captures how well the program is extending its reach by retaining NHSC providers in service after the end of the contract period. The second goal captures how well the program is targeting the most needy communities by measuring the severity of the physician shortage in communities based on their HPSA rating. The third goal captures additional program efforts to help communities by measuring the percentage increase of NHSC vacancies filled through all sources. These sources can include private matches, J-1 visas and other entries to employment.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: NHSC clinicians commit to a period of service in a designated area in return for financial incentives in the form of scholarships and loan repayments. The clinicians are held liable if they breach the contract by failing to fulfill their service commitment. The majority of clinicians continue to serve even after the required period. This commitment to a minimum period of service and often times longer period of service supports the program's annual and long-term goals. Additional partners include the health care delivery sites that are eligible to recruit NHSC supported providers. By definition, these partners share the goal of placing providers in underserved areas.

Evidence: If this contract is breached, participants will be liable to pay the total amount of loan repayments paid and an obligation penalty of up to $24,000. A NHSC scholar who fails to begin or complete service is liable for up to three times the amount received plus interest. In 2000, 75% of NHSC clinicians who fulfilled their service commitment continued to serve under served populations. In addition to the service commitment, the program encourages extended service through newsletters, list serves and personal contact with the providers.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: The NHSC has significant room for increased meaningful collaborations outside of the Federal government, but recent NHSC budget requests have reflected a meaningful budget and management actions in response to the health center initiative. Guidance for this question states a Yes would require that the program show evidence of collaboration leading to meaningful actions in management and resource allocation. Similar management and budget changes within the health centers program have not been made. The program is based on a "one community at a time" approach to improving access to health care, and may be able to further its impact by more aggressively partnering with other entities to encourage providers not receiving NHSC support directly to practice in designated areas. In addition, further collaboration with other Federal activities that share similar goals such as the Health Professions grants may be beneficial.

Evidence: The program is in contact with underserved communities designated as eligible for NHSC providers, consolidated health centers, state-based entities, professional organizations, and academic institutions. The program is associated with the Consolidated Health Centers initiative, and budget formulation and planning seems to reflect the connection between the two programs. An example of budget actions includes an emphasis on directing loan repayments to staff health centers expanded by the health centers initiative.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: Evaluations have been conducted on an average of once every five years. These evaluations include information on program performance and have recommended changes to the program. The agency plans to support additional evaluations in the future to obtain updates on program effectiveness, including retention of NHSC clinicians after the period of required service. More focused evaluations that also include effectiveness information are conducted by third parties on a more ad hoc basis.

Evidence: The latest evaluation was published in May of 2000 and was conducted by the University of North Carolina at Chapel Hill and Mathematica Policy Research, Inc. under contract with the agency. In addition, GAO has reported on the program and provided information on program effectiveness.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program can estimate the associated cost of each field placement, which is directly associated with the program's outcome goals. While the program's annual budget display does not meet all standards of alignment, the program's ability to attribute cost to each output is sufficient to meet the standards of this question. The program budget structure is fairly straightforward and clear and does not vary markedly from program goals. The agency is working to tie budget planning to strategic planning. The program can estimate outputs (number of placements) per increased increment of dollars, and the distribution of funding between scholarships and loan repayments is specifically designated in the authorizing legislation. The program surveys retention rates and can also estimate the impact of funding changes on the total directly supported and retained workforce. Program management funds are budgeted elsewhere.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress, and other information provided by the agency. The annual output is the field strength of the NHSC through scholarships and loan repayment agreements. By statute, the program knows the annual allotment between scholarships and loan repayments.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The purpose of this question is to give credit where programs are not meeting the standards for a Yes to questions in this section, but are taking steps to correct those specific deficiencies. The main deficiency related to this section had been in setting long-term goals. Given the program has adopted meaningful long-term goals, this question is rated as not applicable and the points are redistributed. Related to strategic planning, the agency overall is making organizational changes which will further integrate budget and performance planning. Additional work is also needed to enhance opportunities for meaningful collaboration. The agency reorganized its operations to organizationally fold the program in with the Health Professions. The program adopted a performance measure that tracks the number of community placements filled by other sources. These steps should greatly enhance opportunities for meaningful collaboration between related state and Federal partners, and between the NHSC and Health Professions.

Evidence: The assessment is based on discussions with the agency. The agency's electronic data system can also improve the use of performance information in budgeting and planning. An agency management reform effort transferred the NHSC and the office charged with developing HPSA designations from the Bureau of Primary Health Care to its sister entity, the Bureau of Health Professions. The restructuring puts a single bureau in charge of all health professional programs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects and reports on information annually on the field strength and short-term retention of NHSC clinicians. The program collects information from scholars during training and service and annually collects data from communities regarding services performed. The program uses this information to improve selection and placement, help scholars through the training period, ensure clinicians meet their service requirements, and design efforts to increase retention after the period of required service.

Evidence: Annual performance reports, service verification form, National Health Service Corps Uniform Data System reports.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: The agency's senior managers are held accountable for operations of their programs, including performance results, through their annual performance contracts. Program partners are held accountable through penalties for breach of contract.

Evidence: "The Administrator's performance contract includes an outcome target for the NHSC. If NHSC loan repayment clinicians breach their contract, they are liable to repay their subsidy, plus a penalty of up to $24,000. A NHSC scholar who breaches his/her scholarship service commitment is liable for three times the amount received, plus interest, prorated for partial service. "

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Scholarships and loan repayments are awarded annually with sufficient time to shift awards to alternates in the event a potential recipient declines the award. Scholarship awards are made in August to conform to the school year. Loan repayment contracts are made in September after the new HPSA designation scores are available. The program monitors placements to ensure clinicians remain in eligible service areas.

Evidence: Assessment based on apportionment requests and annual budget submissions.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In general, there is little evidence that the program has incentives and procedures in place to improve efficiency and cost effectiveness in program execution. The program does contract out some services.

Evidence: Contracted services include scholarship support, technical assistance, marketing and outreach, logistics and filing.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The program does not capture all direct and indirect costs borne by the program agency, including applicable agency overhead, retirement, and other costs budgeted elsewhere. The program does not have a procedure for splitting overhead and other costs between outputs, including scholarships and loan repayment, or include informational displays in the budget that present the full cost of outputs.

Evidence: The assessment is based on annual budget submissions to OMB and Congress. The program does not have an agency program budget estimate that identifies all spending categories in sufficient detail to demonstrate that all relevant costs had been included or a report that shows the allocation of overhead and other program costs to the program. Overhead and other program costs, including FTEs, are included in the field budget.

Does the program use strong financial management practices?

Explanation: HRSA received its first clean audit in 1999.The 2000-2001 agency financial statements showed no material weaknesses. HRSA financial statements are conducted by the Program Support Center. The IG found in a 2002 audit of HRSA's travel, appointments, and outside activities that there was no evidence of substantive violations, but that there are technical lapses requiring improvement. The agency disagrees with the breadth of the problem and has re-issued guidance to improve oversight.

Evidence: The assessment is based on agency financial statements and IG audits. The program maintains procedures to detect if NHSC clinicians are out of compliance with program requirements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The main deficiencies in this section include incentives and procedures to improve efficiency, and the development of the full annual cost of operating the program to achieve desired performance. The agency is taking meaningful steps to correct these deficiencies. One potential barrier to the program's efficiency in meeting the goals is the ability of providers not to serve the target population if they take advantage of the national research service award option.

Evidence: The program is working with a consulting firm to reengineer its business processes. The program is also in the process of examining competitive sourcing options. The program anticipates completing the transition to an electronic system for the applications for community sites, scholarships, and loan repayments by the end of the 2002 calendar year. The program is also examining ways to stretch Federal loan repayment investments by adjusting maximum repayment levels for an individual clinician in the second and third years.

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: While not peer reviewed, the process for making loan repayment and scholarship awards is competitive and fair and is based on clear criteria including those established by law. Determining what facilities should be eligible for NHSC providers can be a subject of debate, but the program has a clear and consistent approach for making those designations.

Evidence: "The criteria to determine whether a community is eligible to receive a NHSC supported clinician require that the health care facility be located in a federally designated HPSA, document sound fiscal management, use a sliding-fee schedule or other documented methods to reduce fees that ensure no financial barriers to care exist, accept assignment of Medicare, enter into an agreement with the State agency that administers Medicaid, and produce proof of the capacity to maintain a competitive salary, benefits, and malpractice coverage package. "

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: The NHSC supports an annual recruiting effort through print and radio advertising, direct mail, and communication with schools, communities and other Health Professions programs to encourage new clinicians. The application is open to all scholars and clinicians who meet the legal requirements, and the majority of awards are made to first time applicants.

Evidence: Between 85-90% of scholarships and between 55-70% of loan repayments for each of the last few years have been new awards. The program has found It difficult to recruit a diverse workforce for the NHSC due to the overall composition of the health professions student body.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Scholars are monitored throughout the training process directly and through the school to verify compliance with legislation, regulations and programmatic issues, and checked monthly prior to payment of awards. Loan repayment clinicians are monitored using six month verification checks, periodic phone calls to the site, and site visits from HRSA field office staff.

Evidence: The program conducts financial audits of scholars, including stipends, tuition and other costs expended. Upon completion of study, program participants fulfill the service commitment by obtaining employment at an approved facility for their discipline or through a National Research Service Award.

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NHSC clinicians and partner facilities provide data annually to the agency. The program uses End of Service Surveys and the Uniform Data Set to collect information on the care delivery and retention of NHSC clinicians. Annual performance data are summarized in the performance report and made available on the agency web site. On a less systematic basis, performance data are also presented at conferences and other public presentations. The names of those who breached their contract are not provided to the public.

Evidence: Assessment based on agency GPRA reports and web site (www.hrsa.gov).

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The program has developed new long-term goals for the program to measure outcomes. The program tracks the immediate field size of the NHSC through scholarships and loan repayment agreements and surveys retention rates of those who completed the program. The long-term outcome goals measure the impact of the program based on the amount of care provided by current and retained providers. Once data showing this impact are available, the program can be rated from between a Small Extent to a Yes.

Evidence: The baseline year for these goals is 2001 and no baseline data exists for newly developed goals. The program will adjust its data collection efforts to accurately record and report on new goals. Targets are based on assumptions and will be adjusted, if necessary, once baseline data are available.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program developed new annual goals that will measure progress toward its long-term outcomes. Performance data available from previously held goals and survey information that relate to these new measures are available and indicate results that contribute to the long-term outcomes of the program. Targets for FY 2004 are still under review.

Evidence: Relevant performance data related to the new goals include, in FY 2002, the current NHSC field strength increased 14% to 2,703. In FY 2001, the percent of NHSC clinicians retained in service increased from 75% to 80%. Data are not yet available on HPSA scores and vacancies filled through all sources. A large extent would require data that show progress on these other measures. The program's annual goals capture not only the number of physicians directly supported by Federal investments, but also the number retained after the service contract is complete and the number of communities the program works with that are able to recruit physicians through other channels.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The program received a No in Question 4 of Section III, and according to guidance is not eligible for a full Yes to this question. By the end of FY 2002, the program had converted 14 of its Federal full time equivalent positions previously serving in administrative and other support roles into NHSC providers through a first responders initiative. This change will increase the number of NHSC clinicians within the current year totals. A Yes or Large Extent would be appropriate with additional incentives in place and as this conversion continues, if the conversion of FTE translates into improved cost effectiveness in achieving program goals. As noted previously, the agency finds as its primary barrier to increasing efficiency the inability to shift resources further from scholarships to loan repayment awards. Additional work is also needed to better target NHSC providers in areas of highest need.

Evidence: The program announced in April of this year that it is recruiting clinicians to serve as commissioned officers of the U.S. Public Health Service within the NHSC. The clinicians will be classified as Ready Responders within the NHSC and would eventually include 36 family practice physicians and four dentists who will be assigned for 3 years in a HPSA. These 40 positions are to be absorbed by the program through reduction of FTE for administrative and other program support positions. With respect to the balance between scholars and loan repayments, 79% of NHSC loan repayment clinicians serve in an underserved area after the required period of service compared to 62% NHSC scholars. NHSC loan repayment costs per clinician placement per year of promised service are one half to one third as much as the scholarship costs.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: The program is not involved in the Federal government's Health Common Measures (for information on these measures see www.whitehouse.gov/omb). The NHSC's sister program, the Health Professions, does not provide a direct comparison, but shares the goal of improving the distribution of health professionals. Relative to the Health Professions, the NHSC is a more direct mechanism for improving the distribution of health professionals and based on annual performance data is more efficient in its rate of placements. When considering the ability of the NHSC to show retention of its clinicians in shortage areas, the performance of this program compares favorably.

Evidence: Dollar for dollar, the NHSC is more efficient in placing medical professionals in shortage areas than the Health Professions. According to the most recent data available, in 2000 the average cost per placement was $77,400 for the Health Professions and $47,900 for the NHSC. According to the National Conference of State Legislatures, most state scholarship and loan repayment programs have not been evaluated, and thus have no evidence of their effectiveness.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Recent evaluations indicate the program is effective. A 2000 Mathematica evaluation found the program is effective in providing underserved communities with clinicians. The evaluation found low satisfaction in the matching process, but increasing effectiveness in recruiting individuals motivated by a more altruistic desire to practice in underserved communities, a factor that can improve long-term retention. Earlier evaluations were more mixed. A 1995 GAO report found the program is working, but placed more providers than needed in some areas and none in others, did not have the most effective mix of loan repayments and scholarships, and needed improved coordination with J-1 visa waiver process. A 1995 University of Washington survey of rural scholars found half remain in service long-term. A 1994 HHS OIG report found facilities receiving NHSC clinicians depend on them to adequately serve patients, but certain procedures needed improvement. A 1994 JAMA study found low morale and poor retention among rural NHSC physicians in the 1980s.

Evidence: The 2000 evaluation found long-term retention of up to 15 years of NHSC providers after the required period of service is 52%. The evaluation also found overall NHSC clinicians and alumni reach new patient populations, increase the volume of services, add new services and may often play a role in initiating community-oriented primary care programs. With respect to the mix of loan repayments and scholarships, the GAO report states loan repayments are more cost-effective and produce clinicians more likely to complete their obligation and remain in service, however, the program must dedicate at least 40% to scholarships by statute.