Detailed Information on the
Telehealth Network Grant Program Assessment

Collecting data on a new long-term measure to assess the impact of the program's activities on clinical outcomes in diabetic patients served by grantees of the program

Tracking performance against benchmarks and refining performance goals for mental health services.

Tracking performance against benchmarks and refining performance goals for sustainability.

Create and validate measures for telemonitoring and telehomecare.

Tracking performance against benchmarks and refining performance goals for expanding sites and services

Measure: Access to Mental Health Services: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. The baslines, targets and actual numbers listed below represent COMMUNITIES that have gained access to: pediatric/adolscent and adult mental health services where access did not exist prior to the TNGP grant. (Please note: Because this is a demonstration program, every three years each cohort of TNGP grantees "graduates" from its three-year grant while a new cohort of grantees commences a new three-year cycle of grant-supported telehealth activities. Thus, no change can be expected when a cohort finishes its grants and a new cohort starts up.)

Explanation:Baseline: In February 2005, 101 communities had access to pediatric and adolescent psychiatric services and 60 communites had access to adult psychiatric services that otherwise would not have had them in the absence of the TNGP grant. Target: By 2012, a total of 300 communities will have gained access to pediatric and adolescent mental health services and 180 communities will have gained access to adult psychiatric services that otherwise would not have had them in the absence of the TNGP program.

Measure: Telehealth Chronic Disease Management: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%).

Explanation:Note: This target includes data from the 2006-2009 cohort plus data from the 2009-2012 cohort. In the 2006-2009 cohort, the baseline for entering patients into a new telehealth diabetes control program would be 10%. By the end of the three-year period of the grant, the target will be reached. In 2009, a new cohort of grantees will be funded, with a new cohort of patients. It is anticipated that this new cohort of grantees would have patients similar to those who entered in 2006 in their first year, i.e., that mostly would have out-of-control glycemic levels (otherwise, they would not be enrolled in the program.) As such, we are assuming that the new group of grantees would also start with a baseline of 10% and over the three year period of having grant funds would achieve the 30% target.

Measure: Sustainability of Telehealth Programs: The percent of TNGP grantees that continue to offer services after the TNGP funding has ended.

Explanation:Baseline: 100% of 1997 grantees from a similar predecessor program continue to offer telehealth services in 2005. Target: By 2012, an average of 95% of the TNGP grantees funded in all cohorts since 2003 will continue to offer telehealth services after the grant has ended. The 95% target accounts for unforseen events associated with a more heterogeneous program.

Measure: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%) in each year of the three-year grant cycle.

Explanation:Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%) in each year of the three-year grant cycle. Baseline: 10% of patients initially enrolled in a telehealth diabetes case management program are under ideal glycemic control (HbA1c less than or equal to 7%). Target: For each year of the three-year grant cycle, the population under glycemic control enrolled in a telehealth diabetes case management program for at least 6 months would increase by 45 percent. That is, after the first year, 14.5% of the patients enrolled in a telehealth diabetes control program for at least 6 months would be under control. By the end of the third year, 30% of patients would be under control. In 2009, a new cohort of grantees would be funded, wherein we anticipate the baseline to also be 10% and the process would be repeated for an average of 30% under control for the second cohort.

Measure: Expand the number of telehealth services (e.g., dermatology, cardiology) and the number of sites where services are available as a result of the TNGP program. (This is calculated as the sum of services and sites [service+site factor] across all grantees in each year of a three-year grant.)

Explanation:This measure provides the number of sites and services made available to people who otherwise did not have access to them per million dollars spent. Thus for less than $10,000, a community in which a grantee operates gains access to at least one health care service that it did not have access to prior to the program. (see chart for baseline/historical data and targets.)

Measure: Expand the number of services and/or sites that provide access to health care as a result of the TNGP program per federal program dollar expended. This will be expressed as the "Service-Site Factor per $1 million program dollars," which is calculated as follows: Numerator = the number of new sites+services divided by the Denominator = annual federal cost of TNGP grants.

Explanation:This measure provides the number of sites and services made available to people who otherwise did not have access to them per million dollars spent. Thus for less than $10,000, a community in which a grantee operates gains access to at least one health care service that it did not have access to prior to the program. To calculate the efficiency measure for 2004 and 2005, the program divides the number of sites+ services funded each year by $3.8 million, the amount of funding available for TNGP program activities. This yields the following efficiency measures for those years: 2004: 108 sites and services (S&S) per $1 million (411 S&S/$3.8 million) 2005: 122 sites and services(S&S) per $1 million (463 S&S/$3.8 million) The program is able to demonstrate increased cost-effectiveness from 2004 to 2005 (most recent data available), as the program increased the total number of sites served and services provided per $1 million of federal funding within the cohort of grantees funded for three years. (see chart for additional data.)

Is the program purpose clear?

Explanation: The purpose of the Telehealth Network Grant Program is to demonstrate how technology can be used by telehealth networks* to: 1) expand access to, coordinate, and improve quality of health services; 2) improve and expand the training of health care providers; and 3) expand and improve the quality of health information available to health care providers, and to patients and their families. *A Telehealth Network consists of a group of collaborating entities, (e.g., clinics, hospitals, universities, health departments), that use electronic information and telecommunication technologies to enable the provision of long-distance clinical health care, patient and professional health-related education, public health, and health administration.

Evidence: 1) Authorizing Legislation: Public Law 107-251, Section 330I 2) The Program is in its first three-year cycle of implementation, with the first round of grantees (15) receiving an average award of approximately $250,000 per year for each of three years. The first round of grants is scheduled to close August 31, 2006, with a new three-year cycle scheduled to begin on October 1, 2006.

Does the program address a specific and existing problem, interest, or need?

Explanation: The Program addresses the lack of access to quality health care services in many rural and other under-served communities. These communities also lack access to training for health care providers and access to quality health information upon which providers and patients can make well-informed health care decisions. This problem was recognized by the authorizing legislation and is a component of the Office for the Advancement of Telehealth (OAT) mission. For example, many rural communities do not have access to providers, including specialists such as dermatologists, mental health professionals, and geriatricians. These communities also lack access to other services, such as pharmacy, pathology, rehabilitation, or intensivist services (i.e., services provided by specialists in critical care medicine) that are complementary to and support primary care providers in these communities.

Evidence: The maldistribution of health care professionals in the U.S. has been widely reported. (1) HRSA reports that there are over 10,000 areas of the country that are designated as health professional shortage areas for primary care physicians, mental health and dental providers, approximately 6,700 of which are located in rural areas. (2) Approximately 20 percent of the U.S. population resides in rural communities, whereas only 9 percent of physicians practice there. (3) More than 90% of all psychiatrists and psychologists and 80% of all masters prepared social workers work exclusively in urban areas (Dennis Mohatt, WICHE Mental Health Program, Presentation to President's New Freedom Commission, January 9, 2003 - http://www.mentalhealthcommission.gov/presentations/presentations.html). (4) The health care labor shortage in the U.S. has been widely documented. Almost half of the healthcare workforce will be 45 years old or older by 2008. By 2010, 40% of all registered nurses will be 50 years old or older, and the U.S. will need 2,344,584 nurses, but only 2,069,369 will be available, which equates to a 12% shortage. (5) Shortages of personnel are even greater for highly specialized professionals. For example, in 2000, the Leapfrog Group, a patient safety initiative representing Fortune 500 companies, called for full-time physicians specially trained in critical care medicine ("Intensivists") staffing in ICUs as a way to save more than 50,000 lives per year. Unfortunately, there is a severe shortage of intensivists. Less than 6,000 are actively practicing in the U.S., leaving less than 15% of ICUs receiving dedicated intensivist care.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The activities funded by the Telehealth Network Grant Program, and the Office for the Advancement of Telehealth (OAT) do not overlap with other Federal, state, local or private efforts. While there are other programs designed to supply telehealth-related equipment and support transmission costs, the TNGP is the only one that provides funds that directly support organizational development. The Program's unique emphasis allows entities such as rural clinics, local health departments, universities, home health care providers, long-term care providers, and hospitals to develop systems for putting telehealth technologies to use in underserved communities. In order to promote collaboration across government agencies, OAT chairs the Joint Working Group on Telemedicine. This working group includes representatives from key Federal agencies such as the Food and Drug Administration (FDA), Department of Agriculture, NASA, Department of Defense, Department of Commerce, Rural Utilities Service and the Agency for Health Research and Quality. OAT facilitates this venue for the regular exchange of information across Federal programs and to work together to overcome challenges faced in the telehealth field. The working group helps ensure that OAT grant activities complement and leverage resources made available from other related Federal initiatives. TNGP also seeks to create synergy between its efforts and those of States and localities. Grantees must identify other sources of funding in their grant application, which allow OAT staff to identify any duplication in funding. Moreover, through the American Telemedicine Association, the California Telemedicine and eHealth Center, its grantee network, and other relevant organizations, OAT staff stays current regarding state initiatives that may impact TNGP program priorities and implementation as reflected in the program guidance.

Evidence: 1) The Telehealth Network Grant Program (TNGP) 2006 Program Guidance requires grantees to coordinate their plans with other related entities, details of which can be found in the following documents: - Intergovernmental Review - Program Requirements and Expectations for Grant Recipients - Legislative Funding Preferences (c) Coordination - Conditions for Telehealth Network Grants 2) The Joint Working Group on Telemedicine is discussed on the Telehealth Network Grant Program (TNGP) website:Http://www.hrsa.gov/telehealth/jwgt.htm

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has no major design flaws. Providing competitive grants is a targeted way to reach underserved communities that would most benefit from the use of telehealth. These are often the communities lacking access to services and least likely to have the operational funds for start-up and development of a telehealth program. The TNGP is unique in its approach and operation and has resulted in the dramatic growth of telehealth sites, advances in the field of telehealth, the creation of a collaborative telehealth community, and a trusted vehicle for knowledge exchange. In addition, the funds provided by the Program serve as seed funding that grantees have leveraged to successfully grow their programs. As part of the grant application process, each applicant is required to submit a sustainability plan that outlines its initial strategy on how the program will be sustained after federal funding has ended. Through bi-annual and annual reports, grantees report on their progress towards achieving sustainability, barriers faced, and lessons learned to inform the broader community engaged in telehealth.

Evidence: 1) Draft 2006 Program Guidance: outlines the structure of the program and; 2) Progress Report template: provides the types of information that is tracked by the Program and; 3) Final GPRA Report: outlines the types of data tracked by the grantees for reporting on their progress.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The grant application and award process ensures that TNGP funds are targeted directly and effectively to those who can benefit from the funds to implement telehealth advances in their communities. For less than $1 per person per year, the TNGP provides access to specialized services in underserved rural areas. Grantees are required to perform evaluations of their grant-funded activities and submit semi-annual Progress Reports that demonstrate that they are targeting the appropriate patient populations. In the absence of the TNGP, it is likely that many of these communities would not have adopted telehealth technologies or would have adopted inferior, duplicative, and inefficient solutions. The TNGP, and its predecessor program, the Rural Telemedicine Grant Program, accelerated the adoption of telehealth across the US, which responds directly to the intent of Congress.

Evidence: Final Office of Advanced Telehealth GPRA Report: provides a summary of the information provided by the grantees on the operational impact of their program, e.g., volume of services offered, types of services, settings, technology, and impact data.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Grantees of the TNGP use the grants to support a diverse range of telehealth services operating in diverse communities based on the needs identified by the community. Thus the program has developed two long-term measures using specific health conditions that serve as indicators of performance for the overall program and a third long-term measure that demonstrates the Program's overall impact.

Evidence: Long-term measure 1: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. Long-term measure 2: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%). Long-term measure 3: The percent of TNGP grantees that continue to offer services after the TNGP funding has ended.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long-term measure #1: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. Baseline: In February 2005, 101 communities had access to pediatric and adolescent psychiatric services and 60 communities had access to adult psychiatric services that otherwise would not have had them in the absence of the TNGP grant. Target: By 2012, a total of 300 communities will have gained access to pediatric and adolescent psychiatric services and 180 communities will have gained access to adult psychiatric services that otherwise would not have had them in the absence of the TNGP program. Long-term Measure 2: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%). Baseline: 10% of patients initially enrolled in a telehealth diabetes case management program are under ideal glycemic control (HbA1c less than or equal to 7%). Target: By 2012, 30% of diabetic patients enrolled in a telehealth diabetes case management program for 6 months or longer will be under ideal glycemic control (HbA1c less than or equal to 7%). Note: This target includes data from the 2006-2009 cohort plus data from the 2009-2012 cohort. In the 2006-2009 cohort, the baseline for entering patients into a new telehealth diabetes control program would be 10%. By the end of the three-year period of the grant, the target will be reached. In 2009, a new cohort of grantees will be funded, with a new cohort of patients. It is anticipated that this new cohort of grantees would have patients similar to those who entered in 2006 in their first year, i.e., that mostly would have out-of-control glycemic levels (otherwise, they would not be enrolled in the program.) As such, the new group of grantees would also start with a baseline of 10% and over the three years would achieve the 30% target. Measure 3: Sustainability of Telehealth Programs: The percent of TNGP grantees that continue to offer services after the TNGP funding has ended. Baseline: 100% of 1997 grantees continue to offer telehealth services in 2005. Target: By 2012, an average of 95% of the TNGP grantees funded in all cohorts since 2003 will continue to offer telehealth services after the grant has ended.

Evidence: LT measure 1: Abt Report, Page 17: These ambitious targets take into account the environmental challenges experienced by grantees even when grant funds are available, e.g. difficulty in obtaining services due to the shortage in the supply of mental health specialists and continued low reimbursement for these services. The program typically funds 15 or fewer demonstration grants in three-year cycles - these data include the current cohort ending in 2006 and two additional cohorts ending in 2009 and 2012. Data Sources: 6-month performance monitoring progress reports from grantees. The performance monitoring system that OAT uses is web-based; twice each year OAT grantees submit data using consistent definitions and reporting guidelines. LT measure 2: Susan L. Dimmick, PhD, Adjunct Assoc Professor, UT Health Science Center, Telemedicine Evaluation and Project Manager: "Based on our experience in diabetes disease management using telehealth technology (store-and-forward) and tele-counseling, we have found that approximately 10-15% of diabetics seen in our public health and community health centers have HbA1c figures of 7% or below as a baseline. Approximately 50-60% of those who participate in our diabetes program bring their HbA1c down to 7% or below [with an average entry HbA1c of 9%]." ".....it is reasonable to follow the Indian Health Service guidelines of 27-30% as a national standard, which would take into account the racial diversity of those with diabetes; gender differences; and the type and duration of diabetes in individuals." Data Sources: Data on glycemic control is routinely collected by disease management programs. In each cohort, grantees will begin to collect the data on enrollment of patients and provide a revised baseline in the first 6-month report. For the first cohort, the baseline data will be available March 2007. The number of patients under control for each cohort by the end of each grant cycle will be divided by the number of individuals with diagnosed diabetes participating in the telehealth diabetes control programs for at least 6 months. LT measure 3: Sustainability of grant-funded programs resulting in increased access to health care for more Americans is a key objective of this program. The 1997 cohort of Rural Telemedicine Program (RTGP) grantees, the predecessor program of the TNGP, began their programs in FY 1998, some ended in FY 2000, while others continued into FY 2002. Sustainability of programs was a key objective of this program. Currently, 100 percent of the 1997 cohort of RTGP grantees continues to provide telehealth services. It is the program's intention to maintain the their past accomplishments, with the 2003 grantee cohort and those cohorts following. As presently stated, there is no growth potential for this measure as the current baseline is 100%. Yet, because this measure is the best reflection of the program's real and sustained impact it should be included. However, market variability often results in conditions that are beyond a program's control to ensure sustainability. It has been our experience that grantees have had to shift services to accommodate changing demand. As such, we believe it is reasonable to set a target of 95%, allowing for unforeseen events associated with a more heterogeneous program. Data Source: For the 2003 grantee cohort, data from the close-out report will be available in early FY 2007. For the 2006 grantee cohort, data from the close-out report will be available in early 2010. Each year after a cohort completes their grants, they will voluntarily inform the Program of their operating status via the Telehealth list serve. (Note: At present, all current and past grantees actively communicate and report out on this listserve and we anticipate this will continue.)

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has developed a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term goals.

Evidence: Annual measure 1: Increase annually the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. Annual measure 2: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%) in each year of the three-year grant cycle. Annual measure 3: Expand the number of telehealth services (e.g., dermatology, cardiology) and the number of sites where services are available as a result of the TNGP program. (This is calculated as the sum of services and sites [service+site factor] across all grantees in each year of a three-year grant.)

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The baselines and targets of the Telehealth program's annual measures are as follows: Annual measure #1: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. Pediatric/Adolescent Adult 2003 baseline 47 communities 39 communities 2005 Actual 148 communities 99 communities 2006 Target 148 communities 99 communities 2007 Target 168 communities 114 communities 2008 Target 195 communities 123 communities Annual measure #2: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%) in each year of the three-year grant cycle. Baseline: 10% of patients initially enrolled in a telehealth diabetes case management program are under ideal glycemic control (HbA1c less than or equal to 7%). Target: For each year of the three-year grant cycle, the population under glycemic control enrolled in a telehealth diabetes case management program for at least 6 months would increase by 45 percent. That is, after the first year, 14.5% of the patients enrolled in a telehealth diabetes control program for at least 6 months would be under control. By the end third year, 30% of patients would be under control. In 2009, a new cohort of grantees would be funded, wherein we anticipate the baseline to also be 10% and the process would be repeated to attain a target of 30% under control for the second cohort. Annual measure #3: Expand the number of telehealth services (e.g., dermatology, cardiology) and the number of sites where services are available as a result of the TNGP program. This is calculated as the sum of services and sites [service+site factor] across all grantees in each year of a three-year grant. Baseline: The baseline for this measure is 489 services+sites were made available in 2006, based on the number of sites plus services provided in the last year of the 2003-2006 cohort of grantees. Target Number of Sites+Services for 2006-2007 (first year) = 900 Target Number for 2007-2008 (second year) = 952. Target Number for 2008-2009 (third year) = 978. These numbers/targets are cummulative building on the 489 Sites and Services achieved through the 2003-2006 cohort. This cycle will begin again in 2009 with the new cohort - annual targets would be established based on the 2006-2009 cohort experience. Current targets are ambitious in that grantees continue to face significant barriers to deploying telemedicine and with each new set of grantees, the program funds grantees who add more difficult services, as grantees explore the boundaries of providing these services, e.g, innovative use of telehealth for physical therapy, stroke assessment and post-treatment rehab., teledentistry. etc. Moreover it is anticipated that the program will be awarding 1-2 fewer grants in the 2006-2009 cohort, but is projecting that it will add the same number of sites for each year as in the current cohort. Program management believes that despite the challenges of adding more difficult services, the lessons learned from the current grantees will translate in more effective deployment of services in the future and hence the reason for the optimistic and ambitious projections.

Evidence: Evidence for annual measure #2: 1) This target includes data from the 2006-2009 cohort plus data from the 2009-2012 cohort. In the 2006-2009 cohort, the baseline for entering patients into a new telehealth diabetes control program would be 10%. By the end of the three-year period of the grant, the target will be reached. In 2009, a new cohort of grantees will be funded, with a new cohort of patients. It is anticipated that this new cohort of grantees would have patients similar to those who entered in 2006 in their first year, i.e., that mostly would have out-of-control glycemic levels (otherwise, they would not be enrolled in the program.) As such, we are assuming that the new group of grantees would also start with a baseline of 10% and over the three years would achieve the 30% target. Moreover, in the first year of any grant cycle, grantees are challenged with organizing their program and thus may enroll relatively few patients, with the consequence that the population under care may be small and very few may be under the program for 6 months. As such, we are assuming that they would also start with a baseline of 10% and over the three years would achieve the target for the grantee population. 2) See Evidence for Long-term measure #2 (question 2.2) 3) Data Collection Plan: Data on glycemic control is routinely collected by disease management programs. In each cohort, grantees will begin to collect the data on enrollment of patients and provide a revised baseline in the first 6 month report. For the first cohort, the baseline data will be available March 2007. The number of patients under control for each year will be divided by the number of individuals with diagnosed diabetes participating in the telehealth diabetes control programs for at least 6 months during that grant year. Thus, separate data collection should not be required. The grantees also will be expected to incorporate this measure as one of their objectives, progress on which will be routinely reported as aggregated data in their progress reports. Evidence for annual measure #3: 1) The percent increase in the third year is ambitious as grantees generally add most of their sites in the second year, and devote the third year to solidifying the program in preparation for achieving sustainability once the grant ends. Nevertheless, a 8.7% increase in the number of services and sites from the first to the third year is an ambitious target for any grant given that the baseline achieved is high to begin with. 2) Every three years the cohort of grantees changes, with each cohort expanding services into new realms, often without a secure basis for reimbursement. The purpose of this program is to "push the envelope" and fund innovative proposals to bring new services into communities using telehealth technology. Therefore, there is no reason at this point to assume that the program would achieve different target rates of increase in the 2006 cohort from those of the 2003 cohort, the most recent cohort for which the program has data. As a result of grant-funded activities, some grantees find that the demand for services is dynamic and evolves. To remain competitive and be responsive to community needs, they switch out services, dropping some and adding others. 3) Data Source: Each grantee will provide information on the number of health care services (e.g., dermatology, cardiology, pediatric mental health, etc.), made available and the number of sites that are operational as a result of TNGP grant funding in the annual non-competing continuation applications. These data are routinely reported at the beginning of each grant year and in the 6-month progress reports.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The Telehealth Network Grant Program (TNGP) grantee community has been greatly involved in collaborating with the Program to develop the measures used to date for measuring program results. Upon receiving funding, grantees understand the program requirement to collect data in support of these measures through regular progress reports, wherein the grantee will extract the information from routine logs and case management records. Also, grantees benefit from having the data as it has provided them with a systematic way, by establishing a benchmark, of comparing their program to similar programs funded by TNGP grants. Grantees are provided a feedback report of their measures compared to averages across all OAT grantees. Data collected from its program partners (TNGP grantees) includes 6-month progress reports (including reporting on measures for annual GPRA-related measures), annual reports, and final grant project reports. HRSA also ensures that high quality performance data will be available from grantees by requiring that grant applicants describe the evaluative measures they will be using in their projects. Grantees then report on these measures in their reports to HRSA. Grantees are required to participate in OAT's data collection/evaluation efforts as a condition of accepting OAT funding. All of the data is used to manage the program. The experience and performance of grantees is shared with other grantees as lessons learned. Grantee data also influences how future grants are structured including grant award requirements.

Evidence: 1. Draft Program Guidance for Applying to program (Tab 1.3), 2. Progress Report Template (Tab 1.4). 3. The Guide for Getting Started in Telemedicine 4. The OAT Directory These documents can be found at: http://telehealth.hrsa.gov Once on this page, click on "Telemedicine Technical Assistant Documents."

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The first cohort of TNGP grantees will be completing their project period on August 31, 2006. The Program is contracting with an independent contractor to evaluate this first round of grantees and identify any areas where the Program could improve progress reporting. In January of 2006, Abt. Associates Inc. completed a report on all Telehealth grantees which detailed the grantees' efforts to meet the program's goals and objectives as stated in their GPRA plan. Abt collected data across all the TNGP grantees for a 3-year period providing analyses of the performance on a wide variety of measures. Analyses were provided to individual grantees to measure their performance against that of the entire group of grantees for each 6-month progress reporting period.

Evidence: Final GPRA Report - prepared by Amy Fitzpatrick, Andrea Hassol, Abt. Associates Inc.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budgets are not explicitly tied to accomplishments of annual and long-term goals. The relationship between annual and long-term targets and budget resources is not clear.

Evidence: The budget justifications for the Telehealth program's activities are included in the Health Resources Administration Fiscal Year 2007 Justification of Estimates for Appropriation Committees.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: To date, the Program has not experienced any strategic planning deficiencies. The program has developed new knowledge based on the performance of the first round of grantees. This new knowledge is then incorporated into the FY2006 Program Guidance and is included when new grant applications are solicited. The new applications require greater specificity and establish explicit standards for review that reflect the program's priorities for quality evaluation capabilities among the applicants.

Evidence: Draft Review Criteria for FY 2006 competition for Telehealth Network Grant Program

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: HRSA uses data submitted by the Telehealth Network Grant Program (TNGP) grantees to manage the program and to continue to improve performance. Data collected from grantees includes semi-annual progress reports (including reporting on measures for annual GPRA-related measures), annual reports, and final project reports. HRSA also ensures that high quality performance data will be available from grantees by requiring that grant applicants describe the evaluative measures they will be using in their projects. Grantees then report on these measures in their reports to HRSA. Grantees are also required to participate in OAT's data collection/evaluation efforts as a condition of accepting OAT funding. All of the data is used to manage the program. The experience and performance of grantees is shared with other grantees as lessons learned. A thorough review and assessment of grantee data influences how future grants are structured including grant award requirements.

Evidence: The Telehealth Network Grant Program (TNGP) contains the following reporting requirements (excerpted from the TNGP program guidance): "Reporting": Every TNGP grantee is required to: Submit a Progress Report(s). Specific reporting requirements for the TNGP are as follows: ?? Data Collection, Evaluation, and Reporting Requirements: Applicants accepting this award must, if requested, participate in the Office for the Advancement of Telehealth (OAT) data collection and evaluation of telemedicine activities. At a minimum, grantees are required to participate in the following activities: ?? Progress Reports: Technical progress reports are required at 6-month intervals, with the first due 6 months after award. When the due date coincides with the conclusion of the project, the final report will also serve as the progress report. Reports should include a summary of what has been accomplished during the reporting period and what has been learned, as well as basic information required by OAT to measure the accomplishments of the program across all grantees. A copy of the format to be used will be provided by OAT. Reports are submitted in duplicate the Division of Grants Management Operations. ?? Information on grant performance for OAT's Grantee Directory: Instructions for completing this task are provided to grantees. The current directory is available online at: http://www.hrsa.gov/telehealth/grantee.htm

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: HRSA managers and their program partners, the TNGP grantees, are identified and held accountable for costs, timely schedule and effective implementation of project work plans, and program performance results. Program personnel monitor and evaluate reports as described in item 3.1 above.

Evidence: Accountability for cost, schedule and performance is woven throughout HRSA's day-to-day activities. Once a TNGP grant is awarded, detailed semi-annual reports are submitted to Project Officers. POs carefully review these reports and compare original grant objectives and goals with those stated in these progress reports. When a variance occurs between the grant application and the actual grant performance, the PO provides technical assistance (or procures it on behalf of the grantee from a peer grantee) to remedy any discrepancies. If improvements aren't observed, grantees are then placed on restricted grant draw-down, which means they must seek special permission to use grant funds. This process ensures that HRSA staff has close scrutiny of grantees activities until their performance improves. Concomitant with this process, HRSA's Division of Financial Integrity conducts pre-and post-award review of grant applicants' accounting systems and also ensures that grantees are strictly held to OMB cost principles. All Federal managers are on performance plans that hold them accountable for cost, schedule and performance results cascading from HRSA's Administrator through the Director of the Office of Health Information Technology, to the Director of the Office for the Advancement of Telehealth (OAT), who is responsible for managing all OAT programs, including the TNGP program. The Director supervises all project officers who have day-to-day programmatic oversight of the TNGP grantees. These non-managerial employees are held accountable by the Employee Management System Summary Rating system, which includes the critical job elements used for rating the performance of each program staff. Each year, employees are rated based on their performance in managing the TNGP program, with 6-month reviews conducted in-between. The Program has relieved staff of their duties when performance standards were not met.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Program funds are completely obligated each year as mandated by federal appropriations. Spending plans are developed and approved by Bureau and HRSA leadership, and the plans are monitored by HRSA program and budget officials to track costs and report status of funds. HRSA's Office of Grants Management is responsible for the fiscal oversight of grant awards. Program officials work closely with the Office of Grants Management in this regard, including setting adequate and program-appropriate procedures for reporting actual expenditures, comparing them against the intended use, and taking timely and appropriate action to initiate and audit action should funds not be spent as intended. HRSA ensures that program funds are spent for the intended purpose by requiring budgets as part of the mandatory grantee Progress Reports. These budgets are reviewed for any problems that are then addressed by the grantee's project officer, as necessary. HRSA has established an independent Office of Program Review (OPR) to provide oversight of grant projects. OPR serves as the agency's focal point for reviewing and enhancing the performance of HRSA-funded programs within communities and States. On a regularly scheduled basis, HRSA grantees are required to participate in a performance review of their HRSA funded program(s) by a review team from one of the ten OPR regional divisions. Grantees should expect to participate in a performance review at some point during their grant project period. When a grantee receives more than one HRSA grant, each of the grantee's HRSA funded programs will be reviewed during the same performance review. The purpose of performance review is to improve the performance of HRSA funded programs. Through systematic pre-site and on-site analysis, OPR works collaboratively with grantees and HRSA Bureaus/Offices to measure program performance, analyze the factors impacting performance, and identify effective strategies and partnerships to improve program performance, with a particular focus on outcomes. Upon completion of the performance review, grantees are expected to prepare an Action Plan that identifies key actions to improve program performance as well as address any identified program requirement issues. Performance reviews also provide direct feedback to the agency about the impact of HRSA policies on program implementation and performance within communities and States.

Evidence: The website www.hrsa.gov/performancereview provides the full HRSA performance review protocol and guide (in pdf format) that is used by OPR staff to review each HRSA grantee. The website includes an action plan template that outlines a structure for monitoring performance improvement actions that could be taken by a grantee or by HRSA staff, and an outline for monitoring technical assistance for the grantees should it be found necessary. From the website: "To assure that all HRSA funded programs are accomplishing their intended purposes, HRSA continuously tracks and analyzes the performance of its grantees using this protocol. On a regularly scheduled basis, HRSA grantees are comprehensively reviewed by a review team from one of the ten OPR regional divisions." The schedule of grantee reviews is also posted.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: HRSA has instituted centralized peer review and grants management to provide more efficient management of grants. Formerly grants management was conducted by each component of HRSA and each component managed its own review process. All grants are competitively reviewed by outside experts to ensure that the most competitive projects are selected. HRSA has moved to an entirely electronic application process, requiring all applicants to submit applications in "grants.gov." In addition to regular monitoring of grants by project officers, HRSA has instituted a separate division, the Division of Independent Review, that works across the agency to ensure that grantees are operating efficiently and effectively in light of program goals. This Division provides an independent perspective on grantee performance, while working closely with program offices to improve the efficiency of program monitoring of grantees. In addition, the TNPG program upgraded its grantee data reporting from paper to Excel spreadsheets, and more recently upgraded to a web-based reporting system to improve efficiency of data reporting. Telehealth program Efficiency Measure: Expand the number of sites+services that provide access to health care as a result of the TNGP program per federal program dollar expended. This is expressed as the "Sites+Services Factor per $1 million program dollars," which is calculated as follows: Numerator = the number of new sites plus number of new services divided by the Denominator = annual federal cost of TNGP grant activities.

Evidence: The Telehealth program's efficiency measure captures the fundamental underlying mission of the program: namely to expand access to services in underserved communities at an affordable cost. This is a consistent indicator that can be tracked over time. For example, in the third year of funding for the 2003 Telehealth Network Grant Program cohort, grantees provided 137 services to 274 rural community sites for a Service-Site Factor equal to 411.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Office of the Advancement of Telehealth (OAT) coordinates its Telehealth Network Grant Program (TNGP) with other telehealth-related Federal programs in a variety of ways. It requires its grantees to apply for the FCC's Universal Service Program (USP), which provides subsidies for telecommunications costs, thus allowing grantees to stretch their limited grant dollars to provide more services and/or more effective evaluation of their activities. The TNGP also coordinates with the federal Distance Learning/Telemedicine Program of the Rural Service Program in the Department of Agriculture to maximize use of federal dollars. This program provides grants and loans for telehealth equipment and telecommunications support. As with the USP, OAT coordinates the TNGP program design and implementation with the efforts of this program, which results in a leveraging of resources to promote telehealth adoption by grantees. Some states have also chosen to build on the successes of the TNGP, wherein the Federal dollars are supplemented by state dollars, often to provide telecommunications infrastructure, which the TNGP does not support. OAT chairs the Joint Working Group on telemedicine, a staff level working group that allows representatives from many Federal agencies (e.g., other Department of Health and Human Services agencies, Appalachian Regional Commission, NASA, Department of Defense, Department of Agriculture, Department of Commerce, etc) to routinely share information about their programs and to work together to overcome challenges faced in the telehealth field. This is a vehicle by which OAT works to ensure that it is not duplicating other Federal initiatives or awarding grants to organizations for activities that are duplicative of other programs. Finally, the TNGP encourages its grantees to become active in the American Telemedicine Association, as a means of educating a wider audience about their programs.

Evidence: The TNGP has historically worked closely with other federal programs to ensure that duplication of effort does not take place. In 2003, the first year of funding under the TNGP program, HRSA program officials compared lists of projects to be considered for funding with other Agencies, including the Appalachian Regional Commission, USDA's Rural Utilities Service, Commerce's Technology Opportunities Program, and others, to ensure that no duplication of funding existed with respect to the 15 projects that were ultimately selected. Much of this interaction was accomplished under the auspices of the federal Joint Working Group on Telemedicine (JWGT) which the Director, Office for the Advancement of Telehealth chairs. It also important to point out that the Telehealth Network Grant Program (TNGP) is subject to the provisions of Executive Order 12372, (Intergovernmental Review) as implemented by 45 CFR 100. All applicants are required to submit applications to the single state point of contact as part of the application process. Executive Order 12372 allows States the option of setting up a system for reviewing applications from within their States for assistance under certain Federal programs. ( Executive Order 12372, "Intergovernmental Review of Federal Programs," was issued with the desire to foster the intergovernmental partnership and strengthen federalism by relying on State and local processes for the coordination and review of proposed Federal financial assistance and direct Federal development. The Order allows each State to designate an entity to perform this function.) Moreover, the TNGP authorizing legislation requires applicants to prepare their application in coordination with the relevant state office of rural health or other relevant agency in their states.

Does the program use strong financial management practices?

Explanation: In 2005, HHS received a material control weakness for its financial systems and processes. HRSA contributes to the material internal control weakness identified in the 2005 HHS audit. HHS is in the process of resolving these weaknesses by replacing existing accounting systems within HHS with the Unified Financial Management System (UFMS). UFMS is scheduled to be operational for HRSA in October 2006.

Evidence: Since 2003, HRSA has been not been included in a consolidated HHS audit. In a 2005 audit of HHS, Ernest and Young found a material weakness in HHS financial systems and processes. In particular, the audit found: Documentation regarding significant accounting events, recording of non-routine transactions and post-closing adjustments, as well as correction and other adjustments made in connection with data conversion issues must be strengthened. Processes to prepare financial statements need improvement. Financial systems are not FFMIA compliant. Weaknesses were identified in Department/Operating Division Periodic Analysis, Oversight and Reconciliations In addition, the audit found PSC's DFP CORE accounting system, which supports the activities of HRSA, did not facilitate the preparation of timely financial statements and did not have an efficient mechanism in place to compile accounting statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The UFMS will improve funds control and monitoring and provide real-time data. In addition to streamlining the accounting process, HHS monitors funds received through annual Independent Financial Audits from grantees.

Evidence: To address management deficiencies, HRSA developed a baseline assessment of grantees to provide information about the overall strengths and weaknesses within the program. In 2005, HRSA implemented a web-based data collection system through the Electronic Handbook on the HRSA GEMS site to improve the data quality and elements collected. HRSA also held a TA conference call with consultants presenting elements of health care and business plans to incorporate program planning and provide HRSA program staff concrete information for grantee goals.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: HRSA operates a fair and open TGNP competition. HRSA's Division of Independent Review is responsible for managing objective reviews within HRSA. Applications competing for federal funds receive an objective and independent review performed by a committee of experts qualified by training and experience in particular fields or disciplines related to the program being reviewed. In selecting review committee members, other factors in addition to training and experience may be considered to improve the balance of the committee, e.g., geographic distribution. Each reviewer is screened to avoid conflicts of interest and is responsible for providing an objective, unbiased evaluation based on the review criteria noted above. The committee provides expert advice on the merits of each application to program officials responsible for final selections for award. Applications that pass the initial HRSA eligibility screening are reviewed and rated by a panel based on the program elements and review criteria presented in relevant sections of the TNGP Program Guidance. The review criteria are designed to enable the review panel to assess the quality of a proposed project and determine the likelihood of its success. The criteria are closely related to each other and are considered as a whole in judging the overall quality of an application. Applicants are carefully screened and held to very high standards. On each criterion, they are judged against a specific set of standards on a 4-point rating scale: outstanding, successful, marginal, or weak [See Draft Review Criteria]. Historically, only grantees with average standardized scores of 80 percent or higher have been considered for funding. HRSA also makes an extensive effort to reach out to the telehealth community so as to promote the participation of new grantees into the program. OAT works closely with the American Telemedicine Association (ATA) and other organizations to publicize the availability of funds. The availability of funds is also publicized at meetings where OAT staff members speak.

Evidence: 1) The Telehealth Network Grant Program (TNGP) 2006 Program Guidance - Includes clear criteria used for assessing applicants on a competitive basis. 2) HRSA's Division of Independent Review conducts objective, independent reviews of all applications for HRSA grant programs (refer to DIR policies and procedures.)

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program has a number of financial and programmatic oversight practices in place to provide sufficient knowledge of grantee activities. Through 6-month and annual progress reports, grantees report on their progress towards achieving sustainability, barriers faced, and lessons learned to inform the broader community engaged in telehealth. OAT staff carefully monitors these reports and establish corrective actions when necessary. HRSA also has reporting systems in place at both the program level and in the Office of Grants Management to document grantees' use of funds in eligible activity categories. See sections 3.1 and 3.2 for specific information collected from grantees to facilitate the oversight process. In addition, HRSA has established the Office of Performance Review (OPR). OPR serves as the agency's focal point for reviewing and enhancing the performance of HRSA funded programs within communities and States. On a regularly scheduled basis, HRSA grantees are required to participate in a performance review of their HRSA funded program(s) by a review team from one of the ten OPR regional divisions. Grantees can expect to participate in a performance review at some point during their project period. When a grantee receives more than one HRSA grant, each of the grantee's HRSA funded programs will be reviewed during the same performance review. The purpose of performance review is to improve the performance of HRSA funded programs. Through systematic pre-site and on-site analysis, OPR works collaboratively with grantees and HRSA Bureaus/Offices to measure program performance, analyze the factors impacting performance, and identify effective strategies and partnerships to improve program performance, with a particular focus on outcomes. Upon completion of the performance review, grantees are expected to prepare an Action Plan that identifies key actions to improve program performance as well as addresses any identified program requirement issues. Performance reviews also provide direct feedback to the agency about the impact of HRSA policies on program implementation and performance within communities and States.

Evidence: 1) The Telehealth Network Grant Program (TNGP) 2006 Program Guidance requires grantees to coordinate their plans with other related entities, details of which can be found in the following documents; 2) Progress Report template: provides the types of information that is tracked by the Program and; 3) Final OAT GPRA Report: outlines the types of data tracked by the grantees for reporting on their progress; 4) HRSA has established the Office of Performance Review (OPR). For additional information on performance reviews, please visit: http://www.hrsa.gov/performancereview.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects required program elements from grantees on a semi and annual basis. This information is then compiled into the Telehealth Grantee Directory, which is available on HRSA's website at http://www.hrsa.gov/telehealth/grantee.htm. In addition to publishing the Directory on its website, the program distributes the Directory on CD ROM and makes hard copies available. The Grantee Directory consists of a detailed compendium of grantee performance information. The publicly-accessed Directory provides a comprehensive repository of public data for each individual telehealth grantee that is updated annually. It includes extensive reference charts with specific performance-related data for each grantee project as follows: project components, e.g., project foci -- clinical telemedicine, distance learning, or electronic medical records; major types of clinical services provided in at least 25 categories (e.g., cardiology, mental health, dermatology, etc.); sources of reimbursement for telehealth services provided, (e.g., Medicare, Medicaid, private payer, other); modes of technology and transmission; the types/number of sites where services are provided, (at least 11 categories - e.g., hospitals, nursing homes, community health centers, physician offices); number of health professional shortage areas (HPSAs) and/or medically underserved areas (MUAs) served with the population in those areas; activities with respect to homeland security; and population demographics. The public can turn to a more detailed description for each project that contains network partner information, the project purpose, outcomes expected, the service area, services provided, equipment and transmission. The detailed description also contains contact information with the names of key individuals to contact, address and phone numbers, and email address, as well as the project website link where the public may obtain more detailed project information. The data from the Grantee Directory is used to track grantee performance on a number of parameters. For example, data from the Directory is used to measure the program's efficiency for sites+services provided per the grant dollars expended and is used to track over time the scope of services provided and access to services for underserved populations.

Evidence: Evidence: The Directory can be found at the OAT homepage: http://www.hrsa.gov/telehealth/grantee.htm

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: One of the program's key objectives is to make services available through telehealth that otherwise would not be provided to a community in the absence of the program; and subsequently the program's first long-term measure reflects this: Long-term measure #1: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. The program has provided data collected from Telehealth grantees from 6-month progress reporting periods from October 2002 through February 2005 (the last period for which current data have been analyzed). Number of Communities with Psychiatric Services Available Only Through the Telehealth Network Grant Program Specialty Oct02(P1) Mar03(P2) Sept03(P3) Mar04(P4) Oct04-Apr05(P5) Pediatric/ 50 47 43 81 101 Adolescent Psychiatry Adult 57 39 37 51 60 Psychiatry Periods 1 and 2 reflect the experience of the previous cohort of grantees, whereas beginning in Period 3, the achievements of the current cohort are reported. Thus P3 represents the period September 2003-February 2004, the first reporting period for the current grant cycle. In summary, this data indicates that the Telehealth program has been successful in bringing specific health services (in this case psychiatric services) to increasing numbers of communities that would otherwise be unable to provide such specialized health care services if not for the Telehealth program. This data illustrates the program's progress in achieving its first long-term performance goal. The program's second long-term measure, "increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%)," is new, and as of yet, the program has not been systematically abstracting and synthesizing these data from grantees to demonstrate the program's progress in achieving this particular goal. However, as part of routine clinical practice, these measures are taken by grantees who serve diabetic patients and two current TNGP grantees were able to provide these data: Duke University and the University of Tennessee. At Duke, researchers are finding that the introduction of telehealth technologies as an adjunctive tool results in a higher proportion of diabetic patients attaining HgbA1c control. The program at the University of Tennessee observed wide variability in its ability to effect HbgA1c control of patients enrolled in its program thus confirming the ambitiousness of achieving this measure. In moving forward, the program will be providing technical assistance to grantees to assist them in reporting these data systematically to ensure that the TNGP program can report data on this long term measure. The program has been tracking data for its third long-term measure for many years (by following graduates of the TNGP predecessor program which first received funding in 1997 through to the present) using its listserv. Due to the unique nature of this program, (funding three-year telehealth demonstration grants to improve access to care), this measure illustrates the program's success in achieving sustainability over a long period of time - the percent of TNGP grantees that continue to offer services after the TNGP funding has ended - and presently (at baseline) this percentage is at 100%.

Evidence: Abt Associates Inc. Report, Office for the Advancement of Telehealth - Final GPRA Report. January 12, 2006. Page 17. These ambitious targets take into account the environmental challenges experienced by grantees even when grant funds are available, e.g. difficulty in obtaining services due to the shortage in the supply of mental health specialists and continued low reimbursement for these services. The program typically funds 15 or fewer demonstration grants in three-year cycles - these data include the current cohort ending in 2006 and two additional cohorts ending in 2009 and 2012. Data Sources: 6-month performance monitoring progress reports from grantees. The performance monitoring system that OAT uses is web-based; twice each year OAT grantees submit data using consistent definitions and reporting guidelines.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The same data provided to demonstrate that the program is achieving its first long-term goal, is used to demonstrate the program's progress towards achieving it's first annual performance goal: Annual measure #1: Increase the number of communities that have access to pediatric and adolescent, and adult mental health services where access did not exist in the community prior to the TNGP grant. The program has provided data collected from Telehealth grantees from 6-month progress reporting periods from October 2002 through February 2005 (the last period for which current data have been analyzed). Communities with Psychiatric Services Available Only Through the Telehealth Network Grant Program Specialty Oct02(P1) Mar03(P2) Sept03(P3) Mar04(P4) Oct04-Apr05(P5) Pediatric/ 50 47 43 81 101 Adolescent Psychiatry Adult 57 39 37 51 60 Psychiatry Periods 1 and 2 reflect the experience of the previous cohort of grantees, whereas beginning in Period 3, the achievements of the current cohort are reported. Thus P3 represents the period September 2003-February 2004, the first reporting period for the current grant cycle. In summary, this data indicates that the Telehealth program has been successful in bringing specific health services (in this case psychiatric services) to increasing numbers of communities that would otherwise be unable to provide such specialized health care services if not for the Telehealth program. This data illustrates the program's progress in achieving its annual performance goals. For the program's second annual measure: Annual measure #2: Increase the proportion of diabetic patients enrolled in a telehealth diabetes case management program with ideal glycemic control (defined as hemoglobin A1c at or below 7%) in each year of the three-year grant cycle. As was the case with the second long-term measure, the program has limited historical data that can demonstrate its progress toward achieving their second annual measure at this time. For the program's third annual measure: Annual measure #3: Expand the number of telehealth services (e.g., dermatology, cardiology) and the number of sites where services are available as a result of the TNGP program. (This is calculated as the sum of services and sites [service+site factor] across all grantees in each year of a three-year grant.) The baseline and targets for this measure are as follows: Year Number of Sites-Services 2006 489 Baseline 2006-2007 (first year) 900 Target 2007-2008 (second year) 952 Target 2008-2009 (third year) 978 Target

Evidence: Abt Associates Inc. Report, Office for the Advancement of Telehealth - Final GPRA Report. January 12, 2006. Page 17. These ambitious targets take into account the environmental challenges experienced by grantees even when grant funds are available, e.g. difficulty in obtaining services due to the shortage in the supply of mental health specialists and continued low reimbursement for these services. The program typically funds 15 or fewer demonstration grants in three-year cycles - these data include the current cohort ending in 2006 and two additional cohorts ending in 2009 and 2012. Data Sources: 6-month performance monitoring progress reports from grantees. The performance monitoring system that OAT uses is web-based; twice each year OAT grantees submit data using consistent definitions and reporting guidelines.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program's efficiency measure illustrates the number of sites and services made available to people who otherwise did not have access to them per million dollars spent. This is expressed as the "Sites+Services Factor per $1 million program dollars," which is calculated as follows: Numerator = the number of new sites+services divided by the Denominator = annual federal cost of TNGP grants.

Evidence: The program measures efficiency by measuring the number of services and sites that provide access to health care as a result of the TNGP program per federal program dollar expended. From 2004 to 2005 this measure has increased from 108 sites and services (411/$3.8 million) to 122 (463/ $3.8 million) created per $1 million spent and is anticipated to rise to 129 in 2006. Thus for less than $10,000 per year, a community in which a grantee introduces services via telehealth gains access to at least one health care service (and typically more than one once the telehealth capability is established) that it did not have access to prior to the program.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: While there are other programs designed to supply telehealth-related equipment and support transmission costs, the TNGP is the only one that provides funds that directly support organizational development. Because of the program's unique purpose, there are no other programs (public or private) which can be used to accurately compare the performance of the Telehealth Network Grant Program.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Abt Associates was commissioned by HRSA to study grantee performance based on uniform measures that could be applied across a diverse group of grantees. A draft manuscript of this report has been provided which summarizes the findings of the report, some of which are listed below. Abt collected data across all the TNGP grantees for a 3-year period providing analyses of the performance on a wide variety of measures. The information presented in this report reflects the activity of all 19 TNGP programs funded by the Office of the Advancement of Telehealth (OAT) each year. The report provides a detailed summary of the program's accomplishments and demonstrates that the program has been effective and is achieving results. In particular, the report provides data that demonstrates that the TNGP program has been successful in: 1. Improving access to needed services 2. Reducing rural practitioner isolation 3. Improving health system productivity and efficiency Some key findings of the report are as follows: ?? Between October 2002 to February 2005 (divided into six month periods, 1-5) hospital outpatient and patient home volume grew by over 5,500. ?? Among the more than 200 rural communities served by OAT grantees, many receive basic and specialty services that would not otherwise be available. ?? Pediatric/Adolescent Psychiatry and Dermatology are the most common specialties service provided through OAT that would not otherwise be possible in these communities. o 66 communities had no other way to offer dermatology services in period 1, and adult psychiatry was only available through telemedicine in 57 communities. ?? Adult Primary Care, a basic service, was available to six rural communities only through OAT-funded telemedicine in period 2; with no other source of care available. ?? During the two and a half years, there were 3,232 sessions conducted over OAT grantee networks for the purposes of providing required supervision of trainees in formal educational programs. ?? Over 79% of practitioners who referred patients during the early part of the year continued to refer patients in the latter part of the year. ?? Another 12% of the referring practitioners in the latter part of the year were new and had not referred patients earlier. ?? There were almost no practitioners who stopped referring patients after having done so earlier, other than the few whose practice/location changed (<1%). ?? Hundreds of nurse hours were saved through reduced travel (again, these data reflect only sessions that would otherwise have been conducted in-person). These hours translate into thousands of dollars of salary that are saved. ?? In reality, rather than simply reducing nursing costs, the nurses who spent less time driving were able to care for more patients - stretching the short supply of nurses to improve access for more patients.

Evidence: Office for the Advancement of Telehealth - Final Government Performance and Review Act (GPRA) Report, prepared by Abt. Associates Inc., January 12th, 2006