Detailed Information on the
Protection and Advocacy for Individuals with Mental Illness Assessment

Conducting an independent evaluation of the program.

Refinement of data reporting across grantees

Providing technical assistance to PAIMI Programs in regard to the right to access facilities, consumers, and information through the National Disability Rights Network (NDRN)

Developing baselines and targets for measure, "Percentage of interventions on behalf of groups of PAIMI-eligible individuals that are concluded successfully."

Providing grantees with guidelines as to how to calculate the number of PAIMI-eligible individuals impacted, including a definition of "impacted," so that reporting on efficiency measures across States will be more consistent in the future.

Measure: Percentage of interventions on behalf of groups of PAIMI-eligible individuals that were concluded successfully.

Explanation:This measure is under development and is intended to capture the more systemic impacts of the program. The program is working with grantees to develop a list of outcomes to be included in the annual Program Performance Report that would reflect progress on this measure.

Measure: Increase in the number of persons served by the PAIMI program

Explanation:Data is collected from standardized annual program performance reports.

Measure: Increased percentage of complaints of alleged abuse and neglect, substantiated and not withdrawn by the client, that resulted in positive change for the client in her/his environment, community, or facility, as a result of PAIMI involvement.

Explanation:Data is derived from standardized annual Program Performance Reports in which grantees list each of their cases by type and select from a list of 6 pre-defined possible outcomes. The program calculates this measures by dividing the number of closed cases determined by PAIMI staff as having been resolved in the client's favor by the total number of closed cases. The program is in the process of revising the performance report format so that only one outcome is reported per cleint and that the pre-defined outcomes that grantees can select from are more closely aligned with direct outcomes (i.e. investigations, regulation, and policy changes will no longer considered an outcome for this measure). Actuals and targets may change from 2006 onward to reflect this change.

Measure: Cost per 1,000 individuals served/impacted

Explanation:Data is derived from standardized annual Program Performance Reports in which grantees estimate the potential number of individuals impacted through a pre-defined list of 7 possible interventions (e.g., group advocacy non-litigation, facility monitoring services, class litigation). This measure is calculated by using the total PAIMI allotment as the numerator and the total number of persons served/impacted as the denominator. The program has committed to providing grantees with guidelines as to how to calculate number of PAIMI-eligible individuals impacted so that reporting on this measure across States will be more consistent in the future. The guidelines will include a definition of "impacted".

Measure: Ratio of persons served/impacted per activity/intervention

Explanation:Data is derived from standardized annual program performance reports in which grantees estimate the potential number of individuals impacted through a pre-defined list of 7 possible interventions (e.g., group advocacy non-litigation, facility monitoring services, class litigation). This measure is calculated by using the total number of persons served/impacted as the numerator and the total number of complaints addressed and intervention strategies conducted as the denominator. The program has committed to providing grantees with guidelines as to how to calculate number of PAIMI-eligible individuals impacted so that reporting on this measure across States will be more consistant in the future. The guidelines will include a definition of "impacted."

Measure: Increased percentage of complaints of alleged rights violations, substantiated and not withdrawn by client, that resulted in positive change through the restoration of client rights, expansion or maintenance of personal decision-making, or elimination of other barriers to personal decision-making, as a result of PAIMI involvement.

Explanation:Data is derived from standardized annual Program Performance Reports in which grantees list each of their cases by type and select from a list of 4 pre-defined possible outcomes. The program calculates this measures by dividing the number of closed cases determined by PAIMI staff as having been resolved in the client's favor by the total number of closed cases. The program is in the process of revising the performance report format so that only one outcome is reported per cleint and that the pre-defined outcomes that grantees can select from are more closely aligned with direct outcomes (i.e. regulation and policy changes). Actuals and targets may change from 2006 onward to reflect this change.

Is the program purpose clear?

Explanation: The purpose of Protection and Advocacy for Individuals with Mental Illness (PAIMI) Program is to extend protections to adults with severe mental illness and children with serious emotional impairment through establishing State systems to advocate for their rights. Congress determined that adults with significant mental illness and children with serious emotional impairment were vulnerable to abuse and neglect in their treatment or care and that State systems for monitoring compliance with respect to the rights of these individuals varied widely and were frequently inadequate. The program seeks to protect these individuals from abuse, neglect, and infringements on their civil rights by establishing independent protection and advocacy agencies (P&As) in each of the 50 States and six territories (American Indian Consortium, America Samoa, Guam, Northern Mariana Islands, Puerto Rico, and the U.S. Virgin Islands).

Evidence: The Protection and Advocacy for Individuals with Mental Illness (PAIMI) Act [42 USC 10081] extends the protections of the Developmental Disabilities Assistance and Bill of Rights Act (DD Act) [42 USC 6001] to individuals with a mental illness. The DD Act established a federally mandated network of independent disabilities rights protection and advocacy agencies in States and territories. The PAIMI Act requires the Secretary of Health and Human Services to make allotments to the states and territories to administer protection and advocacy agencies for individuals with mental illness and children with serious emotional impairment. The PAIMI Act does not create any rights to care and treatment; rather, the Act is intended to secure for the mentally ill their rights under existing statute and other forces of law.

Does the program address a specific and existing problem, interest, or need?

Explanation: The PAIMI Program addresses the existing and ongoing problem of the need to protect adults with significant mental illness and children with serious emotional impairment from abuse, serious injury, and neglect, as well as inadequate/inappropriate treatment, nutrition, clothing, health care, and discharge planning. In 2003, an estimated 12 million adults had a serious mental illness and 5 million children had a serious emotional disturbance. States frequently lack independent, unencumbered monitoring systems for these individuals.

Evidence: In 2002, the State P&As investigated 972 deaths. In 1998, the Hartford Courant reported that improper use of seclusion and restraint in State mental health treatment facilities may have contributed to the death of patients (Hartford Courant, October 11-15, 1998). In a review of State seclusion and restraint policies, the Government Accountability Office (GAO) found that States had varying degrees of oversight and regulation (Mental Health: Extent of Risk from Seclusion and Restraint is Unknown, October 26, 1999). Another GAO report found that conflicts of interest in overseeing State-run institutions affected States' ability to conduct objective investigations (Medicaid--Oversight of Institutions for the Mentally Retarded Should be Strengthened, September, 1996). The Department of Health and Human Services (HHS) Office of the Inspector General (OIG) also found that States varied in how they handled, responded to, and reported incidents of neglect and abuse (Reporting Abuses of Persons with Disabilities, May 2001).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: While other Federal, State, and private programs have an impact on the same population that PAIMI serves, PAIMI is the largest, most wide-ranging program that provides protection and advocacy for individuals with mental illness. Unlike other programs, the PAIMI Program operates in every state, is authorized to access public and private treatment facility records, and is authorized to pursue legal remedies, including individual and class actions on behalf of their clients irrespective of age and income.

Evidence: Federal programs that deal with similar issues include: 1) Department of Justice (DOJ), which has the authority to investigate conditions of confinement at State and local government institutions. However, DOJ is not authorized to represent individuals or to address specific individual cases. 2) The Center for Medicaid and Medicare Services (CMS) has oversight authority over institutions that provide services covered by their programs. However, many persons with disabilities do not reside in facilities subject to CMS standards (HHS OIG, May 2001). States also deal with similar issues, but in many states those in charge of investigating violations are not completely independent of the State agency that provides services. Finally, various private groups address advocacy issues, including the American Civil Liberties Union, the Bazelon Center for Mental Health Law, the Legal Aid Society, and the Legal Services Corporation. However, actions by these groups are more targeted and do not have the same scale or scope as the PAIMI Program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Overall, the program is well-designed to meet its objectives and performance goals. The formula grant is suitable for population targeted, affirms that States have primary responsibility for the targeted population and the purpose of the grant, and provides for a vehicle to carryout the task that is independent. The program could improve its effectiveness in reaching its targeted population if States were required to report incidents of abuse and neglect in State facilities. In addition, the program could more effectively use its resources if States had greater understanding of and acknowledged the program's rights to access facilities, consumers, and information involved in incidents they are forced to investigate.

Evidence: HHS OIG found that P&A "advocacy efforts have a major impact on how States and facilities protect the rights of persons with disabilities." The OIG also found that P&As are limited by restricted access to information by State agencies. The OIG noted that "The HHS does not have the authority to establish standards or mandate processes for State agencies to identify, investigate, and resolve incidents involving persons with disabilities." (OIG 2001)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Funds are targeted to meet the program purpose. PAIMI is a formula grant program that allots funds to States based on population and per capita income. There is a minimum allotment to all States, and those States with fewer resources receive more generous allotments. Funds are provided to governor-designated-agencies that are independent from the States. The program utilizes existing P&A State agencies (there are 8 P&A programs that each address identified disbility groups). The PAIMI Act targets specific populations with severe mental illness who are at risk for abuse, neglect, and civil rights violations while residing in care and treatments facilities as well as in the community.

Evidence: The Protection and Advocacy for Individuals with Mental Illness (PAIMI) Act [42 USC 10081] provides for the formula by which the program distributes grants to the States. The Act also lays out the critera of who is eligible for assistance under the PAIMI program.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program currently has two long-term measures that address program impacts on individuals that have been abused, neglected, or have had their civil rights violated. In addition, the program is developing a long-term measure to capture the systemic benefits by measuring the impact of interventions for groups of individuals.

Evidence: The PAIMI Program's three long term measures are: 1) Increased percentage of complaints of alleged abuse and neglect, that are substantiated and not withdrawn by client, and that resulted in positive change for the client in their environment, community, or facility as a result of PAIMI involvement; 2) Increased percentage of complaints of alleged rights violations that are substantiated and not withdrawn by the client, and that resulted in positive change through restoration of client rights, expansion or maintenance of personal decision-making, or elimination of other barriers to personal decision-making. 3) Percentage of interventions on behalf of groups of PAIMI-eligible individuals that were concluded successfully (under development).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established ambitious targets and timeframes for its long-term measures.

Evidence: The program's goal is to successfully resolve 88% of abuse cases and 94% of neglect cases in 2012 (an increase from the respective baselines of 78% and 86% in 2003). The program also has a goal of successfully resolving 97% of civil rights cases by FY 2012 (an increase from the baseline of 78% in 2003). The program will establish a FY 2012 target for its long-term measure on the number of interventions concluded successfully.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has four annual measures. Three of its annual measures are the same as its long-term measures described in question 2.1. In addition, there is an annual measure to increase the number of persons served by the PAIMI program.

Evidence: The annual outcome measures are discussed in question 2.1. The numbers served measure reflects the reach of the program.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baselines and targets for all of its annual measures.

Evidence: The program's FY 2005 goals are: To successfully resolve 83% of abuse cases and 88% of neglect cases (an increase from the respective baselines of 78% and 86% in 2003); To successfully resolve 95% of civil rights cases (an increase from the baseline of 78% in 2003). The program will establish a baseline and targets for its annual measure on the number of interventions concluded successfully.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantees commit to long-term goals of the program in applications and the program's performance data comes directly from grantees annual performance reports. The program recently worked with grantees to develop program goals that more meaningfully reflect the activities of the program.

Evidence: The program recently convened a PAIMI Performance Indicators Workgroup of program officials and grantees to better identify performance measures that reflect program outcomes. The PAIMI Performance Measures Workgroup is a federal-state partnership to develop more meaningful performance measures. The group has reviewed current PAIMI GPRA measures and recommended a three tier approach that would capture 1) impact on individuals, 2) impact on targeted groups, and 3) more systemic outcomes from existing data. In this way a broad range of PAIMI outcomes and the complexities of the program are reflected. These new measures are being implemented among all grantees. Grantees commit to program goals annually in standardized Applications and Program Performance Reports.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There has never been an independent evaluation of the PAIMI Program. The program has recently contracted with the Human Services Research Institute to provide the first in what is expected to be several tri-annual evaluations of the program. The evaluability assessment was received in June 2005, and the independent evaluation is expected in July 2006.

Evidence: In Contract Number 280-03-2100; Task Order 280-03-2102, the contractor agrees to 1) Assess the feasibility of evaluating the PAIMI Program and recommend evaluation design options, including identifying specific performance indicators; 2) Conduct a comprehensive evaluation of the structure, processes, and outcomes associated with the program, including how the program is accomplishing its mission and long-term goals; and 3) Provide a written evaluation report that highlights program strengths, challenges, and opportunities for improvement.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program has made progress in reporting performance information in its budget, but requests are still not explicitly tied to accomplishments of annual and long-term goals. Specifically, the relationship between annual and long-term targets and budget resources is not clear.

Evidence: SAMHSA's 2006 Congressional Justification lays out and discusses the PAIMI Program budget and performance information.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In the past year, the program met with PAIMI grantees to develop more meaningful performance measures. As a result the program developed new outcome-oriented measures on which the the program is currently reporting. In addition, the program is attempting to reduce reporting burdens for P&As Systems by creating one annual Program Performance Report (PPR) for all programs to use. Finally, the program will include all of its outcome and efficiency measures in its next budget submission.

Evidence: Interagency Working Group P&A Common Definitions Workgroup meets monthly to develop a new tool by which all P&A programs can report performance information. In FY 2005, the PAIMI Performance Indicators Workgroup came to the consensus that existing performance measures did not adequately capture the broad range of PAIMI activities and the complexities of the program. In response the group recommended a three-tiered approach that would capture 1) impact on individuals; 2) impact on targeted groups; and 3) more systemic outcomes.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: State P&A systems regularly submit performance plans and reports to the PAIMI Program. Data collected from these grantees are used to set baselines for the program's performance measures and to identify technical assistance needs of grantees. In addition, the program conducts regular on-site monitoring and makes recommendations to grantees on how to improve performance. Grantees that are found not to be in compliance with program rules and regulations are designated "high risk" and are required to submit supporting information and to seek approval for all payment requests. High risk grantees are placed on Corrective Action Plans and are closely monitored by program staff.

Evidence: State P&A systems annually submit an application that describes program priorities, expected outcomes, and proposed budgets. In addition P&As submit annual Program Performance Reports that measures performance and expenditures against these applications. These reports quantify program outcomes/ Performance is monitored by peer review of Program Performance Reports and by SAMHSA site visits. SAMHSA provides grantees with written monitoring reports after each site visit which include recommendations so the grantees can improve program performance and management.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The SAMHSA Administrator and PAIMI program managers' performance contracts establish clearly defined performance standards. Grantees identify key managers in annual Applications and Program Performance Reports. Even though grants are distributed by formula, the program monitors grantees and reserves the right to place grantees on "high risk" status for programmatic and/or fiscal mismanagement. "High risk" grantees are placed on Corrective Action Plans that includes specific actions, responsible individuals, and target dates for remedying program deficiencies.

Evidence: The SAMHSA's Administrator and the PAIMI Program Mangers' contracts include specific goals for improving the program performance including developing evaluations and peer review models to better identify deficiencies and improve program performance. The PAIMI Program manager contract also includes targets for number of site visits (which include the issuance of a program monitoring report that recommends program performance). In FY 2005, one grantee has been designated as "high risk" for financial management reasons and is required to submit supporting information and to seek approval for all payment requests.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated on an annual basis, usually within 3 days after an application has been reviewed and approved. SAMHSA monitors grants through its SAMHSA Grants Information Management System and the CORE accounting system. In the last 4 years there have been no grantees out of compliance with regard to appropriately obligating and expending PAIMI funds. The PAIMI program ensures that grants are spent for their intended purpose through review of annual applications that include estimated expenditures for the fiscal year as well as annual Program Performance Report, which tracks how funds were actually spent. In addition, grantees file an annual audit report that is completed by independent auditors and filed with HHS.

Evidence: State P&As submit proposed budgets in their annual Applications and disclose how funds were actually spent in their annual Performance Report. SAMHSA Grants Management monitors the obligation and expenditure of individual grantee (PAIMI) funds primarily through the annual Standard Form 269, a federal financial status report that is completed by an independent auditor and filed with HHS.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: PAIMI has two efficiency measures: 1) Cost per 1000 individuals served/impacted and 2) Ratio of persons served/impacted per activity intervention. The PAIMI Program is relatively small, requiring only 2.8 FTEs in SAMHSA, but it competitively outsources its limited number of contracts.

Evidence: Efficiency measures will be used by the program to identify grantee outliers in terms of cost and/or efficiency of providing services who may need further training and technical assistance and to identify practices of efficient grantees that may benefit other grantees. The program's FY 2005 targets for these measures are: To spend an average of $2,200 per 1,000 individual served/impacted (a decrease from a baseline of $6,814 in 2001); To impact an average of 390 individuals per intervention. PAIMI also controls program costs through competitive outsourcing. It competitively sources its the peer review process and monitoringand its contract to conduct a program evaluation.

Does the program collaborate and coordinate effectively with related programs?

Explanation: PAIMI collaborates and coordinates its activities with other P&A programs in the federal government. The PAIMI Program has an interagency agreement with the Administration of Developmental Disabilities--one of the 8 P&A programs in the federal government--to provide training and technical assistance. The interagency agreement is expected to maximize fiscal and staffing resources by providing technical assistance through one contractor who will coordinate all technical assistance and training events for P&A systems and serve as a repository for and the disseminator of information and resource materials. The program meets regularly with other P&As and Federal partners to develop a cross agency annual program performance tool. The PAIMI Program also works with the National Association of Protection and Advocacy Systems (NAPAS), the P&A trade association, to review performance data to identify activities/interventions that are under-represented or common difficulties experienced by local programs.

Evidence: There are 8 P&A programs in the federal government. The other 7 P&A programs advocate for other groups of individuals with major disabilities, for example physical and developmental disabilities. Most States centralize advocacy efforts for these 8 groups in one State P&A agency in order to minimize overhead expenses. Each P&A program has its own funding stream, requirements, and fiscal accounting that grantees use to separate the programs for activity purposes.

Does the program use strong financial management practices?

Explanation: SAMHSA receives regular unqualified audit opinions. The program estimates 94-98% of grantees have received unqualified opinions in the preceding two fiscal years. In addition, the program conducts pre-award accounting system reviews on new grantees to determine whether they have an adequate financial management system. This review is conducted before an award may be made and examines the grantees' fiscal policies and procedures to determine overall adequacy/ability to handle federal funds, as well as relevant financial reports to determine the solvency of the organization. Finally, after awards are distributed SAMHSA conducts onsite monitoring of programs to check whether P&As have adequate internal controls to safeguard federal funds.

Evidence: SAMHSA received unqualified audit opinions in FYs 1998-2002. In FY 2003, SAMHSA's financial statement was consolidated with other HHS operating statements. HHS received unqualified opinions for FYs 2003-2004. All grantees are reviewed under the Single Audit Act in accordance with OMB Circular A-133. The audits are reviewed by HHS after they are received at the Clearinghouse. Any problems identified in the audit are dealt with at the Department level for resolution. SAMHSA is notified with regard to problems and resolutions. The PAIMI Program's monitoring tool for on-site visits includes fiscal prompts that helps SAMHSA identify weaknesses in internal controls.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program does not have any significant management deficiencies.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Grantees submit annual applications that describe proposed objectives, performance measures, expected outcomes, and proposed budget expenditures. PAIMI reviews these applications against grantee annual Program Performance Reports and actual objectives, performance measures, outcomes, and budget expenditures. In addition, the PAIMI program conducts regular on-site monitoring and peer reviews.

Evidence: Grantees are required to submit applications and annual reports; failure to do so results in not receiving grant funds. Grantees are monitored on a rotating basis with approximately 13 percent of grantees monitored each year. Program data is assessed by the PAIMI program officer and two PAIMI data analysts. In 2004, PAIMI contracted an independent evaluation. Results are expected in FY 2006.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects grantee performance data annually in Program Performance Reports. Data from these reports are posted on the Program's website.

Evidence: Performance data by State can be found at http://mentalhealth.samhsa.gov/cmhs/PAIMI/

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Data available since 1999, demonstrates that the program is on track to meet its long-term performance goals. It exceeded its FY 2004 targets for abuse and civil rights complaints. Long-term data shows that the program has increased the number of cases it has successfully resolved. The program did not meet its FY 2004 targets for neglect complaints. The program is not yet reporting on its systemic measure.

Evidence: Between 1999 and 2004, the program's measure for the number of complaints successfully resolved did not distinguish between type of complaints. Data for that measure demonstrates that the total number of complaints successfully resolved increased from 75% in 1999 to 89% in 2004. Data by case type only became available in 2003, and the program revised its measure to reflect case type in 2005. The percentage of abuse complaints resolved positively increased from 78% in FY 2003 to 82% in FY 2004. The percentage of civil rights violations resolved positively increased from 78% in FY 2003 to 95% in FY 2004. The percentage of neglect cases resolved positively decreased from 86% to 82%.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program exceeded its FY 2004 targets for positive outcomes from abuse and civil rights complaints, but made negative progress on its annual target for neglect complaints as discussed in question 4.1. The program did not set targets for its two efficiency measures prior to FY 2005, but historical trends suggests overall decreases in costs and increases in numbers served/impacted (with some negative progress made in FY 2004). The program is not yet reporting on its annual measure of systemic impacts.

Evidence: The percentage of abuse complaints resolved positively increased from 78% in FY 2003 to 82% in FY 2004. The percentage of civil rights violations resolved positively increased from 78% in FY 2003 to 95% in FY 2004. The percentage of neglect cases resolved positively decreased from 86% to 82%

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has demonstrated improved efficiencies in its two efficiency measures over time.

Evidence: Average program costs decreased from $6,814 per 1,000 persons served/impacted in FY 2001 to $2,431 in FY 2004. Numbers of persons served/impacted per intervention increased from 129 in FY 2001 to 354 in FY 2004.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The PAIMI Program is unique in that it is the only program which addresses the protection and advocacy of individuals with mental illness.

Evidence: Protection and Advocacy for Individuals with Mental Illness (PAIMI) Act [42 USC 10081] establishes the authorities of the PAIMI Program.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: There have been no independent evaluations of the PAIMI Program.

Evidence: The PAIMI Program is in the process of undergoing an evaluation.