ADD Request for Proposal
SECTION C – DESCRIPTION/SPECIFICATIONS/WORK STATEMENT
C.1. TITLE
- Developmental Disabilities Program Independent Evaluation Project (DDPIE)
C.2 DEVELOPMENTAL DISABILITIES ASSISTANCE AND BILL OF RIGHTS ACT
- The Administration on Developmental Disabilities (ADD) oversees the implementation of the Developmental Disabilities Assistance and Bill of Rights Act of 2000 (DD Act) (P.L. 106-402) (42 USC 15062). The purpose of the DD Act is to assure that individuals with developmental disabilities and their families participate in the design of and have access to needed community services, individualized supports, and other forms of assistance that promote self-determination, independence, productivity, and integration and inclusion in all facets of community life.
- State Developmental Disabilities Councils;
- State Protection and Advocacy Systems to Protect the Rights of Individuals with Developmental Disabilities; and
- The National Network of University Centers for Excellence in Developmental Disabilities, Education, Research, and Service.
As defined in the DD Act, the term “developmental disabilities” means a severe, chronic disability of an individual that is attributable to a mental or physical impairment or combination of mental and physical impairments that is manifested before the individual attains age 22 and is likely to continue indefinitely. Developmental disabilities result in substantial limitations in three or more of the following functional areas: self-care, receptive and expressive language, learning, mobility, self-direction, capacity for independent living, and capacity for economic self-sufficiency. It is estimated that four million people in America have developmental disabilities.
The DD Act authorizes appropriations for three programs in the States to achieve the purposes of the Act:
- These entities, which comprise the “State DD Network”, are administered by ADD. A brief description of the programs follows:
- State Councils on Developmental Disabilities (55 grantees)
The State Council on Developmental Disabilities (State Council) must engage in advocacy, capacity building, and systemic change activities, such as outreach; training; technical assistance; supporting and educating communities; interagency collaboration and coordination; coordination with related councils, committees, and programs; barrier elimination, systems design, and redesign; coalition development and citizen participation; informing policymakers; and demonstration of new approaches to services and supports. These activities must contribute to a coordinated, consumer- and family-centered, consumer- and family-directed, comprehensive system which includes needed community services, individualized supports, and other forms of assistance that promote self-determination for individuals with developmental disabilities and their families. The State Council has a membership appointed by the Governor of the state and comprises at least 60 percent service consumers (persons with developmental disabilities and their parents or guardians). Forty percent of the membership is service providers and policy makers (state and local levels). Funding is set aside for technical assistance to the State Councils. Funding for FY 2005 totals $72.5 million. - State Protection and Advocacy Systems (56 grantees)
The State Protection and Advocacy System (P&As) provides for the protection and advocacy of legal and human rights. The P&As advocate on behalf of, and provide advocacy services to persons with developmental disabilities in issue areas related to their disabilities, including: education, abuse and neglect, institutional and habilitation services, guardianship issues, and housing issues. The P&As pursue legal, administrative, and other appropriate remedies or approaches to ensure the protection of, and advocacy for, the rights of such individuals within the State who are or who may be eligible for treatment, services, or habilitation, or who are being considered for a change in living arrangements, with particular attention to members of ethnic and racial minority groups. They provide information on and referral to programs and services addressing the needs of individuals with developmental disabilities. The P&As provide individual advocacy to about 40,000 clients per year and provided training and information and referral services to additional thousands. Funding is set aside for technical assistance to P&As and for P&As’ services via American Indian Consortium. Funding for FY 2004 totals $38.1 million. - University Centers for Excellence in Developmental Disabilities (61 grantees)
The University Centers for Excellence in Developmental Disabilities Education, Research, and Service (UCEDDs) is a discretionary grant program. Funding is provided to support the operation and administration of 61 UCEDDs in every State and Territory that comprise the nationwide system. UCEDDs engage in four core functions: (1) interdisciplinary training and continuing education; (2) community service and technical assistance; (3) research; and (4) information dissemination. Under the framework of these core functions, UCEDD programs are designed to increase the independence, productivity, and community integration of individuals with developmental disabilities. For example, in 2004 UCEDDs provided interdisciplinary training to more than 4,000 people across more than 25 disciplines, including health care, education, psychology, pediatrics, and dentistry. They provided various community-based services, such as training and intervention services, to more than 800,000 individuals with developmental disabilities, their families, and others. UCEDD personnel conducted close to 4,000 research activities to field test service delivery models and evaluated existing innovative practices. They developed over 4,000 new products that were disseminated to close to one million constituents on a local, regional, state, national, and international level. Funding is set aside for technical assistance to the 61 UCEDDs. Funding for FY 2005 totals $31.5 million.
- The DD Network programs address all elements of the life cycle and focus on areas of emphasis in numerous contexts -- employment, education and early intervention, housing, health, transportation, quality assurance, child care, recreation, and formal and informal community supports. They are charged with carrying out activities that align with the following principles of the DD Act-
- Individuals with developmental disabilities, including those with the most severe developmental disabilities, are capable of self-determination, independence, productivity, and integration and inclusion in all facets of community life, but often require the provision of community services, individualized supports, and other forms of assistance;
- Individuals with developmental disabilities and their families have competencies, capabilities, and personal goals that should be recognized, supported, and encouraged, and any assistance to such individuals should be provided in an individualized manner, consistent with the unique strengths, resources, priorities, concerns, abilities, and capabilities of such individuals;
- Individuals with developmental disabilities and their families are the primary decision makers regarding the services and supports such individuals and their families receive, including regarding choosing where the individuals live from available options, and play decision-making roles in policies and programs that affect the lives of such individuals and their families;
- Services, supports, and other assistance should be provided in a manner that demonstrates respect for individual dignity, personal preferences, and cultural differences;
- Specific efforts must be made to ensure that individuals with developmental disabilities from racial and ethnic minority backgrounds and their families enjoy increased and meaningful opportunities to access and use community services, individualized supports, and other forms of assistance available to other individuals with developmental disabilities and their families;
- Recruitment efforts from disciplines related to developmental disabilities in pre-service training, community training, practice, administration, and policymaking must focus on bringing larger numbers of racial and ethnic minorities into the disciplines in order to provide appropriate skills, knowledge, role models, and sufficient personnel to address the growing needs of an increasingly diverse population;
- With education and support, communities can be accessible to and responsive to the needs of individuals with developmental disabilities and their families and are enriched by full and active participation in community activities, and contributions, by individuals with developmental disabilities and their families;
- Individuals with developmental disabilities have access to opportunities and the necessary support to be included in community life, have interdependent relationships, live in homes and communities, and make contributions to their families, communities, and States, and the Nation;
- Efforts undertaken to maintain or expand community-based living options for individuals with disabilities should be monitored in order to determine and report to appropriate individuals and entities the extent of access by individuals with developmental disabilities to those options and the extent of compliance by entities providing those options with quality assurance standards;
- Families of children with developmental disabilities need to have access to and use of safe and appropriate child care and before-school and after-school programs, in the most integrated settings, in order to enrich the participation of the children in community life;
- Individuals with developmental disabilities need to have access to and use of public transportation, in order to be independent and directly contribute to and participate in all facets of community life; and
- Individuals with developmental disabilities need to have access to and use of recreational, leisure, and social opportunities in the most integrated settings, in order to enrich their participation in community life.
- Many service delivery systems and communities have limited capacity to meet the needs of individuals with developmental disabilities and their families. As such, a number of individuals with developmental disabilities live with their families and, in almost every State, individuals with developmental disabilities are waiting for appropriate services in their communities.
Individuals with developmental disabilities often require lifelong community services, individualized supports, and other forms of assistance, that are most effective when provided in a coordinated manner. Moreover, there is a need to ensure that services, supports, and other assistance are provided in a culturally competent manner that ensures that individuals from racial and ethnic minority backgrounds are fully included in all activities.
The public needs to be made more aware of the capabilities and competencies of individuals with developmental disabilities to enhance their independence and integration and participation in the community. The State DD Network programs are a vital resource to achieving improved outcomes for individuals with developmental disabilities and their families. Each entity contributes unique responses to the needs of individuals with developmental disabilities and their families in a State.
While each entity in the DD Network (e.g., State Council, P&A, and UCEDD) has a distinct structure and purpose, collaboration across the Network is a key aspect of their work in the States. The DD Network is meant to work collectively to address and resolve issues in the States to improve outcomes for individuals with developmental disabilities and their families in terms of their participation in the design of and access to needed community services, individualized supports, and other forms of assistance that promote self-determination, independence, productivity, and integration and inclusion in all facets of community life.
C.3 BACKGROUND
-
ADD seeks to fund Phase I of an independent evaluation of the State DD Network to assess program impact. The independent evaluation will be a high quality, unbiased evaluation to demonstrate actual impact on the well-being of individuals with developmental disabilities and their families that can be attributed to a particular DD Network project or program. It is aimed at providing feedback to ADD to help improve the effectiveness of its programs and policies. It will also serve as a decision-making tool for ADD to promote accountability to the public. Phase I of the independent evaluation project will involve the development of measurement tools for the study and a pilot study.
It is not the purpose of the independent evaluation to analyze ADD’s measurement system currently used by grantees to report on their activities. Instead, the purpose is to have an objective, outside contractor develop and conduct a new and specialized evaluation to determine the impact of the DD Network on individuals with developmental disabilities, state service systems, and on the capacity of service providers and a wide-range of professionals to reach, treat, or assist individuals with developmental disabilities to become more independent and to participate in and contribute to community life along side of other members of the community.
-
Background on Independent Evaluations
The Office of Management and Budget (OMB) provides guidance to Federal agencies for conducting independent evaluations. The guidance from OMB can be found in the Program Assessment Rating Tool (PART) (see http://www.whitehouse.gov/omb/part/ for additional information). In 2003 ADD conducted a PART self-assessment under OMB guidance.
PART is a systematic method of assessing the performance of program activities across the Federal Government. The PART is a diagnostic tool; the main objective of the PART review is to improve program performance. The PART assessments help link performance to budget decisions and provide a basis for making recommendations to improve results.
The PART is composed of a series of questions designed to provide a consistent approach to rating programs across the Federal Government, relying on objective data to assess programs across a range of issues related to performance. The PART holds programs to high standards. Simple adequacy or compliance with the letter of the law is not enough. Rather, a program must show it is achieving its purpose and that it is well managed.
One aspect of the PART focuses on questions related to independent evaluations. The PART review seeks documentation that independent evaluations of sufficient scope and quality are conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need. The purpose of this question is to ensure that the program conducts non-biased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope to improve planning with respect to the effectiveness of the program.
The PART guidelines state that the most significant aspect of program effectiveness is impact—the outcome of the program, which otherwise would not have occurred without the program intervention. PART guidelines recommend that, where feasible, programs should measure program impact using experimental designs, which use random samples representing control and treatment groups. However, the PART guidelines recognize that these studies are not suitable or feasible for every Federal program. A variety of evaluation methods, such as quasi-experimental studies, may need to be considered because Federal programs vary so dramatically. Quasi-experimental studies use techniques such as pre- and post-tests to assess the impact of the intervention.
-
The ADD Independent Evaluation
The structure and intent of the three ADD programs create significant challenges for carrying out independent evaluations based on experimental or quasi-experimental research designs. The structure of the ADD programs does not lend itself to conducting randomized trials or pre- and post-tests. ADD programs exist in every State. Many interventions of the DD Network are meant to impact all individuals with developmental disabilities residing in the State rather than one sector of the developmental disabilities population. Moreover, many DD Network interventions are not experimental. For example, P&As bring lawsuits in response to conditions within the State. Such interventions may be reactive rather than pro-active and their effect is felt overtime, rather than immediately.
Other conditions that affect the design of the independent evaluation include the variability across the programs funded under the DD Act. Given that the DD Network responds to needs of individuals with developmental disabilities and their families in a State, the programs are characterized by great variability. For example, DD Network programs can choose to address one or more areas of emphasis (e.g., quality assurance, education and early intervention, child care, health, employment, housing, transportation, recreation, and other services available or offered to individuals in a community, including formal and informal community supports that affect their quality of life). Some may choose to address one area of emphasis (e.g., child care) while other may address 5 areas (e.g., quality assurance, education and early intervention, health, employment, and housing). This flexibility in the law creates great variability in terms of program implementation and impact on individuals with developmental disabilities and their families. This level of variability across the DD Network programs poses measurement challenges and increases the likelihood of sampling errors.
Finally, the developmental disabilities population is unique and small making it difficult to gather representative, random samples. The nature of the population poses challenges to gathering a national, representative sample that contributes to the validity and reliability of the research design for measuring program impact. Potential threats to internal validity include small sample size, mortality, regression, diffusion, and selection bias. Threats to external validity include selection and program interaction and setting and program interaction.
Design of the ADD Independent Evaluation
The limitations affecting the design of an independent evaluation project require that ADD consider non-experimental methods to determine program impact. The non-experimental design for any independent evaluation conducted to assess DD Network program impact must be rigorous and use techniques that ensure the validity and reliability of the research. ADD believes that for a non-experimental independent evaluation to be rigorous, it must follow performance-based approaches to measuring program impact. Performance-based approaches offers ADD a vehicle for reporting program accomplishments with particular attention on progress towards pre-established goals – the goals being those outlined in the purpose and principles of the DD Act.
As a performance based independent evaluation, a primary objective of Phase I of this project will be the development of measurement matrices. The measurement matrices will be dynamic tools with criterion-referenced performance standards to assess the extent to which a DD Network entity exhibits the purpose and principles of the DD Act and affects outcomes for people with developmental disabilities. Using numerical indices, the measurement matrices will be able to gauge the impact of the individual DD Program and the collaborative work of the Network. To this end, the measurement matrices will function as scoring rubrics for assessing program impact.
-
A major activity in the development of measurement matrices will be the creation of criterion-referenced performance standards for the DD Network programs (e.g., State Council, P&A, and UCEDDs). A set of performance standards will also be developed for collaboration across the DD Network. All performance standards will be new measures for each DD Network program and independent of any current measures used by grantees for reporting purposes and incorporated into the measurement matrices.
- Structural indicators – assess the DD Network’s (individual and collaborative) capability to achieve goals of the DD Act through adequate and appropriate settings and infrastructures, including staffing, facilities and equipment, financial resources, information systems, governance and administrative structures, etc.
- Process indicators – assess the content and quality of the DD Network’s (individual and collaborative) activities, procedures, methods, and intervention supporting practices aligned with the purpose and principles of the DD Act.
- Output indicators – assess immediate results of the DD Network’s (individual and collaborative) policies, procedures, and services that can lead to achieving the purpose and principles of the DD Act.
- Outcome indicators – assess the unique and collective contributions of the DD Network programs to achieving intermediate results for individuals with developmental disabilities and their families relating to the provision of, response to, and results of services. Outcome indicators can be observed within and across the DD Network programs.
Draft performance standards will be promulgated by the contractor based on preliminary data collection activities. The contractor will develop draft performance standards for each DD Network program (DD Council, P&A, and UCEDD) and for collaboration across the DD Network programs. The contractor will have the draft performance standards validated by a panel of experts. There will be a validation panel for each program (DD Council, P&A, and UCEDD) and a validation panel for collaboration. The performance standards, when complete, will serve as a set of benchmarks for measuring program performance in terms of impact.
The performance standards once developed will be incorporated into measurement matrices for each DD Network program and for collaboration. The measurement matrices will be tools for determining program impact. To this end, the measurement matrices will function as scoring rubrics that will yield scores indicating program impact.
To organize and guide the development of the performance standards and related measurement matrices, the following framework of indicators of program impact should be used:
- To summarize Phase I of the independent evaluation, the contractor will develop measurement matrices organized by a framework of indicators. Each program will have measurement matrices with performance standards. There will also be a collaboration measurement matrix with performance standards. Performance standards should be established for each DD Network entity in all indicator areas. In addition, performance standards for collaboration across the DD Network should be generated for each indicator. The measurement matrices will function as scoring rubrics that will generate a quotient indicating the degree of program impact. Following the development of the measurement matrices, the contractor will use it to conduct a pilot study in up to 5 States of the independent evaluation using the measurement matrices as the primary evaluation tools.
C.4 PURPOSE
-
The purpose of the Developmental Disabilities Program Independent Evaluation (DDPIE) Project is to examine through rigorous and comprehensive performance-based research procedures the targeted impact of grantee activities funded under the Developmental Disabilities Assistance and Bill of Rights Act of 2000 (DD Act). The DDPIE is divided into two phases. The first phase will involve the development of measurement matrices for determining program impact and the implementation of a pilot study in up to 5 States. The second phase will focus on full implementation of the independent evaluation. ADD is seeking to fund Phase I of DDPIE.
It is not the purpose of the independent evaluation to analyze ADD’s current measurement system that is used by grantees to report on their activities. Instead, the purpose is to have an objective, outside contractor develop a new and specialized measurement system designed specifically to determine the impact of the programs on individuals with developmental disabilities, state service systems, and on the capacity of service providers and a wide-range of professionals to reach, treat, or assist individuals with developmental disabilities to become more independent and to participate in and contribute to community life along side of other members of the community.
The performance-based evaluation will reveal specific DD Network system structures and supports that have impacted on people with developmental disabilities and their families. A key aspect of the DDPIE will be the development and use of reliable and valid measurement matrices. The contractor will in Phase I develop and pilot the use of measurement matrices that includes performance standards to evaluate the extent to which grantee activities exhibit the purpose and principles of the DD Act and affect outcomes for individuals with developmental disabilities and their families.
C.5 EVALUATION DESIGN AND RESEARCH QUESTIONS
-
The independent evaluation shall use performance-based approaches to measure the impact of State DD Networks on individuals with developmental disabilities. The contractor shall develop in Phase I of the project performance standards that will be incorporated into measurement matrices, which will be used to assess program impact. The measurement matrices will be organized by a framework of indicators (structural, process, output, and outcome). There will be performance standards for each indicator.
- Performance Standards Items:
- What structural elements enable and/or impede the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States to engage in activities that help meet the purpose and principles of the DD Act?
- What structural elements enable and/or impede the collaborative activities of the DD Network in selected States that support achievement of the DD Act purpose and principles?
- Measurement Matrix Items:
- To what extent do the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States have the capacity to achieve the purpose and principles of the DD Act as measured by the structural indicator performance measures?
- To what extent does the DD Network (e.g., State Council, P&A, and UCEDD) in selected States have the capacity to achieve the purpose and principles of the DD Act as measured by the structural indicator performance measures?
To guide the development of all phases of the independent evaluation, the questions below must be addressed within the evaluation done by the contractor. The questions are organized by the framework of indicators and there are questions for development of the performance standards and for measurement matrices.
Structural Indicator questions:
- Process Indicator questions:
- Performance Standards Items:
- What are the activities of the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States that contribute to achievement of the purpose and principles of the DD Act?
- What are the collaborative activities of the DD Network in selected States that support achievement of the purpose and principles of the DD Act?
- Measurement Matrix Items:
- To what extent are the activities of the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States aligned with the purpose and principles of the DD Act as measured by the process indicator performance measures?
- To what extent are the collaborative activities of the DD Network in selected States aligned with the purpose and principles of the DD Act as measured by the process indicator performance measures?
- Performance Standards Items:
- What are the immediate results of the activities of the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States that contribute to achievement of the purpose and principles of the DD Act?
- What are the immediate results of the collaborative activities of the DD Network in a selected State that support achievement of the DD Act purpose and principles?
- Measurement Matrix Items:
- To what extent are the immediate results of the activities of the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States aligned with the purpose and principles of the DD Act as measured by the output indicator performance measures?
- To what extent are the immediate results of collaborative activities of the DD Network in selected States aligned with the purpose and principles of the DD Act as measured by the output indicator performance measures?
- Impact questions:
- What is the level of impact on individuals with developmental disabilities and their families of the activities of the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) in selected States?
- What is the level of impact on individuals with developmental disabilities and their families of the activities produced by the collaborative activities of the DD Network in selected States?
- What is the overall impact of DD Network activities on individuals with developmental disabilities and their families?
For each indicator, the contractor will develop performance standards for the DD Network programs (DD Council, P&A, and UCEDD) and for collaboration across the DD Network programs. There may be multiple performance standards for an entity under any given indicator. The performance standards will be validated by a panel of experts. The panel should include individuals with developmental disabilities, family members of individuals with developmental disabilities, other consumers, advocates, researchers, representatives from the DD Network, policy specialists, etc.
The validated performance standards will be incorporated into measurement matrices. The measurement matrices will be organized by the framework of indicators, which will have corresponding performance standards.
The measurement matrices will guide the collection of data and lead to the development of impact quotients of DD Network programs, at the individual and collaborative level. The impact quotients will be determined using Likert-type scales to indicate the degree to which a performance standard may be observed or developed within a program. The measurement matrices will enable evaluators to use a Likert-type scale to determine the extent to which a program exhibits a performance standard. The individual scores for each performance standard will be aggregated across all items and divided by the total possible score to yield a score indicting program impact on scale of 0 – 100. Figure 1 offers an example of what a measurement matrix might look like for one program in the DD Network.
Figure 1
Sample of a Basic Measurement Matrix for One Program
| INDICATOR |
STATE COUNCIL |
||||
|---|---|---|---|---|---|
| Structural Indicators |
|
Total Number of Points Earned divided by 9 and multiplied by 100 | |||
| Process Indicators |
|
Total Number of Points Earned divided by 9 and multiplied by 100 | |||
| Output Indicators |
|
Total Number of Points Earned divided by 9 and multiplied by 100 | |||
| Outcome Indicators |
|
Total Number of Points Earned divided by 9 and multiplied by 100 |
-
Once developed, the contractor will use the measurement matrices to conduct a pilot study of the independent evaluation in up to 5 States and/or Territories. Project evaluators will visit DD Network programs in a State to gather data for use against the measurement matrices. Data may be collected through document analysis, interviews, surveys, or other appropriate procedures. Any data collection procedures should follow established protocols. The pilot study will be used to test the reliability of the measurement matrices and to inform full implementation of the independent evaluation in Phase 2. ADD recommends that specific areas of emphasis be selected to maintain some standardization across the various programs in the States. Areas of emphasis of particular importance to ADD would be: employment, education and quality assurance (refer to Section 102 of the DD Act for definitions for these areas of emphasis) (P.L. 106-402) (42 USC 15062).
C.6 TA CONTRACT GOALS
-
GOAL 1 - IMPLEMENT PHASE I OF AN INDEPENDENT EVALUATION FOR ASSESSING THE IMPACT OF THE PROGRAMS FUNDED UNDER THE DD ACT.
- Design a performance-based evaluation study that uses standardized procedures and measurement tools for observing and measuring program impact.
- Implement a pilot study of the evaluation study following the proposed protocol.
- Identify impacts on people with developmental disabilities and their families as a result of DD Network program activities.
- Describe specific DD Network activities that stimulate positive change for the people with developmental disabilities and their families.
- Develop draft performance standards for DD Network programs by meeting with ADD Program Staff to discuss DD Network programs and reviewing data and related documents of the DD Network programs.
- Convene panels of experts to validate performance standards.
- Design a reliable measurement matrix that collects data on performance standards to assess program impact.
- Package the measurement matrix for use by ADD staff and others for future evaluations of program impact.
- Synthesize findings from data gathered through DDPIE.
- Organize findings into recommendations for ADD regarding Phase II of DDPIE
- Provide preliminary results of DD Network program performance and impact to improve results.
The primary goal of Phase I of the Developmental Disabilities Program Independent Evaluation is to assess the impact of DD Network (State Council, P&A, UCEDD) programs on people with developmental disabilities and their families. The following are the objectives for goal 1:
The quality of the information gathered through the independent evaluation depends upon the tools used to collect data on the DD Network programs. A key aspect of Phase I of the DDPIE will be the development of reliable measurement matrices with performance standards for assessing program impact. The following are the objectives for goal 2:
Findings from Phase I of DDPIE should be used to provide feedback and recommendations to ADD for Phase II of DDPIE. In addition, Phase I findings may provide preliminary evidence of program performance and impact. Results should be synthesized around a set of recommendations for ADD. The objectives for Goal 3 are to:
C.7 SCOPE OF WORK
- The contractor shall employ qualified and experienced individuals who are able to coordinate and conduct program evaluations measuring impact. Project staff, including the project director, must have advanced training (Ph.D. or similar degree) in research and evaluation. The contractor must also demonstrate that leadership staff with advanced training will spend a sufficient amount of time of the project. Credentials presented by the offeror’s proposed staff shall include a record of successful academic and professional experience in the field of research and program evaluation.
- Develop a rigorous yet practical, ethical, and affordable research design for Phase I of the independent evaluation. The evaluation design must take into consideration the unique nature of the DD Network programs and develop valid and reliable performance-based procedures for conducting the research.
- Develop and validate performance standards that will be a part of measurement matrices that assesses the extent to which DD Network programs in selected States achieve through their various individual and collaborative activities the purpose and principles of the DD Act. The measurement matrices will be organized according to a framework of four indicators (e.g., structural, process, outcome, and outcome). The contractor will convene a panel of experts to develop the performance standards. They will also develop the measurement matrices, test it for reliability, and use them to conduct a pilot study of the independent evaluation in selected States.
- Data collection, including information collected related to the measurement matrices, must come from multiple sources of evidence using standardized procedures and protocols. Data will be analyzed using procedures appropriate to the study purpose and data type.
- Synthesize findings from Phase I of DDPIE and produce a final report for ADD. The report must inform plans for Phase II of DDPIE and include preliminary findings from DD Network activities and outcomes with regard to achieving the purpose and principles of the DD Act and describe in quantitative and qualitative terms the impact on individuals with developmental disabilities and their families. The report must provide recommendations for ADD with regard to Phase II of DDPIE.
The contractor shall be responsible for implementing the independent evaluation (as stipulated in this Statement of Work) of State DD Network programs. All research-related activities will be carried out by the Contractor through face-to-face meetings with Program Staff and representatives of the DD Network Programs, on-site data collections activities, and development of findings from the investigation. All travel (excepting local travel) to collect data will require the prior approval of the Project Officer. The contractor shall be responsible for all activities, and will perform its responsibilities as outlined. Close coordination will be maintained with the contractor and overall joint planning is the responsibility of the Project Officer.
The contractor shall provide all necessary facilities, materials, equipment, qualified personnel, travel and per diem – including consultants – required to administer and implement Phase I of DDPIE according the following tasks:
C.8 Task 1: Project Meetings
-
1.1 Initial Meeting
The Contractor shall meet with the Project Officer (PO) and other ADD staff in Washington, D.C., within ten working days of the contract award. The purpose of this meeting will be to review the proposed work, deliverables, and schedule to ensure mutual understanding and expectations. The Contractor shall provide the PO with minutes of this meeting, along with a summary of any modifications to the schedule of deliverables or other contract items, for approval within 10 working days of this meeting.
1.2 Annual Project Review Meetings
The Contractor shall attend a minimum of two review meetings each year with the PO, to be held in Washington, D.C. At these meetings, the Contractor shall provide a review of activities undertaken to date, problems and barriers to implementation that have arisen, and activities planned for the next year. The Contractor shall provide the PO with minutes of all meetings (face-to-face), along with a summary of any modifications to the schedule of deliverables or other contract items, for approval within 10 working days of each meeting.
1.3 Independent Evaluation Design Meetings
Upon award of the contract, the Contractor shall meet with the PO by the end of the 2nd month of the contract to review current data collected by ADD and discuss research procedures. Any issues related to implementing the research to carry out DDPIE will be resolved.
1.4 Regular, on-going communication
The Contractor shall engage in regular, on-going communication with the ADD PO. This must include monthly conference calls with the PO and other ADD staff to discuss project activities. The contractor shall provide updates on project activities and discuss future directions of the research.
-
Task 2: Independent Evaluation Initial Activities
2.1 Review of ADD and Other Program-related Data
The Contractor shall identify and review data currently collected by ADD from the DD Network Programs. The Contractor shall also review data from other relevant sources that provide information about the DD Network programs. The Contractor shall work with the PO to identify data sources, such as documents, annual reports, etc.
2.2 Preliminary Interviews with ADD Staff and DD Network Members
The Contractor shall conduct preliminary interviews with ADD staff and representatives of the DD Network to better understand the nature of the three DD Network programs. The Contractor shall work with the PO and other ADD staff to identify appropriate people to interview. The Contractor will develop a list of people to be interviewed that will be approved by the PO prior to conducting the interviews.
2.3 Establish Advisory Committee
The Contractor shall establish an advisory committee who will provide guidance for conducting the independent evaluation. Meetings with the advisory committee must take place at least twice a year. The advisory committee should be comprised of individuals with developmental disabilities, family members of individuals with developmental disabilities, other consumers, advocates, researchers, representatives from the DD Network, policy specialists, etc. The Advisory Committee shall be established by the end of the 2nd month of the contract.
-
Task 3: Development of Study Protocols for the Collection of Data
3.1 Develop Study Protocols, as Appropriate
As appropriate, the Contractor shall develop protocols for the collection of data as part of the independent evaluation. A protocol is an original document developed at the outset of the study by the research team indicating procedures to follow in the collection of data. The protocols may be used to collect data during the initial and full implementation phases of the independent evaluation.
3.2 Validate study protocols
For any protocols developed, the Contractor shall test the validity and reliability of the protocol as appropriate.
3.3 Train Research Staff on the Use of Study Protocols
To ensure the reliability and validity of the data that is collected, the research staff shall be trained on the use of any protocols developed for the study. Reliability coefficients should be used, as appropriate, to assess the impact of the training.
-
Task 4: Development of Measurement Matrix
4.1 Convene Validation Panels
The Contractor will establish performance standards validation panels. There should be a performance standards validation panel for each type of grantee (DD Council, P&A, and UCEDDs) and a separate performance standards validation panel for collaboration across all grantees within a State. The Contractor shall identify experts to participate on validation panels for each DD Network program and for collaboration. Members of the validation panel may come from the Advisory Committee described in Task 2. Experts must include individuals with developmental disabilities, family members of individuals with developmental disabilities, other consumers, advocates, researchers, representatives from the DD Network, policy specialists, etc. The Contractor shall have the list of validation panel Committee members approved by the PO.
4.2 Promulgate Draft Performance Standards for the Measurement Matrices
Based on review of documents and data collected initially as described in Task 2, the Contractor shall promulgate draft performance standards for each indicator in the framework (e.g., structural, process, output, and outcome). Performance standards will be developed for each grant program (e.g., DD Council, P&A, and UCEDD) and for collaboration across the grantees. Language in the DD Act with regard to independence and participation shall be part of the draft performance standards. The draft performance standards should be reviewed by the PO and others identified by the PO.
4.3 Validate Performance Standards for the Measurement Matrices
The Contractor shall work with the validation panels to authenticate performance standards for the measurement matrices. Draft performance standards will be reviewed by the validation panels and revised according to consensual feedback.
4.4 Organize Performance Standards into a Measurement Matrices
Once all performance standards have been developed and validated by the panels, the Contractor shall organize the standards into measurement matrices. The matrices must clearly identify the performance standards for each indicator in the framework (e.g., structural, process, output, and outcome), for each grantee (e.g., DD Council, P&A, and UCEDD), and for collaboration across the grantees. The measurement matrices must be completed by the end of Option Year 1.
4.5 Test the Reliability of the Measurement Matrices
The Contractor shall test the measurement matrix for reliability. Through pilot testing procedures, the Contractor shall ensure that the measurement tool obtains a reliability coefficient that meets that the standards of the PO. The pilot test will be conducted in up to 5 States that are chosen with input from ADD and the Advisory Committee.
4.6 Train Research Staff on the Use of Measurement Matrices
The Contractor shall train any staff that will be collecting data using the measurement tool. To the extent possible, the training should be carried out under standardized procedures. The Contractor shall ensure consistency across the research staff using reliability coefficients, as appropriate. Reliability testing should continue as the independent evaluation is implemented in the States. The Contractor shall share with the PO results from reliability tests before entering the field to collect data as part of the implementation of DDPIE described in Task 5.
4.7 Package the Measurement Matrices for Use by ADD
The Contractor will use the measurement matrices to produce a tool that can be used by ADD in future evaluation activities. The tool should include clear directions for use by a variety of people. The tool must be approved by the PO.
-
Task 5: Implement Pilot Study of DDPIE in States
5.1 Identify States in which to Implement DDPIE
The Contractor shall identify with input from ADD staff and members of the Advisory Committee up to 5 States in which to conduct a pilot study of DDPIE. The final list of States shall be approved by the PO.
5.2 Collect Data
The Contractor shall collect data for the independent evaluation according to the plans described in the research proposal. Any protocols developed in Task 3 shall be used as appropriate. The measurement matrices developed under Task 4 will serve as the primary guide for data collection and for measuring program impact.
5.3 Analyze Data on an On-going Basis
As appropriate, the Contractor shall analyze the data as collected to guide future data collection activities.
Task 6: Synthesize Findings and Develop Recommendations
6.1 Synthesize Findings
Upon collecting all data, the Contractor shall complete analysis of the data using the measurement matrices to assess program impact. The data should be summarized to reveal significant findings from the DDPIE that are relevant to the purpose of the investigation. Ancillary findings should be reported.
6.2 Develop Recommendations
After synthesizing the findings from the data analysis, the Contractor shall develop a set of recommendations for Phase I of DDPIE. These recommendations for ADD should inform Phase II of DDPIE. The recommendations must be reviewed by the advisory committee.
-
Task 7: Progress Reports
7.1 Quarterly Progress Reports
The Contractor shall submit a quarterly technical progress report to the PO within 10 calendar days after the end of each federal fiscal quarter. The report shall include a review of activities undertaken to date; problems and barriers to implementation that have arisen and agreed-upon solutions; and activities planned for the next quarter. Specific information shall be provided regarding the development and dissemination of information.
7.2 Annual Technical Progress Report
The Contractor shall submit an annual technical progress report to the PO at the end of each contract year. The report shall include highlights of major accomplishments, problems encountered and future plans. It is due within a month of the conclusion of each contract year.
7.3 Final Report
The Contractor shall develop and submit to ADD a final report of the DDPIE. The final report shall provide an overview of the research design, a summary of major findings, and recommendations to ADD.
C.9 PERFORMANCE MEASURES
| Task or Deliverable | |||
|---|---|---|---|
| 1. Develop list of interviewees | Potential list of people to interview developed in consultation with ADD Program Staff, final list approved by PO, schedule developed with approval from PO within specified timeframe. | Interview subjects approved, interviewees contacted and interviews scheduled within specified timeframe. | Customer feedback |
| 2. Organize Advisory Committee | Committee members selected, proposed meeting schedule developed with approval from PO within specified timeframe | Committee members approved, meetings scheduled developed by PO. Committee members contacted and accepted within specified timeframe. | Customer feedback |
| 3. Develop study protocols for collecting data | Protocols developed for any data collected by the contractor within specified timeframes | Contractor informs PO of the protocols for use in collecting data prior to data collection activities. | 100% inspection Customer feedback |
| 4. Convene Performance Standards validation panels | List of proposed panel members developed by contractor, schedule developed with related timeline for developing performance standards approved by PO within specified timeframe. | Validation panel members approved and confirmed, meetings scheduled within specified timeframe. | 100% inspection Customer feedback |
| 5. Promulgate draft performance standards using the framework of four indicators | Draft performance standards developed within specified time frames and approved by PO | Draft performance standards are based on tenets of the DD Act and the framework of indicators and developed with input from ADD Program Staff and DD Network program representatives | 100% inspection |
| 6. Authenticate performance standards for the measurement matrices | Convene validation panel to authenticate performance standards that will be incorporated into the measurement matrices within specified timeframe | Performance standards that reflect input from the Validation Panel and integrate the framework of indicators approved by the PO within the specified timeframe | 100% inspection Customer feedback |
| 7. Develop measurement matrices that includes performance standards | Using the performance standards, develop measurement matrices to guide data collection and assess impact of DD Network grantee activities within specified timeframe | Measurement matrices will offer a clear format for assessing program impact and will achieve a high level of reliability according to PO approval and within specified timeframe | 100% inspection Customer feedback |
| 8. Produce a DD Network program assessment tool for use by ADD | Using the measurement matrices develops a DD Network program assessment tool that can be used by ADD in future evaluation activities | The tool follows a user-friendly format and is readily accessible to a variety of individuals. | 100% inspection Customer feedback |
| 9. Identify States in which to conduct pilot study of DDPIE | Develops with input from ADD Program Staff list of States in which to conduct DDPIE. From list, identifies up to 5 States in which to implement DDPIE. | The list of States is approved by PO and confirmed by Contractor. Schedule for entering States developed and approved. | 100% inspection Customer feedback |
| 10. Develop recommendations from Phase I for ADD | After synthesizing findings, Contractor develops recommendations for ADD around budgeting, monitoring, technical assistance, and, as appropriate, other activities to improve results. | Recommendations reflect the findings from Phase I of the independent evaluation and inform Phase II of DDPIE as well preliminary findings of DD Network program impact. | 100% inspection Customer feedback |
| 11. Submit technical progress reports | Develops and submits to the PO comprehensive technical quarterly reports on progress achieved under the contract | Technical quarterly reports are submitted in a timely manner and provide evidence of progress achieved and, in cases where progress has not been achieved, explanations for the lack of progress | 100% inspection Customer feedback |