| CFDA Number: | 84.305 - Education Research |
| Goal 8: Transform education into an evidence-based field. |
| Objective 8.1 of 2: Raise the quality of research funded or conducted by the Department. |
| Indicator 8.1.1 of 4: The percentage of new research and evaluation projects funded by the IES that are deemed to be of high-quality by an independent review panel of qualified scientists. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Status: Target exceeded
Progress: For 2003, 20 IES proposals were reviewed by a panel of 10 reviewers. The external reviewers who served this year also served as external reviewers from the previous years. |
Additional Source Information: IES selects a random sample of newly funded research proposals from IES. These proposals are distributed to senior scientists in education for evaluation. Data will be collected annually. This evaluation is separate from the peer review panels used to evaluate applications submitted for research funding. Frequency: Annually. Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.2 of 4: The percentage of new research and evaluation publications by IES that are deemed to be of high-quality by an independent review panel of qualified scientists. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: No new research/evaluation publications were issued in 2003. |
Additional Source Information: IES selects a random sample of new research and evaluation publications from IES. Publications are distributed to senior scientists in the field for review. Data will be collected annually. Frequency: Annually. Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.3 of 4: Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Status: Target exceeded
Progress: For 2003, there were 38 new IES proposals; all included causal questions; 37 used a randomized experimental design to answer causal questions. |
Additional Source Information: IES researchers evaluate all newly funded research and evaluation proposals by IES to identify projects that address causal questions and of those projects, those that utilize randomized experimental designs to answer those questions. Data will be collected annually. The 75% target for 2002-2005 recognizes that some high quality research addressing causal questions will not be able to employ randomized experimental designs. Frequency: Annually. Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the 2 IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there is (a) an experimental (treatment) group and one or more comparison groups and (b) random assignment of either participants to treatment and comparison groups or groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.4 of 4: Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: No new research/evaluation publications were issued in 2003. |
Additional Source Information: IES researchers evaluate all newly funded research and evaluation publications by IES to identify projects that address causal questions and of those projects, those that utilize randomized experimental designs to answer those questions. Data will be collected annually. The 75% target recognizes that some high quality studies will not be able to employ randomized experimental designs. Frequency: Annually. Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the 2 IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there is (a) an experimental (treatment) group and one or more comparison groups and (b) random assignment of either participants to treatment and comparison groups or groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design. |
|||||||||||||||||||||||||||||||
| Objective 8.2 of 2: Increase the relevance of our research in order to meet the needs of our customers. |
| Indicator 8.2.1 of 4: The percentage of new research projects funded by IES that are deemed to be of high relevance to educational practice as determined by an independent review panel of qualified practitioners. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: 2003 data not yet available (10/2003). We don't expect the data will be available before December 2003. |
Additional Source Information: External panel of qualified practitioners will evaluate the relevance of a random sample of newly funded research proposals. Data will be collected annually. The final target of 75% recognizes that some important research may not seem immediately relevant, but will make important contributions over the long-term. Frequency: Annually. Evaluations are only as good as the qualifications of the external review panel. Inclusion of only experienced practitioners and administrators in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.2 of 4: The percentage of K-16 policymakers and administrators who report routinely considering evidence of effectiveness before adopting educational products and approaches. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: Data to be collected in 2005 |
Additional Source Information: Survey of education decision-makers and policymakers. Data will be collected every 3 years. Frequency: Other. Data are valid to the extent that sample includes education decision-makers across high-, low-, and average-achieving districts and states, across urban and rural areas, and from all regions of the country. The sample included district superintendents, chief state school officers, and state higher education executive officers across all of these dimensions. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.3 of 4: The number of annual hits on the What Works Clearinghouse web site. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Status: Target exceeded
Progress: Actual web hits for 2003 registered over 150% of the target 1 million hits. Note: 2004 target has been revised to 2,000,000 and is in the process of being corrected in the data base. |
Additional Source Information: What Works Clearinghouse. Baseline data for number of annual hits is FY 2003. Web-based program will automatically count hits on web site. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.4 of 4: Percent of What Works Clearinghouse web site users surveyed randomly who responded to the question, '' Would they recommend the WWC web site to a colleague or friend'' (by checking ''agree'' or ''strongly agree'') | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: Data will be collected in 2004. |
Additional Source Information: What Works Clearinghouse. Baseline data for web site users who would recommend it is FY 2004. |
|||||||||||||||||||||||||||||||
