| CFDA Number: | 84.305 - Education Research |
| Program Goal: Transform education into an evidence-based field. |
| Objective 8.1 of 2: Raise the quality of research funded or conducted by the Department. |
| Indicator 8.1.1 of 4: The percentage of new research and evaluation projects funded by the IES that are deemed to be of high-quality by an independent review panel of qualified scientists. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Explanation: The scores of one reviewer were extreme outliers - greater than 3.8 standard deviations below the average ratings of the other 12 reviewers. If these scores were removed, the percentage of new projects deemed to be of high quality would be 70 percent. In the future, if the average ratings of a reviewer constitute extreme outliers, these scores will be removed. |
Additional Source Information: IES selects a random sample of newly funded research proposals from IES. These proposals are distributed to senior scientists in education for evaluation. Data will be collected annually. This evaluation is separate from the peer review panels used to evaluate applications submitted for research funding. Frequency: Annually. Collection Period: 2003 - 2004 Data Available: September 2004 Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.2 of 4: The percentage of new research and evaluation publications by IES that are deemed to be of high-quality by an independent review panel of qualified scientists. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: No new research/evaluation publications were issued in 2003 or 2004. |
Additional Source Information: IES selects a random sample of new research and evaluation publications from IES. Publications are distributed to senior scientists in the field for review. Data will be collected annually. Frequency: Annually. Collection Period: 2003 - 2004 Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.3 of 4: Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: This is the third year that targets have been exceeded. |
Additional Source Information: IES researchers evaluate all newly funded research and evaluation proposals by IES to identify projects that address causal questions and of those projects, those that utilize randomized experimental designs to answer those questions. Data will be collected annually. The 75% target for 2002-2005 recognizes that some high quality research addressing causal questions will not be able to employ randomized experimental designs. Frequency: Annually. Collection Period: 2003 - 2004 Data Available: September 2004 Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the 2 IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there is (a) an experimental (treatment) group and one or more comparison groups and (b) random assignment of either participants to treatment and comparison groups or groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design. |
|||||||||||||||||||||||||||||||
| Indicator 8.1.4 of 4: Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: No new research/evaluation publications were issued in 2003 or 2004. |
Additional Source Information: IES researchers evaluate all newly funded research and evaluation publications by IES to identify projects that address causal questions and of those projects, those that utilize randomized experimental designs to answer those questions. Data will be collected annually. The 75% target recognizes that some high quality studies will not be able to employ randomized experimental designs. Frequency: Annually. Collection Period: 2003 - 2004 Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the 2 IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there is (a) an experimental (treatment) group and one or more comparison groups and (b) random assignment of either participants to treatment and comparison groups or groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design. |
|||||||||||||||||||||||||||||||
| Objective 8.2 of 2: Increase the relevance of our research in order to meet the needs of our customers. |
| Indicator 8.2.1 of 4: The percentage of new research projects funded by IES that are deemed to be of high relevance to educational practice as determined by an independent review panel of qualified practitioners. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
|
Additional Source Information: External panel of qualified practitioners will evaluate the relevance of a random sample of newly funded research proposals. Data will be collected annually. The final target of 75% recognizes that some important research may not seem immediately relevant, but will make important contributions over the long-term. Frequency: Annually. Collection Period: 2002 - 2003 Data Available: September 2004 Evaluations are only as good as the qualifications of the external review panel. Inclusion of only experienced practitioners and administrators in education and special education assures the quality of the data. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.2 of 4: The percentage of K-16 policymakers and administrators who report routinely considering evidence of effectiveness before adopting educational products and approaches. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Explanation: Next data collection is scheduled for 2005. |
Additional Source Information: Survey of education decision-makers and policymakers. Data will be collected every 3 years. Frequency: Other. Data are valid to the extent that sample includes education decision-makers across high-, low-, and average-achieving districts and states, across urban and rural areas, and from all regions of the country. The sample included district superintendents, chief state school officers, and state higher education executive officers across all of these dimensions. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.3 of 4: The number of annual hits on the What Works Clearinghouse web site. | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: Actual hits were more than double the target level. |
Additional Source Information: What Works Clearinghouse. Baseline data for number of annual hits is FY 2003. Frequency: Annually. Collection Period: 2003 - 2004 Data Available: September 2004 Web-based program will automatically count hits on web site. |
|||||||||||||||||||||||||||||||
| Indicator 8.2.4 of 4: Percent of What Works Clearinghouse web site users surveyed randomly who responded to the question, '' Would they recommend the WWC web site to a colleague or friend'' (by checking ''agree'' or ''strongly agree'') | |||||||||||||||||||||||||||||||||
| Targets and Performance Data | Assessment of Progress | Sources and Data Quality | |||||||||||||||||||||||||||||||
|
Progress: Note that about two-thirds of respondents found the WWC website useful in 2004 even though the WWC began releasing its reports only during the last 3 months of fiscal year 2004. The number of reports released at this stage is small though growing. Explanation: Note that the performance measure/indicator being tracked was changed in July 2003 to the following: The percentage of WWC website users surveyed randomly who responded to the following statement, 'Evidence provided on the WWC website is useful in making decisions about education programs and practices,'' by checking ''agree'' or ''strongly agree''. |
Additional Source Information: What Works Clearinghouse. Baseline data for the indicator, as revised July 2003, is FY2004 actual. Subsequent targets will be adjusted. Frequency: Annually. Collection Period: 2003 - 2004 Data Available: September 2004 |
|||||||||||||||||||||||||||||||
