1. | General Recommendations |
2. | Requirements at a Glance |
3. | Content Standards |
4. | Academic Achievement |
5. | Statewide Assessment |
6. | Technical Quality |
7. | Alignment |
8. | Inclusion |
9. | Reports |
10. | Conclusion |
Recommendations
Use the "State Index" form provided by USED as the template for organizing the submission.
Rationale
Clearly identify the appropriate reference and where in the reference the evidence appears. Be as specific as possible. All attachments and citations should be clearly labeled and numbered sequentially. It is time consuming and frustrating for reviewers to search through a submission for cited references.
Recommendations
In the section entitled "Evidence" provide a clear and concise description of how the state meets the requirement.
Rationale
Narrative and citations of evidence that do not directly address the element give the impression that the state is not sure its system is compliant with the requirement(s). Many states submit too much material, much of it irrelevant to the questions in the Peer Review Guidance keep the evidence submissions
Recommendations
Provide duplicates of any materials that cannot be photocopied, e.g. CD-ROM, color brochures, etc.
Rationale
A State is required to submit only one copy of all materials, but if the state wishes to ensure that reviewer’s copies are complete and formatted as originally designed, that state may provide 6 complete copies of the submission.
Recommendations
List the assessments currently in use in your State. Components currently under development for future use, such as content standards, performance standards and descriptors, and assessments, should be clearly described with proposed dates for full implementation.
Rationale
In 2005-06 many states will be implementing parts of the assessment system for the first time and may not yet have implemented science assessments. Reviewers need to be clear about which assessments are currently used to calculate AYP and which are planned for future development.
Recommendations
If the state's program does not currently meet the requirement, say so; but give the projected plan and timeline for when the requirement will be met.
Rationale
Do not attempt to obfuscate. If it appears that the state is trying to misrepresent the existing program, reviewers may question the integrity and intent of the submission and give closer scrutiny to all aspects of the documentation provided.
Recommendations
Committees, panels and reviewers cited in evidence as being used in development processes should reflect the demographics of the student population for your state, but at the least should include representatives for students with disabilities, limited English proficient students, and racial/ethnic groups.
Rationale
The participation of representatives that reflect the diversity of the state in all aspects of development is a requirement for several sections of the submission. Documentation should include some indication of panel (committee, reviewer) qualifications.
Recommendations
Participation data should include student enrollment data for the grades tested as well as the number of students assessed.
Rationale
All students must be tested, regardless of the length of time in the school. Reviewers must be able to verify that all students are included in testing so the state should provide accurate enrollment data for the grades tested.
Recommendations
Reports and studies by independent evaluators are valuable types of evidence.
Rationale
Independent evaluations and studies add a level of credibility to findings. This is particularly relevant for studies pertaining to comparability, alignment and validity but it applies to all development processes.
Fulfilling the NCLB requirements for a state's standards and assessment system can be challenging. In preparing the submission states must first decide if their program meets the requirements. Then the appropriate documentation must be attached to substantiate their response. Most state departments house large volumes of reports, memos, policies and regulations. Although extensive amounts of evidence may be submitted, in reviewing submissions we have found that states often overlook, omit or fail to adequately address essential aspects of critical elements. In this part of the Guide for each section of the requirements we cite the critical elements and appropriate evidence. As needed, we also provide reminders under the heading of "Notes".
Critical Element
Reading/language arts and Mathematics: Content standards for each of grades 3-8 and 10-12 grade range or content standards for grade ranges with specific content expectations for each grade.
Evidence
Formal adoption/approval of challenging academic content standards for all public schools and students in the state is required. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval.
If the content standards have been adopted/approved in the past and revisions have been made to the standards, provide evidence of the formal adoption/approval of the revised content standards or a timeline for the approval and implementation.
Critical Element
Science: Content standards for grade ranges 3-5, 6-9, and 10-12
Evidence
Formal adoption/approval of challenging academic content standards for all public schools and students in the state is required by 2005-06. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval. If they have not been approved yet include your plan and timeline for development.
Critical Element
Academic content standards must be challenging with rigorous content and encourage the teaching of advanced skills.
Evidence
Evidence may include aspects of your development process which address higher order thinking skills or studies conducted by an external group. If recommendations have been made for changes to the academic content standards to make them more rigorous, provide documentation on how you addressed these recommendations and how the academic content standards have been changed.
Critical Element
Involvement of education stakeholders in the development of its academic content standards.
Evidence
Stakeholders include educators, parents, community members, higher education representatives, and representatives for student with disabilities and students with limited English proficiency.
*Note: Academic content standards are the basis for the development of performance descriptors, assessments, and achievement standards. Clearly identify any continued development/revisions of the academic content standards and how the revision has impacted the development/revision of the assessment system.*
Critical Element
Reading/language arts and Mathematics: Achievement standards for each of grades 3-8 and 10-12 grade range.
Evidence
Formal adoption/approval of challenging academic achievement standards for all public schools and students in the state is required by 2005-2006. Provide evidence such as state statutes, regulations, State Board minutes or if approved by Chief State School Officer, written documentation of formal approval. The evidence requirement also applies to any revision of cut scores and levels. The evidence must specifically state that the standards apply to all students, unless alternate achievement standards have been developed for students with the most severe cognitive disabilities. In this case, include the formal adoption/approval of the alternate achievement standards.
Critical Element
Science: Achievement standards for each of the grade spans 3-5, 6-9, and 10-12.
Evidence
The standards must apply to all public schools and students in the state. As with reading/language arts and mathematics, achievement descriptors for science are due by 2005 – 2006. However, cut scores for science are not due until 2007-08. Provide evidence such as state statutes, regulations, or State Board minutes. The evidence requirement also applies to any revision of cut scores and levels. The evidence must specifically state that the standards apply to all students, unless alternate achievement standards have been developed for students with the most severe cognitive disabilities. If these are under development, describe the development process and timelines.
Critical Element
Reading/language arts, Mathematics and Science: Alternate achievement standards for students with the most severe cognitive disabilities (if alternate achievement standards have not been developed then the alternate assessment must be based on grade level achievement standards).
Evidence
Alternate achievement standards must be:
Provide evidence such as how the alternate standards were defined and how they are linked to grade level content.
Critical Element
The Academic and Alternate achievement standards must include:
Evidence
The submission must include all four criteria. As evidence, provide official summary reports and documentation of the process (including how impact data was considered).
Critical Element
In addition, for alternate achievement standards the state must document that it has:
Evidence
The submission must include all six criteria. As evidence, provide official summary reports, documentation of the processes (including the involvement of parents in decision making), guidelines for the use of the alternate assessment, training materials and sample test reports.
Critical Element
The academic and alternate achievement standards must be aligned with the content standards.
Evidence
The achievement standards should fully reflect the content standards for each required grade and describe the content based expectations each achievement level represents. Provide the descriptions of the process used to develop the levels, descriptions and cut scores and how the state assures itself that alignment is present.
Critical Element
The development of achievement standards must involve diverse stakeholders.
Evidence
The development of achievement standards must involve a broad range of stakeholders that reflect the diversity and needs of students in the state. Provide descriptions of the group compositions; minutes or summaries from public hearings and/or public reviews; and reports which indicate the degree of involvement. Special attention should be made to include individuals knowledgeable of and concerned about the various categories of special needs students.
Critical Element
The State assessment system may include different types of assessments (e.g., CRT, Augmented NRT, and various types of Alternate Assessments).
Evidence
Complete the Overview chart provided in the Guidance. Clearly indicate what type (s) of assessments are used for each required grade and subject area (s).
Critical Element
If different assessments are used (e.g., CRT, ANRT, native language assessments, simplified English versions), document a rational overall design and coherence, as well as alignment with the academic content and achievement standards.
Evidence
Provide the rationale and blueprint for the design. Also provide documentation of the processes and reports from studies that clearly indicate coherence, alignment, and comparability.
Critical Element
Multiple measures that assess higher-order thinking skills and understanding of challenging content.
Evidence
Provide test blueprints, item specifications, statistical evidence, descriptions of the processes used to judge the cognitive level of the test or item content. The intent of the evidence is to ensure content coverage in terms of depth, breath and cognitive load. It is also useful to provide samples of the measures being used.
Critical Element
The State assessment system may employ a matrix design. If a matrix design is used, the multiple forms within a content area and grade must:
Evidence
Provide technical manuals, state reports and/or reports from independent evaluators on the quality, equivalence, and comparability of the forms.
Critical Element
The State assessment system may include alternate assessments. Student achievement may be measured against grade level standards or against alternate achievement standards.
Evidence
Provide:
Critical Element
Local assessments, if used as part of the State assessment system, must:
Evidence
Provide documentation of the processes used by the State to ensure quality and comparability, including reports done by independent evaluators as well as follow-up plans to address any identified deficiencies.
*Note If the state assessment system is comprised of more than one assessment, the state must provide reports from comparability studies for the various assessments to document that all components are aligned with the same academic content and achievement standards*
Seven categories of validity are noted in the Guidance. To address the first category of validity, provide written documentation, such as State Board policies or state legislative code that defines the purpose of the state assessment system. For the remaining six categories provide written documentation from the studies done that provide evidence of validity. If deficiencies exist provide the plan and timeline to address them.
Critical Element
A full range of reliability must be documented.
Evidence
Three categories of reliability are noted in the Guidance. Provide written documentation of the studies that support the reliability of the assessments with the State's own student population. Also provide documentation of the precision of cut scores and the consistency of student classification. If any deficiencies exist (such as in establishing cut scores or in classification consistency) provide the plan and timeline to address them.
Critical Element
The assessment system must be fair and accessible to all students, including students with disabilities and students with limited English proficiency.
Evidence
The State assessment system must be designed to be valid and accessible for use by the widest possible range of students. Provide evidence that there is an appropriate variety and number of accommodations to meet the needs of students with disabilities and students with limitedEnglish proficiency; that individuals with experience and expertise in working with SWD and LEP students were included in test development processes; that educators are trained on the appropriate selection and use of accommodations and alternate assessments.
Critical Element
The State must document their coherent approach to ensuring alignment between each of its assessments and the content and achievement standards the assessment is designed to measure.
Evidence
Provide documentation of the process used including stakeholder involvement, ongoing quality control reviews, how the alignment is maintained over time, reports of alignment studies and how the results were addressed.
Critical Element
Provide evidence that the assessments and standards are aligned comprehensively including range, degree of cognitive complexity, level of difficulty, and depth.
Evidence
Provide evidence such as assessment plans, assessments blueprints, and item/task specifications. Include information that shows which standards are assessed and which are not along with the weights. Include any reports of independent alignment studies and studies of cognitive complexity of the assessments.
Critical Element
Assessments and standards must be aligned in terms of both content (knowledge) and process (how to do it).
Evidence
Provide evidence that processes and skills required in the content standards are assessed as well as the content.
Critical Element
Assessments and standards must be aligned in terms of degree and pattern of emphasis.
Evidence
Include information on how the assessments reflect the weights of the content standards and their sub-domains.
Critical Element
Assessments must yield scores that reflect the full range of the state's academic achievement standards.
Evidence
Provide evidence that the state's assessment system provides a sufficient number of items to assess students at all levels of achievement--for example, basic, proficient, advanced.
Critical Element
Assessment results must be expressed in terms of achievement standards, not just scale scores or percentiles.
Evidence
In order for members of the school community to see the alignment between the standards and assessments, reports must include information on how the students perform relative to the achievement standards. Provide evidence such as sample student score reports.
Critical Element
Provide evidence such as sample student score reports.
Provide evidence on how it maintains and/or improves alignment of assessments and standards over time.
Evidence
Documentation should be provided on planned alignment reviews, external studies of alignment, and how assessments are modified if content standards are revised.
*Note:
Critical Element
Participation data must be included and show that all students in the required grades/grade ranges are included in the assessments.
Evidence
Documentation must be provided that shows the participation rate is calculated as a proportion of the students enrolled and the students assessed in total and in each subgroup.
Critical Element
Provide guidelines the State has for including all students with disabilities in the regular assessment system.
Evidence
Documentation must be provided to show how the State is making efforts to include students with significant cognitive disabilities in the regular assessment and that if alternate achievement standards are adopted for these students, guidelines and training are provided for the identification and assessment of these students.
Critical Element
If alternate achievement standards have been adopted/approved for students with the most significant cognitive disabilities, provide the following:
Evidence
Documentation on alternate achievement standards must include all five criteria.
Critical Element
Provide the following:
Evidence
For the 2003-04 assessments, students in their first year of enrollment in U. S. schools may be exempt from the State's reading/language arts assessment.
Provide policies, regulations, guidelines related to inclusion of LEP students including information such as allowed accommodations.
Critical Element
Ensure the identification and inclusion of migrant and other mobile students in the assessment system in the required tested grades.
Evidence
Evidence should include policies, guidelines, and practices used by the State to include mobile students.
*Note:
Critical Element
Report participation and assessments results for each of the required subgroups for the required tested grades and grade ranges at the school, LEA, and State level.
Evidence
Required subgroup reporting for assessment reports includes: gender, race/ethnicity, English proficiency status, migrant, students with disabilities, and economically disadvantaged. For LEA and State Report Cards, comparisons of students with disabilities vs. non-disabled and economically disadvantaged vs. non-economically disadvantaged are also required. Provide sample reports for the required grades.
Critical Element
Ensure that personally identifiable information about an individual student is not reported
Evidence
Documentation on alternate achievement standards must include all five criteria.
Critical Element
Provide documentation on how the State ensures student confidentiality.
Evidence
Produce individual interpretative, descriptive, and diagnostic reports following each administration of its assessments. The reports must include the following:
Critical Element
Ensure the identification and inclusion of migrant and other mobile students in the assessment system in the required tested grades.
Evidence
Evidence may include examples of the individual student reports for the required grades tested, interpretive guides, scoring and reporting timelines.
Critical Element
Ensure that student-level assessment data are maintained securely to protect student confidentiality
Evidence
Evidence should include policies and procedures on access to student assessment data.
Critical Element
Produce itemized score analyses so that parents, teachers, and principals can interpret and address the specific academic needs of students.
The itemized score analyses should include results for each of its academic content standards and each of the sub-domains/strands within these standards, to the extent that these sub-scores are based on enough items or score points to be meaningful
*Note:
It is our hope that this Guide will facilitate the process, provide information that will enhance the submission and thus expedite the review process.
Last Modified: 08/08/2005
|