[Federal Register: May 10, 1999 (Volume 64, Number 89)]
[Notices]
[Page 25028-25029]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr10my99-50]
-----------------------------------------------------------------------
DEPARTMENT OF EDUCATION
National Assessment Governing Board; Public Forum
AGENCY: National Assessment Governing Board; Education.
ACTION: Notice of information collection activity.
-----------------------------------------------------------------------
SUMMARY: This notice announces that the National Assessment Governing
Board (NAGB) will submit an Information Collection Request (ICR) to the
Office of Management and Budget for approval.
The ICR is: An Investigation of Alternative Methods for Scale
Anchoring and Item Mapping in the National Assessment of Educational
Progress.
DATES: Public comments must be submitted on or before June 9, 1999.
ADDRESSES: Written comments should be submitted by June 9, 1999. Mail
to Patricia Hanick, NAEP ALS Project Manager, ACT, Inc., 2255 North
Dubuque Road, P.O. Box 168, Iowa City, IA 52243-0168. Copies of the
complete ICR and accompanying appendices may be obtained from the NAEP
ALS Project Manager at the address above.
Comments may also be submitted electronically by sending electronic
mail (e-mail) to Hanick@ACT.org. Electronic comments must be identified
by the title of the ICR. No confidential business information should be
submitted through e-mail.
Information submitted as a comment concerning this document may be
claimed confidential by marking any part or all of that information as
confidential business information (CBI). Information so marked will not
be disclosed except in accordance with the procedures set forth in 40
CFR part 2. A copy of the comment that does not contain CBI must be
submitted for inclusion in the public record. Information not marked
confidential may be disclosed publicly by NAGB without prior notice.
FOR FURTHER INFORMATION CONTACT: Patirica Hanick, NAEP ALS Project
Manager, ACT, Inc., 2255 North Dubuque Road, P.O. Box 168, Iowa City,
IA 52243-0168, Telephone: (319) 337-1452 or (800) 525-6930, e-mail:
Hanick@ACT.org.
SUPPLEMENTARY INFORMATION: Electronic copies of this ICR can be
obtained from the contact person listed above.
I. Information Collection Request
NAGB is seeking comments on the following Information Collection
Request (ICR).
Title: An Investigation of Alternative Methods for Scale Anchoring
and Item Mapping in the National Assessment of Educational Progress.
Affected Entities: Parties affected by this information collection
are persons who served as panelists for pilot studies, achievement
levels-setting (ALS) meetings, and validation research for the 1998
Science NAEP.
Abstract. The purpose of this information collection activity is to
gather research information for NAGB to be used in evaluating the
procedures for the selection of exemplar items to represent student
performance on NAEP. Exemplar items are used as one of the primary
means of communicating student performance on NAEP. The criteria for
selecting exemplar items is critical to the ``message'' portrayed by
the items when reporting the outcomes of NAEP to the public.
Two statistical criteria currently guide the selection of exemplar
items: One based on item difficulty, and the other on item
discrimination. Part One of the study will examine that statistical
criteria used to select exemplar items by comparing various scale
anchoring and item mapping methodologies. The results of these
systematic comparisons will be judged not only statistically, but also
according to the degree to which they agree with informed judgments
about item difficulty. Collecting responses from informed judges is
Part Two of the study. Because these individuals have participated in
the process of setting achievement levels for the 1998 Science NAEP,
they are likely to be keenly interested in the research study.
In Part One of the study, several technical aspects of the
anchoring process will be investigated:
(1) Stringency of difficulty criterion (response probability
criterion);
(2) Point versus interval-based estimates;
(3) Empirical versus model-based estimates;
(4) Type of discrimination criterion.
Before conducting the data analysis of the anchoring process, cross
validation analyses of random half-samples of the data will be done.
Four methods of analyzing the student data are planned:
(1) Empirical/interval estimation;
(2) Empirical/point estimation;
(3) Model-based/interval estimation;
(4) Model-based/point estimation.
Three factors will be considered as the criteria to determine which
anchoring/mapping method is best:
(1) Consistency of results across subsamples;
(2) The degree to which the results are supported by informed
judgment;
[[Page 25029]]
(3) The number of exemplar items produced by the method.
In Part Two of the study each prospective respondent will be sent a
selection of items from the 1990 Science NAEP Physical Science item
pool. All of the items have been ``released'' for public review.
Respondents will be asked to rank order the items according to the
perceived level of difficulty. The ranking task will involve four
overlapping sets of 8-9 items per set. Each item will be displayed on a
separate card. Item sets will include obtained and bogus clusters.
Participants will be asked if the items appear to cluster together, and
if so, what are the common tasks or content areas that form the basis
of the cluster. They will be asked to determine if the clusters reflect
Basic, Proficient, or Advanced performance, based on the NAGB policy
definitions of achievement levels. Finally, participants will be asked
the following question:
If you were told that American students can do item N, what
would you assume this meant? Specifically, what percent of students
would need to be able to answer correctly in order for you to agree
that students can do the item? (40%, 50%, 60%, 70%, 80%, (90%,
100%).
The results of Part One will then be evaluated according to the degree
to which they agree with the informed judgments about item difficulty.
The response rate for the survey used in Part Two is expected to be
80% or higher. Only persons who served as panelists for pilot studies,
achievement levels-setting meetings, and validation research for the
1998 Science NAEP will be invited to participate. These individuals
have shown keen interest in NAEP by their participation in the ALS
process. The mailing list for these individuals is being updated, and
those who can be contacted will be asked in advance if they will agree
to participate in the survey. Only persons who consent will receive the
survey and accompanying materials. Follow-up procedures will include
mailing reminder postcards, making telephone calls, and sending
replacement materials.
No third party notification or public disclosure burden is
associated with this collection.
Burden Statement: The estimated total respondent burden is 228
hours, and the average burden per respondent is 1.5 hours. This is a
one-time survey. Neither small business nor other small entities are
included in the survey.
II. Request for Comments
NAGB solicits comments to:
(i) Evaluate whether the proposed collection of information is
an appropriate method to determine the ``message'' portrayed by the
items regarding student performance on NAEP reported to the public.
(ii) Enhance the accuracy, quality, and utility of the
information to be collected.
(iii) Evaluate whether the design of this survey maximizes the
response rate, i.e. the number of selected persons who will respond.
Records are kept of all public comments and are available for
public inspection at the U.S. Department of Education, National
Assessment Governing Board, Suite 825, 800 North Capitol Street, NW,
Washington, DC from 8:30 a.m. to 5 p.m.
Dated: May 5, 1999.
Roy Truby,
Executive Director, National Assessment Governing Board.
[FR Doc. 99-11660 Filed 5-7-99; 8:45 am]
BILLING CODE 4000-01-M