[Federal Register: January 18, 2000 (Volume 65, Number 11)]
[Notices]
[Page 2596-2598]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr18ja00-45]
-----------------------------------------------------------------------
DEPARTMENT OF EDUCATION
National Assessment Governing Board; Information Collection
Request
AGENCY: National Assessment Governing Board; Department of Education.
ACTION: Notice of Information Collection Activity; Request for
Comment.
-----------------------------------------------------------------------
SUMMARY: In compliance with the Paperwork Reduction Act (44 U.S.C.
3501 et seq.), this notice announced a proposed information collection
request (ICR) of the National Assessment Governing Board (the Governing
Board, or NAGB). The information collection is to conduct two research
and validation support studies related to test development for the
proposed Voluntary National Test (VNT) during Spring 2000. Before
submitting the ICR to the Office of Management and Budget (OMB), the
Governing Board is soliciting comments on the information collection as
described below.
DATES: Comments must be submitted on or before February 17, 2000.
ADDRESSES: Submit written comments identified by ``ICR: VNT Research
and Validation Support Studies (Option Year 2)'' by mail or in person
addressed to: Ray Fields, Assistant Director, National Assessment
Governing Boards, Suite 825, 800 North Capitol Street, N.W.,
Washington, DC 20002.
Comments may also be submitted electronically by sending electronic
mail (e-mail) to Ray__@FieldsED.GOV. Electronic comments must be
identified by the title of the ICR. No confidential business
information should be submitted through e-mail. Comments sent by e-mail
must be submitted as an ASCII file avoiding the use of special
characters an any form of encryption.
Information submitted as a comment concerning this document may be
claimed confidential by marking any part or all of that information as
confidential business information (CBI). Information so marked will not
be disclosed except in accordance with procedures set forth in 40 CFR
Part 2. A copy of the comment that does not contain CBI must be
submitted for inclusion in the public record. Information not marked
confidential may be disclosed publicly by NAGB without prior notice.
All written comments will be available for public inspection at the
address given above from 8 a.m. to 4:30 p.m., Monday through Friday,
excluding legal holidays.
FOR FURTHER INFORMATION CONTACT: Ray Fields, Assistant Director,
National Assessment Governing Board, Suite 825, 800 North Capitol
Street, N.W., Washington, DC 20002. Telephone (202) 357-0395; e-
mail:Ray__Fields@ED.GOV.
SUPPLEMENTARY INFORMATION: Copies of this ICR may be obtained from the
contact person listed above.
I. Information Collection Request
The National Assessment Governing Board is seeking comments on the
following Information Collection Request (ICR).
Title: Voluntary National Tests (VNT): Research and Validation
Support Studies (Option Year 2)
Affected Entities: Parties affected by this information collection
are individuals and State, local, or Tribal SEAs or LEAs.
Abstract: In order to comply with the mandates of PL 105-78, the
National Assessment Governing Board (NAGB) proposes to conduct two
research and validation support studies. Congress vested exclusive
authority in the Governing Board for test development for the proposed
VNT. At the same time, Congress prohibited pilot testing and field
testing of questions developed for the proposed VNT. No test question
developed for the proposed VNT will be used in these research studied.
Instead, test questions used for the National Assessment of Education
Progress (NAEP) will be employed. This is to ensure that the
prohibition on pilot and field testing is not violated, while still
providing for research needed to answer questions related to test
development.
The data collected will serve two purposes: (a) Provide information
on the feasibility of a calibration linkage between the proposed
Voluntary National Test (VNT) and the National Assessment of Education
Progress (NAEP) (more specifically--between a test designed to give
individual results and a survey designed to report group results); and
(b) provide information needed to inform policy and practice related to
test accommodations for students with limited English proficiency,
specifically, to help guide the development of an 8th grade mathematics
test booklet in two languages (i.e., a ``dual language'' booklet in
this case in English and Spanish).
The two research studies will also assist NAGB in making three of
the four determination required by Congress: (1) The extent to which
test items selected for use on the tests are free from racial, cultural
or gender bias; (2) whether the test development process and test items
adequately assess student reading and mathematics comprehension in the
form most likely to yield accurate information regarding student
achievement in reading and mathematics; and (3) whether the test
development process and test items take into account the account the
needs of disadvantaged, limited English proficient and disabled
students.
The first study is directed toward establishing the feasibility of
a calibration linkage between a test form resembling an individual test
and a survey of group results--the National Assessment. Research
questions to be answered include the following: What are the effects on
the measurement of student performance of an individually administered
test that shares a framework with NAEP but which differs somewhat from
NAEP in content coverage, administration, and unit of analysis? It is
possible to establish a strong link between the group-focused results
of NAEP and such an individually administered test? What inferences can
be supported by such a link?
4800 students from Grade 4 and 4800 students from Grade 8 are
expected to participate in this study. The 9600 students will be
divided equally across three conditions.
Students in the first condition will take a ``NAEP Special Form''
booklet, consisting of NAEP items constructed to be as parallel as
possible to the proposed VNT forms. This parallelism would include
content coverage, timing, and shape of the test information function
(TIF), which has been proposed to be flatter than the TIF for NAEP.
Because empirical information on each item is needed to construct a
form with a specified TIF, the items would come from the previous NAEP
administration in the respective subjects.
Students in the second condition would take ``Extended NAEP''
booklets, which are based on blocks of items from the 2000 NAEP
administration and would be constructed to be representative of the
content and statistical specifications (TIF) of NAEP. The forms for
Grade 8 mathematics would consist of six intact 15-minute blocks
administered in two 45-minute sessions. The forms for Grade 4 reading
would consist of four NAEP reading blocks, also administered in two 45-
minute sessions. (Because the reading blocks are timed at 25 minutes
each, some items will have to be deleted to fit into the reduced
testing time.) The administration of these forms would be
[[Page 2597]]
under conditions proposed for the VNT. To avoid the circularity of
linking the same items to themselves, the items used in the extended-
NAEP forms should be distinct from those used in the NAEP Special
Forms.
In the first two conditions of this proposed study, the two types
of forms would be spiraled together and administered to equivalent
samples of students. Because the NAEP Special Forms and the Extended-
NAEP forms would be administered under the same conditions, issues of
administration, timing, and motivation become moot. If the content
match between the NAEP Special forms and the simulated VNT forms could
be made sufficiently close, a linking study between the two types of
forms would approximate a linkage study between actual VNT forms and
Extended-NAEP. If a calibration were successful, the resulting linkage
interpretations would be in terms of student performance on NAEP when
NAEP is given under VNT conditions.
Students in the third condition differ from the other two in that
they would be taking the ``NAEP Special Form'' under motivated
circumstances. It is quite plausible that the same student would
perform at a higher level under a motivated situation such as the VNT,
where individual scores are obtained under a low motivation situation
such as the NAEP. This differential effect of motivation could impact
achievement level cut-points (among other things) in ways that cannot
be assessed in the two conditions described above. Consequently, the
third condition of this study involves paying students $1 for every
item they answer correctly. This procedure is directly modeled after
research conducted on motivational interventions for the NAEP. A
comparison of item parameters and test characteristic curves for the
NAEP Special Forms under motivated and unmotivated conditions would
provide information on the differential impact of motivation and how to
adjust results for any subsequent linking study between the VNT and
NAEP.
The second study involves a series of subtasks directed toward
informing NAGB's inclusion and accommodation policies regarding LEP
students. These tasks are:
Subtask A. Writing an issues paper covering theory and research
related to the development of a dual language test. This paper would
inform procedures to be used in the translation of items into the
second language (i.e., Spanish) (Subtask B).
Subtask B. Using released and secure NAEP 8th grade mathematics
items to construct simulated VNT-M test booklets (dual language and
English-only versions). The English language version of this booklet
will be the same as the one for the ``NAEP Special Form described
earlier.
Subtask C. Evaluating the psychometric equivalence of the dual
language and English-only booklets via traditional quantitative
analyses. Six hundred bilingual and LEP students will be recruited and
randomly assigned to complete either the dual language or English-only
version of the test booklet. Quantitative analyses will be conducted to
examine the psychometric equivalence of the two test versions (mean
differences; differential item functioning; correlations).
Subtask D. Conducting focus groups of students immediately after
they take the VNT-M to document students' overall experience with the
two types of booklets. Sixty students will be recruited to do these
focus groups, in order to obtain their insights and general reactions
to the booklets.
Subtask E. Conducting cognitive laboratory studies to obtain in-
depth information on the validity of the translation and about how
students use the dual language test. An additional nine LEP and nine
English-speaking students will be asked to participate in this study,
in order to explore the performance of both Anglo and Hispanic LEP
students to identify solution pathways that students choose to use.
Subtask C through E will allow for a thorough investigation into
the cognitive processes that bilingual and limited English proficient
(LEP) students employ when using the dual language version of the VNT-
M. In addition, they will provide information about factors other than
mathematical knowledge and problem-solving ability that may have an
effect on their performance on the test.
The five subtasks listed above will offer answers to the following
research questions to examine the quality of the dual language test,
taking into account several features of the items:
Cognitive: Do students understand the native language version of
the test questions as a vehicle for assessing mathematics? (Subtasks C,
D, E)
Content: Is the content of the native language version of the test
questions the same as the English version? (Subtasks B, C, D, E)
Format: What considerations should be given to how the test
questions appear on the pages of the test booklet? (Subtasks A, B)
Cultural: Is the native language version clear and acceptable to
the various communities in the United States for whom this is the
native language? (Subtasks A, B, C, D, E)
Academic: Are the grammar and language structure used in the native
language version correct? (Subtasks B, D, E)
Scoring: What considerations need to be made for scoring dual
langauge test booklets?
(Subtask A)
Psychometric Equivalence: Is there a psychometric equivalence
between the dual language version and the English only versions of the
test? (Subtask C)
A total of 10,128 students is expected to participate in the two
studies (4800 4th graders and 4800 8th graders in the calibration
linkage feasibility study; 510 LEP and bilingual students taking the
dual language or English-only math test (from which there will be 60
focus group participants); and 18 cognitive laboratory participants).
These students will be recruited from 300 schools. Students in the
motivated condition of the calibration linkage study, focus group
participants and cognitive laboratory participants will receive a token
monetary incentive. Also under consideration is a modest monetary
inventive for each participating school.
Burden Statement: Assuming a 2 hour burden for each of the 10,128
students expected to participate in the two studies, a total of 20,376
hours is estimated. An additional 300 hours of school burden (one hour
per participating school) is expected, reflecting the time it would
take to collect student background data for our research purposes.
Participation in this study is voluntary. State, local, and non-public
education agencies will not be mandated or required to participate.
II Request for Comments
The National Assessment Governing Board solicits comments to assist
it:
(a) Evaluate whether the proposed collection of information is
necessary for the proper performance of the functions of the Governing
Board, including whether the information will have practical utility;
(b) Evaluate the accuracy of the Governing Board's estimates of the
burden of the proposed collection of information;
(c) Enhance the quality, utility and clarity of the information to
be collected;
(d) Minimize the burden of the collection of the information on
those who are to respond, including through the use of appropriate
automated, mechanical or other technological collection techniques or
other forms of
[[Page 2598]]
information technology, e.g., permitting electronic submission of
responses.
III. Public Record
A record has been established for this action. A public version of
this record, including printed, paper versions of electronic comments,
is available for inspection from 8:30 a.m. to 5 p.m., Monday through
Friday, excluding legal holidays. The public record is maintained at
the National Assessment Governing Board, 800 North Capitol Street NW,
Suite 825, Washington DC, 20002.
Dated: January 12, 2000.
Roy Truby,
Executive Director, National Assessment Governing Board.
[FR Doc. 00-1072 Filed 1-14-00; 8:45 am]
BILLING CODE 4000-01-M