As schools in the United States are increasingly using digital technology in the classroom to teach and assess students, the National Assessment of Educational Progress (NAEP) has moved forward to align with these practices. NAEP’s transition from paper-based to digitally based administration provides an engaging assessment experience for students and aligns with the delivery mode of many other large-scale assessments. Importantly, this transition to digitally based assessment (DBA) also allows NAEP to use tools available in digital platforms to measure content in new ways; to use assistive technology to provide enhanced accommodations for students with special needs; and to collect new types of data that deepen our understanding of what students know and can do, including how they engage with new technologies to approach problem solving.
In 2017, the
NAEP mathematics assessment was administered for the first time as a DBA at grades 4 and 8. The digital platform allowed for the collection of new data within the testing system, including information on how students used onscreen tools to develop their responses to the assessment questions. These new data are called
response process data. To further enrich our understanding of what students know and can do in the digital environment, response process data from the 2017 NAEP grade 8 mathematics assessment are now available for secondary analysis.
Digitally Based Assessment in NAEP
NAEP DBAs offer far more flexibility in meeting the needs of different students. The DBAs include
Universal Design Elements, or built-in features that make it possible for more students to participate without special accommodation sessions. Onscreen tools are also available for students to use in their problem solving. The goal is for all students to have a seamless assessment administration, regardless of their ability.
At the beginning of each assessment, students take a brief, interactive tutorial designed to teach them about the testing system and the tools they will use to take the assessment. Experience one of the
NAEP DBA tutorials and learn more about the tools available to students. Some of the universal design elements and tools available in the NAEP grade 8 mathematics assessment are described below.
![Graphic of a math assessment item with two points, A and B, plotted on a number line that has the numbers 0 through 3 below it. There are hash marks on the line and arrows at both ends. The following questions are below the number line: What number corresponds to point A? What number corresponds to point B? What number corresponds to the point that is located halfway between point A and point B? Below each question, there is an answer box. There is a gray bar across the top of the image with the following icons inside it, from left to right: A question mark in a black circle. A color contrast symbol that is rectangular and has a lowercase white letter a in a black square on the left and a lowercase black letter a in a white square, which is labeled Color Contrast/Change Theme. [Note for Rick: Can we force the screen reader to read the letter “A” twice in this sentence that describes the color contrast/change theme tool?] A light gray magnifying glass with a minus sign in the middle, and a black magnifying glass with a plus sign in the middle, with the label Zooming underneath and two red arrows pointing up at the magnifying glasses. A capital letter T with a blue speech bubble next to it that has three white dots inside, with a Text-to-Speech label above that has two orange arrows pointing down at the capital letter T and the blue speech bubble with three white dots. A black square with three rings at the top and a scribble on it, with a Scratchwork/Highlighter label below it that has two green arrows pointing up at the black square. A green dotted square box above a black line with the number 2 below the same black line and the label Equation Editor above it that has a purple arrow pointing down at the box, line, and number 2. A gray icon that looks like a calculator, with a small screen at the top and three rows of buttons beneath it. A white, star-shaped icon behind the number VH139196. A gray icon that looks like a stop watch, with one button on the top, one button on the right side, a minute hand, and an hour hand. The word Progress. A gray arrow pointing left. A dark gray arrow in a light gray circle that is pointing right.](https://webarchive.library.unt.edu/web/20201218062310im_/https://www.nationsreportcard.gov/subject/process_data/images/process_data_toolbar.jpg)
 Color Contrast/Change Theme | Students have a choice of three color contrast options, including one high-contrast option. |
|---|
 Zooming | Text size options are provided for short stand-alone items (e.g., multiple-choice and short constructed-response items) but not for scenario-based tasks. Students have a choice of four zoom options, or “states” (100%, 125%, 150%, and 200%). |
|---|
 Text-to-Speech
| Directions within the assessment can be read aloud using the text-to-speech function. Students can select some or all text to be read aloud.
|
|---|
 Scratchwork/Highlighter Capability | A scratchwork/highlighter tool is available for short standalone items (e.g., multiple-choice and short constructed-response items). This tool contains an embedded pencil mode and highlighter mode for annotating figures, performing computations, drawing diagrams, and highlighting portions of a question.
|
|---|
 Equation Editor
| An onscreen equation editor is provided for entering numbers and expressions using the correct mathematical symbols.
|
|---|
 Calculator
| For item blocks in which a calculator is allowed, an onscreen calculator is provided.
|
|---|
Try the elements and tools in a
sample test containing released questions from the 2017 NAEP grade 8 mathematics assessment. Note, zooming and text-to-speech tools are not functional in the sample test.
About NAEP Response Process Data
The NAEP digital platform collects different types of data during a student’s assessment session. In addition to logging students’ responses to items in the cognitive assessment and their responses to survey questions, the testing system logs
response process data—specifically, the actions or events initiated by students as they complete the digital assessment. This response process data includes information such as students’ use of the onscreen calculator, clicking of response choices, elimination of response choices, and key presses.
The response process data can be transformed further into variables called
features, which summarize meaningful actions or events that occur during the assessment. Examples of features include the number of times that a student opened the onscreen calculator, the number of times that a student used the highlighter tool, or the amount of time that a student spent on an item.
The graphic below provides a high-level overview of the data captured within the digital platform and where these data are organized in the files available in the response process dataset from the 2017 NAEP grade 8 mathematics assessment.
2017 NAEP Grade 8
Mathematics Assessment |
| Restricted-Use
Response Dataset |
| Restricted-Use
Response Process Data |
| Features
Data |
|---|
Data collected from each digital assessment session:
- Student responses to cognitive items
- Student responses to survey questions
- Response process data
| Data from students who took one or both released blocks:
- Respondent data
- Documentation
- Software syntax
| - Response process data
- Response data
- Student demographics and accommodation information
| - Summarized response process data at the item and block levels
- Response data
- Student demographics and accommodation information
|
The 2017 NAEP grade 8 mathematics assessment included two 30-minute blocks of cognitive items in each digital test form, followed by a 15-minute
survey questionnaire that collected information on students’ demographic characteristics, opportunities to learn in and outside of the classroom, and educational experiences. There were 10 cognitive blocks in the assessment; each block was paired with every other block, resulting in 50 unique digital test forms. After the assessment, the NAEP program released two blocks of cognitive items used in 2017. One of the released cognitive blocks was adapted from the paper-based assessment, and the other released cognitive block was designed for the digitally based assessment. Ten of the 50 digital test forms included the digitally designed released item block and one of the 50 digital test forms included both released blocks.
The current dataset includes restricted-use response process data and associated files for respondents who took the digitally designed released block (approximately 28,000 students) and respondents who took both blocks of released cognitive items (approximately 2,800 students). The following information is provided for each set of respondents:
- process data (“observable data”) text file, containing logs of the response process data collected from each student;
- response data file, containing students’ raw response data and scored response data for cognitive items and block-level timing data; and
- student demographics (7 variables) and accommodation information (35 variables).
The dataset also includes features files for respondents who took the digitally designed released block (approximately 28,000 students) and respondents who took both blocks of released cognitive items (approximately 2,800 students). The features files contain demographics, accommodations, response data, and scored response data along with summarized response process data at the item and block levels. The raw response process data capturing each student’s interactions with selected DBA tools in a block (i.e., color contrast/change theme, zooming, and text-to-speech tools) are summarized as variables representing tool usage features. Some examples of the variables in the files include time spent in seconds, the predominant color contrast/theme used, the predominant zoom state used, the number of times the text-to-speech tool was used, and the number of times the calculator tool was opened.
Finally, this dataset also includes information about the assessment that has previously been released, including the released cognitive items, survey questionnaire files, and the full suite of restricted-use response data product materials for the 2017 NAEP grade 8 mathematics assessment. The
restricted-use response data files include respondent data, documentation, and software syntax limited to students who took both of the released blocks or one of the released blocks. Note that for research activities utilizing the NAEP response process data, plausible values are not included in the
restricted-use response data files. Empirical scores (i.e., percent correct or number correct) are included to evaluate response process data related to mathematical ability.
Unique pseudo IDs are exclusively used as the student identifiers in all data files in this suite of products. The pseudo IDs are 9 digits in length and include a 3-digit digital test form number followed by a random 6-digit number. These pseudo IDs provide a higher level of security for student respondents, including eliminating the ability to match the restricted-use response process data, features data, restricted-use response data, and associated files to the operational restricted-use data products.
To explore this response process dataset, interested researchers should apply for a
restricted-use license and request access to the files through the National Center for Education Statistics (NCES) website.
Protecting Personally Identifiable Information of Students, Teachers, and Schools in NAEP Data
Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP tests do not question test takers about personal or family beliefs or make information about their personal identity publicly available. By design, Personally Identifiable Information (PII) is not available at the individual student or school levels. By law, NAEP is not allowed to report results at the individual or school levels.
Direct identifiers (i.e., names of students and schools) are not available in the restricted-use response process dataset. Each student is represented with a unique pseudo ID that cannot be directly linked to PII. The restricted-use process data files include a full log of student constructed responses, which could potentially contain PII if a student included PII in a response. However, if this occurs, researchers must remember that they have promised under penalty of law to keep these identities confidential. Any re-identification of students or schools through the use of this dataset is prohibited by law. Violations are subject to prosecution as a Class E Felony, with penalties of up to five years imprisonment and/or a $250,000 fine.
