Skip Navigation
small header image
U.S. Department of Education
Institute of Education Sciences
NCES Kids' Zone
  
What's New?  What's New?

The Nation's Report Card — Results from the 2005 NAEP Trial Urban District Assessment in Science
Dr. Peggy G. Carr Good afternoon, and welcome to our StatChat on the NAEP 2005 Trial Urban District Assessment in science. I hope you?ve had time to examine the results and that I can answer any questions you may have. The TUDA provides a unique look at the performance of large urban districts across the country. There are many findings in all of the districts and I?m interested to hear which ones you want to talk about. So, let?s get right to your questions?
Amy from Portland, Oregon asked:
What do the scores reflect in terms of racial/ethnic comparisons when compared to past Math scores.

Dr. Peggy G. Carr 's response:
NAEP science and math assessments are not designed so that cross-subject comparisons can be made--in fact, science is reported on a scale that does not allow cross-grade comparisons. One unarguable fact, unfortunately, is that unaccectably large gaps persist between racial/ethnic groups in all subjects NAEP assesses.


Kelly from Washington, DC asked:
Why was DC included in the 2002 Reading & Writing, 2003 Reading and Mathematics, and 2005 Reading and Mathematics, but not included in the 2005 NAEP Science? Thank you.

Dr. Peggy G. Carr 's response:
Kelly, unfortunately, Washington, DC does not have enough students to allow for three assessments and still produce reliable and valid scores. Therefore, in 2005, DC was only able to participate in reading and mathematics assessment.


charles from austin, texas asked:
How do you select the districts that participate in this study?

Dr. Peggy G. Carr 's response:
Hi Charles, NCES worked collaboratively with the Governing Board and the Council of Great City Schools to identify different types of large urban districts who were willing to participate. Because this is still considered to be a "trial assessment," the selection of districts is, in part, meant to represent a variety of types of districts (e.g., location, proportion of low income, and minorities and English language learners). The Governing Board is currently exploring a definition of "urban district" that could be used in the future to identify districts for future assessments should funds become available.


Allie from Westfield, NJ asked:
How is it decided which districts are included in the assessment?

Dr. Peggy G. Carr 's response:
Hi Allie-- Please see my response to Charles from Austin, Texas. Thank you.


Bill Noxon from Orange, VA asked:
I wasn't able to see the webcast today, although I reviewed much of the report. I would like your short "bottom-line" view of what this NAEP assessment reveals today, in light of overall student performance and the inner city gaps that persist in science (as it does in math). Thanks.

Dr. Peggy G. Carr 's response:
Hello Bill, Because this is the first TUDA in science, the results are really a baseline for future comparisons. Clearly, the performance in the participating districts trails the nation, but the report does show that when results are disaggreated by race/ethnicity and income status, some districts outperform the average for peer groups in large central cities.


Jon McChesney from San Diego, CA asked:
Hello Dr. Carr. Thank you for fielding questions today. In past NCES releases, such as the recent private school and charter school studies, NCES has performed "secondary analyses" in an attempt to make apples-to-apples comparisons between cohorts. At the time of those releases Commissioner Mark Schneider commented that NCES would discontinue such studies, and, if I understood the implication correctly, the research community would pick up where NCES had left off. Obviously the characteristics and challenges of each participating TUDA district are varied. Do you believe NAEP has value in terms of comparing, weighing, and evaluating these characteristics, or educational inputs? Or do you consider NAEP to be an isolated snapshot?

Dr. Peggy G. Carr 's response:
Dear John, Commissioner Schneider was referring to reports using a particular analysis methodology. NCES anticipates research using a variety of analytic techniques from the active community of NAEP researchers. I do believe that NAEP has tremendous value in facilitating comparisons of student performance across districts. In fact, NAEP is the only common measure these districts share. However, NAEP is a cross-sectional survey, and the data are not appropriate bases for causal inference--we can't answer "why" questions. NAEP data and results can provoke insights and inquiries within and across districts. More in-depth studies investigating inputs and characteristics of the differences and similarities, weaknesses and strengths, often follow a release of NAEP results.


Claudia Wallis from New York, NY asked:
1. Could you please explain why the comparison is made between the large central cities and the TUDA cities? Are the TUDA cities different in any significant way from the central cities? How were they chosen? 2. Any insights as to why some TUDA districts greatly outperformed others? What, for example, is Austin doing that's making a difference? Do they have better qualified science teachers?

Dr. Peggy G. Carr 's response:
Hi Claudia. The results for large central cities (those with a population of 250,000 or more) are compared with the TUDAs because these are the most appropriate peer group. The TUDA cities are not strictly representative of large central cities, but they were chosen to represent a diversity in size, geography, and student populations. The NAEP data cannot answer the "why" question. However, the information in the NAEP Data Explorer (NDE) on the website can help you address whether Austin differs from other districts on such variables as teacher certification, years of experence, times per week science is taught, etc. In addition, I suggest you consult the experts at the Council of Great City schools, who have conducted contextual analyses of NAEP background variables, as well as other data sources, to address such questions. Please see cgcs.org.


Mary from Washington, D.C. asked:
Do we know how Title I schools fared? How about schools identified for improvement? Thank you!

Dr. Peggy G. Carr 's response:
Dear Mary, The NAEP assessments do not provide direct information about Title I schools. There is indirect evidence, as we ask school principals to indicate the approximate percentage of their students who are eligible for the National School Lunch Program, a measure of low income. The report shows that low income students score relatively low on the assessments. Regarding your second question, NAEP does not identify schools identified for improvement.


To from Reno, NV asked:
Hello. I was wondering what the plans are regarding the expansion of the number of districts that will be included in data collection/analysis and district-level reporting. Thank you.

Dr. Peggy G. Carr 's response:
Plans are still being formulated. Senior level officials are considering plans to expand. In fact, on November 15, the National Assessment Governing Board will discuss criteria for defining urban districts. TUDA, up to this point, has been conducted on a trial basis, and districts have been chosen to participate on that basis.


Marianne from Dover, NH asked:
I read that this is the last national assessment under the existing science framework and that a new framework will be used in 2009. How does the new framework differ from the current one? Is it more in line with the recent NRC recommendations that science curriculum should focus on a smaller, core group of scientific topics?

Dr. Peggy G. Carr 's response:
Hello Marianne, The NAEP assessment frameworks are actually developed by the National Assessment Governing Board. I do know, however, that the NRC's National Science Foundation Standards was one of several sources used in guiding the development effort--along with the AAAS Benchmarks. There are several differences between the new and existing science frameworks. Perhaps one of the more exciting science innovations will be the use of computer-based interactive tasks to assess student scientific inquiry skills. You can access the new framework at nagb.org/pubs/naep_science_framework_2009.doc.


Dan from Washington DC asked:
Can we summarize by this report that race and income are the primary determining factors in achievement?

Dr. Peggy G. Carr 's response:
Hello Dan, NAEP is not designed to assign courses of achievement. Achievement is the result of many complex factors. The data, however, do indicate an association between race, eligibility for the National School Lunch Program (measure of low income), and average performance. But there are high-achieving minority and low-income students, so these factors should not be viewed as determinants of educational performance.


Deinya from New York, NY asked:
Are there enough respondents in NYC's community school districts for a disaggregation? Would it give reliable estimates?

Dr. Peggy G. Carr 's response:
Deinya, Thank you for your question. The sample for each district is drawn and weighted to be representative of the entire district. I am aware of the community school districts in New York City--but our procedures do not include sampling at that level. We can only report results for the entire NYC school district.


Gregory from Washington, DC asked:
How many students in Washington D.C. would be needed in order to participate in the science assessment?

Dr. Peggy G. Carr 's response:
Gregory, We generally require students for TUDA per subject/grade combination (e.g., grade 4 reading), so in a three-subject assessment year, a district needs at least 3,000 students in grade 4 and 3,000 in grade 8 to participate in the NAEP TUDA. However, DC is a special case. All other districts are part of a state NAEP sample as well, so the 1,000 student minimum is the additional sample--over and above the state sample--we require for participation. State samples are typically 3,000 students per grade/subject. DC is, obviously, not part of a state sample, and generally we treat DC's data as we treat state data. Even to get acceptably reliable and precise results for two TUDA subjects (reading and math), all grade 4 and grade 8 students in DC are sampled for NAEP.


Courtney from New York, New York asked:
How and why were the specific kinds of questions listed under "grade 8 item performance" chosen (e.g. "explain relative motion of two vehicles", "list three uses for human-made satellites", etc.)?

Dr. Peggy G. Carr 's response:
Courtney, Thank you for your question. We used several criteria to choose the sample questions in district profiles. First, the questions had to come from the released item pool. Second, every district had to have sufficient data. Third, sample questions needed to cover the range of achievement levels. And fourth, both multiple-choice and and contructed-response questions were included. More detail on the released questions is available in the NAEP Questions Tool at http://nces.ed.gov/nationsreportcard/itmrls/.


Steve from Chapel Hill, NC asked:
It appears that public schools in large cities don't have much to be proud of in terms of Science scores. If this is indeed the case, from what geographical areas can we attribute the higher national average?

Dr. Peggy G. Carr 's response:
Hi Steve, As Michael Casserly, executive director of the Council of the Great City Schools, said today at the press conference, "Nobody has bragging rights. In fact, as a nation, these results are discouraging and students in all geographic areas need to improve." Casserly also noted that "large central cities face significant challenges. For example, students in large central cities have a 69 percent poverty rate compared to 45 percent for the nation. The 2005 NAEP results in science show that, on average, public school students eligible for free or reduced-price lunch score at the 31st percentile while students who are not eligible score at the 63rd percentile." Casserly's analysis of our data also show similar patterns for English proficiency, parent education level, and teacher preparation. Consult their website for more insight at www.cgcs.org.


Tara from New York, NY asked:
How were the standards for student "science achievement levels" decided? What information or data was used to create these standards?

Dr. Peggy G. Carr 's response:
I have time to answer a few more questions. Tara, the National Assessment Governing Board develops the achievement levels. They gather committees of educators, professors, subject-matter experts, business and industry representatives, and informed members of the general public. These committees use a consensus process to arrive at cut-points on the NAEP scale that define the boundaries between levels. It is largely a judgmental process, informed periodically by student performance data.


Brian from Indianapolis asked:
What schools were in the comparison group and were any attempts made to determine if they were equivalent?

Dr. Peggy G. Carr 's response:
I'm assuming you are referring to the "large central city" comparison group. Based on the Census definition, schools are considered to be in large central cities if the population size of their city is 250,000 or more. Throughout the TUDA report, the 10 participating districts are compared to the average performance of large central city students nationwide because it represents a comparison group that is more reflective of the student populations in these 10 districts. Nearly all of the schools in our participating districts are in large central city locations.


Mary from Los Angeles, CA asked:
Why are these science results released so much later than the National/State science results?

Dr. Peggy G. Carr 's response:
The report production and review process gives priority to subjects addressed by No Child Left Behind (i.e., reading and mathematics). At this point, NCLB does not address science NAEP scores. And so, there was no requirement to report results within six months, as there is for reading and mathematics.


Rima from Los Angeles asked:
Is the new Science framework for 2009 going to allow comparisons to the data released today?

Dr. Peggy G. Carr 's response:
Hi Rima, The new science frameworks for 2009 will update what is being assessed in science NAEP to reflect current teaching practices and content. As a result, the NAEP science results in 2009 will not allow direct comparisons to the data released today. However, there are plans to conduct a bridge study to allow for comparisons between the two types of assessments.


Lauren from New York asked:
Why are there separate reports for National and State Reading and Math, National and State Science, TUDA Reading and Math, and TUDA Science instead of one comprehensive report that covers all 2005 data?

Dr. Peggy G. Carr 's response:
Hi Lauren, We produce separate reports so we can make results for the NCLB subjects (reading and mathematics at grade 4 and 8) as soon as possible. In addition, various audiences are interested in some subjects but not others.


Robert Buckley from Atlanta, GA asked:
Dr. Carr, thank you for taking questions today. What are some of the measures taken to ensure the reliability and validity of NAEP TUDA?

Dr. Peggy G. Carr 's response:
Robert, Our assessments are conducted according to the highest standards of reliability and validity. I don't have time to give you a full answer here, but our website has descriptions of our methodology.


Mary from Los Angeles asked:
I understand your response about why Science is released after the Reading/Math results, but why is the TUDA Science report out 6 months after the National/State Science report?

Dr. Peggy G. Carr 's response:
Mary, I appreciate your question. Of course, our goal is always to release data and publish reports as soon as possible. NCES is administering a variety of assessments and surveys (including NAEP) continuously. Each report is custom developed, thoroughly QC'd, and reviewed--and this process does take some time. This report was released as soon as we were able to complete all of these processes.


Dr. Peggy G. Carr :
Thanks for all the excellent questions. Unfortunately, I could not get to all of them, but please feel free to contact NAEP staff or myself for more information. For additional information, contact Sherran Osborne at Sherran.Osborne@ed.gov. I hope that you found this chat helpful and the report interesting.

Pubs/Products|Surveys/Programs|DataTools|Tables/Figures|FastFacts|School/LibrarySearch|Annuals|What's New|Kids Zone
|Institute of Education Sciences|NCER|NCEE|NCSER
NewsFlash|Staff|Contact|Site Index|Help|RSS|Privacy Policy
  
1990 K Street, NW
Washington, DC 20006, USA
Phone: (202) 502-7300 (map)
Statistical Standards|FedStats.gov