Assessment of Academic Literacy Skills: Preparing Minority and Limited English Proficient (LEP) Students for Postsecondary Education

Purpose

Despite important recent strides in diversifying the means of instruction, college students still acquire most of their education by listening to lectures and reading textbooks. Yet the language of lectures and textbooks is radically different from that spoken on a daily basis by most students and their families. This difference is often at the root of serious academic difficulties. Academic language, which includes words and phrases common to all fields, is used in lectures and texts to teach the technical information of each discipline. Because the elimination of books and lectures is neither feasible nor desirable, we can expect that students who come to campus already familiar with academic language will continue to enjoy a clear advantage over those who do not.

Students with academic language deficiencies come from varied backgrounds. Some are native English speakers, while others speak a different language at home and, despite having had all their education in the United States, are only marginally proficient in English. Whatever the reason for the problem, colleges and English as a Second Language (ESL) programs generally rely on standardized multiple choice tests to determine whether to accept these students, where to place them, and when they have satisfied exit criteria. The native English speakers are likely to be placed in remedial courses, whereas the U.S.-educated non-native speakers are usually steered into ESL classes intended for international students, whose needs are entirely different from theirs.

The standardized tests, for which students may prepare by undergoing expensive coaching and memorizing lengthy word lists, are unrelated to the actual language tasks required in college and have only low to moderate correlations with college grades. They also underpredict success for minority students, women, and ESL students. The composite numbers reported as test results give no information about the nature of the strengths and weaknesses of the individual.

The goals of this project were to develop an assessment of students’ academic language proficiency and a curriculum to improve their academic language skills.

Innovative Features

The assessment and the curriculum developed for this project are both based on videotaped college lectures and excerpts from college textbooks. The curriculum focuses on the major language differences between under-prepared and well-prepared students. These differences were determined through detailed analyses of field tests of 700 college and high school students, representing a wide range of background and preparation levels.

To create the pre- and posttests the project director first studied the nature of the language used in college courses, and defined the language tasks required in college. Project staff videotaped eight 30-minute lectures and selected one for the tests, while others became the basis for the curriculum. The test lecture focuses on a topic, theories of learning, that the students are not likely to have encountered before. The test lecture is difficult enough—in its proportion of academic vocabulary, speed of delivery, and sentence complexity—to avoid a ceiling effect in grading, and it is well organized, with cues for important topics and repetition of key points.

The tests consist of the following elements:

1. Lecture notes. Students watch a lecture excerpt (with an introduction that includes an overview of the topic, key vocabulary, and instructions for note taking) and take notes. These are scored holistically for inclusion of the most important information and for evidence of such skills as using abbreviations and indicating relationships among topics.

2. Short answer questions (based on the lecture). Scored for precision, vocabulary, grammar and spelling, this task indicates comprehension and requires students to synthesize information from the notes, using their own words.

3. Dictation of lecture sentences. This exercise is scored for correctness of academic vocabulary and for word endings, which correlate with other indicators of academic language proficiency (whereas minor misspellings do not).

4. Self assessment of academic vocabulary (reading and dictation). Field test results revealed this as a valid and reliable method for assessing vocabulary knowledge. Students rate their level of knowledge of each of 25 read or dictated words on a one-to-five scale. Students are generally more confident about the words they read than the words they hear, perhaps because exposure to academic language in high school occurs more through reading than through oral use. This indicates that practice with lectures is important for students with academic language deficiencies.

5. Sentence completion (reading). This exercise evaluates the ability to recognize the correct morpheme endings of common words. Students read simplified passages related to the lecture and for certain words select the correct ending that fits the sentence.

6. Reading importance. In passages from textbooks related to—but not repetitive of—the lecture topic, students underline the twelve most important propositions.

7. Notes from reading. After the underlining task, students take notes from an excerpt of the reading selection. The notes are scored for the presence of important information, evidence of note-taking skills and absence of verbatim copying.

8. Short answer questions (based on the reading). The reading excerpt is collected, and students answer a general question using only their notes. This writing sample and the lecture writing sample are scored on a ten-point rubric for writing correctness and for content accuracy.

After the tests are scored, each student receives an individual report which shows his or her strengths and weaknesses and, in the posttest, relative progress. There is no combined, overall or passing score, because this would result in the loss of specific information about students’ strengths and weaknesses. Normative information for interpreting test results is provided on the student score reports.

The curriculum emphasizes academic vocabulary because the initial field tests revealed that its absence constitutes by far the most important barrier to the comprehension of lectures and text. Students learn prefixes, suffixes and roots that indicate meaning. They also learn the nominal, verbal, adjectival and adverbial forms of a word and markers that indicate plural, past tense, or gerund forms. To help remember the meaning of new words, each is marked with an icon representing one of eleven semantic categories such as person characteristics; cause, effect and change; evaluation and description, and so on. Each lesson has a dictionary in which new words are defined in non-academic language, and used in context.

The lessons also provide instruction on note-taking, on recognizing the markers of lecture organization, and on reading strategies such as identifying important information and analyzing complex sentences. Because many students who need this kind of assistance are also the first in their families to attend college, the curriculum includes information on academic culture.

Evaluation and Project Impact

The assessment and curriculum have been implemented at various levels at 19 sites, including high schools, community colleges, and universities.

One group of community college students served as controls by taking the pretest and the posttest before and after their first two semesters of college with no exposure to the project curriculum. The results showed that merely attending college classes has no measurable effect on the academic language skills development of underprepared language-minority students and that practice on the tests alone, without benefit of the curriculum, does not increase scores. Academic language scores for these students showed a moderate negative correlation with the number of units dropped during the two semesters. In other words, the lower the academic language proficiency, the more units students withdrew from during the year. This suggests that low academic language skills are related to low persistence and low achievement for beginning college students.

The longest use of the assessment and curriculum so far took place in a large, predominantly Hispanic, rural comprehensive high school with a 50 percent graduation rate. During 1996-97, 103 students took the pretest, and 84 the posttest. The results show that, after exposure to only three of the eight lessons—taught part-time over five months—the students made significant progress, especially in taking notes from lecture and reading and answering a reading question using their notes. There were also significant increases in self-ratings of vocabulary knowledge and small increases in other areas of the test.

In a college remedial reading class, attendance and retention changed from extremely poor to perfect when the project curriculum was introduced, and students found the curriculum valuable and interesting. Most importantly, scores in the posttest showed significant increases in academic language skills.

Results from an Upward Bound program site showed statistically significant increases in academic language test scores. Interviews with students at this site revealed positive changes in their attitude about and knowledge of academic language and academic culture. Students reported that the course helped them improve their grades and skills in their high school classes.

Students enrolled in an anatomy and physiology course at one four-year institution were given academic language assessment tasks and the results were compared with their course grades. Correlations ranging from .35 to .50 were found between the various academic language tasks and the course grades. Students who received passing scores in the course had significantly higher academic language scores than those who received a D or F in the course. All students who completed the course had higher academic language scores than did students who withdrew from the class during the semester.

In summary, in all contexts where the assessment and curriculum were used, students made significant improvements in their academic language proficiency and had positive attitudes toward the course. Evaluation evidence suggests that academic language proficiency is related to persistence and academic achievement in postsecondary education.

Lessons Learned

Before the assessment and curriculum could be implemented, project staff had to overcome certain obstacles related to prevalent theories and traditional attitudes toward language learning and remedial instruction.

Study skills teachers generally are not keenly aware of the actual requirements of regular college courses, and most base their teaching on the book and on their intuition—as opposed to factual information about what students need to know. In ESL classes, there is often even less awareness of what students need to succeed in college. Often, as ESL students become more orally fluent they are mainstreamed, without assessment of their writing and other academic proficiencies, which likely lag behind.

The whole-language approach, which holds that literacy, like spoken language, develops simply through exposure, has often resulted in the neglect of specific instruction in reading, phonics, spelling and writing skills. Although many teachers are moving away from this approach, there is still resistance to such techniques as vocabulary instruction and skills-based approaches based on expository text.

Academic language development is a long-term process. The curriculum and assessment created during this project aim to give students the tools to continue learning new vocabulary and overcoming their weaknesses long after the course is over. The project should continue to bear fruit as these students advance through the academic ranks.

The results of this project confirm the notion that academic language is, in a sense, a second language for most students. On the pre- and posttests, the profiles of native speakers of English and of students classified as fully English proficient and limited English proficient were almost indistinguishable. In fact, in some tasks students with limited proficiency had slightly higher scores than the other groups—possibly because of better schooling in their first language.

The curriculum is effective only with a teacher who understands the language needs of students and who is enthusiastic and effective. A teacher who does not engage and motivate students, showing them the relevance of academic language tasks, will not see much improvement.

Project Continuation and Dissemination

The curriculum continues to be disseminated and its effectiveness documented based on teacher and student reactions. The materials have been used by, among others, several university Upward Bound programs and at sites in several school districts.

The project received a new grant under FIPSE’s 1997 Comprehensive Program competition to disseminate and evaluate the program’s effectiveness more widely. The new evaluation efforts include looking for long-term effects on students. At some sites the course may be expanded to a two-semester sequence.

Available Information

Further information may be obtained from:

Phyllis A. Kuehn
Department of Educational Research
California State University, Fresno
Fresno, CA 93740-8025
Telephone: 559-278-0323

[Western Interstate Commission for Higher Education (WICHE)] [Table of Contents] [III. Improving Teaching and Learning]

FIPSE Home