 | |||||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
|
| | |||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
Detailed Evaluation of a Novel Approach to Curricular Software
Purpose
Exploding numbers of underprepared students and shrinking budgets are only some of the factors that make teaching on American campuses today increasingly difficult. Under these conditions, it is only natural for faculty and administrators to turn to the computer for the solution to problems ranging from cost effectiveness to remediation.
Educational software holds immense promise for solving these dilemmas and improving teaching and learning, but it is difficult to design, requires costly equipment, and demands considerable adjustment in the way that faculty teach. Given the weight of these investments and the growing popularity of computers in the classroom, it is imperative that new software receive a thorough and uncompromising evaluation.
From 1987 to 1990, with FIPSE funding, the Curricular Software Studio at Tufts developed ConStatS, a software package to help introductory statistics students to assume an active, experimental style of learning. The designers of ConStatS believed that developing this learning style was essential to the acquisition of a deep conceptual understanding of the subject. Every ConStatS program is divided into many screens, each of which confronts the student with a small number of closely related decisions. The student's choices lead down different pathways, some of which loop into each other. At any point, the student can back up along a pathway, or use the WHY and HELP buttons available on every screen to obtain additional information.
Although, during the original project, formative evaluation showed that ConStatS was free of bugs and students felt comfortable using it, its creators wanted to find out whether the experimental approach that the software fosters made a difference in learning. They also wanted to find out how students used the software and which portions of it worked with what type of students, especially underprepared ones. Finally, they wanted to provide an example of good software evaluation, a model that would focus on comprehension and retention rather than on student attitudes and speed of learning, as much technology assessment does.
Innovative Features
Several factors made it imperative that the evaluation include a number of sites. Project staff wanted to find out whether ConStatS would help students in different courses, different departments and different institutions to learn statistics. Besides, if only one or two sites were used, factors unique to the course or the institution could overly influence the results, a common defect of technology evaluations.
Accordingly, staff recruited faculty at the University of New Hampshire, Gallaudet University, Colorado College, and Bowdoin, in addition to Tufts, to use ConStatS as they saw fit. This flexibility in how the software was used made faculty at all five sites eager to cooperate. Unfortunately, it was not possible to recruit a community college, perhaps because these institutions offer fewer statistics courses and have less technology. Overall, 20 introductory statistics courses participated in the project. Sixteen used ConStatS and four did not.
Because faculty use ConStatS in many different ways (Tufts alone has eight introductory statistics courses in various departments, all of which employ ConStatS differently), the evaluation needed to include a record of how the program was used. Project staff used a multivariate assessment measure to capture the effectiveness of each part of the program and performed detailed testing and extensive monitoring of implementation.
During the first year of the project, staff developed a pretest of the skills necessary for students to use ConStatS and a detailed test of conceptual understanding that assessed every part of the program, to be taken by students in the twenty participating courses. They also designed a feature in the program for recording how each student used the software. The test of conceptual understanding required a detailed analysis of over 200 ConStatS screens, which yielded over 1,000 comprehension points. Because it was impossible to test all the points, they were grouped into clusters, which in turn yielded 103 concepts. For each concept, a question testing understanding was formulated and then reviewed by two outside statistical experts and teachers of quantitative methods.
Designing the feature for recording interaction required describing the kinds of interactions offered by ConStatS and linking them to the educational purpose that motivated each one—such as executing an experiment, asking for help, or studying a result. The next step involved engineering a software tracing function that would capture each interaction, interpret it, and include it in a database. This permitted project staff to account for differences in performance based on differences in use.
All participating faculty were given a one-day workshop before the evaluation began. Project staff communicated biweekly with all the sites, and made visits to install the program and perform related tasks. In the second and third years of the project, students in the 20 participating classes took the test of conceptual understanding, which consisted of questions on 103 statistics concepts taught both in the software and in the classes that did not use the software.
Project Impact
The evaluation results strongly suggest that ConStatS makes a big difference—students using ConStatS did better on 94 out of the 103 questions testing understanding of statistical concepts. These results do not appear to be due to increased learning time, because at least two courses substituted a computer laboratory for a class. Nor are they tied to a particular way of integrating ConStatS into the curriculum, because the results were comparable among classes with different approaches
Students with poor mathematics backgrounds achieved fewer gains than better-prepared students. A ten question pretest examining basic mathematics proficiency (middle- and high-school levels) predicted accurately which students would gain most from ConStatS at all sites, perhaps because low scores on the test were symptomatic of serious academic deficits. It is possible, however, that if students had been tutored in fractions and ratios (the two most commonly missed items on the pretest) they would have shown greater gains from ConStatS.
The trace data enabled staff to connect differences in class scores and software use. Some classes made more extensive use of certain portions of the program than others. Also, trace data allowed the linkage of student performance to the different ways in which individual students used the software. In general, students who tended to stay in one spot in the program or moved slowly were less likely to be engaged by ConStatS and learned less from it.
The evaluation confirmed the project staff's impression that most students leave introductory statistics with little conceptual understanding of the material. Students who did not use ConStatS averaged only 31 percent on the test of conceptual understanding. Even though scores were substantially higher in classes that used ConStatS, students were still not learning statistical concepts as well as they should have.
Lessons Learned
Project staff had expected the evaluation to reveal ConStatS to be an unqualified success. Instead they learned that, although the software is undoubtedly useful, students failed, for unknown reasons, to learn a number of concepts. Some of the handsomest illustrations in the program, the ones that received the most acclaim at conferences, did not promote learning. It is difficult, staff found out, to anticipate how students will learn from a given illustration.
The hope that all students would engage in active experimental learning with the help of well-designed software was not completely fulfilled, because students with learning deficits showed only a marginal gain. For students whose mathematics background was appropriate for college-level work, however, ConStatS was helpful.
Project Continuation
The software continues to be used at Tufts in psychology, economics, biology and civil engineering courses.
Dissemination and Recognition
ConStatS has been the subject of many conference presentations and a number of articles in refereed journals. It was published by Prentice-Hall, Inc., in 1996.
Available Information
For further information, contact:
Richard Chechile
Psychology Department
Tufts University
Medford, MA 02155
Telephone: 617-627-3765
[Southwest Texas State Univ.] [Table of Contents] [West Virginia Univ.]
|
||||||||||||
|
|
|||||||||||
| |
||||||||||||
