The Institute of Museum and Library Services
(IMLS) is a federal agency that fosters leadership, innovation,
and a lifetime of learning through grants to museums and
libraries. Please see IMLS’s Web site at www.imls.gov
for additional information about IMLS and its grant programs.
If outcome-based and other formal program
evaluation methods are new to your institution, many helpful
publications are available to introduce them. IMLS offers
this list as a resource; it is not intended to be limiting
or exclusive, and it is not an endorsement of any particular
publication or web site.
These resources draw most of their examples
from educational and social service settings, but many are
readily applicable to typical goals of library and museum
programs. URLs change frequently and it is very difficult
to keep pace. If you find a closed or incorrect link, try
entering the full title of the publication or resource in
a good search engine. This will often direct you to the
new URL.
Manuals, Guides, and Other Resources
Many titles below are available at no cost
online. While terminology differs from publication to publication,
basic concepts are very similar. All are designed for use
by organizations who want to know the results of their programs
in terms of human benefits, whether those are called “impacts,”
“results,” or “outcomes.”
Administration on Children, Youth, and Families,
Department of Health and Human Services (nd). The Program
Manager’s Guide to Evaluation. Washington, DC:DHHS,
http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/pmguide_toc.html.
This excellent introduction was developed for grantees of
this program and provides very concrete, practical explanations.
It is accompanied by additional guides for specific kinds
of human services programs funded by the agency. Available
as of 1/16/08.
Bond, Sally L., Boyd, Sally E., and Rapp,
Kathleen A. (1997). Taking Stock: A Practical Guide
to Evaluating your own Programs. Chapel Hill, N.C.:
Horizon Research, Inc., http://www.horizon-research.com/publications/stock.pdf.
This manual was developed for community-based science education
initiatives through funding from the DeWitt Wallace-Readers
Digest Fund. Participating advisors included the Association
of Science-Technology Centers and the National Science Foundation.
Available as of 1/16/08.
California State Library (nd). Outcome
Measurement Information at http://www.library.ca.gov/grants/lsta/grants.html
includes an excellent bibliography referencing a number
of journal articles about outcomes-based evaluation in the
library world (see http://www.library.ca.gov/grants/lsta/docs/OutcomeMeasurementBibliography.pdf).
Available as of 1/16/08.
Diamond, Judy (1999). Practical Evaluation
Guide: Tools for Museums and Other Informal Educational
Settings. Walnut Creek, CA: Alta Mira Press, 192 pp.
Available from American Association of State and Local History,
1717 Church Street, Nashville, TN 37203-2991. This guide
provides a concrete guide to tools and approaches for understanding
how well programs and exhibits communicate the intended
messages to museum audiences. It includes samples of numerous
strategies for collecting information on museum learning,
and describes how to construct and use them.
Joan C. Durrance and Fisher, Karen E. with
Marion Bouch Hinton (2005). How Libraries and Librarians
Help: A Guide to Identifying User-Centered Outcomes.
Chicago: American Library Association, 183 pp. Chapters
include Introduction and Overview; Organizational Readiness
Survey; Preparing for Outcomes; Logic Model; and Data Collection
Plan. See the closely related Information Behavior in Everyday
Contexts (IBEC), below.
Hernon, Peter and Robert E. Dugan (2002).
Action Plan for Outcomes Assessment in Your Library.
Chicago, IL: American Library Association, 192 pp. This
recent title provides data collection tools to measure learning
and research outcomes linked to user satisfaction. This
practical, how-to manual, with detailed case studies from
actual outcomes assessment programs, is intended to help
academic librarians: plan an outcomes assessment strategy
in line with accrediting guidelines; identify user needs,
collect and analyze data; present findings; and measure
the value of services, and identify areas for improvement.
The Plan includes a chapter designed to help public libraries
apply its concepts.
Information Behavior in Everyday Contexts
(IBEC), University of Washington Information School and
University of Michigan School of Information joint venture
(2004). http://ibec.ischool.washington.edu/ibecCat.aspx?subCat=Outcome%20Toolkit&cat=Tools%20
and%20Resources. . A simple, flexible effective methodology
for evaluating outcomes, targeting libraries and community-focused
services. It includes worksheets and examples, and the method
is in the process of being piloted by a group of public
libraries from large to small for a variety of typical library
programs. Available as of 1/16/08.
Innovation Network provides an online "workstation"
approach to program development and evaluation planning
at http://www.innonet.org/index.php?section_id=64&content_id=185
that allows users to build a program plan, an evaluation
plan, and a budget, essentially most of the pieces important
to a competitive proposal or good program management. You
must register (no charge) to access the tool. It fully integrates
the program planning and evaluation planning pieces and
provides tools (under resources) for developing data collection
strategies and instruments. Available as of 1/16/08.
Korn, Randi, and Laurie Sowd, (1999). Visitor
Surveys: A User’s Manual, Professional Practice
Series Nichols, Susan K. (Compiler); Roxana Adams (Series
Editor), Washington, DC: American Association of Museums,
164 pp. Available from AAM Technical Information Service,
1575 Eye Street, NW, Suite 100, Washington, DC 20005. This
thorough introduction to visitor surveys in museums may
also be useful in library and evaluation contexts; includes
worksheets and samples.
Joseph R. Matthews (2004). Measuring for
Results: The Dimensions of Public Library Effectiveness.
Westport, CT: Libraries Unlimited, 240 pp.
New York State Library (2006). Outcome-Based
Evaluation, http://www.nysl.nysed.gov/libdev/obe/index.html.
Resources developed as part of a multi-year training plan
to develop outcomes-oriented planning and measurement capacity
for New York’s library personnel. Available as of
1/16/08
Rubin, Rhea (2005). Demonstrating Results:
Using Outcome Measurement in Your Library. Chicago,
IL: ALA, 176 pp. This latest addition to the PLA-sponsored
Results Series uses familiar task breakdowns along with
key terms in a step-by-step, service-oriented format so
that readers can master the outcome measurement process.
Applying these concepts in examples and in two running case
studies, an Internet class for seniors, and a teen mother-tutoring
program brings the model to life. The “Toolkit”
includes tips on creating evaluations, coding data, and
selecting a sample. Rubin suggests that by assessing the
impact of services in users’ lives, public library
directors and program managers demonstrate to funding bodies
their accountability and the effectiveness of programs,
thus positioning their libraries to secure maximum funding.
State Library and Archives of Florida (2004).
Library Services and Technology Act Outcome-Based Training
Toolkit (LSTA Toolkit), http://www.lstatoolkit.com.
Includes an Outcomes Plan "wizard" and data report
forms for Florida LSTA grant reports; point-of-need instruction
in outcome-based evaluation, instruction on data collection,
tools for data analysis, guidance in reporting project progress,
and strategies for reporting project successes and results
Taylor-Powell, Ellen, Sara Steele, and Mohammed
Douglah (1996), Planning a Program Evaluation.
University of Wisconsin-Extension, Madison, WI. This is
an excellent suite of text-based resources for program development
and evaluation that allows individual "what I want
to know, when I need to know" approaches. While examples
come from the community service context, this is well constructed
and well written, and provides a significant amount of detail,
including report examples, sampling protocols, and other
elements of a genuinely comprehensive all-text "toolkit."
Available online at http://www.uwex.edu/ces/pdande.
Available as of 1/16/08.
Texas State Library and Archives Commission
(nd), Austin, TX, Outcomes., http://www.tsl.state.tx.us/outcomes.
An excellent, wide-ranging resource developed primarily
for local library users, this includes a bibliography, examples,
and an online forum. Includes many resources also identified
here. Available as of 1/16/08.
W.K. Kellogg Foundation Evaluation Handbook
(January 1998). Available via Acrobat PDF at http://www.wkkf.org/pubs/Tools/Evaluation/Pub770.pdf
as of 1/16/08. An excellent and tested guide for social
service and educational programs, from a foundation that
provides significant funding for not-for-profit organizations.
See also http://www.unitedway.org/Outcomes/Library/pgmomres.cfm.
Sage Publications, Inc., 2455 Teller Road,
Thousand Oaks, CA 91320, 805-499-0721 or http://www.sagepub.com
is a commercial publisher that specializes in publications
on evaluation and related subjects. They offer many titles
that cover aspects of evaluation in detail.
Finding and Making Good Use of a Consultant
OBE is still new to library and museum management,
so as yet very few consultants combine knowledge of program
evaluation, outcomes measurement, and the library or museum
context. The best source of contacts is others in your field.
If you cannot identify someone with all the experience you
need, it is probably good strategy to hire a well-recommended
generalist and to invest in orienting the consultant to
your institution and its culture.
The Nonprofit Resource Center. Management
Resources and Consultants for Nonprofits. Available
at http://www.not-for-profit.org,
this source is designed for administrators, board members
and volunteers of nonprofits; provides a listing of individuals
and firms that offer consulting services to 501(c)(3) organizations.
Click on "Consultants" under the Support Organizations
menu heading. Available as of 1/16/08.
The Foundation Center Learning Lab. Working
with a Consultant or Technical Assistance Provider: A Resource
List, http://foundationcenter.org/getstarted/topical/consult.html.
Available as of 1/16/08.
TechSoup.org. Managing a Consultant.
Short and pithy tips for non-profits, specific to technology
consultants, but applicable to program evaluation and many
other needs. Available at http://www.techsoup.org/articlepage.cfm?articleid=53&topicid=8
as of 1/16/08.
Langford, Linda, and William DeJong. How
to Select a Program Evaluator (2001). The Higher Education
Center for Alcohol and Other Drug Prevention. Available
at
http://www.edc.org/hec/pubs/prev-updates/prog-eval.pdf
as of 1/16/08.
Department of Health and Human Services. How
Do You Hire and Manage an Outside Evaluator? Tips for
finding and managing program consultants, available at http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/
chapter_4_pmguide.html, Chapter 4 from the DHHS The
Program Manager’s Guide to Evaluation. Available as
of 1/16/08
Baselines, Outcome Indicators, Data
Collection Tools, and Related Resources
A number of web sites offer additional guidance
for outcome-oriented evaluation of community service and
educational programs. You may find the following useful:
American Evaluation Society, http://www.eval.org.
Links, publications, and information about evaluation training.
Links are provided to consultants offering evaluation services;
none of these have been assessed by IMLS and no endorsements
are implied. Available as of 1/16/08.
Arts Education Partnership, the President's
Committee on the Arts and the Humanities and others have
produced Champions of Change: The Impact of the Arts
on Learning (2000), http://artsedge.kennedy-center.org/champions,
a publication edited by Edward B. Fiske that reports findings
of a number of research projects designed to measure concrete
impacts on educational achievement from sustained activities
in the arts. It provides examples of significant indicators
and a model of how evaluation (in this case systematic research)
can demonstrate impact. Available as of 1/16/08 (6/4/04).
Association of College and Research Libraries,
http://www.ala.org/acrl/sacguid.html.
Bibliography of Information on Performance and Outcome,
assembled and annotated by the Assessment Standards
and Accreditation Committee. Available as of 1/16/08.
The Finance Project http://www.financeproject.org
is a non-profit organization formed in 1994 to “support
decision making that produces and sustains good results
by developing and disseminating information, knowledge,
tools and technical assistance for improved policies, programs,
and financing strategies.” See in particular its publications,
as an example Evaluation and Performance Measurement
Capabilities for Internet Technologies in Human Services
(Jim Frech, January 2003), http://prostores2.carrierzone.com/servlet/financeprojectorg/Detail?no=2663
($10.00). Available as of 1/16/08.
Free Management Library
http://www.mapnp.org/library/evaluatn/evaluatn.htm provides
extensive information about evaluation including materials
developed by Carter McNamara for The Management Assistance
Program for Nonprofits, plus links to manuals for evaluation
and other resources Available as of 1/16/08.
The Harvard Family Research Project http://www.gse.harvard.edu/hfrp/eval.html
publishes a free newsletter for evaluation professionals
(The Evaluation Exchange) and has a helpful Web
site that includes full-text publications on evaluation.
Available as of 1/16/08.
Independent Sector, http://www.independentsector.org/issues/accountability.html
offers pointers to resources and guides. Available as of
1/16/08.
Library Research Service, Colorado Department
of Education,
http://www.lrs.org. This invaluable resource provides
social and economic statistics, technology statistics, data
about libraries and communities, information about research
methods, and tools for statistical analysis. Designed primarily
for library planning, but provides baselines against which
outcomes can be measured for indicators appropriate to many
library and museum educational programs. See especially
Counting on Results, a report of IMLS-funded research
to develop common library outcomes measurement tools. Available
as of 1/16/08.
National Assessment of Educational Progress
(NAEP), http://nces.ed.gov/surveys/SurveyGroups.asp?Group=4,
provides access to statistics applicable to measuring the
educational impact of library and museum programs. It provides
research data on educational indicators in the arts, science,
math, and history among other topics, which suggest useful
indicators and baselines against which program outcomes
can be measured. Available as of 1/16/08.
National Center for Educational Statistics
National Education Longitudinal Study of 1988 (NELS:88),
http://nces.ed.gov/surveys/nels88
reports findings of major longitudinal effort designed to
provide trend data about critical transitions experienced
by students as they leave elementary school and progress
through high school and into post-secondary institutions
or the work force. Data includes indicators of student learning,
early and late predictors of dropping out, and school effects
on students' access to programs and equal opportunity to
learn. These can provide baselines applicable to library
and museum educational and outreach programs. Available
as of 1/16/08.
National Science Foundation Directorate for
Education and Human Resources, http://oerl.sri.com.
The Online Evaluation Resource Library (OERL)includes plans,
instruments, and reports that have been used to conduct
evaluations of projects funded by the Directorate for Education
and Human Resources of the National Science Foundation.
OERL also contains glossaries of evaluation terminology,
criteria for best practices, and scenarios illustrating
how evaluation resources can be used or adapted. Examples
and tools are in the context of science and related applications,
but are transferable. Available as of 1/16/08.
The Staff Room for Ontario’s Teachers,
Rubrics, http://www.cmsdnet.net/allianceplus/dillka01/rubrics2.htm,
provides a broad array of sample rubrics (tools for comparing
progress towards common outcomes) with links to several
no-cost online programs for creating your own rubrics. Available
as of 1/16/08.
United Way Outcome Measurement Resource Network,
http://www.unitedway.org/Outcomes/Library/pgmomres.cfm,
provides a broad array of readings, resources, and methods
applicable to evaluation. Available as of 1/16/08
University of Michigan Institute for Social
Research Survey Research Center, Monitoring the Future,
http://monitoringthefuture.org
reports findings of an ongoing study of the behaviors, attitudes,
and values of American secondary school students, college
students, and young adults. It provides baseline data for
numerous indicators of program impact on education, behavior,
and attitudes for a large sample of these groups. Available
as of 1/16/08.
Western Michigan University Evaluation Center’s
http://www.wmich.edu/evalctr/index.html.
Principal activities are research, development, dissemination,
service, instruction, and leadership for advancing program,
personnel, and student/constituent evaluation, applied primarily
to education and human services. This site includes a directory
of evaluators http://ec.wmich.edu/evaldir/index.html
that can be searched by country, region, state, organization,
or area of specialty. Note that the listed professionals
represent many approaches to evaluation, and do not all
have expertise in outcomes measurement or in library or
museum contexts. Available as of 1/16/08.
Automated Online Survey Tools
The number of commercial products and services
for this purpose is increasing steadily, and those below
are given as examples only, with no implied recommendation.
Others can be found by searching for “Online Survey
Software” using Internet search engines. All are designed
to make the job of creating and using online surveys simple
for users. They typically provide a variety of templates
and standard surveys, as well as tips for building effective
surveys. Information is automatically tabulated as it is
entered by the responder, with the data automatically compiled
by the product/service for the user to interpret. Costs
and fee structure vary from product to product. A reminder–surveys
are not the best tool for all outcomes.
WebSurveyor http://www.websurveyor.com.
The fee structure can be fee-per-survey or a multi-survey
license arrangement.
SurveyMonkey http://www.surveymonkey.com.
You can try this site at no charge for up to 100 responses
to up to 10 questions. Easy to use. Full access is available
by monthly or annual subscription.
Zoomerang, http://info.zoomerang.com/prodserv/ps.htm.
The fee structure is an annual subscription.
1/16/08
|