Outreach Evaluation Resource Center
Annual Report
Year One May 2006-April 2007

Table of contents

• Executive summary
• Web site
• Dissemination of effective practices
• Subcontractor update N/A
• Other staff activities
• Appendix One: Classes and consultations
• Appendix Two: Evaluation liaisons feedback session

Executive summary

• NN/LM National Emergency Preparedness Project

  Using the Empowerment Evaluation approach, the OERC facilitated the eight regions' selection of emergency preparedness as the focus for national collaboration; led subsequent planning meetings; and activated the Emergency Preparedness page of the NN/LM StaffWiki. By the end of April 2007, the core group had developed goals, strategies, major plan components, a logic model, and potential cross-regional workgroups for presentation and discussion at the May 18 RML Directors' Meeting in Philadelphia. See “Other staff activities, Section V.” of this report for more details.

• Evaluation Capacity-Building

  The OERC initiated regular communication among the regions' evaluation liaisons through an email discussion list (rmleval@u.washington.edu) and monthly Evaluation Liaisons teleconferences. Susan Barnes began to put in place a model for expanding evaluation teaching capacity in the regions through the popular workshop on using evaluation for library advocacy, “Measuring Your Impact.” Ms. Barnes helped staff from three regions prepare for and teach this workshop during this contract year; staff members from five regions are now available to provide instruction on this topic. Cindy Olney completed the first two classes in the OERC's new “Outreach Evaluation Series:” “ Community Assessment: Creating a Community Profile” and “Developing an Outcomes-Based Project.” These classes have each been approved for three MLA CE contact hours. The OERC joined the group of Macromedia Breeze/Adobe Connect early adopters within NN/LM by using Breeze/Adobe Connect for two all-NN/LM teleconferences, evaluation overviews for hospital librarians in two regions, for the Evaluation Liaisons' teleconferences, and for NN/LM Emergency Preparedness Project teleconferences. The Adobe Connect web meeting software is proving to be very valuable in reaching groups of people who are scattered across great distances. The OERC also sponsors an evaluation blog that posts summaries of informative articles related to evaluation (with links to the articles) and announcements of evaluation conferences and training opportunities.

• Evaluation Assistance

  OERC staff provided consultation for almost all regions (both to RML staff and network members), the NTCC, Web-STOC, and MLA. OERC staff members have presented (or are scheduled to present) training sessions on a number of evaluation topics including evaluation overviews, logic models, focus groups, logic models and “shoestring evaluation methods”. They also have provided technical assistance to RML staff on evaluation plans, logic model design, focus group studies, questionnaire design, sampling, and response rates. The final quarter in particular saw an upswing in requests for technical assistance on evaluation projects.

• Publications

  The OERC published, made available in full text on the web, and distributed the new “Planning and Evaluating Health Information Outreach” series, composed of three booklets:

  1. Getting Started with Community-Based Outreach
  2. Including Evaluation in Outreach Project Planning
  3. Collecting and Analyzing Evaluation Data

Cathy Burroughs' guide “Measuring the Difference” remained available in full text on the web and was also sent to requestors during the year. Through Web-STOC's Usability Evaluation and Review project, OERC staff made extensive revisions to the OERC web site and participated in a usability study of the site. The OERC and Web-STOC developed the new OERC Blog, and the OERC took responsibility for the NN/LM StaffWiki page on Emergency Preparedness. The OERC distributed 376 sets of “Planning and Evaluating Health Information Outreach” booklets and 55 copies of “Measuring the Difference” to requestors in 23 states, the District of Columbia , two Canadian provinces, the UK , and Peru .

• Year One by the Numbers

  	Evaluation Consultations	National Collaboration	Total
  Q1	12	1	13
  Q2	17	8	25
  Q3	10	7	17
  Q4	24	13	37
  Total	63	29	92
  Note: The numbers represent consultations where OERC staff talked with clients. It does not reflect preparation time for each consultation, which took from a few minutes to, in the case of the national collaboration, days of preparation

  

  Training Date	Class	Region	Attendance
  8/30/2006	Evaluation Overview	SE/A,	28
  8/17/2006	MYI	GMR,	16
  3/5/2007	MYI	PSR,	25
  3/10/2007	MYI	SE/A,	6
  4/18/2007	Evaluation Overview	PNR	24
  		Total	99

  Note: see Appendix One for all Classes and Consultations, May 1, 2006 - April 30, 2007.

Web Site

Susan Barnes served as the OERC representative to Web-STOC's Usability Evaluation and Review (UER) group. As part of the UER project, she migrated the OERC web site into the revised organization, new navigational structure, and new templates of the overall nnlm.gov site that had been developed from AIR's recommendations. In spring 2007, Ms. Barnes began work with AIR on its OERC web site usability study, conducted as part of Web-STOC's contract with AIR. After finalizing the test plans and collecting a pool of study participants, AIR conducted four usability sessions (on 4/2/07, 4/17/07, 4/18/07, and 4/19/07). Study participants were drawn from these groups of RML staff: Outreach Coordinators, Consumer Health Coordinators, and Evaluation Liaisons. RML staff was used because at present that is the principal user population for the site. Ms. Barnes observed the sessions, which employed the UserVue software from Techsmith (similar to Adobe Connect). Each session was about 90 minutes in duration and resulted in a large amount of valuable feedback. Reactions to the site were generally positive. AIR's final report was still in progress at the end of the contract year, but while observing the sessions, Ms. Barnes collected ideas for updates to the web site, including adjustments to navigational categories (links to publications could be more obvious; difference between "publications" and "resources" is not clear; link to "national collaborations" is confusing), streamlining contents (fewer blocks of text; move internal reports to NN/LM StaffWiki), and improvements to workshop contact information.

OERC staff collaborated with Web-STOC to create the OERC Blog at http://nnlm.gov/evaluation/blog/, announced it to all eight regions via the nnlmall@u.washington.edu email list, and began to use it for communication with the regional evaluation liaisons and the NN/LM Emergency Preparedness Project core group. The NN/LM StaffWiki was also used for communication about this project and served as a repository for planning documents and a referral point for links to resources. The wiki and email discussion list proved more successful than the OERC Blog for project communication and management. The blog format seems better suited for brief news items, so the decision was made to restructure the OERC Blog to feature practical advice and lessons learned, and to retain the News category.

Staff added the full text of the three booklets in the Planning and Evaluating Health Information Outreach series to the OERC web site's “Publications” section at http://nnlm.gov/evaluation/guides.html#A2. The full text of Cathy Burroughs' Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach was restructured for improved usability at http://nnlm.gov/evaluation/guides.html#A1.

Dissemination of effective practices

Susan Barnes participated as a member of the TC4C Effective Practices team, with specific responsibility for compiling a bibliography of publications about effective practices in health information outreach. This committee is entering effective practices on the Effective Practices Collection web site at http://nationalserviceresources.org/epicenter/. As part of Ms Barnes' participation on this team, she developed an articulation of the difference between “best practices” and “effective practices”:

“Effective practices" implies health information outreach strategies that have been successful, useful, or promising, based on experience and evidence. Although the terms "effective" and "best" practices are often used interchangeably, the OERC differentiates between them. "Best practices" have been rigorously evaluated and represent the "Gold Standard" of activities to support specific objectives. In this context, "effective practices" indicates experiences with what has worked in a given situation. ( This concept is from the Johns Hopkins INFO Project, http://www.infoforhealth.org/practices.shtml.)

The OERC has implemented a plan for disseminating information about effective practices through the monthly Evaluation Liaison teleconferences. The Learning Circles, which started in February, now will be designed to showcase an evaluation project in one of the regions, with that region's evaluation liaison providing a report of “lessons learned” about the Learning Circle topic. This practice began in April, with Karen Vargas beginning the Learning Circle by talking about focus groups she had conducted in the NN/LM SCR.

The OERC distributed 376 sets of “Planning and Evaluating Health Information Outreach” booklets and 55 copies of “Measuring the Difference” to requestors in 23 states, the District of Columbia , two Canadian provinces, the UK , and Peru .

Subcontractor update

Not applicable. The OERC has no subcontracts at present.

Other staff activities - OERC goals for the 2006-2011 contract

1. Develop awareness, skill, and capacity in use of evaluation methods.

   The OERC joined the group of Macromedia Breeze/Adobe Connect early adopters within NN/LM by using Breeze/Adobe Connect in its August 1 all-NN/LM teleconference, “What the OERC Can Do for You,” in its August 30 presentation about evaluation to 28 hospital librarians in the Southeastern/Atlantic Region of the NN/LM, in the October 10 all-NN/LM teleconference about the new “Outreach Evaluation Series” of classes, for the RML Evaluation Liaisons' and NN/LM Emergency Preparedness Project teleconferences, and in its April 18 presentation about evaluation to 24 hospital librarians in the Pacific Northwest Region of the NN/LM. The Adobe Connect web meeting software is proving to be very valuable in reaching groups of people who are scattered across great distances.

   Through assisting Ruth Holst and Jacqueline Leskovec of NN/LM GMR in their co-teaching “Measuring Your Impact” at the University of Minnesota Biomedical Library in Minneapolis, Susan Barnes began to develop a model in which this class about using evaluation for library advocacy would provide a way to involve NN/LM staff in teaching evaluation concepts. Ms. Barnes co-taught “Measuring Your Impact” with Jacqueline Leskovec in San Francisco (to a group primarily comprising academic librarians from the University of California, San Francisco and from affiliated institutions) and with NN/LM SEA's Dale Prince in Young Harris, Georgia (to a group primarily comprising hospital librarians in the Georgia Health Sciences Library Association). Dale had attended the San Francisco class and an earlier outreach evaluation class in preparation for the Georgia class and possible future classes in NN/LM SEA; Heidi Sandstrom and Julie Kwan attended the San Francisco class in preparation for possible future classes in NN/LM PSR. The “Measuring Your Impact' class was originally developed—and is still being taught—by Betsy Kelly of NN/LM MCR and Maryanne Blake of NN/LM PNR, so now there are eight NN/LM staff members from five regions prepared to teach evaluation concepts.

   Cindy Olney began the new OERC workshop series based on what was learned from the workshops that were piloted in January and February 2006 (at the end of the previous contract). As part of developing this new series, Dr. Olney and Ms. Barnes held a Macromedia Breeze meeting with the regions' two most experienced evaluation liaisons, Maryanne Blake and Betsy Kelly. Ms. Blake's and Ms. Kelly's f eedback led to the decision to cut the series back from five sessions to four half-day sessions (Community Assessment, Planning Outcomes-Based Outreach Programs, Data Collection for Health Information Outreach, and Data Analysis for Health Information Outreach). The series title “Outreach Evaluation Series” was developed—eliminating the earlier title of “Measuring the Difference” that had been in use since the original outreach evaluation classes were taught during the last contract by Maryanne Blake and Cathy Burroughs. At the same time, the OERC renamed the “Measuring Your Impact: Using Evaluation to Demonstrate Value” workshop to “Measuring Your Impact: Using Evaluation for Library Advocacy” in order to better differentiate it from the outreach evaluation sessions. The OERC hopes that these titles will eliminate most of the confusion that had begun to appear regarding the difference between the library advocacy workshop and the outreach evaluation workshops. Dr. Olney finalized course materials for the first two workshops in the “Outreach Evaluation Series,” Community Assessment and Planning Outcomes-Based Outreach Projects. These were approved in by MLA in April for 3 contact hours each.

   The OERC is also building skill and capacity through consultations. Staff has provided consultation on a wide range of issues and has served as coaches on projects including survey development, sampling, focus group studies, logic models, and evaluation planning. Either through consultations on specific projects or through monthly teleconferences with evaluation liaisons, the OERC has provided support to every region. Not including consultations related to the national collaboration, the number of consultations per region ranged from 8 to 16 with a median number of 14 consultations per region. Consultations were also provided to the Web-STOC, NTCC, MLA, and NLM fellows.

2. Develop mechanisms to disseminate and use evaluation findings to help NLM and the NN/LM document and demonstrate their accomplishments.

   The “Web site” and “Dissemination of effective practices” sections of this report summarize the bulk of the OERC's activities toward this goal. A questionnaire sent to the evaluation liaisons showed that most are using the web site and referring others to it. In addition, some mapping activities took place during the year. The OERC worked with the Polis Center to produce maps for the RML Directors' Meeting in Phoenix on May 20, 2006. Susan Barnes participated in initial consultations regarding electronic reporting and mapping with representatives from NNO, OCCS, and Web-STOC. Ms. Barnes also participated in discussions regarding the development of a GIS component to the Community Health Status Indicators resource. This resource is being developed through a collaboration between CDC, HRSA, and NLM, and at present the NN/LM does not have a role in its creation, but that may change as the project progresses because the resource is anticipated to be a very useful one.

3. Strengthen and build an evaluation culture throughout NN/LM.

   The OERC began communication and sharing among the regional evaluation liaisons through establishment of the rmleval@u.washington.edu email discussion list and monthly Evaluation Liaison teleconferences. An average of 6 evaluation liaisons participate monthly; and often at least one other person also participated (i.e., a representative from NTCC, NLM fellows, other RML staff, or network members). Initial teleconferences featured discussion of the disparate views held by regions of what their evaluation liaisons' roles will be. In most regions the roles for this new responsibility area are still being developed and defined. The third teleconference featured a presentation from Mellanye Lackey about her NLM Associate Fellowship project in which she analyzed the types of evaluation being used in NNLM subcontract projects described in the CHOS Outreach Project Database conducted between 2001-2006 with funding levels of $2500 - $70,000. Subsequent teleconferences featured “Learning Circles” as a way of teaching ourselves to be better evaluators; using the popular “Measuring Your Impact” library advocacy class to build capacity for evaluation teaching in the NN/LM regions (with the model that prospective instructors would attend the class and an experienced instructor would then co-teach the class with the new instructor); and experiences at the American Evaluation Association meeting. The February 26 session was the first Learning Circle , with logic models as the topic. In April, another Learning Circle was held, with focus groups as the topic and featuring Karen Vargas, who described a project from NN/LM SCR in which regional staff conducted focus groups with health professionals working with teens. Future Learning Circles will feature highlights of regional projects. In March, the OERC collected feedback from the evaluation liaisons about the OERC web site; their use of OERC publications in training and outreach; evaluation training and help that has been provided by OERC; and their use of community-based assessment. Cindy Olney followed this discussion in March 2007 with a SurveyMonkey questionnaire to evaluation liaisons to collect quantitative information about evaluation activity in the regions.

   Responses to the evaluation liaison questionnaire indicated that the “evaluation culture” is developing. (See Appendix Two for the Evaluation Liaisons Feedback Report.) Most of the evaluation liaisons reported that they have conducted or assisted with evaluation activities in their regions and more than half have arranged for others to get evaluation support. At least six of the evaluation liaisons have sought training in evaluation since the beginning of the contract. Other RML staff members also are engaging in evaluation activity. Evaluation liaisons in more than half the regions reported that other RML staff members are promoting the OERC evaluation publications, referring others to the OERC web site, and participating in evaluation projects in their regions. Some regions also reported that other staff members have received evaluation training, used the OERC web site for planning or evaluation information, and provided training in their regions.

   Feedback from the AIR usability study of the OERC web site and the feedback session with the evaluation liaisons provided substantial direction for improvement of the web site, the OERC Blog, and the OERC section of the NN/LM StaffWiki. Cindy Olney designed and implemented an “Events Data” spreadsheet for tracking and counting OERC activities. She also began planning evaluation of OERC based on goals in the approved contract proposal. Dr. Olney and Ms. Barnes compiled measurable objectives, indicators, and targets to be used in this evaluation.

4. Provide consultation and direction in implementing evaluation systems, structures, and resources to help the NN/LM's efficacy and efficiency.

   Consultations are listed in Appendix One.

5. Enhance and refine use of community-based approaches to evaluation.

   Cross-regional collaboration: Using the Empowerment Evaluation approach with the communities that it serves, the OERC facilitated the eight regions' determination of how they will collaborate nationally during the 2006-2011 contract. With assistance from Susan Barnes and Cindy Olney, the regions selected emergency preparedness as the focus for national collaboration. Subsequently, the OERC worked with NN/LM PSR to organize and hold a two-day NN/LM National Emergency Preparedness Project planning meeting in Los Angeles , February 1-2, 2007. Hosted by the UCLA Biomedical Library and the NN/LM PSR, this meeting was attended by the Associate Directors from all eight regions, plus staff from the National Library of Medicine and experts from four academic medical libraries and two university library systems. During the rest of the winter and into the spring, the project moved forward through discussions on the OERC-maintained rmlcollab@u.washington.edu email list and OERC-led teleconferences facilitated by Adobe Connect—two in March and four in April. Meanwhile, the OERC activated the Emergency Preparedness page of the NN/LM StaffWiki for use in this project. By the end of April 2007, the core group (composed of the eight associate directors plus Angela Ruffin from NLM's National Network Office and Becky Lyon from NLM's Division of Library Operations, facilitated by Susan Barnes and Cindy Olney of the OERC) had developed goals, strategies, major plan components, a logic model (with resources, activities, outputs, and outcomes), and potential cross-regional workgroups for presentation and discussion at the May 18 RML Directors' Meeting in Philadelphia. The national collaboration work was probably the most time-consuming project for the OERC in year 1. OERC staff provided a total of 29 consultations on this one project in the year 1 contract period, most of which required hours to days of preparation.

Appendix One

Classes and consultations May 1, 2007—April 30, 2007

Date	Responsible staff (last name)	Class/consultation topic	Location (city, state) of class or last name of consult recipient
5/31/2006	Barnes	Consult: TC4C effective practices	Claire Hamasu, MCR, Salt Lake City , UT
6/13/2006	Barnes	Consult: Community health status indicators	Angela Ruffin, NNO, Bethesda , MD
6/14/2006	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
6/15/2006	Barnes	Consult: Community health status indicators	Angela Ruffin, NNO, Bethesda , MD
6/22/2006	Barnes	Consult: TC4C effective practices	Claire Hamasu, MCR, Salt Lake City , UT
7/13/2006	Barnes	Consult: MAR needs assessment	Kathel Dunn, MAR, New York , NY
7/17/2006	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
7/17/2006	Barnes	Consult: PSR outreach award announcement	Heidi Sandstrom, PSR, Los Angeles, CA
7/11/2006	Barnes	Consult: PNR outreach award announcement	Cathy Burroughs, PNR, Seattle , WA
7/26/2006	Barnes	Consult: Presentation quality maps	Angela Ruffin, NNO, Bethesda , MD
7/26/2006	Barnes	Consult: Staff recruitment	Renee Bougard, SCR, Houston , TX
7/26/2006	Barnes	Consult: Regional training	Ruth Holst, GMR, Chicago , IL
8/1/2006	Barnes,	Train: All RML teleconference. “What the OERC Can Do for You”	NN/LM staff, NNO, national
8/2/2006	Barnes, Olney	Consult: Subcontract proposal	Erinn Aspinall, UNM HSLIC, Albuquerque , NM
8/4/2006	Barnes	Consult: TC4C effective practices	Claire Hamasu, MCR, Salt Lake City , UT
8/10/2006	Barnes	Consult: TC4C 2006 proposal	Claire Hamasu, MCR, Salt Lake City , UT
8/14/2006	Barnes, Olney	Consult: Cross-regional Collaboration Breeze teleconference	RML ADs and Directors, NNO, national
8/17/2006	Barnes (with Holst and Leskovec)	Train: Measuring Your Impact	Univ of Minn BioMed Lib, Minneapolis , MN
8/22/2006	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
8/25/2006	Barnes	Consult: PSR LSTA grant	Heidi Sandstrom, PSR, Los Angeles, CA
8/29/2006	Barnes	Consult: Electronic reporting system	Michael Boer, Web-STOC, Seattle , WA
8/30/2006	Barnes	Train: Evaluation Overview	NN/LM Southeastern Atlantic Region, from Seattle via Breeze
8/30/2006	Barnes	Consult: Staff recruitment	Renee Bougard, SCR, Houston , TX
9/6/2006	Barnes	Consult: Staff recruitment	Renee Bougard, SCR, Houston , TX
9/7/2006	Barnes	Consult: Staff recruitment	Renee Bougard, SCR, Houston , TX
9/8/2006	Barnes	Consult: Training assessment	Sharon Brown, NTCC, New York , NY
9/18/2006	Barnes	Consult: TC4C final report	Claire Hamasu, MCR, Salt Lake City , UT
9/25/2006	Barnes	Consult: Presentation quality maps	Angela Ruffin, NNO, Bethesda , MD
9/26/2006	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
10/2/2006	Barnes	Consult: Review of 8/14 teleconference about Cross-regional collaboration and process so far	RML ADs and Directors, NNO, national
10/5/2006	Barnes	Consult: Exhibit evaluation	Erinn Aspinall, UNM HSLIC, Albuquerque , NM
10/6/2006	Barnes	Consult: Review of 8/14 teleconference about Cross-regional collaboration and process so far	RML ADs and Directors, NNO, national
10/12/2006	Olney	Train: OERC workshop series overview	NN/LM staff, NNO, national
10/16/2006	Barnes	Train: OERC services and publications overview at RML update for four western MLA chapters	Heidi Sandstrom, PSR; Cathy Burroughs, PNR; Seattle , WA
10/17/2006	Barnes	Consult: Electronic reporting system	Angela Ruffin, NNO, Bethesda , MD
10/26/2006	Barnes	Consult: MLA/NLM Health Information Literacy Research Project	Jean Shipman, MLA, Chicago , IL
11/2/2006	Barnes, Olney	Consult: OERC support for RMLs' evaluation plans	Evaluation Liaisons, NNO, Portland , OR
11/7/2006	Barnes, Olney	Consult: Core group teleconference for national collaboration planning	Core group, NNO, national
11/15/2006	Barnes, Olney	Consult: Core group teleconference for national collaboration planning	Core group, NNO, national
11/28/2007	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
11/29/2007	Barnes	Consult: PNR program planning	Cathy Burroughs, PNR, Seattle , WA
11/29/2006	Barnes, Olney	Consult: Core group teleconference for national collaboration planning	Core group, NNO, national
11/30/2006	Barnes, Olney	Consult: OERC support for proposal writing workshop	Heidi Sandstrom, PSR, Los Angeles, CA
12/7/2006	Barnes	Consult: PNR network members' survey	Cathy Burroughs, PNR, Seattle , WA
12/12/2006	Barnes, Olney	Train: NN/LM National Emergency Preparedness Project	RML Dirs and ADs, NNO, Bethesda , MD
12/13/2006	Barnes, Olney	Consult: Assessment of NTCC and MEDLARS Management training	Sharon Brown, NTCC, Bethesda , MD
12/18/2006	Barnes, Olney	Train: Teaching the “Measuring Your Impact” workshop	Evaluation Liaisons, NNO, national
1/23/2007	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
1/26/2007	Barnes, Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
1/29/2007	Barnes, Olney	Consult: Evaluation teaching in PSR	Heidi Sandstrom, PSR, Los Angeles, CA
1/30/2007	Barnes	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/1/2007	Barnes, Olney	Consult: NN/LM National Emergency Preparedness Project planning meeting	Core Group, NNO, Seattle , WA
2/2/2007	Barnes, Olney	Consult: NN/LM National Emergency Preparedness Project planning meeting	Core Group, NNO, Seattle , WA
2/6/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/13/2007	Olney	Consult: GMR network member survey	Jacqueline Leskovec, GMR, Chicago , IL
2/14/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/20/2007	Barnes	Consult: OERC services and publications	Curioso and Cervantes, PNR, Seattle , WA
2/20/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/22/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/22/2007	Olney	Consult: GMR network member survey	Jacqueline Leskovec, GMR, Chicago , IL
2/23/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
2/27/2007	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
2/28/2007	Barnes, Olney	Consult: PNR network member survey	Cathy Burroughs, PNR, Seattle , WA
3/1/2007	Barnes, Olney	Consult: MLA Health Information Literacy project	Jean Shipman, MLA, Richmond , VA
3/2/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
3/5/2007	Barnes, Leskovec	Train: Measuring Your Impact	Sadie Honey, LAUC-SF, San Francisco , CA
3/8/2007	Barnes, Olney	Consult: Core group teleconference	Core Group, NNO, Seattle , WA
3/10/2007	Barnes, Prince	Train: Measuring Your Impact	Ros Lett, GHSLA, Young Harris , GA
3/13/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
3/15/2007	Barnes, Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago , IL
3/20/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY
3/27/2007	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
3/29/2007	Barnes, Olney	Consult: Core group teleconference	Core Group, NNO, Seattle , WA
3/30/2007	Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago , IL
4/4/2007	Barnes, Olney	Consult: SCR Year 2 evaluation plans	Renee Bougard, SCR, Houston , TX
4/12/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle , WA
4/13/2007	Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago , IL
4/13/2007	Barnes, Olney	Consult: MLA Health Information Literacy Project	Sabrina Kurtz-Rossi, MLA, Boston , MA
4/16/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle , WA
4/17/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle , WA
4/18/2007	Olney	Consult: PNR network member survey	Maryanne Blake, PNR, Seattle , WA
4/18/2007	Barnes	Train: Evaluation overview	Maryanne Blake, PNR, Seattle , WA
4/19/2007	Barnes, Olney	Consult: Evaluation Liaisons' Teleconference	Evaluation Liaisons, NNO, national
4/20/2007	Olney	Consult: GMR focus group study	Tammy Mays, GMR, Chicago , IL
4/24/2007	Olney	Consult: SCR outreach coordinator meeting	Karen Bulow, SCR, Houston , TX
4/26/2007	Barnes	Consult: Core group teleconference	Core Group, NNO, Seattle , WA
4/27/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York , NY

 

Appendix Two

Evaluation Liaisons Feedback Session
March 27, 2007

The OERC sought feedback from evaluation liaisons to gauge the evaluation activity in the regions and get feedback about how to better support evaluation in the network. A group discussion was held during the March monthly evaluation liaison teleconference, with the following evaluation liaisons in attendance: Susan Hunter (NN/LM MAR); Jacqueline Leskovec (NN/LM GMR); Betsy Kelly (NN/LM MCR); Karen Vargas (NN/LM SCR); Maryanne Blake (NN/LM PNR); and Heidi Sandstrom (NN/LM PSR). Lisa Boyd (NNO) also was present at the meeting, but did not participate. Participants described some of their activities as evaluation liaisons and provided many examples of evaluation projects in their region, use of the OERC for consultation, and promotion of OERC products and services to their network members. They also offered some specific suggestions for the OERC to enhance its support to the regions.

After the feedback session, a short questionnaire was sent to evaluation liaisons to get a more systematic assessment of how much activity was occurring in the regions and to get feedback from those who did not attend the teleconference. The questionnaire confirmed some of the results from the discussion plus provided information about evaluation activities by other RML staff in the regions. Seven of 8 evaluation liaisons responded. Findings from the questionnaire are discussed in appropriate places throughout the text, with summary tables presented in at the end of this report.

Evaluation Liaison Activities

Table 1 presents questionnaire results showing the self-reported activities of the evaluation liaisons. Six of seven respondents said they have conducted or assisted with evaluation activities in their regions and five have arranged for others to get evaluation support. Half of the evaluation liaisons have provided evaluation training to others. Five have used the Survey Monkey online survey software. (Some use the OERC account and some have created their own regional accounts.)

At least six of the evaluation liaisons have sought training in evaluation since the beginning of the contract, according to responses to the questionnaire. In the feedback discussion, the majority of these evaluation liaisons received their training at the American Evaluation Association meetings. One person attended a non-AEA evaluation workshop about three participatory evaluation techniques: appreciative inquiry, open-space, and world café. Another person reported attending the “Measuring Your Impact” workshop, which she said was helpful and may be sponsored in her region.

The Midcontinental Region (NN/LM MCR) has provided informal evaluation assistance to the regions as well. Betsy Kelly has shared her experience with network member surveys with several regions . Betsy said their web meetings for network members are open nationally and sometimes evaluation topics are offered. During the feedback session, several evaluation liaisons said they would like more opportunities to network with and provide support to each other. They suggested (a) posting regional surveys and other materials online for regions to share; (b) providing time at each evaluation liaison monthly teleconferences to share the evaluation activities in their regions; and (c) as more RML staff begin teaching “Measuring Your Impact” classes, posting information about how they were using and personalizing the course in their regions.

Types of evaluation support sought from the OERC

Both the questionnaire and feedback discussion showed that evaluation liaisons are using the OERC resources to fulfill their roles. Five of the seven have used the OERC web site to locate planning or evaluation information. In the feedback session, evaluation liaisons gave examples of the support they have sought and received from the OERC, including the following:

• Help with recruiting an evaluation liaison for the NN/LM SCR
• Review of a focus group report
• Conducting or co-teaching “Measuring Your Impact” workshops
• Providing customized workshops to meet needs of specific audiences in specific regions, like a session on shoestring evaluation techniques to be conducted June 28 for outreach coordinators from Resource Libraries in the NN/LM SCR
• Consultations on survey development and survey sampling
• Consultations on focus group study design
• Use of the OERC web site's online logic models both for planning an RML office project and for an in-house logic model class.

In the questionnaire, all seven evaluation liaisons reported referring others to the OERC web site as well. Some examples of their promotion of OERC materials reported in the feedback session are:

• Promotion of OERC directly or via their regional evaluation web site
• Sending the booklets to outreach libraries
• Giving booklets out at library association meetings. (One evaluation liaison said the booklets are very well received at meetings).

Regional Evaluation Activities – Other RML Staff

The questionnaire also asked evaluation liaisons to report the evaluation activities of others in their regions. Evaluation liaisons from 6 regions said other RML staff members are promoting the OERC evaluation publications and referring others to the OERC web site. Six also said other RML staff has participated in evaluation projects in their regions. The next most popular evaluation activity is use of Survey Monkey, with 5 evaluation liaisons reporting its use by other staff. In half of the regions, staff members are arranging for others to get evaluation support. In three regions, staff members have received evaluation training and used the OERC web site for planning or evaluation information. In two regions, staff members have provided training in their regions.

Requests for Additional Support

In the feedback discussion, evaluation liaisons gave examples of different types of services that the OERC could provide to further support and strengthen evaluation in the region:

Information exchange

• Make the OERC Blog more active . All of the evaluation liaisons who completed the questionnaire have used the blog to varying degrees (see Table 2 for responses). In the feedback session, the regular users said they have lost interest because there has been little activity since the end of 2006. Those who were not frequent users said they might visit it more if they received email alerts when new information was added.
• Make better use of the OERC section of the StaffWiki . The StaffWiki is not being used much at all (in fact, evaluation liaisons indicated so little use of the StaffWiki that it was not addressed in the questionnaire). The StaffWiki would be an ideal place for evaluation liaisons to share information about the evaluation courses (like Measuring Your Impact) and evaluation tools (like membership surveys). However, one person talked about the difficulty she has in navigating the StaffWiki in general (beyond the OERC section). Use might increase if the OERC created a table of contents and alerted evaluation liaisons when something new is posted.
• Include more cross-regional sharing per teleconference . The liaisons would like to know what other regions are doing. One person suggested having each evaluation liaison doing a short report at the start of each meeting. (Another mentioned that not all the evaluation liaisons “wear the same hat” and may not always have something to report every week; so reports should be voluntary.)

Training

• Evaluation liaisons made the following suggestions for how the OERC could provide evaluation training across the NN/LM:
• Develop motivational classes to get network members excited about doing evaluation.
• Develop a session on “small-ball” evaluation or “How to do a needs assessment when you don't have time to do a needs assessment” for network members.
• Work with evaluation liaisons to help them develop classes to meet specific needs.
• Shorten and modify the OERC's face-to-face classes so they can be offered using Adobe Connect.
• Invite Keith Cogdill to do a presentation about his paper from JMLA about the contributions of library and information services in hospitals and academic health sciences centers.
• Develop a menu of short (15-minute) mini-lessons that can be presented at meetings with network members.

Miscellaneous

• Conduct a national survey of network members for issues that cut across all regions, such as “historical and special collections,” “emergency preparedness,” or “health literacy.” That way, the regional membership surveys can eliminate topics that are asked across all regions from their regional surveys.
• Post a list of evaluation classes that are taught and their locations. This would allow instructors to see how they are “touching the rest of the country.”

Conclusion

One of the OERC's main goals for the 2006-2011 contract is to “strengthen and build an evaluation culture within the NN/LM.” A primary strategy for this goal has been to work intensively with evaluation liaisons and some of the indicators we are using to assess this goal are (a) requests for consultations on evaluation projects throughout the region (b) evidence of use and promotion of OERC publications and the OERC web site and (c) evidence of evaluation activities occurring in the regions. The OERC consultation records show that the OERC has provided evaluation consultation or training sessions in 7 of 8 regions and the feedback session indicated that evaluation liaisons who sought help found our consultations extremely helpful. The OERC materials also are proving to be useful: all evaluation liaisons report promoting OERC publications and the web site; and most report using the OERC web site for planning and evaluation. Finally, the evaluation information is being applied: most of the evaluation liaisons are engaged in conducting evaluation or teaching evaluation methods.

Another main goal for the OERC is to develop mechanisms for disseminating and using evaluation findings to help NLM and the NN/LM document and demonstrate their accomplishments. One strategy has been to use online resources like the OERC web site, StaffWiki, and OERC Blog. The OERC web site seems to be helpful, but the feedback shows that the OERC could make more effective use of the StaffWiki and blog. In general, more activity on the blog must occur to maintain the evaluation liaisons' interest in it and some navigational elements like a table of contents are needed to make the StaffWiki easier to use.

Based on the feedback from evaluation liaisons, the OERC has listed some action items it intends to take to further support the evaluation culture in the region:

• Create a place online for regions to share their surveys and other evaluation materials
• Increase activity on the OERC Blog and encourage participation from evaluation liaisons
• Provide time at each evaluation liaison monthly teleconference for evaluation liaisons to share evaluation activities in their regions
• Have a place online for RML staff teaching “Measuring Your Impact” classes to post information about how they used and personalized the course in their regions
• Post email alerts to the RML evaluation liaison discussion list when additions are made to the OERC Blog and OERC section of the StaffWiki
• Create a table of contents to the StaffWiki to make it easier to navigate
• Create an online list of OERC-sponsored classes that are taught and their locations.
• Discuss with evaluation liaisons the possibility of doing a national network member survey
• Develop a “small-ball” evaluation training session to present to SCR and, if it is well received, offer it to other regions
• Explore with evaluation liaisons the best venue for a presentation by Keith Cogdill (i.e., a learning circle; a presentation open to the entire network).
• Develop a menu of “mini-lessons” to present to the evaluation liaisons for their feedback.
  

Questionnaire Responses

(Summary)

The evaluation liaisons were asked if they had engaged in eight different activities that the OERC identified as evidence of liaisons' engagement in their role as the “go to” person for evaluation in their region since the start of the 2006-2011 contract. We also asked them to report whether other RML staff had engaged in these activities. Questionnaire results are summarized below in Tables 1 and 2. Table 1 shows the activities and findings (because our group is so small, frequencies are reported rather than percentages.)

Table 1: Please indicate if you/others in your region have done the following since the start of the 2006-2011 contract.
	Evaluation Liaisons	Others in Region
Promoted the OERC evaluation publications at outreach and training activities	7	6
Referred others to the OERC web site	7	6
Received training in evaluation	6	3
Conducted or assisted with evaluation projects in your region	6	6
Used the OERC web site to locate planning or evaluation information	5	3
Used Survey Monkey	5	5
Arranged for others to get evaluation support (e.g., referred them to the OERC or organized an online training about evaluation)	5	4
Provided evaluation training to others in your region	4	2

 

Table 2: Which of the following best describes your use of the OERC Blog?
I use the RSS feed and try to keep up with new entries	2
I read it occasionally, when I have time	2
I have looked at it a few times (or less)	4
I have never been to the OERC blog.	0