Outreach Evaluation Resource Center

Quarterly Report: February - April, 2007

Executive summary

Once again, the lion's share of the quarter was spent by the OERC on facilitating the NN/LM National Emergency Preparedness Project. This was the most active period for this project so far, beginning with the planning meeting in Los Angeles, February 1-2, 2007. Hosted by the UCLA Biomedical Library and the NN/LM PNR, this meeting was attended by the Associate Directors from all eight regions, plus staff from the National Library of Medicine and experts from four academic medical libraries and two university library systems. The project moved forward through teleconferences facilitated by Webmeeting-two in March and four in April. By the end of the quarter, the core group (composed of the eight associate directors plus Angela Ruffin from NLM's National Network Office and Becky Lyon from NLM's Division of Library Operations, facilitated by Susan Barnes and Cindy Olney of the OERC) had developed goals, strategies, major plan components, a logic model (with resources, activities, outputs, and outcomes), and potential cross-regional workgroups to present at the RML Directors' Meeting in Philadelphia. AIR conducted a usability study of the OERC web site this quarter, more sets of the "Planning and Evaluating Health Information Outreach" booklet series and the "Measuring the Difference" guide were distributed, and Learning Circles were initiated as part of the Evaluation Liaisons' Teleconferences in order for the group members to learn from each other. Two onsite, daylong workshops were conducted, one in the NN/LM PSR and the other in the NN/LM SEA. Each was co-taught by Susan Barnes and a regional Evaluation Liaison, following the path of using the "Measuring Your Impact: Using Evaluation for Library Advocacy" as a way for librarians to begin teaching evaluation content in their regions. This quarter also saw an upswing in the requests for technical assistance on evaluation projects, with requests coming primarily from three regions (the MAR and PNR on their membership surveys, and the GMR on sampling for their membership survey and on a focus group project with community-based organizations) and from MLA (related to the MLA Health Information Literacy Project). (See table 1, below, for details). The OERC also consulted with the SCR on evaluation planning for next year and is developing a capacity-building session to be held in June at the SCR's outreach coordinator meeting.

Classes and consultations

Date	Responsible Staff (Last Name)	Class/Consultation Topic	Location (city, state) of Class or Last Name of Consult Recipient
2/1/2007	Barnes, Olney	Consult: NN/LM National Emergency Preparedness Project planning meeting	Core Group, NNO, Seattle, WA
2/2/2007	Barnes, Olney	Consult: NN/LM National Emergency Preparedness Project planning meeting	Core Group, NNO, Seattle, WA
2/6/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
2/13/2007	Olney	Consult: GMR network member survey	Jacqueline Leskovec, GMR, Chicago, IL
2/14/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
2/20/2007	Barnes	Consult: OERC services and publications	Curioso and Cervantes, PNR, Seattle, WA
2/20/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
2/22/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
2/22/2007	Olney	Consult: GMR network member survey	Jacqueline Leskovec, GMR, Chicago, IL
2/23/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
2/27/2007	Barnes, Olney	Consult: Evaluation Liaisons' teleconference	Evaluation Liaisons, NNO, national
2/28/2007	Barnes, Olney	Consult: PNR network member survey	Cathy Burroughs, PNR, Seattle, WA
3/1/2007	Barnes, Olney	Consult: MLA Health Information Literacy project	Jean Shipman, MLA, Richmond, VA
3/2/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
3/5/2007	Barnes, Leskovec	Train: Measuring Your Impact	Sadie Honey, LAUC-SF, San Francisco, CA
3/8/2007	Barnes, Olney	Consult: Core group teleconference	Core Group, NNO, Seattle, WA
3/10/2007	Barnes, Prince	Train: Measuring Your Impact	Ros Lett, GHSLA, Young Harris, GA
3/13/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
3/15/2007	Barnes, Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago, IL
3/20/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY
3/27/2007	Barnes, Olney	Consult: Evaluation Liaisons' teleconference	Evaluation Liaisons, NNO, national
3/29/2007	Barnes, Olney	Consult: Core group teleconference	Core Group, NNO, Seattle, WA
3/30/2007	Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago, IL
4/4/2007	Barnes, Olney	Consult: SCR Year 2 evaluation plans	Renee Bougard, SCR, Houston, TX
4/12/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle, WA
4/13/2007	Olney	Consult: GMR focus group study	Ruth Holst, GMR, Chicago, IL
4/13/2007	Barnes, Olney	Consult: MLA Health Information Literacy Project	Sabrina Kurtz-Rossi, MLA, Boston, MA
4/16/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle, WA
4/17/2007	Olney	Consult: Core subgroup teleconference	Core Group, NNO, Seattle, WA
4/18/2007	Olney	Consult: PNR network member survey	Maryanne Blake, PNR, Seattle, WA
4/18/2007	Barnes	Train: Evaluation overview	Maryanne Blake, PNR, Seattle, WA
4/19/2007	Barnes, Olney	Consult: Evaluation Liaisons' teleconference	Evaluation Liaisons, NNO, national
4/20/2007	Olney	Consult: GMR focus group study	Tammy Mays, GMR, Chicago, IL
4/24/2007	Olney	Consult: SCR outreach coordinator meeting	Karen Bulow, SCR, Houston, TX
4/26/2007	Barnes	Consult: Core group teleconference	Core Group, NNO, Seattle, WA
4/27/2007	Olney	Consult: MAR network member survey	Sue Hunter, MAR, New York, NY

Web site

Susan Barnes began work with AIR on its OERC web site usability study. This was conducted as part of Web-STOC's contract with AIR, an unanticipated benefit of NTCC not yet being ready for usability study of their web site. After finalizing the test plans and collecting a pool of study participants, AIR conducted four usability sessions (on 4/2/07, 4/17/07, 4/18/07, and 4/19/07). Study participants were drawn from these groups of RML staff: Outreach Coordinators, Consumer Health Coordinators, and Evaluation Liaisons. RML staff were used because at present that is the principal user population for the site. Susan Barnes observed the sessions, which employed the UserVue software from Techsmith (similar to Adobe Connect). Cindy Olney was unable to observe the sessions live because she was facilitating substantial work on the NN/LM National Emergency Preparedness Project during April (see below). Each session was about 90 minutes in duration and resulted in a large amount of valuable feedback. Reactions to the site were generally positive. AIR's final report was still in progress at the end of the quarter.

Dissemination of effective practices

The OERC distributed 26 sets of "Planning and Evaluating Health Information Outreach" plus 6 copies of "Measuring the Difference."

Susan Barnes continued work on a bibliography of publications about effective practices for health information outreach. As part of her participation on the TC4C Effective Practices team, she developed this articulation of the difference between "best practices" and "effective practices":

"Effective practices" implies health information outreach strategies that have been successful, useful, or promising, based on experience and evidence. Although the terms "effective" and "best" practices are often used interchangeably, the OERC differentiates between them. "Best practices" have been rigorously evaluated and represent the "Gold Standard" of activities to support specific objectives. In this context, "effective practices" indicates experiences with what has worked in a given situation. (This concept is from the Johns Hopkins INFO Project, http://www.infoforhealth.org/practices.shtml.)

The OERC also has a plan for disseminating information about effective practices through the monthly evaluation liaison teleconferences. The Learning Circles, which started in February, now will be designed to showcase an evaluation project in one of the regions, with that region's evaluation liaison providing a report about "lessons learned" about the Learning Circle topic. This practice initiated in April, with Karen Vargas starting the learning circle by talking about focus groups she had done in the region. (These learning circles are open to other attendees.)

Subcontractor update

Not applicable. The OERC has no subcontracts at present.

Other staff activities-this section is organized according to OERC's goals for the 2006-2011 contract

1. Develop awareness, skill, and capacity in use of evaluation methods.

   Training: Susan Barnes co-taught "Measuring Your Impact" with Jacqueline Leskovec in San Francisco (to a group primarily comprising academic librarians from the University of California, San Francisco and from affiliated institutions) and with Dale Prince in Young Harris, Georgia (to a group primarily comprising hospital librarians in the Georgia Health Sciences Library Association). Dale had attended the San Francisco class and an earlier outreach evaluation class in preparation for the Georgia class and possible future classes in NN/LM SEA; Heidi Sandstrom and Julie Kwan attended the San Francisco class in preparation for possible future classes in NN/LM PSR.

   Feedback about the OERC's plan to supplant the day-long outreach evaluation workshop (piloted in winter, 2006) with a series of five half-day, targeted workshops led to the decision to cut the series back from five sessions to four half-day sessions (Community Assessment, Planning Outcomes-Based Outreach Programs, Data Collection for Health Information Outreach, and Data Analysis for Health Information Outreach). The series title "Outreach Evaluation Workshop Series" was developed-eliminating the earlier title of "Measuring the Difference" that had been in use since the original outreach evaluation classes were taught during the last contract by Maryanne Blake and Cathy Burroughs. At the same time, the OERC renamed the "Measuring Your Impact: Using Evaluation to Demonstrate Value" workshop to "Measuring Your Impact: Using Evaluation for Library Advocacy" in order to better differentiate it from the outreach evaluation sessions. The OERC hopes that these titles will eliminate most of the confusion that had begun to appear regarding the difference between the library advocacy workshop and the outreach evaluation workshops.

   Cindy Olney finalized course materials for the first two workshops in the new "Outreach Evaluation Workshop Series," Community Assessment and Planning Outcomes-Based Outreach Projects. She also drafted applications for MLA CE credit, which Susan Barnes finalized and submitted to the Medical Library Association. These were approved in April for 3 contact hours each.

   Cindy Olney met with NN/LM SCR staff to plan a June Webmeeting for outreach librarians in that region on "shoestring evaluation" (ie, evaluating your work when you lack time and money.

   Susan Barnes gave a presentation, "Do You Make a Difference? Providing Evidence of Program Value," to 24 NN/LM PNR network members (mainly hospital librarians) via Webmeeting. The presentation was about evaluation and its importance for library advocacy and project planning.

2. Develop mechanisms to disseminate and use evaluation findings to help NLM and the NN/LM document and demonstrate their accomplishments.

   Publications: The OERC has received feedback that the OERC Blog is not updated with new postings often enough. In addition, the "Collaboration" and "Liaisons" sections are not being used by those groups for communication. The blog format seems better suited for brief news items, so the decision was made to restructure the OERC Blog to have a category about lessons learned, perhaps titled "Practical Approaches" plus the main News category.

3. Strengthen and build an evaluation culture throughout NN/LM.

   Evaluation liaisons: Three Evaluation Liaisons' Teleconferences took place this quarter. The February 26 session was the first Learning Circle, with logic models as the topic. In March, the OERC collected feedback from the Evaluation Liaisons about the OERC web site; their use of OERC publications in training and outreach; evaluation training and help that has been provided by OERC; and their use of community-based assessment. (Cindy Olney followed this discussion with a SurveyMonkey questionnaire to collect information from anyone who did not participate, and the summary report of this feedback session is still in progress pending completion of this survey.) In April, another Learning Circle was held, with focus groups as the topic and featuring Karen Vargas, who highlighted a project from NN/LM SCR in which regional staff conducted focus groups with health professionals who work with teens. Future Learning Circles will feature highlights of regional projects.

   Evaluation of the OERC: Feedback from the AIR usability study of the OERC web site and the feedback session with the Evaluation Liaisons provided substantial direction for improvement of the web site, the OERC Blog, and the OERC section of the NN/LM StaffWiki. Final reports are pending.

4. Provide consultation and direction in implementing evaluation systems, structures, and resources to help the NN/LM's efficacy and efficiency.

   Consultations: Consultations are listed in Table 1, above.

5. Enhance and refine use of community-based approaches to planning.

   Cross-regional collaboration: On February 1 and 2, Cindy Olney and Susan Barnes led a two-day NN/LM National Emergency Preparedness Project planning meeting in Los Angeles, attended by all eight ADs plus, from NLM: Becky Lyon, Angela Ruffin, Lori Klein, Stacey Arneson; and expert advisors: Marty Thompson (U of OK), Mary Ryan (U of AR), Eva Seltzer (U of MS), Susan Parker (UCLA), Julie Page (UC San Diego) and Deborah Halsted (HAM-TMC). The UCLA Biomedical Library and NN/LM PSR hosted the meeting in their excellent facilities, and made all participants feel very welcome. Cindy and Susan compiled a report of the meeting and circulated it to the group for their additions and corrections. Susan and the core group scheduled teleconferences to work on the NN/LM National Emergency Preparedness Project, beginning in March.

   The process began with two NN/LM National Emergency Preparedness Project core group calls in March, in which goals were finalized and strategies were agreed upon. In April, Cindy met with three subgroups to take the results of the core group's work in March and develop logic models for the three NN/LM National Emergency Preparedness project goals. Based on these logic models, Cindy developed five NN/LM National Emergency Preparedness Project components; Susan distributed and discussed these with the core group in its April 26 teleconference. This session resulted in project component revisions, a project logic model, and possible cross-regional workgroups, which were compiled and distributed by the OERC in preparation for presentation at the May 18, 2006 RML Directors' Meeting in Philadelphia.