Outreach Evaluation Resource Center
Annual Report
Year Four May 2004-April 2005

Training and Continuing Education

• For the up coming NLM Community-Based Outreach Symposium , C. Burroughs prepared a literature review, posted on the CBO Symposium website: ttp://medstat.med.utah.edu/symposium/articles.html. Topics of the review included Library Science: Selected perspectives or evaluations of consumer health collaborations; and Public Health and Health Promotion: Selected perspectives about principles and evaluation of collaborative or community-based interventions
• Presented a paper titled Outcomes Assessment in Community-Based Health Information Outreach (with bibliography) at the American Evaluation Association annual conference, Atlanta, GE. The paper was presented on a panel shared with Cynthia Olney, who presented a paper titled Overcoming the barriers -- successful outcomes assessment in community-based outreach projects. The panel's focus was health information outreach with examples from NLM-funded outreach project conducted in South Texas. (November 2004)
• C. Burroughs attended the NLM Community-Based Outreach Symposium in Bethesda, Maryland as chair of the Assessment & Outcome panel that featured evaluation perspectives and white papers from 4 panelists: Charles P. Friedman, Ph.D; Lawrence W. Green, Ph.D.; Judith M.Ottoson, Ed.D.; and Cynthia A. Olney, Ph.D. (December 2004)

To increase her expertise in various areas of program evaluation, the assistant director of the OERC completed several professional development experiences:

• Seminar titled What's a Community Partner To Do by Ira Harkavy, Ph.D., Founding Director of the Center for Community Partnerships at the University of Pennsylvania at the University of Washington School of Public Affairs.

Consultation with Network members

C. Burroughs was involved in the following activities:

• Developed a concept memo describing the purpose and design of the training impact survey to share and discuss with all staff at the RML Director's meeting
• Participated as a member of the NLM Listening Circle trip in Alaska to generate input and feedback on the role of NLM in dissemination of health information to tribal communities
• Continued maintaining and updating the OERC website
• Continued participation in the Hospital Internet Access Task Force
• Continued developing plans for a context evaluation process with potential community-based organization partners in the PNR
• Continued to review and provide input on evaluation sections of proposals to implement strategies developed in the first phase of the Community Outreach Partnership Planning Awards (COPPA)
• Contacted two RMLs about their interest in hosting a new evaluation workshop that will pilot updated educational materials about evaluating community-based outreach efforts
• Based on a request by NNO, worked with C. Olney to develop a survey to assess RML-wide progress on objectives of the Public Libraries and Public Health Outreach Initiatives. The underlying goals of these initiatives are to increase awareness, skill and access to NLM resources and needed health information by public librarians and the public health workforce. The survey was distributed to RML staff to assess progress on selected objectives included in the logic models developed for the initiatives. Results will be presented at the RML Directors meeting in San Antonio, TX in May.

Assistance with analysis, synthesis, and reporting of evaluation results

• A major effort in this area involved the PubMed/MedlinePlus Training Follow-up Survey. C. Burroughs assumed leadership and responsibility for developing the a follow-up survey to assess overall impact of RML-conducted trainings/demonstrations about PubMed and MedlinePlus. She also oversaw the data analysis and reporting of survey results. The project progressed in three steps:
  • The first step in the process was working with Aron Beal of Web-STOC to design online data collection methods that included integrating information from the Patient Information Sheets with the follow-up survey data. UW Catalyst software was considered as a platform for the web-based surveys, but A. Beal found that the UW software did not allow merging of OARF data with survey responses. Thus, he designed a local database with customized functions required for the web-based survey. The development process included the following activities:
    • Finalizing the survey items, based on feedback collected at the May RML Director's meeting. C. Burroughs piloted an email invitation and the web-based survey with NN/LM and NNO staff and several other representative Network member respondents in September 2004 C.
    • Get consent from the RMLS to revise the Participant Information Sheet to collect voluntary email addresses of outreach participants willing to participate in the follow-up survey. Specifically, on the Participant Information Sheet, "Section 3. Special Populations Served" was replaced with "Section 3. My email address."
    • Installing the revised PI sheet in the OARF system, which was posted on http://nnlm.gov/evaluation/datacollection/ starting August 19
    • Writing a program for automated survey invitations to emails collected on the Participant Information Sheet
    • Developing a database to match respondents'answers with mirror data from OARF records, such as the date, length, type, and content of the session attended by the survey respondent. When respondents fill out the web-based survey, the database populates their responses, based on the code identifier (included in the email inviting them to the survey), with the OARF data that identifies the class they took (e.g. location, date, length, content, etc). The database also links survey responses with information from the class signup sheet, such as zip code at work or home, and the survey respondent's role in attending the class (member of general public, health professional, librarian, etc)
    • Designing an interface to view the data, called the Survey Results Browser. This capacity allows for a rich descriptive analysis of the survey data that can be viewed dynamically by NN/LM staff.
  • The second step was administering the follow-up survey, which began during the 3rd quarter of the year after several pilot tests and refinements. Any NN/LM training session with PubMed or MedlinePlus content and email addresses provided from Participant Information Sheets email were included in the follow-up survey. In addition to public libraries and public health staff, results included feedback provided by trainees from many spheres: members of the general public, patients, families or friends of patients, health educators, health care providers, researchers and scientists, health office workers, health administrators, and students. C. Burroughs compiled and reported 8 weeks of survey data at the November 2004 Directors Meeting in Houston. She also led a discussion about next steps for the survey. The group decided that survey had not been running long enough for clear decisions on its usefulness and group recommended another review of survey results at the May RML Director's meeting
  • The third step was analyzing follow-up survey data collected from Summer 2004 through Spring 2005 (which included the data reported in November). C. Olney provided the survey analysis, which was added to a report that included other evaluation data collected via the Outreach Activity Reporting Forms and the Participant Information Sheets. The report will be presented at the May 2005 RML Directors' in San Antonio.
• As a second activity in this area, C. Burroughs provided input to NOMC about purpose and specifications of mapping to be presented at the RML Director's meeting in May.

Communications and collaboration

• C. Burroughs presented maps showing activities in Year 3 by RMLs in reaching public libraries and the public health workforce at the RML Director's meeting in Washington, DC.
• Also at the RML Director's meeting in Washington D.C., C. Burroughs provided an overview of plans and preliminary survey questions for the training impact evaluation and facilitated discussion of the plans and questions during the breakout session of the Associate Director and Directors.
• Leading up to the Symposium, C. Burroughs continued membership on the steering committee for the Community-Based Outreach Symposium:
  • As chair of the Outcomes & Assessment Panel, C. Burroughs finalized commitments by the panelists, who were invited to submit white papers as well as give presentations at the symposium. C. Burroughs held several phone conferences with the panelists. She developed goals for the panel and identified the following key areas to be addressed: In underserved communities, what are effective ways to evaluate community-based interventions to
• • • Improve use of health information resources
    • Enhance capacity for healthier communities
    • Improve health outcomes
  • C. Burroughs also participated in conference calls for the CBO Perspectives session at the symposium.
  • C. Burroughs reviewed and commented on a draft white paper, for presentation at the CBO Symposium, about NLM's Native American portfolio.
• M. Blake investigated options for subscribing to SurveyMonkey for NN/LM-related surveys. It was determined that OERC would propose a trial subscription during Year 5.

Utilization and promotion of the evaluation guide

• NN/LM PNR continued to distribute the guide upon request. At the teleconference presented as part of NN/LM Center Review in September 2004, C. Burroughs stated that a total of 3,800 copies of the manual had been distributed to date.
• C. Olney developed concept outlines and drafts of booklets 1 and 2 of a publication series to complement and update the current evaluation guide. NN/LM PNR staff reviewed and commented on the first booklet in the (to be named) series. C. Burroughs also asked C. Olney to consider co-developing a new evaluation workshop to pilot the new booklets.

Administrative

• For the NN/LM Centers Review requirement, C. Burroughs developed documentation and a presentation about OERC's progress in the current contract. The review was conducted via teleconference on September 9, 2004.
• C. Burroughs developed an agreement and scope of work for Cynthia Olney related to her OERC consultation and other special projects to be assigned.
• C. Burroughs developed a list of recommended goals for the OERC in the upcoming 2006-2011 contract, submitted to NNO in Dec 2004, per request from Angela Ruffin.
• C. Burroughs developed a Year 5 budget and narrative for the OERC and submitted it to NLM.
• C. Burroughs was invited to be a panelist at the MLA annual meeting, in a session sponsored by the Research Section called: "Yes! You can do Research." She declined due to time constraints in holding her concurrent positions as the OERC Assistant Director, and the Acting Associate Director of the NN/LM PNR. Instead, on her recommendation, C Olney, consultant to the OERC, was confirmed to present at the MLA annual conference.
• Susan Barnes, Resource Sharing Coordinator in the NN/LM PNR, agreed to become Acting Assistant Director of the OERC, as of May 1, 2005 to replace C. Burroughs, who was recruited to become Associate Director of the NN/LM PNR.
• C. Burroughs submitted answers to questions by NLM related to the Year 5 budget and narrative for the OERC.