National Network of Libraries of Medicine
nnlm.gov
Archives
This page was archived on: May 02, 2008 | View page metadata
Document content is not current. Links may be broken.
| National Network of Libraries of Medicinennlm.gov |
Archives |
|
This page was archived on: May 02, 2008 | View page metadata | Document content is not current. Links may be broken. |
||
DRAFT 2 - October 1997
Prepared for the Project:
Planning and Evaluating Information Among Minority Communities:
Model Development Based on Native Americans in the Pacific Northwest
By:
Joanne G. Marshall, Ph.D.
Dean, School of Information and Library Science
100 Manning Hall
CB 3360
University of North Carolina
Chapel Hill, NC 27599
marshall@ils.unc.edu
Project Sponsor:
National Library of Medicine
Office of Health Information Programs Development
Bethesda, MD, USA
Since 1989, the National Library of Medicine has collaborated with Regional Medical Libraries (RML's) in its National Network of Libraries of Medicine (NN/LM) to conduct outreach to health professionals, in particular those in rural, minority or underserved areas. The five objectives of the NLM outreach program are to: 1) increase health professionals' awareness of NLM's products and services; 2) facilitate health professionals' access to and use of biomedical information; 3) provide training to health professionals about searching electronic databases; 4) assist health professionals in implementing new information-seeking behaviors; and 5) help improve the quality of services provided by health professionals to their clients (National Library of Medicine, 1989).
From 1990 to 1995, NLM supported almost 300 outreach projects across the United States and a five-year review of the outreach program was completed in 1996. (Wallingford, Ruffin, Ginter, Spann, Johnson, Dutcher, et al., 1996). The present paper is part of collaborative project to develop a framework for outreach planning and evaluation that is being undertaken by the NN/LM's Pacific Northwest Regional Medical Library at the University of Washington Health Sciences Library. While other commissioned papers explore outreach efforts in related fields, such as community development, technology diffusion, Native American culture and health communication, this paper focuses on activities in the health sciences library community. Particular emphasis is placed on "best practices" in health sciences library outreach and evaluation.
Although the term "outreach" is used frequently in the library and information science literature, it is difficult to find a concise, agreed upon definition. Instead, outreach tends to be defined by the specific activities that are undertaken by librarians as they attempt to reach beyond or surpass the boundaries of their traditional on-site services. Outreach has always been considered to be among the most innovative aspects of library service, as librarians use such activities to make their services more accessible to information users. At the same time, outreach has led to considerable controversy over the years because outreach pushes the boundaries of regularly provided library services and often competes with regular services for scarce resources.
With the growing capability of libraries to deliver information to remote sites using electronic information storage and retrieval technologies, an increasing proportion of library activities fall into the category of outreach. Health sciences librarians have a pioneering history of providing innovative outreach services to meet the information needs of clinicians, with and without the use of technology (Marshall, 1993). These services include Literature Attached to the Chart or LATCH (Sowell, 1978); clinical librarian programs in which librarians are attached to health care teams as information specialists (Cimpl, 1985); circuit librarian programs providing service to underserved health professionals (Pifalo, 1994) and end-user training programs in the use of computerized databases (Sewell and Teitlebaum, 1986).
More recently, health sciences librarians have also joined other health professionals in the development of clinical information management systems that integrate internally generated patient care information such as the patient record with the external knowledge base of health care available through library and information services (e.g. Anderson, Haddix, McCray and Wunz, 1994). Librarians are showing that their skills are essential in continuous quality improvement programs that use literature searches as part of the process of prospectively improving patient outcomes (e.g. Howell and Jones, 1993). Librarians have identified an important role in knowledge-based health care as it is defined in the standards of the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO) (e.g. Doyle, 1994). Librarians are also finding key roles in teaching critical appraisal of the literature (e.g. Dorsch, Frasca, Wilson and Tomsic, 1990) and in providing access to the evidence from the literature required for evidence-based health care practice (e,g, Lefebvre, 1994).
The activities of the National Library of Medicine (NLM) and its National Network of Libraries of Medicine (NN/LM) continue to have a profound effect on the outreach efforts of health sciences libraries in the United States and elsewhere. NLM has provided the leadership, training, search tools, research funding and resources required to build a world-leading system of access to health information for health care providers. NLM's products and services complement and empower the local outreach efforts of health sciences librarians. In 1997, NLM also began to support the provision of consumer health information through the availability of free MEDLINE on the Internet and the addition of consumer health literature to the MEDLINE database.
NLM's outreach activities in the last decade have been guided by a 1987 report of the Senate Appropriations Committee that encouraged NLM to develop an outreach program "to transfer the latest scientific findings to all health professionals". A Planning Panel chaired by Dr. Michael E. DeBakey recommended that NLM focus on four areas in its outreach program:
(National Library of Medicine, 1989)
In response to the report, NLM staff developed a comprehensive program to make health professionals aware of the information resources available to them, to facilitate their access to these resources and to link health professionals with local library resources whenever possible. This program has largely defined the scope and capabilities of health sciences library outreach activities to health care providers in recent years, although many health sciences librarians have also conducted outreach activities independent of NLM, particularly in the area of consumer health information services. In addition to outreach to individual health professionals, NLM has supported the following types of projects: HIV/AIDS community outreach; toxicology information outreach; improving access for hospitals; Internet connections; and telemedicine.
While the efforts of NLM and health sciences librarians in general to improve access to health information for health care providers and consumers are laudable, it is important to establish the extent to which the specific objectives of these health sciences library outreach activities are achieved. Burroughs (1997) has described the objectives of NLM outreach to individual health professionals as cognitive, affective and skill-based. The cognitive objective is to increase awareness of health information resources by health professionals. The affective objective is to persuade health professionals of the value in accessing health information. The final objective is to develop the knowledge and skills of health professionals in using NLM or other resources to access health information. Other possible objectives (but ones that are not explicitly stated as objectives of the NLM initiative) are to examine the impact of information access and use on changing health care practice behavior and to measure the impact of information use on patient outcomes.
Since the definition of outreach used by NLM is very broad, studies were examined for this paper in a variety of areas. Literature searches covering the last five years were conducted in the MEDLINE, Library and Information Science Abstracts (LISA) and Library Literature databases on outreach to the following groups:
Both general library evaluation literature and general health sciences library evaluation literature were selectively examined. Since the retrieval was broad, only key articles are cited in this paper and outreach to Native Americans, minorities and individual health professionals are focused upon. Although some review articles were identified, no meta-analyzes that combined data from multiple studies were found.
In the searches on outreach to specific groups, no evaluation articles were found specifically on health sciences library outreach to Native Americans. There was one article found that focused on information seeking behavior of African American women (Gollop, 1997), one on services to British Black and minority ethnic groups (Presley, 1995) and one on information-seeking behavior by multicultural students in a university environment. All of these articles noted the particular difficulties that minority groups experience in accessing information sources. Efthimiadis (1996) describes the difficulties of accessing medical research on minority populations in the major indexing and abstracting services of the health sciences literature. He concludes that population groups are not represented adequately in the index languages of health sciences databases.
Many more publications were identified on health sciences library outreach to rural populations, almost all of them based on project grants received from NLM. The ongoing work of Josephine Dorsch, Victoria Pifalo, Trudy Landwirth and Cheryl Dee is particularly notable in this literature and is reflected in the references cited in this paper. Several articles were found on health sciences library outreach services to the AIDS/HIV community, again based on project grants from NLM. An increasing number of articles in the health sciences library literature are deal with consumer health information services, an area that has been of interest to health sciences librarians for some time.
As with other practicing professions, the idea of research and evaluation is relatively new to health sciences librarians; however, interest and commitment are strong. The need for research skills by librarians was identified in the Medical Library Association's (MLA) educational policy statement, "Platform for Change" in 1992. Subsequently, MLA produced a research policy document, "Using Scientific Evidence to Improve Information Practice" in 1995. MLA has a very active Research Section that is striving to create a community of health sciences librarian researchers and, in particular, to increase the research content at the annual meeting. The fact that health sciences librarians practice in a research intensive environment in the health field means that they have the potential to play a leadership role in improving research standards in the profession of librarianship as a whole.
Despite this strong interest by MLA members, health sciences librarians have a ways to go to fully integrate research and evaluation into their professional practice. In reviewing the literature on education services in health sciences libraries published between 1987 and 1994, Dimitroff (1995) found that the percentage of articles that include research and evaluation of educational services is considerable lower than the percentage of such articles on other topics in the library and information science literature. Marshall (1995) has suggested that the traditional evaluation model that includes needs assessment, program planning, formative evaluation and summative evaluation has many applications for health sciences library practice and can be applied in the context of continuous quality improvement. The guide to outreach evaluation being undertaken in this project will be of great value to health sciences librarians who are seeking guidance on how to implement various research and evaluation models and techniques. The fact that this project gathers approaches and models from other fields for application to health sciences library outreach is particularly innovative and welcome.
Selecting the variables to measure in research and evaluation studies is fundamental. In evaluating overall services, health sciences librarians have traditionally emphasized the collection of input variables, or those resources that form the basis of service provision, such as staff, facilities, collection and equipment. In the past, such input measures have appeared in the form of library and information service standards (Medical Library Association, 1994; Canadian Health Libraries Association, 1995). Processes, or the activities and programs that form the basis for service provision, such as reference service, training or interlibrary loan, are also frequently described in the library literature, although often without an evaluation component.
The measurement of output variables that count the products of the processes involved in library service, such as number of reference questions answered, number of clients trained or number of interlibrary loans sent and received, have also been routinely collected in the form of library statistics. Compendiums such as the "Annual Statistics of Medical School Libraries in the U.S. and Canada" collect and publish a valuable combination of input, process and output data. These general purpose data can sometimes be used to glean information about outreach activities and should be considered as a source when planning an outreach evaluation framework. Alternatively, encouraging the addition of outreach variables to these general purpose instruments could be considered part of an overall outreach evaluation data collection strategy.
More recent evaluation measures in health sciences libraries are looking at the perceived importance of library services and levels of satisfaction of library users with those services, often in a continuous quality management or benchmarking context (Marshall and Inglis, forthcoming). Still other studies are extending the output measures to include the perceived value and impact of information provided by the library (Marshall, 1992; Urquhart and Hepworth, 1995). The remainder of this paper will review input, process and output measures used to date in health sciences library outreach studies as background for the development of a more substantive evaluation framework being undertaken in this project.
Demographic variables are a type of input variable that is essential for understanding the clientele being served as well as for analyzing data for evaluation purposes. In the health sciences library outreach literature, the most commonly gathered demographic variables include: geographic location; occupational group; type of work activity (e.g. clinical, research, teaching and learning, etc.) and type of employer.
Information needs are also frequently explored through variables such as types of information required; purpose of the information; frequency of information needs; sources of information used; most useful sources; modes of accessing information; library and information services used; and problems experienced in locating and retrieving information. Since primary care practitioners frequently seek information about pharmaceuticals, some studies ask specific questions about information sources and use in this area.
Since biomedical databases are a key information resource for health professionals, many studies include specific variables dealing with the use of MEDLARS. Specific questions include: reasons for searching MEDLARS; how health professionals learned how to use MEDLARS; how they obtain documents identified through MEDLARS; and reasons for not searching MEDLARS.
In outreach studies particularly, investigators have included questions on availability and usability of the technology required to access databases. These variables may be related to telecommunications access as well as hardware and software availability and reliability. Connected to these variables are questions about the barriers to technology and information use. Respondents are often asked to identify factors that could improve access and use of databases and the information they contain. Questions about the use of document delivery systems such as Loansome Doc usually accompany the database questions.
A variety of activities or interventions designed to improve health professionals' access to information are described in the literature. These range from promoting regularly provided hospital library services to off-site clients to specially tailored programs designed to meet the particular needs of rural or minority groups. Dorsch and Pifalo (1995) have summarized the traditional and technological modes that have been used to provide health sciences library outreach. Their list includes:
Each of these interventions may have an array of characteristics associated with it that may form the basis for additional variables. For example, end-user training make take different forms: one-on-one interaction, hands on training, or brief lectures and demonstrations. The provision or loan of equipment and technical support may also be part of the package. Training may be integrated into continuing education programs in the institution or given through a professional association for credit. Sessions may be linked to specific clinical problems and local information needs. Various types of phone or electronic help desks may be available. Follow-up refreshers and other types of ongoing support and communication may also be used in specific interventions. Posters may be used to advertise sessions and advance information packets may be sent to participants. Some interventions have experimented with information sent to the hospital administration as well as press releases to gain visibility and support for the initiative.
Standard output variables used in health sciences library outreach reflect traditional practice, namely counting the number of times the client uses a given service. These services can include database searches, document delivery, attendance at training sessions and consultation with the librarian. For specific training programs, the number of health professionals trained and the number of searches conducted are most frequently documented.
Client satisfaction with training, searching and search results form another set of output variables. Since health sciences outreach is constantly concerned with implementing new technologies and new methods of access, variables that identify the obstacles encountered by health professionals in using these services are seen as especially important. Questions about where health professionals go for assistance when they need it are also included in some studies.
As the outreach efforts sponsored by NLM have matured, there has been a move away from studying the effect of single training interventions to monitoring the continued use of tools such as Grateful Med and Loansome Doc, as well as local library services. The expansion of the interventions to include additional follow-up and support have been based on observations made in the initial evaluation studies.
In addition to use of specific tools and services, second generation output measures have attempted to gather more in-depth data on changes in information seeking and use. For example, asking whether the interventions have resulted in health professionals consulting more books and journals; seeking out more information; or recommending information resources to others. Another group of studies examines information search strategies in depth in an attempt to gauge the quality of end-user searches and contribute to improvements in training techniques.
Some studies have begun to add additional value or impact measures to determine not only if the library user was satisfied with the service, but also whether the information contributed to decision-making in key areas such as diagnosis, treatment or, in the case of hospital care, length of stay. Some of these studies also ask health professionals whether they changed their behavior as a result of applying the information received (Marshall, 1992; Klein, Ross, Adams and Gilbert, 1994; Burnham and Perry, 1995). In a "Value Toolkit" developed in the U.K., Urquhart and Hepworth (1995) provide data collection instruments for a range of evaluation purposes from assessment of user needs to information services provision and service outcomes. Their outcome measures include support for clinical decision-making in the areas of diagnosis, therapy and management and quality of care.
The results of the research and evaluation studies on health sciences library outreach to date provide some consistent messages. First is that health professionals in rural communities are extremely busy practitioners who want practical information that can be easily located and applied in their own settings. They often find the medical literature too fragmented and complex for these purposes and not easily applied to patient care. These health professionals do not have time to do extensive literature searches, let alone critically appraise the findings. The amount of information contained in MEDLARS is overwhelming to most primary care providers and specialized databases that organize and synthesize information in a time-saving manner, particularly for pharmaceuticals, would be welcomed particularly by rural practitioners.
The second result of note is that there is great diversity among the users of health sciences library outreach. Research-oriented health professionals may well want access to the complete family of MEDLARS databases and be willing to spend the time to critically appraise their search results. However, adoption of end-user searching is not universal -- some health professionals prefer to use search intermediaries such as librarians or other traditional means of obtaining information such as consulting a colleague or using their own department or personal collection of journals and books (Dorsch, 1997). Health sciences library outreach efforts that have as an objective to change health professionals' behavior would do well to note findings that the personal contact between health professionals and the librarians was found to be essential in developing and sustaining changes in information seeking habits.(Dorsch, 1997;Burnham and Perry, 1995).
Dee and Blazek (1993) found that rural physicians in Florida were also open to information-seeking behavior change interventions supported through personal contact with librarians. In a needs assessment phase, the investigators used the case study method to determine the number, type and urgency of patient care questions. Colleagues were cited as the primary source of answers; however there were still, many unanswered questions and the information sources used by the physicians were not sufficient to answer their questions.
Lundeen, Tenopir and Wermager (1994) conducted interviews and sent a mailed questionnaire in their study of information needs of rural Hawaiian health care practitioners. They found that practitioners used journal articles most frequently as information sources. Personal files or colleagues were most common sources used for accessing such material. Two types of information needs were identified: case specific and more general requirements related to health care management and practice issues. Health professionals expressed problems in locating and retrieving information.
Bowden, Kromer and Tobia (1994) sent a questionnaire to physicians in five Texas counties and found differences between physicians who had access to established medical libraries and those who practiced in remote areas. There was a statistically significant difference in the use of library services by the two groups, with the remotely located physicians having the lower use levels. Ely, Burch and Vinson (1992) found differences in the number of questions raised regarding patient visits between rural and urban physicians. Urban physicians and those with most colleagues in their practice tended to ask the most questions.
These results suggest that rural health professionals continue to be disadvantaged and that outreach efforts need to be continued and strengthened. The major barriers to effective information seeking and use that have been identified to date are:
Need for other types of information in additional to literature, e.g. networking with colleagues, statistical data for grant applications, program planning, directory and referral information
Increased demand on local resources without increased support]
There was also some concern expressed that when Grateful Med training programs were successful, that they put additional strain on local resources without additional support. This must also be considered as a potential obstacle.
A study by Dorsch and Pifalo (forthcoming) demonstrated the complex information needs of rural health professionals and the difficulty of building core collection to support them. The authors found that Abridged Index Medicus and Brandon-Hill core list titles filled no more than 30% of the requests in their study. Additional requests were from a wide variety of clinical, administrative and allied health areas.
Although a variety of methodologies were used in the health sciences library outreach studies, survey questionnaires were by far the most common. Surveys were sometimes used at the start of projects to identify needs and existing resources. They were also used for summative and follow-up evaluation purposes. In most cases, surveys were mailed, but questionnaires were also used as part of personal interviews. Sometimes mailed and interview questionnaires were used in the same study in a complementary fashion. While the majority of questionnaires were sent to health professionals, several studies also sent a questionnaire to the trainers and facilitators (e.g. Burnham and Perry, 1995)
Some studies analysed existing data such as document delivery requests. Only one case study of rural practitioners that involved in-depth observation in clinical settings, documentation of information sources used, examination of patient records and interviews was found (Dee and Blazek, 1993). Many studies relied on self-report of health professionals rather than other objective sources of data. Studies of database searching behavior sometimes made use of transaction logs (e.g. Nelson, 1992).
A few studies employed focus groups to gather qualitative data. The critical incident technique is also a research method that has proven useful in earlier studies of the impact of MEDLINE searches conducted by NLM (Lindberg, Siegel, Rapp, Wallingford and Wilson, 1993).
Although not reporting on an evaluation study in strict terms, Fuller (1995) describes the importance of an environmental scan in determining the climate for introducing new information technology and raising awareness among key personnel in health care institutions. Such activities may contribute to a climate of "readiness" for outreach interventions.
While a number of elements that will be useful in building an effective model of outreach and outreach evaluation are beginning to appear in the health sciences library outreach literature, additional conceptualization and refinement are required. The research that has been funded to date by NLM and the evaluation efforts of health sciences librarians over the years are all contributing to this effort. Unfortunately, many of the program descriptions in the library and information science literature lack an evaluation component and most studies are descriptive in nature; however the number of reports that reflect a more in-depth research approach is increasing.
The cumulative work of Dorsch, Pifalo and Landwirth on Grateful Med outreach points towards the need for varied training formats, reexposure to end-user searching, repeat training and the importance of "readiness" (Dorsch 1997). These experienced researchers in the field suggest that reaching equity in information access between rural and urban practitioners is emerging as a multifaceted, long term effort.
Lundeen, Tenopir and Wermager (1994) suggest a possible model for rural health care information agents based on the idea of agricultural extension agents. It would be useful to compare the agricultural model with previous outreach librarian roles such as circuit librarians and clinical librarians to identify the similarities and differences in these approaches.
The need for further studies that include adequate numbers of health professionals in different occupational groups is noted, so that information preferences and barriers of different occupation groups can be compared.
Burnham and Perry (1996) raise questions about the effectiveness of Grateful Med training projects on the basis that user-friendly access to the medical journal literature in itself does not meet the information needs of health professionals. The authors call for additional efforts to make training more relevant to local needs by using examples that draw heavily on patient care and demonstrate resources that meet local needs. The success predictors cited by Burnham and Perry (1996) are:
This review of the substantive research that has been done on health sciences library outreach suggests that a more contextualized model of outreach is required that will take into account the many challenges that exist for health professionals in the complex, changing and uncertain environment of health care. Gaining a greater understanding of the multiple pressures that practitioners face from consumers, third party payers, government, professional groups and other stakeholders can help to illuminate the information needs and motivations of health professionals.
Work conducted by members of the Biomedical Library Review Committee (BLRC) of NLM also contributes some useful ideas for a framework for evaluation health sciences library outreach (Stead, Haynes, Fuller, Friedman, Travis, Beck, et al.1994). Focusing on medical informatics research and library resource projects, the committee points out that such projects can present unique evaluation problems and that it may not always be possible to adapt evaluation methods from the natural and social sciences in some cases. The committee suggests that problems in evaluating medical informatics may be tackled by planning system development in the context of testable hypotheses, subdividing complex projects into modules and using qualitative studies in situations where quantitative approaches are impractical or would provide incomplete information.
The traditional model of evaluation research with its steps of needs assessment, program planning, formative evaluation and summative evaluation coupled with an environmental scan may also be useful as the basis for a tailored model for health sciences library outreach. Multiple research methods that take advantage of a wide variety of quantitative and qualitative approaches also appear to be appropriate as a means of providing numeric data together with in-depth description that forms the basis for understanding of complex issues. It appears that personal contact with competent and responsive librarians who can deliver a range of training and services to meet the varied needs of the rural practitioner is part of the ideal model. Seeing specific interventions such as Grateful Med training in this broader context and with longer term goals for sustainability will help to ensure future success of such efforts.
Anderson, R.K., Haddix, A., McCray, J.C., Wunz, T.P. Developing a health information infrastructure in Arizona. Bull Med Libr Assoc 1994; 82(4): 397-400.
Bowden, VM, Kromer, ME, Tobia, RC. Assessment of physicians' information needs in five Texas counties. Bull Med Libr Assoc 1994; 82(2): 189-96.
Burnham, JF, Perry, M. Promotion of health informamtion access via Grateful Med and Loansome Doc: Why isn't it working? Bull Med Libr Assoc 1996; 84(4): 498-506.
Burroughs, C. Planning and evaluating information outreach among minority populations. Paper presented at the Medical Library Association Annual Meeting, Seattle, WA, May 1997.
Canadian Health Libraries Association. Standards for library and information services in Canadian health care facility libraries. Toronto: CHLA, 1995.
Cimpl, K. Clinical medical librarianship: A review of the literature. Bulletin of the Medical Library Association 1985; 73(1): 21-28.
Dee, C., Blazek, R. Information needs of the rural physician: a descriptive study. Bull Med Libr Assoc 1993; 81(3): 259-64.
Dimitroff, A. Educational services in health sciences libraries: A content analysis of the literature. Bull Med Libr Assoc 1995; 83(4): 420-24.
Dorsch, J.L. Document needs in a rural Grateful Med outreach project, Bull Med Libr Assoc 1994; 82(4):357-62.
Dorsch, J.L. Equalizing health professionals' information access: Lessons from a follow-up outreach project. Bull Med Libr Assoc 1997; 85(1):39-47.
Dorsch, J.L. Frasca, M.A., Wilson, M.L., and Tomsic, M.L. A multidisciplinary approach to information and critical appraisal instruction. Bull Med Libr Assoc 1990; 78(1):38-44.
Dorsch J.L, Landwirth T.K. Rural Grateful Med outreach. Project results, impact and future needs. Bull Med Libr Assoc 1993; 81(4):377-82.
Dorsch, J.L., Pifalo, V. Rural health professionals and information access: A follow-up study. Proceedings of the 7th International Conference on Medical Librarianship, Washington, DC, May 10-12, 1995: 6-10.
Dorsch, J.L., Pifalo, V. Rural health professionals and information access: A retrospective use study. Bulletin of the Medical Library Association. Forthcoming.
Doyle, J. Knowledge-based information management: Implications for library services. Med Ref Serv Q 1994; 13(2):85-87.
Efthimiadis, E.N. Population groups: indexing, coverage and retrieval effectiveness of ethnically related health care issues in health sciences databases. Bull Med Libr Assoc 1996; 84(3):386-96.
Ely, JW, Burch, RJ, Vinson, DC. The information needs of family physicians: Case-specific questions. J Fam Pract 1992; 35(3): 265-69.
Fuller, S. Internet connectivity for hospitals and hospital libraries: Strategies. Bull Med Libr Assoc 1995; 83(1):32-36.
Gollop, C.J. Health information-seeking behavior and older African American women. Bulletin of the Medical Library Association 1997: 85(2): 141-46.
Howell, P.B., Jones, C. A focus on quality - The librarian's role in occurrence screening. Med Ref Serv Q 1993; 12(2):83-89.
Klein, H.S., Ross, F.V., Adams, D.L., Gilbert, C.M. Effect of online literature searching on length of stay and patient care costs. Acad Med 1994; 69(6): 489-95.
Lefebvre, C. The Cochrane Collaboration: The role of the UK Cochrane Centre in identifying the evidence. Health Libr Rev 1994; 11(4): 235-42.
Lindberg, D.A., Siegel, E.R., Rapp, B.A., Wallingford, K.T., Wilson, S.R. Use of MEDLINE by physicians in clinical problem-solving. JAMA 1993; 269(24): 3124-29.
Liu, M. , Redfern, B. Information-seeking behavior of multicultural students: A case study at San Jose State University. College and Research Libraries 1997; 348-54.
Lundeen, GW, Tenopir, C, Wermager P. Information needs of rural health care practitioners in Hawaii. Bull Med Libr Assoc 1994; 82(2): 197-205.
Marshall, JG, Inglis, J., Benchmarking Task Force Co-chairs. Benchmarking toolkit. Toronto: Canadian Health Libraries Association. Forthcoming.
Marshall J.G. The impact of the hospital library on clinical decision making: The Rochester study. Bull Med Libr Assoc 1992; 80(2): 169-78.
Marshall, JG. Issues in clinical information delivery. Library Trends 1993; 42(1):85-107.
Marshall, JG. Using evaluation methods to improve quality. Health Libr Rev1995; 12(3):121-23.
Medical Library Association. Platform for change. Chicago: MLA, 1992.
Medical Library Association. Using scientific evidence to improve information practice. Chicago: MLA, 1995.
Medical Library Association, Hospital Library Section. Minimum standards for hospital libraries. Chicago: MLA, 1994.
National Library of Medicine. Long Range Plan. Improving Health Professionals' Access to Information. Bethesda, MD. National Library of Medicine, August 1989.
Nelson, JL. An analysis of transaction logs to evaluation the educational needs of end users. Med Ref Serv Q 1992;11(4):11-21.
Pifalo, V. Circuit librarianship: A twentieth anniversary appraisal. Med Ref Serv Q 1994; 13(1): 19-33.
Presley, F. Sharing Black and minority ethnic health information. Assignation 1995; 12(2): 46-47.
Sewell, W., Teitlebaum, S. Observations of end-user online searching behavior over eleven years. J Amer Soc Info Sci 1986; 37(4):234-45.
Sowell, S.L. LATCH at the Washington Hospital Center, 1967-1975. Bull Med Libr Assoc 1978; 66(2):200-209.
Stead, WW, Haynes, RB, Fuller, S, Friedman, CP, Travis, LE, Beck, JR, et al. Designing medical informatics research and library-resource projects to increase what is learned. J Am Med Informatics Assoc 1994; 1:28-33.
Urquhart, CJ, Hepworth, JB. The value of information services to clinicians: A toolkit for measurement. British Library and University of Wales, Aberystywth, 1995.
Wallingford, K.T., Ruffin, K.A., Ginter, M.L., Spann, F.E., Johnson, G.A., Dutcher, G.A., et al. Outreach activities of the National Library of Medicine: A Five-Year Review. Bulletin of the Medical Library Association 1996; 84(2):1-60.