National Information Center on Health Services Research and Health Care Technology (NICHSR) | |
Conducted by AcademyHealth
Funded by the National Library of Medicine
February 2006
Long-Term Value of Grey Literature
Archival Practices of Producing Organizations
AcademyHealth is defining the scope of grey literature in health services research/health policy for the National Information Center on Health Services Research and Health Care Technology (NICHSR) at the National Library of Medicine. In considering the scope and status of grey literature, it is important to know who the major users are and what they are using grey literature for. In addition, knowledge of what grey literature is being produced and what the preservation procedures are for major producers is critical. Only by better understanding current practice can we make recommendations about what can be done to archive and preserve grey literature.
Grey literature is "that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers” according to the Luxembourg convention. This category includes reports, memoranda, conference proceedings, standards, technical documentation, and government documents.” (National Library of Medicine Web site, accessed November 2004). However, others may define grey literature more broadly to include everything except peer-reviewed books and journals accepted by Medline. Examples of grey literature publications include reports, theses, non-commercially published conference papers, government reports, translations, committee reports, working papers, technical specifications and standards, and bibliographies.
To better identify and evaluate the grey literature of health services research and health policy, AcademyHealth convened an Advisory Committee to identify and prioritize relevant disciplines, subject areas, and types of grey literature. To supplement the expertise of the Advisory Committee, a small number of grey literature-producing organizations were contacted to determine their preservation policy
The importance of grey literature lies in its ability to communicate complex information in simple terms and to disseminate results more quickly. Much of the grey literature is derivative and designed to make technical material or research findings easily understood by a lay audience. For example, policy briefs, issue briefs, and fact sheets are often produced by summarizing more technical reports and providing context. They are also a way for organizations to get results out to their target audience more quickly. Many organizations, including research institutes like the Leonard Davis Institute, the Sheps Center, and the Urban Institute, produce working papers and issue briefs. Some organizations, like the Alliance for Health Reform, write policy briefs of interest to their audiences that summarize the findings of a number of individual research studies. Many of these research studies are included in the published literature.
A number of producers indicated that the material included in grey literature is ultimately incorporated into the published literature, although the content may differ somewhat from the grey literature. For example, AHRQ reported that some of the material produced under contract and grants is ultimately published by the researchers who do the work. Likewise, much of the research funded by entities such as The Robert Wood Johnson Foundation is published in health services journals. Although much of the grey literature is sourced in published literature, Advisory Committee members deemed this characteristic to be irrelevant since much of the published literature is likewise derivative in nature. Rather than compete with the published literature, grey literature has the potential to complement and communicate findings to a wider audience.
Grey literature is also a valuable barometer of public interest, reflective of current priorities and indicative of future ones. Many grey literature producers, including IOM, and GAO, cite congressional agendas and public interests as the driving force behind the determination of subject areas. The Alliance for Health Reform likewise expects management to determine specific topics within broad subject areas determined by the congressional and funding priorities. Accordingly, the focus of subject areas changes as priorities change. At any given time, the most popular items are topical and largely determined by what is currently in the news. Most recently, producers report an emphasis on topics including bioterrorism, disparities, information technology, patient safety, and obesity.
To identify the most frequent users of grey literature, AcademyHealth, with the assistance of the Advisory Committee, first identified the most prolific producers of grey literature in the fields of health services research and health policy. Since grey literature covers a wide range of subject areas within HSR, AcademyHealth reviewed web sites and meeting agendas to identify subject areas; Advisory Committee members then prioritized subject areas by relevance.
Access to Care* |
Management & Organization |
Behavioral Health |
Maternal & Child Health* |
Child Health |
Medicaid* |
Chronic Care and Chronic Care Delivery |
Medicare* |
Cost Containment* |
Medical Records |
Coverage |
Mental Health* |
Disparities in Health & Health Care |
Minority Health* |
Econometrics |
Patient Safety |
Emergency Preparedness/Bioterrorism* |
Pharmaceuticals & Emerging Technologies* |
Evaluating New Technologies |
Post Acute Care |
Gerontology* |
Privacy |
Health Care in the Workplaces |
Private Insurance Markets* |
Health Care Workforce |
Program Evaluation/Evaluation Research |
Health Insurance Markets* |
Public Health* |
Health Policy |
Quality |
Infectious Disease* |
Substance Abuse/Addiction |
International |
Technology, Innovation & Evaluation |
Long-Term Care* |
Women’s Health |
Managed Care* |
|
* Voted most important by committee member
Advisory Committee members also identified various types of grey literature and their relative importance. The items used, in order of relevance, with 5 being most frequently used by committee members :
5 |
4 |
3 |
2 |
1 |
|
|
|
|
|
There was consensus among committee members, and the producers contacted confirm, that the most common target audiences for grey literature products are policymakers, reporters, federal and state agencies, foundations, researchers and grantees.
Overwhelmingly, producers cited policy makers and decision makers as their first priority. Decision makers include members of Congress and congressional staff, federal and state agencies and regulators, and policy analysts. In the case of policymakers, grey lit is used more than non-grey lit, because grey lit is generally more up to date and is more accessible. Cost effective, timely data is often simply not available in published literature. In addition to this, policy documents are not generally published in peer-reviewed journals. Grey literature materials of greatest relevance to policymakers are sourcebooks, chartbooks, evaluations, consensus reports, fact sheets, briefings, transcripts, and issue briefs.
Reporters and the media in general were the next group most cited by producers. That decision makers and reporters comprise the main target audience is not surprising when coupled with the fact that most producers described their grey literature products as a means of communicating complex issues to lay persons for the purpose of creating public support and awareness for a particular issue. The materials of greatest relevance include summaries, issue briefs, chart books, source books, fact sheets, transcripts, testimonies, and conference proceedings.
Foundations and funding agencies rank third in terms of priority audiences. Foundations use grey literature to see where research needs are and to determine the course of their research. Technical reports and chartbooks are often produced at the request of funders to summarize the body of work produced under a research contract or grant, and many of these are disseminated widely and are available on their websites. Likewise, most producers cited a high degree of policy relevance as necessary for consideration. Materials of greatest relevance include grants results reports, evaluations, research reports, syntheses, newsletters, and news summaries.
Researchers rank last in terms of audience importance for producers of grey literature. Researchers use grey literature as a resource to support their research, as well as to identify funding priorities. Likewise, grantees and prospective grantees look to grey literature to determine what is being funded by whom to help determine the course of their research. Materials of greatest relevance include datasets, evaluations, research reports, training materials, working papers, surveys, newsletters, program and issue briefs, and listserves.
Grey literature has long-term value, particularly because it provides policy context and implications that may not be found in the published literature. In fact, advisory committee members believed that the value of grey literature is on par with that of traditional published literature. Relevancy, progress, and how debate changes over time on a particular topic can be assessed from these materials. Another use of grey literature is to establish historical documentation. The progress of a document to its finished form can sometimes be as valuable as the finished product, and the various drafts of a document can fill in gaps in the historical record.
Notably, published literature and grey literature alike cite each other. As described earlier, grey literature is derivative in nature, summarizing and communicating complex issues to a lay audience. What is perhaps not as well understood is the corresponding role of grey literature in developing published literature. For example, journal articles, in particular, make frequent use of grey literature to support claims. Likewise, briefs and summaries often rely on disparate sources of information found in grey literature. Even textbooks incorporate new findings and perspectives derived from grey literature when updating new versions. Grey literature often illustrates the progression of knowledge and supports the development of new published works. To lose certain grey literature products would be to lose many of the very sources of published literature.
The grey literature producers did not always have clear and consistent policies regarding what to preserve and how to preserve materials. Although technology has made it less costly to store information electronically, and much information is archived online, material may be accessible only as long as the organization is in existence. Every producer interviewed acknowledged having hard-copy archives either in an on-site library, off-site storage, or in the care of a storage and fulfillment service such as the National Technical Information Service (NTIS). In general, producers internally archive materials electronically in order to fulfill requests for materials with reprints rather than original copies. Many of the producers indicate that they save materials on-line. All indicated their willingness to share material with researchers who make requests, but acknowledge that it may be difficult for researchers to know exactly what materials are available, especially if they are not available on-line.
Federal agencies follow regulatory guidance regarding preservation; this guidance includes Generally Accepted Accounting Standards and National Archives Standards for the General Accounting Office, NARA Schedule of Retention Regulations, and NTIS. AHRQ and other federal government agencies send reports and other final grant and contract products to NTIS for archiving and preservation. Some of the material in these reports ends up in the published literature in the form of journal articles. More transient materials, such as fact sheets and issue briefs, are not preserved in any organized fashion. Many of these materials, such as fact sheets, are updated periodically and so the content may change over time.
All reports produced by the Institute of Medicine receive Library of Congress numbers, and are preserved according to the Library of Congress standards. In general though, the producers interviewed did not pursue or receive Library of Congress numbers for their grey literature products.
Grey literature is a significant resource in the fields of health services research and health policy. Preserving or archiving it in some way is important, and there may also be a role for an organization to take the lead in preserving this material in a way that will act as a quality seal of approval. This can be accomplished either through directly archiving the material, or creating and providing guidance to producing organizations about appropriate preservation procedures. Transparency is also needed about the creation and review process of grey literature to enable users to better assess the products.
Archiving and Preservation
Ensuring Uniform Quality of Grey Literature
Last reviewed: 08 September 2008
Last updated: 08 September 2008
First published: 22 June 2006
Metadata| Permanence level: Permanent: Dynamic Content