OSTIblogs Keyword: [federated-search]

Just as science progresses only if knowledge is shared, accelerating the sharing of knowledge accelerates science. All of us engaged in disseminating science knowledge have the opportunity and obligation to do our jobs better, for to do so accelerates science itself. 

To this end, I propose a grand challenge—to make more science available to, and searchable by, more people than ever before. A momentous milestone will be achieved once we enable everyone with web access the ability to search with unparalleled precision a billion pages of authoritative science. Already, considerable progress has been made. 

Today, all of OSTI's information products are on the web. This is in sharp contrast to the situation as recently as the mid-1990s, when OSTI had no products on the web.

First becoming popular in 1994, the web quickly emerged as a transformational technology, and its potential for reshaping OSTI was apparent. Recognizing the opportunity to advance the OSTI mission, OSTI set out to capitalize on it as quickly as resources would allow by producing web applications to disseminate all manner of scientific and technical information (STI). A steady progression of new OSTI products addressed the various forms of STI: technical reports, e-prints, conference proceedings, accomplishments, patents, and project descriptions . To make it easy for users who want to search through all these products at once, we introduced the DOE Science Accelerator, which is powered by our special web architecture called federated search. Reaching out beyond DOE, we initiated a collaboration with other agencies to allow users to search their R&D results along with DOE's; thus emerged Science.gov. Most recently, we took collaboration world wide by federating the best information sources from governments around the world, WorldWideScience which makes searchable about the same quantity of science as does Google.

This is the third, and final, article in a series. The first article provided an overview of the E-print Network. The second article discussed the special harvested component of the E-print Network in depth. This article provides a tour of the E-print collections which are federated. Hopefully, once you finish reading this article and this series, you will appreciate the innovation and hard work that has gone into producing the premier federated search application for searching E-prints.

This is the final article in a series about the limitations of the crawl and index approach to searching scientific content and the advantages of federated search. Part 1 identified a number of issues with Google and the other crawlers, and showed why researchers and the science attentive citizenry don't rely on "Googleable" content to meet their needs for quality scientific and technical information. Part 2 explained how federated search, by providing access to "non-Googleable" content, overcomes the Google limitations. This article highlights three important applications, developed and maintained by OSTI, that demonstrate how federated search is going beyond crawling to advance science.

This is the second in a three part series of articles about the deficiencies of web crawling and indexing, the superiority of federated search to the serious researcher, and the value of OSTI federated search applications in advancing science. Part 1 identified a number of serious limitations of Google and the other crawlers. This article shows how federated search overcomes these limitations. The final article in the series highlights a number of federated search applications and databases that OSTI makes available to the public.

The web is growing.

For providing searchable access to the content that matters the most to scientists and researchers, Google and the other web crawlers can't keep up. Instead, growing numbers of scientists, researchers, and science attentive citizens turn to OSTI's federated search applications for high quality research material that Google can't find. And, given fundamental limitations on how web crawlers find content, those conducting research will derive even more benefit from OSTI's innovation and investment in federated search in the coming years.

In Part 1 of this series I provided an overview of the technology that drives the E-print Network. In this article I will provide some detail about how the harvested collection, the "E-prints on Web Sites" component of the E-print Network, is constructed. In Part 3, I will discuss the technology of the portion of the E-print Network that relies on federated search of databases.

 Part of OSTI's R&D aims at understanding how scientists use information. This goal was originally articulated by OSTI's Thurman Whitson, who has since retired. To that end we have begun to look at the different kinds of information provided by the different Web-based science resources. Different kinds of information imply different uses. It is not that one resource is better than another overall, it is that they are very different and support different uses.

The E-print Network is one of OSTI's most popular and powerful research offerings yet few of its users know about the advanced technology that drives it and makes it simple to use. Professional researchers in basic and applied science are able to access over 5 million e-prints gathered from nearly 28,000 world-wide databases and web-sites. Numerous OSTI innovations ensure that the E-print Network's documents are of extremely high quality, are highly relevant to researchers, and are easy and quick to find. This is the first in a series of articles about the technology behind this very important component of the Science Accelerator. This article serves as an overview; subsequent articles will provide more technical information.

Federated search is very much at the heart of OSTI's ability to realize its mission. OSTI provides a simple description of what federated search is and how it works in the OSTI environment. The best way to experience the tremendous value of federated search at OSTI is to try several of OSTI's flagship applications:

• Science Accelerator


• Science.gov


• WorldWideScience.org