Analytics for Data at Massive Scales

The Department of Defense (DoD) maintains one of the largest computer networks in the world. The network follows DoD personnel across the globe collecting, transferring and processing information in forms as diverse as data warehouses, in-the-field mobile devices and mission computers on board F-18’s. This network is also constantly changing in size and shape as new missions are undertaken and new technology is deployed. In military terms, that means the cyber terrain of the DoD network is constantly shifting.

DARPA held a multi-program performer meeting for researchers to hear presentations on the latest innovations and promising approaches in the area of Big Data and data analytics. Speakers during the day-long event included representatives from the White House, FBI, universities from across the country and leading companies from the private sector who are focused on the potential efficiencies and advantages that can be gained in Big Data.

Military operations depend upon the unimpeded flow of accurate and relevant information to support timely decisions related to battle planning and execution. To address these needs, numerous intelligence systems and technologies have been developed over the past 20 years, but each of these typically provides only a partial picture of the battlefield, and integrating the information has proven to be burdensome and inefficient.

Bonnie Dorr (left), program manager in DARPA’s Information Innovation Office (I2O), shakes hands with Henry Kautz, past president of the Association for the Advancement of Artificial Intelligence (AAAI), upon her recent induction as an AAAI Fellow. Each year, AAAI bestows the lifetime honor of Fellow on only a handful of researchers for their exceptional leadership, research and service contributions to the field of artificial intelligence.

Today's web searches use a centralized, one-size-fits-all approach that searches the Internet with the same set of tools for all queries. While that model has been wildly successful commercially, it does not work well for many government use cases. For example, it still remains a largely manual process that does not save sessions, requires nearly exact input with one-at-a-time entry, and doesn't organize or aggregate results beyond a list of links. Moreover, common search practices miss information in the deep web—the parts of the web not indexed by standard commercial search engines—and ignore shared content across pages.