NetLab Workshop Report

Note: The views and comments contained in this document are not necessarily those of the National Science Foundation, but are exclusively those of the workshop participants.

Contents:

Executive Summary

Introduction

The NetLab Workshop

Activities Identified by the NetLab Workshop

The Promise of Knowledge Network Research

NetLab Activities

Opening New Methodological and Infrastructural Possibilities

Single-Site Large-Group Experiments

Large-Scale Web-Mediated Experiments

Collaboratories

Purposively Sampled Experimental Designs

"Virtual Group Reality" Experiments

Facilitating Conventional Research Methodologies and Infrastructures

Laboratory Development

Software Development

Software Repositories/dd>

Information Resources

Human Resources

Enhancing Educational Opportunities

Package NetLab Demonstrations

Social Science Training

Student Participation and Replication

Undergraduate, Graduate and Postdoctoral Training

Student Funding

Challenges

Collaboratory Mechanisms

Long-Term Support

Hardware/Software Support

Funding Commitments

Conclusion

Workshop Participants

________________________________________________________________

EXECUTIVE SUMMARY

The Workshop reached several conclusions:

• The Social and Behavioral Sciences are fundamentally concerned with "knowledge networks." Such networks, which may include widely distributed individuals, are ubiquitous. Examples of such networks include markets, which aggregate diffuse individual behavior into prices; election systems, which take pieces of information embodied in individual votes and aggregate to yield a leader; and social systems that aggregate many individual characteristics into social hierarchies.
• Many researchers are establishing networked laboratories -- "NetLabs" -- to test theoretical constructs pointing to knowledge networks. The laboratories themselves are designed as knowledge networks.
• The convergence of new computational power, software tools and the extension of broad-band networks presents new opportunities for NetLab experimenters.
• • Experiments can now be "scaled up" to include hundreds or even thousands of subjects.
  • Experiments can now cross many boundaries, bringing new population samples into the laboratory.
  • Experiments can now mimic lengthy time periods in which subjects interact with one another over long intervals.
  • Laboratory experimentation can now become part of the routine education of undergraduates.
    
• There are serious challenges posed by the use of NetLabs
   
  • The infrastructure for NetLabs is still under development and will require a substantial infusion of intellectual and fiscal resources.
  • Large-scale centers and "collaboratories" are necessary to help push the development of these new tools and approaches.
  • Training and technical support are lacking.

The Workshop participants concluded that the social and behavioral sciences are in an enviable position but at a critical juncture. These sciences are enviable because they are all centrally focused on understanding knowledge networks and have enjoyed substantial success in building theoretical models to explain how networks succeed and fail. These sciences are poised at a critical juncture because the capacity to empirically test many of these models is only now being made possible through the development of broad band computer networks. If the nascent laboratory experimental approach is encouraged and is coupled with new technological innovations, then the SBE disciplines will be primed for major scientific advances.

________________________________________________________________

INTRODUCTION

I.1 The NetLab Workshop

The convergence of an explosion in computing power, the expansion of networks and rigorous experimental methods, combined with a new NSF initiative on Knowledge and Distributed Intelligence (KDI), breaks open the science frontiers for the social and behavioral sciences. These sciences are uniquely positioned both to use and to study networked knowledge systems. NetLabs, which in their simplest sense are networked laboratory computers, constitute a promising technology for pressing forward research in the social and behavioral sciences.

NetLabs, especially when tied to newly developing technologies, promise to free experimentalists in the social sciences from old boundaries. No longer will human subjects need to be drawn from the venerable population of college sophomores: NetLabs provide the possibility of cross-national samples which can reflect the population characteristics of many different countries. No longer will experiments be confined to a networked laboratory of 10 to 20 personal computers: NetLabs potentially can connect hundreds, if not thousands, of subjects who can participate in real time. Finally, because they can be easily accessed at many different locations and times, NetLabs promise the possibility of long-term experiments that can take place across many days, months or even years. In short, NetLabs promise to significantly expand what experimentalists can do and promise to expand the questions that experimentalists can answer.

This report summarizes the fruits of the NetLab Workshop's efforts: a compilation of the challenges, opportunities and barriers to understanding the array of knowledge networks within which people are immersed.

The Workshop identified three major realms of scientific activity that would benefit directly from Knowledge Networking funding opportunities: (1) spurring the development of innovative methodologies; (2) expanding conventional laboratory research; and (3) developing the pedagogy to enhance undergraduate, graduate and postdoctoral training experiences. These three areas are briefly enumerated in the following section. A full discussion of these points follows.

I.2 Activities Identified by the NetLab Workshop

Opening New Methodological and Infrastructural Possibilities

• Single-Site Large-Group Experiments
• Large-Scale Web-Mediated Experiments
• Collaboratories
• "Virtual Group Reality" Experiments

Facilitating Conventional Research Methodologies and Infrastructures

Research Funding

Laboratory Development

Software Development

Software Repositories

Information Resources

Human Resources

Enhancing Educational Opportunities

• Packaged NetLab Demonstrations
• Social Science Laboratory Training
• Student Participation and Replication
• Undergraduate, Graduate, and Postdoctoral Training

I.3 The Promise of Knowledge Network Research

Knowledge networks operate through a wide range of human activities. Not only is much of the knowledge held in complex forms in the minds of humans; the transfer of such knowledge involves the complete range of human motivations including incentives, strategic behavior, perceptions of facts, perceptions of others, and so on. The disciplines represented by the SBE Directorate confront critical problems when testing their theories with human subjects under laboratory conditions. While they may differ in their topics of study, many of these problems are common across fields and have been handled through the use of networked PC's. When properly designed, experiments using lab networks facilitate randomization and subject assignment procedures, provide engaging interfaces and uniform procedural instructions to subjects, control information flows, coordinate the presentation of contingent behavioral options, manage data acquisition and storage, and even offer real-time analysis and presentation of results.

Thanks to the Internet, experimental networks no longer are bound by the four laboratory walls of the local network. The "NetLab" concept is intended to encompass both local and extended networks used for research purposes. However, problems of randomization and control endemic to laboratory research networks also will extend to non-local network research, demanding new solutions to some vexing problems. Encouraging proposals that address these issues should be an immediate priority for the NSF programs involved in this initiative.

Many forms of interaction among individuals, groups and organizations can be understood as knowledge networks. In economics, markets are mechanisms that gather information -- things like consumer preferences and factor costs -- and convert that information into knowledge -- such as a price. Political systems capture information about local needs and concerns from widely distributed individuals and groups, and integrate that information into political decisions. Social psychologists study the informal organizations of friends and acquaintances, and this too, constitutes a knowledge network. The "knowledge" distributed in knowledge networks also may take varying forms, from highly structured statistical information to tacit cultural understandings. All of these phenomena should be viewed as sources of insights about the ways in which disaggregated data are transformed into knowledge.

If indeed knowledge networks are so pervasive, then a basic scientific understanding of them should have far-reaching practical implications and unexpected social benefits. It may be possible to design policies that attack social problems by changing the nature of the network through which individuals become informed. For instance, research has shown that public health officials can introduce sanctioning systems and capitalize on the acquaintance networks of intravenous drug addicts to quash the spread of needle-borne diseases. Might it be possible to educate young people about AIDS, gangs and teenage pregnancies by recognizing them as specialized populations and structure the system to allow them to educate each other? Given the substantial cost of social problems such as teenage pregnancy, the benefits of the research loom much greater than any measure of the cost.
 

II. NetLab ACTIVITIES In the three sub-sections to follow, we will recount and describe activities that NetLab Workshop panelists identified in their discussions as likely to be enhanced or to be made possible with KDI support.

II.1 Opening New Methodological and Infrastructural Possibilities

In this section we summarize a number of the exciting new possibilities for NetLab research that were discussed at the Workshop. Importantly, although it is likely that every participant had in mind some version of a "dream facility," the activities and facilities discussed below were judged to be within reach in the near-term given adequate support from NSF.

The NetLab concept suggests at least three kinds of dramatic expansions for SBE research: increasing the sizes of groups of real-time interactants that may be studied; increasing the sizes of subject pools by experimenting via the World Wide Web (WWW); and capitalizing upon economies of scale by combining the efforts of multiple smaller laboratories to form "Superlabs." In addition to these, we also note two striking possibilities that enhanced NetLab facilities may offer: a completely new kind of methodology combining the control of experimental methods with the sampling power of survey research; and a radical enhancement of group-processes experimental research that would permit interactions between human subjects and computer-generated "simulants" using "virtual reality" (VR) methodologies.

Single-Site Large-Group Experiments. Currently the scientific infrastructure in place to study knowledge networks can produce data on small numbers of interacting individuals, but cannot yield extensive data on large-scale groups. At present much of our knowledge is limited to the interaction of fewer than 20 individuals. A fundamental change in the way we conduct our science -- constructing experiments that allow the participation of large numbers of subjects interacting together -- would allow the exploration of principles that govern groups. Theory occasionally demands experiments with large numbers of subjects. Many areas of experimentation in game theory and political science require large numbers of subjects, as do many market experiments. In sociology, the exchange networks thus far investigated have been limited to ten or fewer nodes. But there are theoretically interesting structures that occur only on a substantially larger scale. In political science one is frequently concerned with the effect of the number of voters in elections. Even in experiments that deal with small numbers of subjects, the matching protocols used to get multiple observations require that subjects be matched with the same other subject no more than once. At present, a combination of physical constraints on the size of the laboratories, and economic constraints -- the fact that subjects usually are paid -- effectively prohibit experiments with more than 25-30 subjects.

To break through these limitations, all that really is needed for each subject is a visual display and an input device. An affordable solution already exists that offers compact individualized display and input hardware: virtual reality headgear provides high quality audio, stereoscopic LCD imaging and a microphone. The system can be used in conjunction with a "VR mouse" or a "Cyberpuck" 3D mouse. In most experiments, subjects' communications are controlled in order to satisfy experimental design needs. Virtual reality headsets can provide this control, even for subjects seated adjacent to one another. An entire auditorium or lecture hall could be wired with network connections so that subjects would only need to plug in their own equipment. This opens the possibility of running experiments with subjects numbered in the multiple hundreds rather than in the tens now possible. NSF should consider funding at least one NetLab facility where experiments with large numbers of subjects can be run. This laboratory would have facilities for 100 or more subjects at the same room. This room should be set up so that it could be partitioned into smaller rooms for conducting smaller experiments.

Large-Scale Web-Mediated Experiments. In principle, any software that now controls the administration of experiments with human subjects in laboratory settings can be upgraded for use in larger-scale web-based experiments. This includes both designs in which subjects work in social isolation, and designs in which subjects interact with real or simulated others. The difference is that such experiments no longer would require the presence at a single site of all of its subjects, but rather Internet access. While such designs can be implemented "in principle" there remain enormous software hurdles. Many experiments in the social and behavioral sciences require complex visual representations, involve subjects engaged in activities that are sensitive to timing and require substantial redundancy in hardware to ensure that no part of the experiment goes "off line." As to this latter, the loss of a single node on a NetLab can mean that the entire experiment must be scrapped -- a very expensive proposition for a large experiment. Beyond the technical question of getting experiments to work on the Web there are important science design questions. Foremost are questions relating to ensuring controls over subjects, effectiveness of experimental manipulations and security of the data. These, and other problems detailed below, present challenges to the rich promise of Web-base laboratory experiment.

Collaboratories. The underdeveloped infrastructure in the SBE sciences erects a number of barriers to scientific progress. Specialists in any given topic are geographically disbursed. Very few locations house a critical masses of specialists with shared interests and experimental research agendas. Even under such conditions, however, NetLabs with relatively minimal hardware capacities and physical spaces may be coordinated such that their resources create virtual "Superlabs." With proper hardware and software development, a local laboratory may have just a few PCs in a room, or several VR headsets and input devices. The power of a Superlab resides not in the individual facilities, but in the coordination of multiple facilities sharing programmers, software, and subject pools. A Superlab staff, consisting of professional programmers, experimental scientists, research assistants and data specialists, may be distributed in various locations and even engaged in independent research. For purposes of NetLab research, however, they work together as though operating a single large facility.

Thus, we envision multiple NetLabs linked electronically to each other and to laboratories at an array of locations such that experiments can be run using subjects at each of the separate locations simultaneously. Such Superlabs even may connect laboratories across nations. These connections will help to solve the subject pool problem which is a part of international recruitment required for cross-cultural comparison studies.

This is the SBE disciplines' version of "big science," capable of sharply increasing the number of active experimentalists and, for each experimentalist, the number and magnitude of experiments. In addition to more large-scale experiments, the pooled resources of Superlabs also will facilitate medium-sized and small experiments. We would expect to see an across-the-board increase in rigorous, theory-driven experimental research in each of the SBE fields, with a concomitant burst of new discoveries and applications.

Workshop participants also favored a bottom-up approach to NetLab and Superlab development. Such organizational structures must be organized so as to represent the needs and interests of groups of researchers sharing interests in particular research programs, particular laboratory configurations, and/or particular substantive issues. At the same time, the distributed nature and bottom-up construction of these laboratories would help to avoid large investments in centralized infrastructures prone to obsolescence.

Purposively Sampled Experimental Designs. Some types of experiments benefit by increasing both the size and the density of the interacting human subjects. Other kinds of experiments benefit simply by increasing the sheer number of subjects, without a need to accommodate any increased volume of interaction among participants. For example, "factorial surveys" can gather responses to information administered in the form of descriptive vignettes, while varying systematically the levels of a number of variables contained within. This method has been used for, among other purposes, studying Americans' beliefs about the relationships between social/demographic characteristics and income scales. The number of unique vignettes is equal to the grand product of the number of levels across all variables. Factorial surveys with larger numbers of variables provide more information about the main and interaction effects of those variables, but also can generate large numbers of "conditions" that, in turn, demand a large number of respondents to ensure adequate statistical power. In the past, this method has required either that respondents be in the physical presence of administrators, or else the use of relatively costly mail surveys with much lower response rates. Capitalizing on web-based administration would tap into subject pools far more varied and sizable than those employed currently. The cooperation of only a relatively few universities, corporations, internet service providers or other organizations would suffice for any given factorial survey.

Aside from the development of new forms of survey-based research, Web-based technologies offer NetLab experiments the possibility of involving new samples of subjects. One domain that appears to be ripe for integrated experiment-survey methodologies is cross-cultural research. NetLabs have not only have the potential for high levels of control over administration of stimuli and assignment to experimental conditions, but they have the geographical reach and relative ease of administration normally associated with survey techniques. Because the world-wide-web extends across national boundaries it will become easier to include subjects from different countries in experiments. This can only broaden the generalizations that social scientists can make about their findings from the laboratory.

"Virtual Group Reality" Experiments. One final possibility for NetLab innovation is the "virtual group reality" experiment. Many experiments in today's social science laboratories control contextual factors by simulating interactants presumed by the subject to be real. Always, however, only minimal information is provided on the behavior and/or characteristics of the simulated others, and always in relatively sterile contexts. The reasons, of course, are to achieve control. The more "realism" that enters into the experimental context, the greater the opportunity for confounding effects. VR technology promises far more realistic simulations while still allowing the experimenter to maintain full control. The subject may interact with multiple "simulants" who appear as responsive, quasi-realistic 3-D video figures -- far richer than the limited information (e.g., a single numerical response) to which interaction partners oftentimes are reduced in present research. But in the same way as current research, simulants can provide responses that either actually do come from other human subjects, or are generated by a computer program. The key difference is that a more veridical "group reality" -- albeit a virtual group reality -- should facilitate the difficult process of moving from basic to applied scientific theory. Basic theories could be tested in more naturalistic settings without relinquishing experimental control.

II.2 Facilitating Conventional Research Methodologies and Infrastructures

Although much discussion at the Workshop centered on exciting new possibilities of NetLab research, it also was evident that various important lines of ongoing research stand to benefit as well. Additionally, an infusion of new resources would facilitate the development of infrastructural features of NetLab research such as discipline-wide information resources and software pools. Next, without delving into particular research programs, we indicate several activity areas that conform to the spirit of KDI and the emerging conception of NetLab research, and at the same time entail incremental steps beyond activities already underway.

Laboratory Development. Laboratory infrastructure is almost nonexistent among the SBE sciences including Economics, Political Science, Sociology, Social Psychology and Decision Sciences. A substantial proportion of universities in the U.S. do not house laboratories in most or all of these fields. Typically, when such laboratories do exist, they are outfitted more poorly than a bottom-rung high school laboratory in any of the traditional laboratory sciences. More laboratories of even quite modest sizes and budgets would have a synergistic effect on the experimental branches of each discipline. Not only would they increase the quantity of research output and the speed of theoretical development, but also they would create broader disciplinary appreciation for the generally unsung benefits that can accrue through the use of such methods.

A problem with current practice is that, within the structure of funding opportunities, each researcher has had to develop his/her own software. One result is that different researchers frequently reinvent wheels in setting up their experiments. It is not unusual for a three-year research project to expend two years on software development. Under pressures of limited time and resources, software ends up being tailored to problems at hand, becoming specialized and inflexible. We envision the possibility of experimenters choosing their subjects' interfaces, instruction formats, and response modalities from toolkits of existing options, with the availability of programmers who can make minor modifications if need be. Similarly for data manipulation, analysis, and archiving functions.

As it stands, more generalized code that might save months of work on other projects never gets written. One NetLab objective is to replace short term objectives with longer term objectives in the area of software development. The purpose is to develop more modularized software, capable of serving multiple purposes at low marginal costs.

Software Repositories. Even if software is developed in a more modular fashion, this does not ensure portability across researchers and laboratories. This is an additional objective that NetLabs should be able to accomplish by establishing open-use software repositories, along with the documentation required to implement program modules on different systems. Currently, limited access to these basic tools impedes potential growth and imperils knowledge by blocking replications.

In the few instances where funding has permitted the development of more generalized software tools, significant benefits have been realized. In Economics, for example, a handful of general-purpose programs produced by individual laboratories has served as the foundation for the development of new laboratories around the world. Currently in almost every major country there is an experimental economics laboratory built from some basic, shared elements. Though still far short of a presence in every university of every country, such an impact is not beyond reach.

Information Resources. In addition to the potential of serving their disciplines as repositories for computer software, some NetLabs already are capitalizing on the WWW by providing information resources. For example, we have begun to see the emergence of electronic journals. Current Research in Social Psychology at the University of Iowa publishes peer-reviewed articles on a regular basis. It continues to grow in stature and features publication lags of as little as one month from the date of (electronic) manuscript submission. The new journal Experimental Economics has similar aspirations. We also have seen scattered use of web sites for posting working papers and conference presentations, newsletters, discussion groups and collections of hyperlinks to web sites of related interest. Not far off, we would hope, is the use of web-sites not only for sharing software as discussed above, but also for the routine sharing of data sets and fororganizing entire data bases, thereby facilitating cumulative developments in different substantive research areas.

Human Resources. One of the reasons for the slow growth rate among the experimental social and behavioral sciences is that most of its laboratories can afford to fund very few if any graduate research assistants or postdoctoral fellows on any regular basis. In a direct and immediate way, this hinders the efficiency with which research is conducted. As noted above, there is also a growing need to employ computer programmers for more than just sporadic and ad hoc purposes. Researchers need to develop working relationships with programmers in order to optimize programs vis-à-vis researchers' needs, and to capitalize on the creativity and knowledge of both parties.As software and hardware capabilities continue to expand and evolve, there has emerged a need for the specialized skills of systems experts. While virtually all experimentalists are experienced end-users of mass-market computers and software products, few have the more advanced capabilities needed to install and maintain a system of laboratory computers. Most universities already have such specialists available through their computer centers, but at present they only offer very basic no-cost consultation services , or more elaborate but costly services on a contractual basis. Ideally -- especially in larger NetLabs -- there should be at least a part-time systems specialist, familiar with the particular configuration and needs of the NetLab through which s/he is employed.

II.3 Enhancing Educational Opportunities

An explicit goal of the NSF is the enhancement of science education. Arguably, theory-driven experimental research is the most rigorously scientific work within the SBE disciplines. Overall, however, the minority status of the experimental branches generally has made training in these areas spotty and unorganized. Below we discuss some of the educational opportunities that would be enhanced by the development of NetLabs.

Package NetLab Demonstrations. etLab materials will greatly enhance undergraduate and graduate instruction. Multimedia classrooms are becoming the norm in colleges and universities across the nation, equipped with personal computers capable of demonstrating and displaying the same programs actually employed in NetLabs, and displaying results of experiments -- even experiments involving the students themselves moments before. Unfortunately, these facilities are monopolized by other uses and they lack the necessary software and personnel to contribute much to experimental social and behavioral science

A key problem today for most instruction in the SBE sciences is that there is no capacity for instructional experiences in the laboratory. The requisite knowledge and instructional strategies exist but the laboratory support facilities do not. NetLabs promise to provide students with laboratory experiences in SBE sciences, just as they do in other laboratory sciences. Experimental designs built around NetLabs become real, not virtual laboratories. NetLabs would be capable of immersing students in rich simulations where the basic principles being taught are made transparent. In those locations fortunate to have some form of laboratory facility, the impact on teaching has been highly successful. Students learn from their experiences, enabling them to translate abstract theoretical concepts into operational terms, and back again. Such educational benefits have been demonstrated. At Caltech, University of Arizona, Indiana University, University of Virginia, and in nearly all business schools, students participate in experiments prior to covering the relevant material in class. They receive raw data from the experiment and are asked to develop a "model" as homework. Later the relevant theory and formal research are covered in class and applied to the same phenomenon. Not only do students learn the operational aspects of the theory, but also they learn about themselves and the limitations of their common-sense but ad hoc theorizing.

The NetLabs can make this wealth of experience widely available. Students around the world can participate in the same experiment and see the same data. Classes at different universities can participate together in, say, market experiments, where they see the operations of the law of supply and demand, the nature of interdependent markets, and the impact of market structure on economic performance. A good example is the teaching effort at the University of Iowa where the Iowa Political Stock Market began as a class project and since has become institutionalized as a powerful teaching tool for economics and political science.

Social Science Laboratory Training. Social science NetLabs will bridge between research and instruction. Today most social science instruction has no laboratory component whatsoever. This is a serious problem, especially in contrast to the recent development of virtual laboratories for physical science instruction. Introducing a laboratory component for social science instruction will be a major advance in science education. Students can conduct real experiments while learning the most advanced methods in current use.

The development of NetLab infrastructure would greatly reduce the marginal costs of experimental replications. This opens the possibility of students designing and conducting experiments that otherwise never would have been possible. Student replication of experimental results as part of a standard teaching program is within the realm of possibility.

While NetLabs should revolutionize the knowledge acquired by students, this impact on education in the many distinct disciplines represented by SBE will not be automatic. The full potential of NetLabs for science education, great as they are, will require an outreach program. Summer institutes housed at NetLabs and funded through them will need to reach teachers at all levels, from junior high schools to graduate schools. Funds will be needed to support the development of new courses at all levels.

Student Participation and Replication. he same NetLab facilities described earlier for auditorium-sized experiments would permit students to participate in classroom demonstrations that mirror actual experiments in every significant respect. Students could read about "classic" experiments, and then experience first-hand what it would have been like to be an actual subject. Experimental protocols could be played in "slow motion" for the students, their various components dissected and related to theoretical literatures covered in readings. This type of exposure is far more likely to capture the imaginations of students than just reading about the studies in journal articles. In the long-run, this would produce an infusion of new interest in the experimental branches of SBE sciences.

Undergraduate, Graduate and Postdoctoral Training. Most researchers have had the experience of having first piqued the interests of their graduate and undergraduate research assistants through describing some of their own research projects when pertinent to course sections. Naturally, the more dynamic and relevant that research appears in presentations, the more likely outstanding students will be motivated to take more specialized courses and directed reading credits and, ultimately, to serve as valued research assistants and even conduct their own projects. The more assistants available, the quicker the pace of research projects, and the more active the NetLab. Providing opportunities for postdoctoral training also would boost NetLab activities: Often more accessible than research faculty, post-docs play an important role in the laboratory training of graduate and undergraduate students. At the same time, the intensive research experience helps to launch the post-doc's own career.

Student Funding. Although occasionally students are willing to volunteer their time or to work for academic credit, most labs function well only when there is a degree of continuity provided by multi-year fellowships and assistantships for pre-doctoral students and post-docs. Along with the obvious infrastructural benefits of such continuity, it also provides a more protracted learning experience for students, and a fuller degree of professional socialization. We have seen great benefits when even a small group of students spends two or three years together on fellowships and/or assistantships. These included increased rates of collaboration and publications among students, smoother development of masters and dissertation projects, greater continuity of faculty research programs, and the production of new Ph.D.'s who are qualified to set up their own laboratories. In short, heightened levels of NetLab support will enhance the educational experiences and increase the numbers of those who will be designing and conducting future experiments in the social, behavioral and economic sciences.

III. CHALLENGES

Despite the promise of NetLabs in all their varied forms, there remain many important barriers to scientific progress, including:

• Absence of collaboratory mechanisms
• Absence of long-term support
• Absence of hardware/software for large-scale experiments

KN funding opportunities offer the potential to help break down those barriers.

III.1 Collaboratory Mechanisms

At present there are no examples of ongoing "collaboratory" mechanisms to enhance interactions among social scientists. Such mechanisms promise unprecedented new opportunities and will increase the integrity of knowledge acquisition. In the social sciences, replications of previous experiments are almost unknown. Oddly, as the quality of experimental work increases, there have been diminishing opportunities to check results by replication. Seeking greater experimental control, scattered laboratories have developed specialized software for local use only. The result is to restrict entry into the experimental arena, making social science research increasingly private. This is a serious problem that collaboratories will address. Once collaboratories are well established, norms will emerge to integrate new software for experiments. With software available on the web, replication will be a common exercise for graduate students and, when achieving professional standards, results will be publishable on the collaboratory's electronic journal.

Social science collaboratories for experimental research will be unique: They will be active knowledge-generating and knowledge-distributing networks. Knowledge networks in other sciences have more limited capabilities. Physical and biological collaboratories only allow remote use of research instruments. In contrast, social science collaboratories are themselves research instruments -- and more. For the knowledge networks of other sciences, special efforts are required to standardize and distribute scientific results. By contrast, knowledge distribution is an automatic byproduct of research in social science collaboratories. Data is automatically produced in standardized form. After an agreed-upon time period -- during which the scientist running the experiment has first use -- results automatically may be made public at the NetLab web site. Announcements that new data are available can be routinely distributed to interested scholars. These unique capabilities assure that social science collaboratories will have a very substantial impact on the social sciences.

The time for collaboratories for experimental research in the social sciences has come. It is encouraging to note that, with very limited funding, individual researchers already are struggling to develop collaboratories. We assert that larger group efforts will have substantially greater payoffs in knowledge development. There is now an opportunity to set the conditions which will speed the development of social science knowledge and revolutionize social science education for the foreseeable future. To do so will require a substantial infrastructure investment in collaboratories. The time has come for that investment to be made.

III.2 Long-term Support

There has been no tradition of providing long-term support to highly technical fields in the social sciences. As researchers make greater use of complex networked systems in their research, the need grows for technicians to conduct experiments. Like any large-scale laboratory in engineering or natural sciences, technical support is necessary. Traditionally this has not been the case in the social sciences (and only somewhat common in the behavioral sciences). In order to integrate current computational and networked tools into social science experimentation, technical support must be forthcoming.

III.3 Hardware/Software Support

As experiments are scaled up to incorporate many more subjects or as experiments are distributed across a number of sites, hardware and software innovations are needed. The needs of NetLab researchers are quite different from those of other engineers and scientists. As a consequence, hardware and software development is going to have to be directed towards those special needs, rather than relying on what has been developed for other sciences.

One of the barriers to current NetLab work is the relatively slow speed of the Internet. Experiments involving "real-time" interactions between hundreds of subjects, scattered across a variety of sites, are nearly impossible. Many of these experiments require that all subjects are brought up-to-date within 500 milliseconds of any action, and that many different actions may be taking place nearly simultaneously. If subjects are all tied to the same server, this is a relatively trivial problem. However, if subjects are widely distributed, then "real time" interaction becomes difficult. Moreover, server "crashes," backlogs, bottlenecks and other threats to subject connectivity must be addressed. These constitute fundamental challenges to our capacity to scale up experiments.

A second barrier concerns massive data storage, handling and retrieval for large-scale experiments. Many experiments require that linkable, heterogeneous data be transmitted from individual sites and merged together. However, there are enormous problems with linking data that may include behavioral actions, physiological measurement and visual images. Moreover, if such data are collected for each subject and the number of subjects is very large, then the resulting data set will be extremely large. Transmitting that data will be difficult. For instance, consider 100 subjects engaged in a 60-minute experiment in which information is collected on: the mouse location in 10 millisecond slices; all mouse clicks; physiological measures such as respiration, galvanic skin conductance; EEG measures; and the complete video of the individual's facial expressions throughout the experiment. Such data, digitally linked, will be extremely valuable, but their size alone will produce major difficulties for researchers.

III.4 Funding Commitments

Like many areas in the social sciences, funding levels for NetLab settings has been scattered. In turn this has encouraged scattered development of laboratory-based experimentation in the social sciences. Unlike many areas of social science research where data to test theoretical ideas is readily available from government documents, public reports or mass opinion polls, work surrounding NetLabs requires funding. These data are not free. Subjects are usually motivated by some form of financial reward and, as a consequence, experiments can be costly. Scaling up experiments can be very costly, but may be necessary if we are to answer questions that are fundamental to large-scale social groups.

More importantly, an infusion of directed funds will have an enormous impact on propelling this promising area of inquiry. Laboratory experiments are common in almost all of the engineering, biological and natural sciences. They are much less common in the social sciences. Nonetheless, in order to disentangle causal linkages in complex social phenomena, laboratory experimentation is necessary. NetLabs promise a fruitful linkage between new scientific tools and useful experimental methods for the social sciences.

IV. CONCLUSION

While aspects of knowledge networking (and KDI) have important implications for a large proportion of the sciences represented by the National Science Foundation, this breadth of potential relevance should not be restricted to the level of grants. At the level of scientific working groups and of the NetLabs themselves, there should be no funding constraints mandating multidisciplinary involvement. While multidisciplinary efforts certainly should be encouraged, by no means should they be required at the most basic level of NetLabs. Perhaps some sciences are so mature that the most impressive gains can only come from abandoning their central mission and redefining it in terms of the objectives of some other science. However, this is not the case with the SBE disciplines in which the full power of experimental scientific method is being applied for the first time. In the SBE disciplines related to knowledge networking, the synergy is not so much from the merging and clashing of multiple disciplines and the merging of knowledge that it generates; the synergies result from merging experimental methodologies with traditional methods. New sources of data and new views of theory provided by laboratory experimental methods are completely changing research directions and creating fresh ideas. The funding policies of NetLabs should recognize the special nature of the opportunities by supporting the mining and pursuing the veins of intellectual gold that are just being discovered in the independent SBE disciplines. Funding policies should not require that the digging start over in untested areas that lie in the expanse between the SBE disciplines and other sciences.

Web-based laboratories for experimental research are poised to conduct advanced experiments across all the social sciences including Economics, Sociology, Political Science, Social Psychology, and Decision Science. Central to the mission of NetLabs is the development of new software, hardware, measurement and design tools. By virtue of being Web-based, these new tools will be accessible to multiple users. In the physical and biological sciences, the web is limited to assisting joint use of laboratory facilities. By contrast, in the social sciences, the web site becomes the laboratory. Because they have capabilities beyond those possible elsewhere, social science NetLabs will have a massive impact on knowledge development and instruction.

---

Knowledge Networking NetLab Workshop

Oct. 30-31, 1997

Co-Chairs:

Charles R. Plott
Division of Humanities and Social Sciences
California Institute of Technology

David Willer
Department of Sociology
University of South Carolina

Participants:

Peter Arzberger
San Diego Supercomputer Center
University of California, San Diego

Phillip Bonacich
Department of Sociology
University of California, Los Angeles

Paul Brewer
Department of Economics
Georgia State University

Noshir Contractor
Annanberg School of Communication

Robert Forsythe
Dept. of Economics
University of Iowa

Chuck Huff
Psychology Department
St. Olaf College

James Hung
The Rockefeller Foundation

Eric Johnson
Marketing Department
The Wharton School

Alaina Kanfer
National Center for Supercomputing Applications
University of Illinois at Urbana Champaign

Michael W. Macy
Department of Sociology
Cornell University

David Mark
Department of Geography
State University of New York at Buffalo

Barry Markovsky
Department of Sociology
University of Iowa

Richard McKelvey
Division of Humanities and Social Sciences
California Institute of Technology

Reagen Moore
San Diego Supercomputer Center
University of California, San Diego

Mark Olson
Department of Economics
University of Arizona

Cecilia Ridgeway
Department of Sociology
Stanford University

Shyam Sunder
Graduate School of Industrial Administration
Carnegie-Mellon University

Henry Walker
Department of Sociology Cornell University

National Science Foundation:

William Bainbridge
Sociology Program

Cheryl Eavey
Methodology, Measurement and Statistics

Catherine Eckel
Economics Program

Rick K. Wilson
Political Science Program