Contents

---

Executive Summary

Introduction

Setting the Stage for Innovation Metrics Development

Advancing Measures of Innovation

Conclusion

Suggested Citation, Acknowledgments

Setting the Stage for Innovation Metrics Development

Research Challenges

Innovation research seeks to understand the sources, mechanisms, and effects of innovation and technological change and to measure its inputs (people and the training they receive, physical and financial resources, and how they change over time). It is also important to understand the intermediate products of the process of technological innovation, such as knowledge spillovers and research tools. Outputs (e.g., scientific papers that directly result from projects or programs) and outcomes (broader social impacts, such as improved productivity, income, and well-being) are also important to understand. From an economic viewpoint, spillover—social returns to innovation in excess of the private returns—may be the most important research topic because, in many cases, it is the basis for public policy.

Part of the workshop discussion centered on improved understanding of what is happening at different stages of the innovation process. It was generally agreed that the innovation process can be characterized as complex and nonlinear. In this context, there remain identifiable players and activities that need intense study and data development on such topics as activities of firms working at different stages of innovation and links between firms and universities at different levels of aggregation. Many research and policy questions require data at less-aggregated levels, such as the industry or firm level, than do broad, national-level indicators. Several analysts expressed an interest in gaining insight into public-private partnerships.

Scholars who study different parts of the innovation system have different data needs. There is little consensus on what to measure at each level and each stage of innovation, how to measure it, how often to measure it, or what would be an appropriate point of comparison. There was discussion about the definition of innovation, ranging from a comment that innovation occurs when a new product is first sold, to considering innovation as a new way of doing something even if it does not put a new product on the shelf. For example, improvements in services and designs can be innovative and enhance productivity and should also be considered innovation. This is part of the set of questions that participants thought should be addressed by NSF-supported research—considering what constitutes innovation.

There was a sense at the workshop that innovation research is constrained by data limitations. The problem may not be only the lack of data, but also the underutilization of data, the lack of connection between the data available and the problems to be solved, and the difficulty of linking existing datasets. For example, if it is desirable to link R&D surveys to data or surveys on innovation and diffusion so that researchers can follow product development through the stages of R&D and innovation, these surveys and databases should be designed with potential links in mind.

Activities that are not traditionally thought of as part of the innovation system may actually contribute a great deal to the innovation process. Technical services, industrial designs, quality assessment, and training are all part of the innovation process, but these activities are not fully integrated into existing theory or models, and little, if any, data are available for them.

Companies have very different ways of thinking about metrics and defining innovation. Because firms are at the forefront of innovation, researchers would benefit from working more closely with firms to determine how they define innovation, research, and productivity. Can innovation metrics used by industry be scaled up for use as national-level indicators, or do they suggest sector-level indicators? Conversely, what kind of innovation metrics do firms need from government? What might be an optimal mix of government and industry roles in the development of data on innovation?

Research questions raised in workshop discussions included the following:

 Back to Top

Innovation Theory Needs

Although the workshop discussion focused primarily on data needs, there was also recognition of the need for new or improved models, theories, or conceptual frameworks. Theories are needed to interpret the data; theories can also suggest the kinds of data that need to be collected.

Marburger has challenged the science policy research community to generate new and better indicators and models in support of a new science of science policy. Some workshop participants discussed the relative merits of multiple models addressing different sectors or policy objectives.

There was discussion about the development of a general equilibrium model based on existing endogenous models of economic growth. Such a model would sew together disparate evidence and would allow policy experiments. The model would be tightly linked to evidence from microeconometric studies on R&D and technological change. The aim would be to accumulate the wisdom from an ever-expanding set of empirical studies into a unified whole. The model would speak to firm-level evidence, but would also aggregate up to the economy-wide level, with an international dimension. One could use it to give advice on formulating R&D policy. When an input is changed, the model would show what would happen. The model should embody international trade and technology diffusion. As more and better data are assembled, the agenda of developing a general equilibrium model seems more realistic.

Developing a set of logic models for different innovation mechanisms (such as partnerships or grants), like the logic modeling done for program evaluations, was another approach discussed. Logic models could reveal the key research questions and what is known relevant to those questions for the various mechanisms. This approach might make research results more understandable and useful to policy makers.

It was pointed out that industrial R&D firms have developed the Technology Value Pyramid, a conceptual framework that links innovation investments with important outcomes, such as increases in shareholder value, in a way that managers can influence. It was suggested that something like this might be developed at the federal level. It was also observed that the broad range of social, economic, behavioral, and cognitive sciences collectively can inform the development of innovation models and suggest what kinds of data should be collected. For example if researchers can understand the cognitive dimensions of how scientists come up with innovations, it may be possible to fit that into innovation models. One participant noted specifically that "a healthy dose of interaction of data and models—that is, a mix of inductive and deductive approaches is very useful."

 Back to Top

Examples of Policy Issues to be Addressed by Research

Ultimately, it is the policy-making community that will define the issues that the science of science policy should address. That said, on the basis of current experience, workshop participants mentioned a number of issues that have served as a backdrop to their research on innovation. They include the following:

Other key policy-research areas were mentioned. These include studying/measuring the difference between manufacturing and nonmanufacturing in terms of employment and their R&D intensity and determining whether U.S. and overseas innovation are substitutes or complements.

 Back to Top

Innovation Data, Analysis, and Policy

The issue of how innovation data and analysis relate to policy was discussed. Some workshop participants expressed concern that there is, at best, a very weak connection between innovation studies and analyses and the questions that policy-makers concerned with innovation confront. Although research has steadily increased our knowledge about innovation, what has been learned is, by and large, not informing policy. For example, for some participants the relationship between relevant research findings and recent policy proposals in the areas of research joint ventures and science parks was unclear. In the absence of systematic research findings, public policy tends to rest on "common wisdom." However, as pointed out more than once at the workshop, research on innovation often shows common wisdom to be wrong.

It was noted that the challenge is to make innovation research inform policy better, while maintaining its independence and objectivity. Researchers can fruitfully focus on specific questions of interest to policy makers, but the research community should articulate clearly which questions can be answered in the near term and which cannot. The research community should lay out a realistic set of expectations for the science of science policy.