Prepared Remarks by Raymond G. Kammer
Director, National Institute of Standards and Technology

before the

Symposium on the Advanced Technology Program: Challenges and Opportunities
Board on Science, Technology, and Economic Policy
National Research Council

March 29, 1999

We measure things at NIST.

We have a favorite quotation -- it shows up sporadically on bulletin boards and in footnotes -- attributed to William, Lord Kelvin. He said that -- I will paraphrase freely -- when you can measure something and put some numbers to it, then you know something about it, and if you can?t your understanding of it is of a "meager and unsatisfactory kind."

Since 1901 that statement or something very like it has been a touchstone for us. We are dissatisfied with things we cannot measure, and we work hard to remedy those unhappy situations.

Obviously, it is the philosophical basis for the NIST Measurement and Standards Laboratories, the research core of our agency. But it plays a strong role as well in the activities of our Manufacturing Extension Partnership and the Baldrige National Quality Program.

So of course when we received the assignment to create and manage the Advanced Technology Program, one of the first thoughts was, "Fine. How do we measure it?"

• The ATP presents a unique measurement challenge, not just for NIST but for government, industry and the economics community.
  
  
• We know that technological innovation, by and large, is a good thing.
• Advanced technology has sustained our leadership in the global community.
• It accounts for over 50 percent of U.S. economic growth since the Second World War.
  
  
• The ATP was established by Congress to help spur this economic growth.
• They created a mechanism to encourage and nourish economically valuable technologies that might otherwise go untried -- or exploited by our global competitors -- because the R&D risks were too high to be supported entirely by the private sector.
• The ATP practices a sort of economic jujitsu, applying relatively small, highly leveraged investments against the potential for major economic impacts.
• Can it work? Well, sure. The classic existence proof is the Internet -- the relatively modest early investments by the Defense Department and universities utterly dwarfed by today's economic impact.
  
  
• But how do you measure the worth of technological innovation?
• How do measure its impact on the economy of a nation?
• How do you predict that impact five years out? Ten years out?
• Ten years ago few people would have been sufficiently insane to predict the current role the Internet plays in the national economy.
  
  
• The ATP measurement challenge is this:
• The ATP bottom line is long-range, broad-based economic benefits
• Long-range means sometime out in the future. Though we ultimately will do retrospective analyses, we can't wait until then.
• Broad-based economic benefits means tracking and measuring impacts that go well beyond the individual company or companies that did the original work.
• We need to measure or credibly estimate the macroeconomic effects of a fundamentally microeconomic program.
• We need to make credible predictions based on those measurements and estimates.
  
  
• How do we assess our work?
  
  
• Measurement and evaluation has been part of the ATP from the beginning.
• First major report appeared in 1993, a survey examining the early impacts on participating companies of  the first 11 ATP projects after a year of work.
• The ATP established its  Economic Assessment Office to oversee wide range of data gathering and analysis efforts including basic data collection, case studies, surveys, and macroeconomic modeling
• The ATP's business reporting system, for example uses an innovative electronic survey form to gather quarterly information on active projects and follow-up data after completion, tracking their progress toward achieving business and economic goals.
  
  
• Our research universe, if you will, encompasses the 431 projects selected by the ATP for funding to date. We can tell a lot about the ATP just from the project profiles:
• We know that industry commits significant resources to the ATP, that it?s a true partnership. The ATP committed to $1.386 billion in funding for the projects selected to date, less than half the projected $2.783 billion budget for the planned research.
• We know that diversified joint ventures play an important role in the ATP. These 431 projects involve more than 1,000 participating organizations, including more than 125 universities and nearly 20 national laboratories.
• We know that small business competes effectively in the ATP?s merit-based selection process.  More than half these projects are led by small businesses.
  
  
• Surveys and the business reporting system tell us more about the ATP?s near-term impact on participants:
• The ATP is stimulating collaboration, encouraging more and more research joint ventures.
• It is accelerating the development of high-risk technologies. Eighty-six percent of project participants in one study said they were ahead in their R&D cycle after just one to two years of ATP funding. Thirty-nine percent of those were ahead because they wouldn?t have undertaken the project at all without the ATP.
• The ATP is fostering the development of leap-frog technologies that go well beyond simple incremental advances in the state of the art. Thirty-seven percent of ATP-developed technologies in a recent study were "new to the world" innovations.
• These technologies have broad potential applications. Companies report identifying an average of 4.5 applications per project.
• And they have already committed their own resources to commercialize more than 100 new products, processes or services based on ATP technologies.
  
  
• And that's just the beginning. The ATP is probably the most thoroughly studied program of its kind in the world.
  
  
• My current stack of reports, studies, analyses, and analyses of analyses on the ATP is about 9- and-a-half centimeters deep, or about a centimeter or so of report for every year of the program's existence. I'm not sure the collection's complete.
  
  
• In addition to near-term results, we have applied macroeconomic models in an attempt to look at potential long-range impacts.
  
  
• In 1996 CONSAD Research Corporation used a 53-sector macroeconomic model developed by  REMI [Regional Economic Models, Inc.] to estimate potential long-range impacts of a single ATP project.
• This was a joint venture under the Auto Body Consortium, a coalition of small and medium-sized auto industry suppliers working with Chrysler and GM as well as the University of Michigan and Wayne State University..
• The project developed a suite of process monitoring and control technologies to reduce dimensional variations -- fit and finish problems -- in auto bodies on the assembly line.
• These technologies led directly to:
• lower production costs
• lower product maintenance costs
• improved product quality, and
• reduced cycle time to launch new models
• Based on their modeling, CONSAD estimated the total impact of this project on the U.S. economy ? counting in all the industrial sectors affected by the auto industry ? at more than $3 billion in the year 2,000.
  
  
• Last year the Research Triangle Institute Center for Economics Research completed a study that was intended to develop a prototype methodology for predicting the long-term social benefits of public investment in technology development.
• RTI analyzed probable partial outcomes for seven ATP projects in the field of tissue engineering.
• These were technologies, for example,
• to make "template" prostheses derived from biomaterials to assist the body in rebuilding lost tissue, or
• to produce implantable cells that generate key human hormones and other bioactive agents, such as insulin for diabetics.
• RTI used a procedure to calculate the expected social return on public investment?the extent to which the nation is better off as a result of the ATP investment.
• This includes not only the private return to the innovating company, but also the value of the technology to the nation at large--lower health-care costs, for example, or  improvements in medical outcomes and quality-of-life for the patients.
• The ATP expects to invest a total of $15.5 million in the seven projects covered in this study.
• RTI calculated the expected social return on the ATP's investment in these technologies at $34 billion in net present value over a 20-year period.
• That is a predicted social rate-of-return on the ATP investment of 115 percent annually over 20 years. This is not a bad return.
  
  
• You might be interested to know that in both cases we urged the researchers to use conservative assumptions. Dedicated optimists could come up with even higher figures. I have asked the ATP to be careful about touting these results because they are so striking. 
  
  
• I cite these results not because I necessarily expect you to accept them at face value. Rather I want to emphasize the magnitude, and the difficulty, of the task we face. 
  
  
• There are major issues involved in the use of macroeconomic models on a program the size of the ATP. But we need to ask these predictive questions. These reports represent the sort of analyses we need to have. 
  
  
• Today we are adding a new report to the stack, this summary study by economist William F. Long of the first set of completed ATP projects. 
  
  
• This is the first comprehensive review of the outcomes to date of the ATP portfolio, the first of a series of studies following up on the results of ATP investments. 
  
  
• Dr. Long's report covers the 38 projects which were completed as of the end of March 1997 documenting research accomplishments, subsequent work by the companies to commercialize the results, and near-term outlooks for the successful technologies. 
  
  
• It also looks at 12 projects which were terminated or canceled during that same period (out of 280 projects selected). This is a respectable rate for a program which concentrates on high-risk research, and we?ve learned some useful things from the failures. 
  
  
• It is, as Commerce Secretary Daley said in his announcement, "a portrait of a program that works."
  
  
• We cover the high-technology terrain pretty thoroughly:
• There is a process that merges tissue-engineering and textile-weaving to help regenerate lost or damaged tissue in the body.
• There is an application of high-temperature superconductors to improve cellular phone service.
• There is a desktop bioreactor capable of growing large amounts of human stem cells isolated from bone marrow for cell replacement therapy -- now in clinical trials.
• There is a computer programming tool to simplify the task of writing software for parallel-processing computers -- already in commercial use.
• There is a novel technology for processing very large semiconductor wafers -- you'll probably hear more about it today from Manny Sieradzki one of the innovators -- that made the U.S. the first in the market with processing equipment for the next generation of 300-mm semiconductor wafers.
• These are technologies that the nation has, here and now, because of the ATP. 
  
  
• As the ATP enters its tenth year, we face some significant challenges. 
  
  
• The ATP, as Secretary Daley says, is working. We are, without a doubt, having an impact. Dr. Long's report is only the most recent testament to that. 
  
  
• We need to grow the ATP to a level consistent with the program's fundamental mission of economic growth for the nation.
• Just what that level should be is a matter for debate, but for an object example, I would point to the only other major technology funding program with demonstrable success and national impact.
• That is the DoD Advanced Research Projects Agency, with a budget in the neighborhood of $2 billion. 
  
  
• To support this, we need the data. The challenge is to document how the ATP impacts the economy, and by how much? The challenge is to do so in a rigorous and credible way that will satisfy the requirements of the public policy makers in Congress and the Administration. 
  
  
• Our goal in this workshop is to examine those measurement and assessment issues. How can we best assess the ATP? What data do we need to do that? How can we get it? 
  
  
• I'll leave you to it. We appreciate your help and your insights. Thank you.