Measuring Research

Approximately 90 million people will watch the Superbowl on Sunday. The New England Patriots will be playing for their fourth Championship in seven years. That’s amazing to some of us who remember the Patriots from the late 1960s and early 1970s. They were one of the worst teams in the League. Out of 142 games they played in ten years starting in 1967 they won less than a third (44). I still remember getting an ‘electronic football’ game as a gift during that time and being bitterly disappointed that one of the two teams I’d been randomly allotted was the Patriots. I quickly bought a replacement team.

Going from doormat to dynasty doesn’t happen by accident. I noted yesterday that EPA had achieved five-greens, the highest possible score, on our management scorecard. What I didn’t say is we started in 2001 with three reds and two yellows.

One of our persistent problems was coming up with good measures, particularly for our research programs. We simply could not find an acceptable way to measure the efficiency of our science programs.

Research is an inherently unpredictable endeavor with unpredictable consequences and outcomes. Albert Einstein once pointed out, “If we knew what we were doing, it wouldn’t be called research, would it?” Indeed, particularly with basic research, sometimes ‘failure’ is success. 3M’s slightly sticky ‘Post-It’ notes came out of a failed effort to come up with a super strong adhesive. Unfortunately, our failure to identify good efficiency measures for research was keeping us from getting high scores on the management scorecard.

Over two years ago we made a commitment to go from mediocre to first in measuring research. We spent a lot of time looking at what other agencies were doing and came up with the best efficiency measures we could. We then submitted those measures to an expert panel at the National Academies of Science and, along with the Office of Management and Budget, asked them, “What do you think? Is this the best we can do, that anyone can do, in measuring research?” They issued their report late yesterday.

There are four major recommendations in the report, but here is my favorite finding:

Among the metrics proposed [by EPA] to measure process efficiency, several can be recommended for wider use by agencies.

In my view, EPA has now become a leader. Many of our research efficiency measures are a model for others to use. It’s easy to measure success on the football field. EPA is now at the cutting edge of being able to measure it in the laboratory.

By the way, based on my research, Patriots 27, Giants 17.

Comments

[Jillian Fischer]

I tuned in to this page to see what EPA is up to...what does Amy Winehouse or football games have to do with the Environmental Protection Agency?? Sorry, but I find these types of blog topics to be inappropriate for the Web site of a government agency.

[Dick Sumpter]

You got the Giants right.

[Marcus]

Jillian, thanks for coming here in the first place. However, this blog is not an EPA-wide blog (as opposed to what the Department of State does, for instance). It's a blog by the Chief Operating Officer and deals with the work I do. I typically avoid repeating information one can find elsewhere on EPA's website. If you want to see what EPA is generally up to, I encourage you to go to the EPA home page: http://www.epa.gov/

[Kate]

It's laudable that you've designed metrics to track the efficiency of the EPA's research. It would be even better if Administrator Johnson would actually heed the advice of EPA's scientists and analysts on global warming, air and water pollution. Conducting the research efficiently is a great first step; now the EPA needs to take action to protect the environment and our health.

[David Salih]

Thanks you for having this blog- I think it's an awesome way to be in conversation with the public interactively. Government politics too often occur as something 'over there', as opposed to something that directly reflects my interests. Kudos on being a leader in this respect.
Reguarding your current blog: It's great that the EPA is developing and expanding it's efficiency and accuracy in its research. It's equally important that we apply the findings made and use them to make better policies. We've seen blatant dismissal of the scientists and analysts in the past, and, perhaps even more disturbingly, quite recently. I would love to read your input reguarding your commitment to actively changing this trend. Of course, addressing this issue would mean beginning with the negligence we've seen of Administator Johnson. We must take advantage of the support of an educated public that finally supports legislation to protect our environment- we certainly have no time to waste.
Thanks you for taking the time to read this. It makes a difference for me and for those who have concerns and thoughts similar to mine.

[Nick jett]

I agree i am dissapointed that i also have reached this blog, hoping to have a disscussion about EPA. And i get football!