I recently undertook a sojourn with CSID director Robert Frodeman and asst. director Britt Holbrook across the Ditch (my first ever!) for a workshop at Brunel University just outside of London. Our hosts, Claire Donovan and HERG (the Health Economics Research Group at Brunel), invited a mix of academics, policy makers, and public figures to discuss the state of the art of assessing research impact.
While commenting on a presentation by a member of RAND Europe, John Brewer, sociology professor and president of the British Sociological Association, raised five key questions about impact after boldly asserting that it was “a sheep in wolf’s clothing”:
1) What is impact?
2) Impact for whom?
3) In what areas is impact displayed?
4) What indicators are there of impact?
5) How is it to be measured?
All of these questions, no doubt, remain contested and are important topics for further investigation, but it is the final one that interests me most. His wording would seem to imply that measuring impact cannot be separated from ethical considerations in doing so – what does the ideal measure of impact look like? In other words, there is a right way and a wrong way to measure impact, both in the methodological and moral sense.
This has always been my sense of metrics – that the ethical concerns in using metrics as such are important. And I have found that this additional facet often gets shoved to the side in favor of bickering over the methodology. But simply refining an unfair measure does not ameliorate the aspects that make it unfair; it simply makes the measure more statistically sturdy.
The simple fact is that, as political tools that embody value judgments, metrics often exact undesirable and politically-charged consequences. But what can be done about this? I have attempted to begin formulating what I describe as a code of conduct for metrics. For now, this will be limited to the appropriate use for bibliometric methods for the purposes of assessing academic research or academic careers. I think it is long overdue that ethical considerations of best practices be infused into the assessment/accountability debate.
But one question will, I think, plague me throughout the project: can one exact a positive cultural shift via external influence? In other words, can I influence a change in the culture of metrics use in academia by those who develop the metrics (scientometricians) despite that I am not a part of that specific culture? Will the code of conduct flop because it didn’t come from inside that discipline itself?