Tag Archives: Impact

Measuring the Impacts of Science | AAAS Forum on Science and Technology Policy

Posted on by

I’m looking forward to moderating a panel on day 1 of the AAAS Forum on Science and Technology Policy.

2:00 Current Issues in S&T Policy (Breakout Sessions) 
 
(A) Measuring the Impacts of Science   
• What are the policy relevant challenges, tools, and approaches to measuring the social impact of scientific research? • How can improved indicators capture change in science, technology, and innovation? • Are altmetrics the solution to measuring social impacts? 
  
Moderator: J. Britt Holbrook, Visiting Assistant Professor, School of Public Policy, Georgia Institute of Technology; and Member, AAAS Committee on Scientific Freedom and Responsibility
 
Kaye Husbands Fealing, Professor, Center for Science, Technology and Environmental Policy, Humphrey School of Public Affairs, University of Minnesota; Senior Study Director, National Academy of Sciences, Committee on National Statistics; and Member, AAAS Committee on Science, Engineering, and Public Policy
 
Gil Omenn, Director, Center for Computational Medicine and Bioinformatics, University of Michigan
 
Mike Taylor, Research Specialist, Elsevier Labs

 

Blue skies, impacts, and peer review | RT. A Journal on Research Policy and Evaluation

Posted on by

This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.

Blue skies, impacts, and peer review | Holbrook | RT. A Journal on Research Policy and Evaluation.

On being tweeted to the top

Posted on by

Our results show that scientists who interacted more frequently with journalists had higher h-indices, as did scientists whose work was mentioned on Twitter. Interestingly, however, our data also showed an amplification effect. Furthermore, interactions with journalists had a significantly higher impact on h-index for those scientists who were also mentioned on the micro-blogging platform than for those who were not, suggesting that social media can further amplify the impact of more traditional outlets.

via Opinion: Tweeting to the Top | The Scientist Magazine®.

I’m surprised there’s no mention of altmetrics in this piece. Results are relevant for altmetrics, though. Hoping they’ve published the study in more detail somewhere so I can check it out.

In any case, this is a quick and thought-provoking read — though it appears that it’s more about being tweeted to the top than tweeting to the top (two are interestingly related, however, I suspect).

Other infrequently asked questions about impact

Posted on by

Here are some other infrequently asked questions about impact that didn’t make it into the final cut of my piece at the LSE Impact of Social Sciences Blog.

Why conflate impact with benefit?

Put differently, why assume that all impacts are positive or benefits to society? Obviously, no one wants publicly supported research not to benefit the public. It’s even less palatable to consider that some publicly supported research may actually harm the public. But it’s wishful thinking to assume that all impacts are beneficial. Some impacts that initially appear beneficial may have negative consequences. And seemingly negative indicators might actually show that one is having an impact – even a positive one. I discuss this point with reference to Jeffrey Beall, recently threatened with a $1 billion lawsuit, here.

The question of impact is an opportunity to discuss such issues, rather than retreating into the shelter of imagined value-neutrality or objectivity. It was to spark this discussion that we generated a CSID-specific list – it is purposely idiosyncratic.

How can we maximize our impact?

I grant that ‘How can we maximize our impact?’ is a logistical question; but it incorporates a healthy dose of logos. Asking how to maximize our impacts should appeal to academics. We may be choosey about the sort of impact we desire and on whom; but no one wants to have minimal impact. We all desire to have as much impact as possible. Or, if we don’t, please get another job and let some of us who do want to make a difference have yours.

Wherefore impact?

For what reason are we concerned with the impact of scholarly communication? It’s the most fundamental question we should be asking and answering. We need to be mindful that whatever metrics we devise will have a steering effect on the course of scholarly communications. If we are going to steer scholarly communications, then we should discuss where we plan to go – and where others might steer us.

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that | Impact of Social Sciences

Posted on by

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that | Impact of Social Sciences.

In which I expand on ideas presented here and here.

Nigel Warburton’s negative vision of what philosophy isn’t

Posted on by

Philosopher Nigel Warburton, of philosophy bites fame, has just resigned his academic post at the Open University to pursue other opportunities. The Philosopher’s Magazine conducts an extended interview with Warburton here. Much of what he reveals in this interview is both entertaining and, in my opinion, true.

But one aspect of the interview especially caught my attention. After offering several criticisms of academic philosophy today with which I’m in total agreement (in particular the tendency of hiring committees to hire clones of themselves rather than enhancing the diversity of the department), Warburton offers what he seems to view as the ultimate take down of academic philosophy. I quote this section in full, below. If you’ve been paying any attention to this blog or our posts at CSID, you’ll understand why, immediately.

He reserves particular venom for the REF, the Research Excellence Framework, a system of expert review which assesses research undertaken in UK higher education, which is then used to allocate future rounds of funding. A lot of it turns on the importance of research having a social, economic or cultural impact. It’s not exactly the sort of thing that philosophical reflection on, say, the nature of being qua being is likely to have. He leans into my recorder to make sure I get every word:

“One of the most disturbing things about academic philosophy today is the way that so many supposed gadflies and rebels in philosophy have just rolled over in the face of the REF – particularly by going along with the idea of measuring and quantifying impact,” he says, making inverted commas with his fingers, “a technical notion which was constructed for completely different disciplines. I’m not even sure what research means in philosophy. Philosophers are struggling to find ways of describing what they do as having impact as defined by people who don’t seem to appreciate what sort of things they do. This is absurd. Why are you wasting your time? Why aren’t you standing up and saying philosophy’s not like that? To think that funding in higher education in philosophy is going to be determined partly by people’s creative writing about how they have impact with their work. Just by entering into this you’ve compromised yourself as a philosopher. It’s not the kind of thing that Socrates did or that Hume did or that John Locke did. Locke may have had patrons, but he seemed to write what he thought rather than kowtowing to forces which are pushing on to us a certain vision, a certain view of what philosophical activities should be. Why are you doing this? I’m getting out. For those of you left in, how can you call yourselves philosophers? This isn’t what philosophy’s about.”

Please tell us how you really feel, Dr. Warburton.

In the US, we are not subject to the REF. But we are subject to many, many managerial requirements, including, if we seek grant funding, the requirement that we account for the impact of our research. We are, of course, ‘free’ to opt out of this sort of requirement simply by not seeking grant funding. Universities in the UK, however, are not ‘free’ to opt out of the REF. So, are the only choices open to ‘real’ philosophers worthy of the name resistance or removing oneself from the university, as Warburton has chosen?

I think not. My colleagues and I recently published an article in which we present a positive vision of academic philosophy today. A key aspect of our position is that the question of impact is itself a philosophical, not merely a technical, problem. Philosophers, in particular, should own impact rather than allowing impact to be imposed on us by outside authorities. The question of impact is a case study in whether the sort of account of freedom as non-domination offered by Pettit can be instantiated in a policy context, in addition to posited in political philosophy.

Being able to see impact as a philosophical question rests on being able to question the idea that the only sort of freedom worth having is freedom from interference. If philosophy matters to more than isolated individuals — even if connected by social media — then we have to realize that any philosophically rich conception of liberty must also include responsibility to others. Our notion of autonomy need not be reduced to the sort of non-interference that can only be guaranteed by separation (of the university from society, as Humboldt advocated, or of the philosopher from the university, as Warburton now suggests). Autonomy must be linked to accountability — and we philosophers should be able to tackle this problem without being called out as non-philosophers by someone who has chosen to opt out of this struggle.

Quick thoughts on Challenges of Measuring Social Impact Using Altmetrics

Posted on by

As altmetric data can detect non-scholarly, non-traditional modes of research consumption, it seems likely that parties interested in social impact assessment via social reach may well start to develop altmetric-based analyses, to complement the existing approaches of case histories, and bibliometric analysis of citations within patent claims and published guidelines.

This and other claims worth discussing appear in this hot-off-the-presses (do we need another metaphor now?) article from Mike Taylor (@herrison):

The Challenges of Measuring Social Impact Using Altmetrics – Research Trends.

In response to the quote above, my own proposal would be to incorporate altmetrics into an overall narrative of impact. In other words, rather than have something like a ‘separate’ altmetric report, I’d rather have a way of appealing to altmetrics as one form of empirical evidence to back up claims of impact.

Although it is tempting to equate social reach (i.e., getting research into the hands of the public), it is not the same as measuring social impact. At the moment, altmetrics provides us with a way of detecting when research is being passed on down the information chains – to be specific, altmetrics detects sharing, or propagation events. However, even though altmetrics offers us a much wider view of how scholarly research is being accessed and discussed than bibliometrics, at the moment the discipline lacks an approach towards understanding the wider context necessary to understand both the social reach and impact of scholarly work.

Good point about the difference between ‘social reach’ and ‘social impact’. My suggestion for developing an approach to understanding the link between social reach and social impact would be something like this: social reach provides evidence of a sort of interaction. What’s needed to demonstrate social impact, however, is evidence of behavior change. Even if one cannot establish a direct causal relation between sharing and behavior change, demonstrating that one’s research ‘reached’ someone who then changed her behavior in ways consistent with what one’s paper says would generate a plausible narrative of impact.

 

Although altmetrics has the potential to be a valuable element in calculating social reach – with the hope this would provide insights into understanding social impact – there are a number of essential steps that are necessary to place this work on the same standing as bibliometrics and other forms of assessment.

My response to this may be predictable, but here goes anyway. I am all for improving the technology. Using Natural Language Processing, as Taylor suggests a bit later, sounds promising. But I think there’s a fundamental problem with comparing altmetrics to bibliometrics and trying to bring the former up to the standards of rigor of the latter. The problem is that this view privileges technology and technical rigor over judgment. Look, let’s make altmetrics as rigorous as we can. But please, let’s not make the mistake of thinking we’ve got the question of impact resolved once altmetrics have achieved the same sort of methodological rigor as bibliometrics! The question of impact can be answered better with help from technology. But to assume that technology can answer the question on its own (as if it existed independently of human beings, or we from it), is to fall into the trap of the technological fix.