As altmetric data can detect non-scholarly, non-traditional modes of research consumption, it seems likely that parties interested in social impact assessment via social reach may well start to develop altmetric-based analyses, to complement the existing approaches of case histories, and bibliometric analysis of citations within patent claims and published guidelines.
This and other claims worth discussing appear in this hot-off-the-presses (do we need another metaphor now?) article from Mike Taylor (@herrison):
The Challenges of Measuring Social Impact Using Altmetrics – Research Trends.
In response to the quote above, my own proposal would be to incorporate altmetrics into an overall narrative of impact. In other words, rather than have something like a ‘separate’ altmetric report, I’d rather have a way of appealing to altmetrics as one form of empirical evidence to back up claims of impact.
Although it is tempting to equate social reach (i.e., getting research into the hands of the public), it is not the same as measuring social impact. At the moment, altmetrics provides us with a way of detecting when research is being passed on down the information chains – to be specific, altmetrics detects sharing, or propagation events. However, even though altmetrics offers us a much wider view of how scholarly research is being accessed and discussed than bibliometrics, at the moment the discipline lacks an approach towards understanding the wider context necessary to understand both the social reach and impact of scholarly work.
Good point about the difference between ‘social reach’ and ‘social impact’. My suggestion for developing an approach to understanding the link between social reach and social impact would be something like this: social reach provides evidence of a sort of interaction. What’s needed to demonstrate social impact, however, is evidence of behavior change. Even if one cannot establish a direct causal relation between sharing and behavior change, demonstrating that one’s research ‘reached’ someone who then changed her behavior in ways consistent with what one’s paper says would generate a plausible narrative of impact.
Although altmetrics has the potential to be a valuable element in calculating social reach – with the hope this would provide insights into understanding social impact – there are a number of essential steps that are necessary to place this work on the same standing as bibliometrics and other forms of assessment.
My response to this may be predictable, but here goes anyway. I am all for improving the technology. Using Natural Language Processing, as Taylor suggests a bit later, sounds promising. But I think there’s a fundamental problem with comparing altmetrics to bibliometrics and trying to bring the former up to the standards of rigor of the latter. The problem is that this view privileges technology and technical rigor over judgment. Look, let’s make altmetrics as rigorous as we can. But please, let’s not make the mistake of thinking we’ve got the question of impact resolved once altmetrics have achieved the same sort of methodological rigor as bibliometrics! The question of impact can be answered better with help from technology. But to assume that technology can answer the question on its own (as if it existed independently of human beings, or we from it), is to fall into the trap of the technological fix.
Thanks for the quick response, I really appreciate the analysis.
I have been refining my thoughts regarding the measurement of social reach (as a potential predictor of social impact) using altmetrics over the weekend, and as a consequence of having read this piece.
As I try to show, the background to any attempt to compute a value of reach / impact is hugely complex – and the fact that so many areas of society have attempted (and failed) to quantify a value for social impact should be a big warning sign. If we are to equate social reach with social impact, we need that research in order to evaluate a computational approach, both against “real world data” and in order to benchmark it against the opinions of experts.
I do not think it is any coincidence that much of the technical progress in scholarly communication has been in fields where meaningful information can be conveyed in highly condensed terms. Unfortunately one of the inadvertant take-aways (particularly for non-experts) from this is the idea that everything can be thus condensed. Consequently there is a downplaying of the importance of argument and language, let alone metaphor and polemic.
In fact, much of our greatest scientific progress has involved language. It certainly appears that while Bohr and Heisenberg used maths and research to explore the validity of their ideas, their ideas emerged from argument and conversation. In fact, I often say (as part of a polemic) that the day a vice-chancellor or a funding agency requires an “altmetric index of 34″ for advancement / funding / tenure, that that is the day that we have failed to communicate the deeply nuanced complexity of evaluating research.
To conclude: I hope that altmetrics (and other elements) can provide some insights into computing a value for social reach, but that it needs to be highly contextualized, and that there is a degree to which we can measure how well ideas are communicated and passed on – but to a large extent, this is measuring the pass-on-ability of research, rather than the implicit value of research.
On a side-note, a worrying outcome of any complete reliance on computed values might be an inbuilt conservativism: if X has outcome F, then X + 10% will have the outcome F + 10%, so don’t fund Y, which has an unknown outcome. To a large extent, this is perfectly reasonable: once we’ve assessed one star for exoplanets, there are another billion to follow. However, an incremental, conservative approach will not necessarily reward research that breaks boundaries and pushes forward human knowledge, and as a community we should not adopt a closed, computerized vision of the world, but rather embrace computed knowledge as providing valuable insights into performance.
Thank you for the quick and thoughtful reply to my quick reply!
We agree. On rereading my own post, I realize that I should have made clearer the fact that the last paragraph was not directed at you. I think we all have certain tendencies toward efficiency and conservatism. I think these tendencies are often exacerbated by technology and groupthink. So, in this regard, I find altmetrics — like many technologies — dangerous.
This is by no means to say that I am against the development of altmetrics. Indeed, I’m a fan. I think that altmetrics are a huge improvement over bibliometrics, even if altmetrics are not yet either as rigorous or as widely used/accepted. So, even though it was a response to your article, part of the reason I wrote the post was to remind ‘us’ — including myself — not simply to become carried away by enthusiasm.
I’m giving a talk this Thursday as part of the UNT Open Access Symposium (http://openaccess.unt.edu/symposium/speakers/2013) in which I address some of these concerns. So the other, selfish, reason for my post was to practice my rap. I really enjoyed your article, and it has helped my thinking. Thank you again for your comment.
I got that the last paragraph was more of a crie de coeur, aimed at us all. I completely agree, the fetish of the single number alarms me considerably. As much as I always prefer more information, it comes with the responsibility of interpretation and criticism.
Good luck with the presentation! and think of me as I start another article, this time for “Information Standards Quarterly”
Thanks, and good luck with the writing!