Publishers withdraw more than 120 gibberish papers : Nature News & Comment.
Thanks to one of my students — Addison Amiri — for pointing out this piece by @Richvn.
Category Archives: Peer Review
What a difference a day makes: How social media is transforming scientific debate (with tweets) · deevybee · Storify
This is definitely worth a look, whether you’re into the idea of post-publication peer review or not.
Apparently NSF Grant Applicants Still Allergic To Broader Impacts
The Consortium of Social Science Associations held its Annual Colloquium on Social And Behavioral Sciences and Public Policy earlier this week. Amongst the speakers was Acting National Science Foundation (NSF) Director Cora Marrett.* As part of her remarks, she addressed how the Foundation was implementing the Coburn Amendment, which added additional criteria to funding political science research projects through NSF.
The first batch of reviews subject to these new requirements tookplace in early 2013. In addition to the usual criteria of intellectual merit and broader impacts, the reviewers looked at the ‘most meritorious’ proposals and examined how they contribute to economic development and/or national security. For the reviews scheduled for early 2014, all three ‘criteria’ will be reviewed at once.
Since researchers don’t like to be told what to do, they aren’t happy. But Marrett asserts through her remarks that this additional review will not really affect the…
View original 183 more words
PLOS Biology: Expert Failure: Re-evaluating Research Assessment
Do what you can today; help disrupt and redesign the scientific norms around how we assess, search, and filter science.
via PLOS Biology: Expert Failure: Re-evaluating Research Assessment.
You know, I’m generally in favor of this idea — at least of the idea that we ought to redesign our assessment of research (science in the broad sense). But, as one might expect when speaking of design, the devil is in the details. It would be disastrous, for instance, to throw the baby of peer review out with the bathwater of bias.
I touch on the issue of bias in peer review in this article (coauthored with Steven Hrotic). I suggest that attacks on peer review are attacks on one of the biggest safeguards of academic autonomy here (coauthored with Robert Frodeman). On the relation between peer review and the values of autonomy and accountability, see: J. Britt Holbrook (2010). “Peer Review,” in The Oxford Handbook of Interdisciplinarity, Robert Frodeman, Julie Thompson Klein, Carl Mitcham, eds. Oxford: Oxford University Press: 321-32 and J. Britt Holbrook (2012). “Re-assessing the science – society relation: The case of the US National Science Foundation’s broader impacts merit review criterion (1997 – 2011),” in Peer Review, Research Integrity, and the Governance of Science – Practice, Theory, and Current Discussions. Robert Frodeman, J. Britt Holbrook, Carl Mitcham, and Hong Xiaonan. Beijing: People’s Publishing House: 328 – 62.
Funny Stuff — but also Serious — from Michael Eisen on the Science OA Sting
This post really starts off well:
My sting exposed the seedy underside of “subscription-based” scholarly publishing, where some journals routinely lower their standards – in this case by sending the paper to reviewers they knew would be sympathetic – in order to pump up their impact factor and increase subscription revenue. Maybe there are journals out there who do subscription-based publishing right – but my experience should serve as a warning to people thinking about submitting their work to Science and other journals like it. – See more at: http://www.michaeleisen.org/blog/?p=1439#sthash.7amnYjlK.dpuf
But I question what Eisen suggests is the take home lesson of the Science sting:
But the real story is that a fair number of journals who actually carried out peer review still accepted the paper, and the lesson people should take home from this story not that open access is bad, but that peer review is a joke. – See more at: http://www.michaeleisen.org/blog/?p=1439#sthash.7amnYjlK.dpuf
I think that message is even more dangerous than the claim that open access journals are inherently lower quality than traditional journals.
Blue skies, impacts, and peer review | RT. A Journal on Research Policy and Evaluation
This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.
Blue skies, impacts, and peer review | Holbrook | RT. A Journal on Research Policy and Evaluation.
Broader Impacts and Intellectual Merit: Paradigm Shift? | NOT UNTIL YOU CITE US!
On the one hand, this post on the VCU website is very cool. It contains some interesting observations and what I think is some good advice for researchers submitting and reviewing NSF proposals.
Broader Impacts and Intellectual Merit: Paradigm Shift? | CHS Sponsored Programs.
On the other hand, this post also illustrates how researchers’ broader impacts go unnoticed.
One of my main areas of research is peer review at S&T funding agencies, such as NSF. I especially focus on the incorporation of societal impact criteria, such as NSF’s Broader Impacts Merit Review Criterion. In fact, I published the first scholarly article on broader impacts in 2005. My colleagues at CSID and I have published more than anyone else on this topic. Most of our research was sponsored by NSF.
I don’t just perform research on broader impacts, though. I take the idea that scholarly research should have some impact on the world seriously, and I try to put it into practice. One of the things I try to do is reach out to scientists, engineers, and research development professionals in an effort to help them improve the attention to broader impacts in the proposals they are working to submit to NSF. This past May, for instance, I traveled down to Austin to give a presentation at the National Organization for Research Development Professionals Conference (NORDP 2013). You can see a PDF version of my presentation at figshare.
If you look at the slides, you may recognize a point I made in a previous post, today. That point is that ‘intellectual merit’ and ‘broader impact’ are simply different perspectives on research. I made this point at NORDP 2013, as well, as you can see from my slides. Notice how they put the point on the VCU site:
Broader Impacts are just another aspect of their research that needs to be communicated (as opposed to an additional thing that must be “tacked on”).
I couldn’t have said it better myself. Or perhaps I could. Or perhaps I did. At NORDP 2013.
Again, VCU says:
Presenters at both conferences [they refer to something called NCURA, with that hyperlink, and to NORDP, with no hyperlink] have encouraged faculty to take the new and improved criteria seriously, citing that Broader Impacts are designed to answer accountability demands. If Broader Impacts are not carefully communicated so that they are clear to all (even non-scientific types!), a door could be opened for more prescriptive national research priorities in the future—a move that would limit what types of projects can receive federal funding, and would ultimately inhibit basic research.
Unless someone else is starting to sound a lot like us, THIS IS OUR MESSAGE!
My point is not to claim ownership over these ideas. If I were worried about intellectual property, I could trademark a broader impacts catch phrase or something. My point is that if researchers don’t get any credit for the broader impacts of their research, they’ll be disinclined to engage in activities that might have broader impacts. I’m happy to share these ideas. How else could I expect to have a broader impact? I’ll continue to share them, even without attribution. That’s part of the code.
To clarify: I’m not mad. In fact, I’m happy to see these ideas on the VCU site (or elsewhere …). But would it kill them to add a hyperlink or two? Or a name? Or something? I’d be really impressed if they added a link to this post.
I’m also claiming this as evidence of the broader impacts of my research. I don’t have to contact any lawyers for that, do I?
UPDATE: BRIGITTE PFISTER, AUTHOR OF THE POST TO WHICH I DIRECTED MY DIATRIBE, ABOVE, HAS RESPONDED HERE. I APPRECIATE THAT A LOT. I ALSO LEFT A COMMENT APOLOGIZING FOR MY TONE IN THE ABOVE POST. IT’S AWAITING MODERATION; BUT I HOPE IT’S ACCEPTED AS IT’S MEANT — AS AN APOLOGY AND AS A SIGN OF RESPECT.
