A Response to: The REF doesn’t capture how academic impact on policymaking takes place | Impact of Social Sciences

Jane Tinkler presents the latest case against the UK’s Research Excellence Framework (REF) here:

The REF doesn’t capture what government wants from academics or how academic impact on policymaking takes place | Impact of Social Sciences.

At the risk of coming off only as a staunch defender of the REF, I want to suggest that there’s something like a category mistake that keeps getting made in criticisms of it. Or maybe it’s a combination of a sort of category mistake and a sort of reactive attitude.

Although there are exceptions, researchers in the UK generally are resistant to the REF. This resistance usually takes the form of criticizing one or another aspect of the REF, or protesting its very existence as unfair to academics, or a combination of both. Jane Tinkler’s objections fall into the third category.

Citing a description from guidance to ‘Panel C’ — the panel that will be in charge of assessing the impact of research in the social sciences — Tinkler suggests that the REF is inadequate for capturing the real essence of the sort of impact research in these disciplines might have, especially on policy making.

Although she acknowledges that policy makers often want advice regarding particular policy issues, she insists that what policy makers really want is an academic’s expertise, rather than the results of a specific research project. It is this expertise that government seeks when it places academics on advisory panels, for instance. Yet, according to Tinkler, the REF unfairly excludes participation on an advisory panel as evidence of impact.

Tinkler’s criticisms thus are two-fold: (1) the REF incorrectly expects evidence of impact from particular research projects, which is more a matter of luck than skill; and (2) insofar as it is incapable of capturing the serendipity of expertise, the REF is unfair to academics.

I suggest that there’s something wrong-headed in expecting the REF to capture the serendipity of expertise. What academics ought to expect from the REF is that it provide enough room for a university to maneuver in making a case for impact. And this is how the REF ought to be judged.

But it seems that many critics make a category mistake in judging the REF to be inadequate. The category mistake involves expecting the REF to capture the impact of every project of every academic researcher. But the REF is designed for a higher level of analysis than that. It looks at universities over a period of time, and it allows universities to pick the best examples of impactful research. It does not look at each individual bit of research or each and every university.

The reactive attitude is more subtle. Evidence of this attitude, however, can be found in the idea of expecting the REF to capture impact. The idea is something like insisting on an assessment framework capable of capturing all that’s good about academic research when we academics do what we do. But this has things backwards — it looks only through the lens of autonomy, without any attempt to see through the lens of accountability.

In contrast to the ‘leave us alone to do what we do — and if you want to know the impact of what we do, develop a better instrument to capture it’ attitude, viewing the REF through an accountability lens will allow academics to take ownership of it. That’s good, because this will allow academics to exercise (rather than merely insisting on) their autonomy while satisfying demands for accountability.

So, another way to put the point about the REF allowing room to maneuver is to ask: does the REF allow for the exercise of academic autonomy? If the answer is ‘yes’, the academics ought to stop resisting it and start figuring out how to own it.

Let’s try to answer what I think are the right questions about the REF. In what follows, I’ll make reference to pages in this presentation on the REF, since it seems to have the latest information. Rather than asking what does the REF require, I suggest we ask: What does the REF allow us to do?

There are three things about the way the REF is organized that allow us (universities) room to maneuver:

  1. The REF defines impact loosely.
  2. The REF utilizes peer review.
  3. The REF asks universities to provide impact case studies in the form of narrative accounts .

Taking these one at a time, the first thing the REF allows us to do is to offer our own accounts of impact:

In drawing up its assessment criteria and the advice to submitting institutions, the main panel agreed that providing HEIs [Higher Education Institutions] with detailed lists of impacts and evidence and/or indicators for those impacts would be unhelpful because these could appear prescriptive or limiting.

Thank you very much, REF! This provides us with room to maneuver when presenting our cases for impact. This is an opportunity for those universities willing to seize it.

Second and third, the REF uses peer review of narrative accounts in order to judge impact:

Within their narrative account in the case study, institutions should provide the indicators and evidence most appropriate to the impact(s) claimed, and to support that chain. The sub-panels will use their expert judgement regarding the integrity, coherence and clarity of the narrative of each case study, but will expect that the key claims made in the narrative to be supported by evidence and indicators.

The main panel anticipates that impact case studies will refer to a wide range of types of evidence, including qualitative, quantitative and tangible or material evidence, as appropriate. Individual case studies may draw on a variety of forms of evidence and indicators. The main panel does not wish to pre-judge forms of evidence. It encourages submitting units to use evidence most appropriate to the impact claimed. (p. 60)

Thanks again, REF! The fact that you don’t wish to prejudge forms of evidence, say, by limiting impact to some “objective” indicator is wise. It’s also an opportunity for universities to present the best cases they can. Might the skillful use of certain indicators help certain cases? Of course! But the point is that no one-size fits all indicator is imposed. Universities get the chance (this is the point) to make their case for each case on a case by case basis. It hardly gets any better than that. But it could get a lot worse.

What about the fact that, as Tinkler brings up in her article, merely serving on an advisory committee is insufficient to demonstrate impact? Well, to be honest, that strikes me as fair. After all, it doesn’t make for much of a story to say that researcher X served in such a capacity. Did it make any difference? No? Then I would choose a different case — one that allowed me (the university) to take advantage of the opportunity the REF presents.

If it did make a difference, then I would want to ask what the difference was, how big it was, and how important it was. Moreoever, I would want to ask how it happened. And I would want to answer all of these questions in my case study — that is, I would want to generate the best narrative account I could. The point is not to exclude participation on an advisory panel as a viable impact. Instead, the point is to ask for a richer descriptions (a narrative account) in place of a simple description (researcher X served on advisory panel y).

The REF represents an opportunity, particularly for universities that cannot rely on a reputation for expertise, to demonstrate that they are making a difference in the world. Imagine if all that were required were a simple listing of all the prestigious advisory panels your faculty were appointed to? My guess is that this would unfairly favor the Oxfords and Cambridges over the rest. One last point — my guess is that Oxford and Cambridge are clever enough to treat the REF as an opportunity. I wonder whether the rest will catch on in time?

This entry was posted in Accountability, Broader Impacts, Future of the University, Metrics, Peer Review. Bookmark the permalink.

13 Responses to A Response to: The REF doesn’t capture how academic impact on policymaking takes place | Impact of Social Sciences

  1. I’m in the interesting position of agreeing both with most of what Jane wrote and with most of what you say here. I don’t think that Jane’s piece does fall into the category of the dominant critique of RAE/REF type research assessment, which I agree often takes the form of a kind of unreflective whining (which does nobody much good). The RAE existed, and the REF exists because of ideas most academics subscribe to – that “research excellence” exists and can in some sense be objectively (more or less) assessed by peers. These exercises are essentially asking us, as a community, to put our money where out mouths are.

    But I don’t detect any of this in Jane’s piece. All I can see in there is, on the one hand, a legitimate question about whether the mechanisms by which “impact” is assumed to be achieved are conceived so narrowly as to rule out many kinds of impacts, and on the other, a legitimate worry that the REF – as with the RAE before it – will tend to reinforce prevailing incentives (at least in the social sciences, the field to which Jane was referring) to produce a particular kind of scholarly output – single-authored, disciplinary, not-too-risky journal articles – at the expense of other kinds of output which may be more associated with impactful (social science) research.

  2. Thanks, Kieron (who should be followed on Twitter @kieronflannagan), for the response.

    I certainly agree that Jane’s article lacks that tone of unreflective whining — and so I do not at all mean to paint her with that brush.

    I do think, however, that protesting against the narrowness of the REF’s impact requirement (as currently formulated) is likely a mistake, both in terms of the interpretation of the REF as a document and in terms of the response to it as a policy.

    If a research program has societal impact, then one ought to be able to construct a plausible narrative of what that impact is and how it is achieved. This is not the same as (or as simple as, or as potentially misleading as) counting citations as a proxy for academic impact. If the REF encourages a certain type of research output (single-authored, disciplinary, not-too-risky journal articles, as you say) AND requires that one draw a connection between that sort of output and societal impact — well, then, that is a huge problem.

    If, on the other hand, the REF’s impact requirement encourages us to rethink the relationship between so-called academic impact and so-called societal impact, then that’s an opportunity we should seize.

  3. The difficulty I have with the REF is that is it a rather tightly circumscribed ‘opportunity’ since, if I have understood correctly, all instances of impact must relate to a particular piece of published research. In many cases — and I think this is Jane’s point — this will discount activities that have genuine impact (never mind trying to tackle the thorny question of providing evidence). I might cite my science blogging and activities within the Science is Vital campaign as impactful but since these are generic activities relate to my role as a scientist but not to a particular piece of research, they cannot be counted in the REF. Which is frustrating.

    • Well, unless I am misreading something badly, I may have a cure for your frustration (from Part 3, Section 3 of this: http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf):

      “146. The REF aims to assess the impact of excellent research undertaken within each submitted unit. This will be evidenced by specific examples of impacts that have been underpinned by research undertaken within the unit over a period of time, and by the submitted unit’s general approach to enabling impact from its research. The focus of the assessment is the impact of the submitted unit’s research, not the impact of individuals or individual research outputs, although they may contribute to the evidence of the submitted unit’s impact.”

      I think you should be able to construct a narrative according to which Science is Vital counts for the REF.

      • Belated thanks for the clarification. I’m not closely involved in the early preparation for our departmental return so haven’t read the documentation as closely as some. Let’s see if SiV counts for anything…! ;-)

  4. Yes, people have been very focused on the requirement to trace back and provide proof of “impact” stemming from specific research outputs of at least a certain quality level. This is certainly being taken far more seriously in my institution than the secondary aspect of constructed a more general narrative about how a research group engages or otherwise seeks to be “impactful”. And to be clear, I think the REF may be pushing social scientists to stick with the traditional “cottage industry” model – single-authored, not-too-risky, etc etc. papers – that some/many of them are already too comfortable with.

    • See my reply to Stephen, above. I think the idea that an impact case study must provide proof that a particular, narrow journal article is the source of the impact is a misinterpretation of the REF.

  5. Also, for the record, I do agree with anything that “encourages us to rethink the relationship between so-called academic impact and so-called societal impact”!

  6. One more post about the notion of ‘underpinning research’. Paragraphs 160 and 161 of this document (http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf) also are relevant.

    What one needs is some research output that is of good quality (at least 2 star) that has been produced by the unit in question during the period from January 1993 to December 2013 to ‘underpin’ the impact. The very beginning of paragraph 161 reads: “There are many ways in which research may have underpinned impact, including but not limited to …” followed by a list of examples that begins with the phrase “Research that contributed directly or indirectly to an impact.”

    I don’t know about anyone else, but I see lots of room here.

    Here’s an important question: how are the peer evaluators being instructed? Well, I looked, and this document give really helpful information that (I think) backs up my interpretation: http://www.hefce.ac.uk/research/ref/pubs/2012/01_12/01_12_2C.pdf.

  7. Yes, I think you are right that there is potentially more flexibility in the proposed approach than is generally recognised, but of course behaviour is based on risk perception rather than risk analysis, and the stakes – reputational as much as financial – are high!

    • In which case Oxford and Cambridge might want to play it safe — although I’ve a hunch they’ve got good people on this who might very well think that what counts as ‘safe’ is a wider margin than those with more to gain from the REF.

      Were I president of a low-to-middling HEI, I’d bet on the best impact I could narrativize. Of course, I’m not — so candy and nuts and all….

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>