Email
The government should be more rigorous in assessing the impact of its policies to make sure they work and are cost-effective, a new Cabinet Office report by the behavioural insights team, or "nudge unit", has advised.
Better evidence on whether new policies work, gathered through more robust trials, could save public money and stop inadvertently harmful policies, the report entitled Test, Learn, Adapt cautioned. It comes as the government has come under fire from experts for ignoring evidence on some of its flagship policies on education and welfare reform.
New drugs and medical treatments are required to pass a randomised controlled trial (RCT), showing they are better or more cost-effective than any existing medicine, before they are prescribed by the NHS. But policies, such as on unemployment schemes, education, tax reminders or parenting programmes, are not subject to the same checks and thus there is often no evidence as to whether they actually work.
The report said these random trials could be applied to "almost all aspects of public policy".
An RCT is a test in which individuals who might be subject to a new programme – for example a "back to work" training scheme – are selected at random to receive the proposed intervention. They are then compared with another group of individuals who are treated as normal (a "control" group). The two are then compared to see which scheme worked best.
The use of these trials has rocketed in medicine, up from fewer than 20,000 in the 1960s to more than 200,000 in the last decade, but for other programmes the increase was much smaller. The report recommended starting the trials in uncontroversial areas – such as wording on tax reminders (one trial suggested different wording could improve effectiveness by around 30%) or frequency of visits to jobcentres – before working up to more contentious issues.
Dr Ben Goldacre, a science blogger and research fellow at the London School of Hygiene and Tropical Medicine who is one of the study's external co-authors, said as the government was required to evaluate its policies, it might as well do it well.
"Almost all new policies get assessed, but currently they're assessed rather badly," he said. "We should do it right. In the 1970s, there were lots of doctors and surgeons around who aggressively opposed random trials on drugs, as they knew what was best for their patients."
Goldacre warned that politicians too often picked policies for reasons of ideology rather than whether they would prove effective or not.
"An important first step is humility," he said. "To recognise you don't know if an intervention will be effective. There's a lack of humility in policy-making. People often have to state things in very melodramatic, black-and-white terms.
"RCTs let you check whether your policy is having the outcome you want. This allows you to identify policies which don't work – and if things don't work, we should probably stop them. It's not just about saving money, it's about reducing harm."
The government has come under fire in recent weeks for expanding some of its policies despite an absence of evidence to suggest they work. The Department for Work and Pensions expanded a scheme requiring selected unemployed people to do up to 30 hours unpaid work a week for up to four weeks from a target of 10,000 people a year to 70,000 – despite its own study concluding the programme had "no impact on the likelihood of being employed".
Similarly, sweeping reforms to the UK's national curriculum introduced last week by Michael Gove were attacked by one of his expert advisers for introducing "overly prescriptive" changes, while under the previous government drugs adviser Professor David Nutt was dismissed after he made blunt public remarks that the UK's drugs rating system had no basis in evidence.
The nudge unit's report suggested that the financial crisis might help create a good climate for introducing randomised trials – if interventions were already being phased in small areas at a time due to resource shortages, this provided a good opportunity for "natural" experiments to better see which worked.
Comments
20 June 2012 1:11PM
Clinical trials are "double blind"; neither the patient nor the person administering the test knows who is in which group, the test group or the control group. It's a bit hard to see how this feature, essential to prevent self-deception, would translate to policy trials.
I fear that we would just get a new layer of spurious self-justification that the politicians will claim validates their hare-brained policies.
20 June 2012 1:31PM
There are already consultation periods when upopular / dangerous policies can be dropped; whenever they are the media collectiveley shout U-Turn the very next day. If the government abandoned a policy because statistical mathematic analysis from blind trieals proved it was not sensible, they would still be berated for indecisiveness.
What is the point in adding another layer of analysis if the negative ramifications prevent acting on the results of that analysis?
20 June 2012 1:44PM
I'm all for more evidenced based policy making but this works for some policy areas and interventions better than others - particularly where you can get reasonably matched samples and measurable outputs for the regressions. Can't be applied to all areas though - both practical and ethical considerations. The US have been doing this for years particularly for education and prison rehabilitation (probably many other areas too).
20 June 2012 2:23PM
I think the kind of interventions they're talking about are at least single-blind (e.g. if you get a letter, then you're unaware of the existence of other letters), and this at least takes account of the placebo effect; the paper itself (have just read it here http://www.cabinetoffice.gov.uk/sites/default/files/resources/Final-Test-Learn-Adapt.pdf) discusses some problems relating to observer bias, the other "blind". The bottom line is that trials need to be well designed to be effective and this requires some kind of expert oversight and external scrutiny and this is what the paper seems to be arguing for. Good, because this kind of approach seems to be *seriously* lacking in policy evaluation. Something is better than nothing in my book, and this actually appears to be a a decent "something"!
20 June 2012 2:31PM
I don't think anyone could be against getting better evidence about policies, but I suspect all that would happen is that politicians would have more reliable evidence to ignore.
20 June 2012 2:45PM
I'm entirely in favour of testing before leaping, and I've argued for it myself, but I've seen too many instances in organisations, where the testing just becomes a smokescreen to divert criticism: "We did all the tests and we can argue that they revealed no problems. Just keep quiet."
Trials do need to be well designed, and that means good statisticians. How long will it be, though, before the statisticians' control starts to get resented and murmurings about "unelected faceless men" standing in the way of enlightened policy begin?
The best solution is probably to have a Mycroft Holmes, but they are rare.
Thanks for the link. I shall read it and try to assuage my cynicism.
20 June 2012 2:45PM
I'd actually be in quite strongly in favour of taking politicians to court and suing them if they implemented policies shown (with good evidence!) to be a complete waste of public money, especially if there was an alternate effective policy. That might focus the mind a bit ;-)
But, yes, point taken, there also needs to be a shift in the civil service's, politicians' and the media's attitudes.
20 June 2012 2:51PM
A good point regarding "faceless men". I think that needs very careful thought regarding separation of roles and incentives in the civil service, e.g. you try an remove self-interest from the evaluation process, and enforce some kind of separation from the decision maker. It's an interesting challenge. This is managed well in other fields (e.g. medicine, engineering), so there is hope!
20 June 2012 3:29PM
Just as an afterthought, most of the British public appears to have no idea what a statistician is. They think it's a man with a clipboard talking in percentages.
20 June 2012 4:44PM
There's 'regulatory capture' and/or 'lobbyocracy' too. IP policy-making has often made me wonder what healthcare policy-making might look like if we had a stronger 'CAM' industry.
[link]
[link]
20 June 2012 7:14PM
You say "what is the point in adding another layer of analysis" but most consultations lack any kind of rigorous scientific analysis. The authors are talking about introducing proper analysis to policy making, rather than the current system where the government does a U-turn whenever the media starts shouting loudly enough.
20 June 2012 11:57PM
The Geek Manifesto has been sent to all MPs. Maybe some will read it, and be inspired by articles like this and the recent Nutt thing to give it a go.
21 June 2012 3:53AM
A good idea that should have been implemented already. First mentioned in the book "Poor Economics" by Abhijit Bannerjee and Esther Duflo, 2011
21 June 2012 4:39PM
It will never happen.