Education secretary Michael Gove'scontribution to rational, evidence-based policy making was a declaration that anyone against his version of academy schools is a deluded left-wing ideologue, antagonistic to improving children's chances in life.

In other words (and he is not the first minister of any party to behave this way), policymaking is a matter of faith. Gove knows giving power to academies is right – never mind what the National Audit Office says about value for money and accountability, never mind there's been little or no prior analysis and certainly never mind what the permanent secretary of the Department for Education may think, since he's on his way out of the door.

It's wise, then, to read the Institute for Government report on reasons for policy success with a sceptical eye. "Build a wider constituency of support … take time and build in scope for iteration and adaptation … open up the policy process … understand the past and learn from failure." Is this really how policies ever get made? Even its own examples have to be stretched to support the formula – the Labour government had to be dragged into implementing a ban on smoking in public places, for instance, long after Scotland and the Republic of Ireland had provided good evidence. Another, the Climate Change Act, is the subject of tense argument within the coalition government, so hardly a done deal.

The institute's own evidence suggests policy making can be definitely non-iterative, subject to the vagaries of ministerial and MPs' enthusiasms. While devolution looks permanent, it was sketchily prepared for, has been poorly monitored (inside Whitehall at least) and, on the pre-Christmas testimony of Sir Gus O'Donnell, may have dramatic and unwished for consequences.

Of course, some policies provide lasting success. Seat belts, for example. Or equal rights. Or the proscription of racism. The latter two are examples of law and policy change meshing with long-term changes in sensibility. Over Christmas, the distinguished social statistician Sir Roger Jowell died. His British Social Attitudes survey, begun in 1983, captures the remarkable liberalisation of views on, for example, same-sex relationships. The Tories' attempt to ban mention of homosexuality in schools and Labour's enactment of civil partnerships played a part – but in what proportion was legislation cause and changing sensibility the effect, and vice versa?

That's hardly a new question. It was posed in AV Dicey's classic Law and Public Opinion in England in the 19th century, which was published before the first world war. Maybe it only has temporary, shifting answers, congruent with the oddly shifting nature of the British constitution, which once we were taught was so fixed and immutable.

If Dicey has an heir, he might look a bit like the director of the Institute for Government, Peter Riddell, who has just taken over. Riddell has a formidable track record as a political reporter and commentator on the Financial Times and The Times, to which he has added books and his chairmanship of the Hansard Society – meriting his elevation to membership of the Privy Council.

Riddell has shed light on the fuzzy boundary between party politics, inside and outside the House of Commons and the "constitution" – the arrangement of decision making between Westminster, Whitehall, the monarchy and, now, the devolved jurisdiction.

At the institute, however, he has to deploy his skills as a mapmaker to another disputed borderland, between "the constitution" and public management. David Sainsbury, the former Labour science minister, put his charitable weight behind the institute, hoping it would improve the quality of the civil service and – in parallel – that of ministerial decision making. Under Michael Bichard and then Andrew Adonis, the institute has published valuable reports but has it yet been able to persuade ministers to redesign the engine?

The organisation faces the same problem as the National Audit Office. The literature identifying civil service shortcomings is as prodigious as the shelves full of reports recommending improvements – in training, recruitment, management and finance and, of course, in the calibre and competence of ministers.

Change, which many civil servants would themselves welcome, depends on secretaries of state and prime ministers, but David Cameron is proving even less interested in structural reform than his Labour predecessors, despite fiscal pressures.

Perhaps fundamental change in Whitehall has, yet again, to await the arrival of a new prime minister. Riddell's task is to keep stacking up the supply of ammunition, in the hope it might one day get fired.

This article is published by Guardian Professional. Join the Guardian Public Leaders Network free to receive regular emails on the issues at the top of the professional agenda.