Meet the man who turned David Cameron onto open data

Interview: If you're not in Downing Street you may never have heard of Rohan Silva. But this key advisor to Britain's Prime Minister is also responsible for helping create Britain's transparent open data regime. As he prepares to leave Westminster, this is his first full interview

Rohan Silva Photograph: Guardian

"We will unleash a tsunami of data."

This was a quote from a "government source" I reported back in May 2010. It was written just after the Coalition had been formed, and had promised a new era of open data and transparency. This was also a year after we launched the Guardian Datablog. Following on from the Free Our Data campaign, we called for government data to be open transparent and available.

It's a good quote and one we've used several times since, but never revealed who said it. Up until now, that is. That man was Rohan Silva, the 32-year-old senior policy advisor to David Cameron who recently resigned.

He will be leaving Downing Street in June to set up his own internet education business. Conservative Home blogger Paul Goodman wrote: "A true radical is leaving the building - perhaps one of the last". And, whether or not you support the government, many open data campaigners may find themselves agreeing.

A true radical is leaving the building - perhaps one of the last

It's certainly a moment: Silva has been a driving force inside the UK government for the open data agenda, chivvying and pushing and crucially making things happen.

It started for him in 2006, inspired by the Federal Funding Accountability and Transparency Bill proposed by, among others, Barack Obama. That Bill became an act - it led directly to data.gov in the US.

He drafted the key May 2010 letter from David Cameron to every government department which promised unprecedented transparency. In that moment the then-new government pledged to release a lot of data - and even set itself deadlines. It pledged to open up all government spending over $50,000. It turns out this is why the UK's government publishes all data over £25,000.

"It really caught my imagination and I managed to persuade George Osborne we should make the same commitment," recalls Silva. "I just did a crude comparison - and it was back when the pound was strong - which is why we ended up with a £25,000 commitment in the UK."

That commitment followed the party through the election and into government, although it was not universally popular. "The reaction we got from the greybeards was 'that's a mad commitment because it'll cost a fortune to implement." So he tested that objection, calling a friend at Windsor Council, Liam Maxwell, asking him to see how long it would take to make the local authority's data transparent. "He did it in an afternoon". Maxwell - who is now the government's Chief Technology Officer - became part of a core team, which also included MySociety's Tom Steinberg.

It's important to note that the process didn't begin with the Conservatives. Asked by then Prime Minister Gordon Brown what the UK could do to use the internet properly, the man credited with inventing the worldwide web, Tim Berners-Lee, replied that the government should put all its data online. He later reported Brown saying, "OK, let's do it." Brown's government launched data.gov.uk and members of that first team stayed on with the new administration, most notably Berners-Lee himself and Southampton University's Nigel Shadbolt.

But it took MPs' expenses to push the Conservatives into adopting Silva's agenda. "We kept setting out these different open data commitments and had no coverage at all. It was proving increasingly challenging in truth to persuade George [Osborne] and David [Cameron] to create more commitments. No-one was writing about this stuff. But our repsonse to MPs expenses was to push for greater transparency. That was great - the challenge can be in opposition that its kind of abstract. You're committing and you can't make it real. MPs expenses converted a lot of Tory ministers to the cause."

"There is this much government data". Rohan Silva (left) with David Cameron Photograph: Steve Back

But how could that enthusiasm be maintained? "My trick when we got in was that I inserted as much ambitious and bombastic language as possible on open data into the speeches I wrote for George and David. Opposition tends to be very ambitious on open data and then back down in government. I was keen we avoided that cycle. I put all these ambitious verbal statements into the speeches - things it would be hard to back away from when we got into government."

One of the first big releases was Coins. The Coins (Combined Online Information System), data was going to change the world. The government's spending database, we reported at the time that The Treasury had repeatedly refused FoI requests for it in the past (it is 24m items long). This is what we said about it then:

"It is the mother lode for central government," says Rufus Pollock, the director of the Open Knowledge Foundation and one of those behind Where Does My Money Go? – a site that breaks down government spending. He says it could change local reporting for journalists. "The big deal with Coins is that when you get a figure like £6bn-worth of cuts it is useful, but what you really want to know is much more granular – how much is spent on police in your parish, for instance. Coins is that kind of data – the lowest and most granular level that government collects."

In fact, the data was a mess - tricky to use even for those in government, for those outside it's almost impossible. It showed to the world that the data the government relies on can be pretty poor at that. But Silva was keen to get it released, in the teeth of stiff civil service resistance. To make matters worse it almost didn't get released at all amid the realisation that the site it would be published on was not secure.

Our Coins navigation page Photograph: Guardian

"We'd always known that coins was a bit of a dogs breakfast of a dataset. At the same time we knew we could get it out relatively quickly - internally showing that we could release it and the world didn't fall apart was really important," says Silva. "We were always careful to say that Coins is not the be all and end all - but the beginning of the process. "

The same thing happened with the crime map - which showcased how many police officers would simply put their police station as the location of a crime, even if it had occurred miles away. Interestingly, the crime map is now the Government's most popular open data initiative - and it broke the rules too. The rules were that 'if you release it, they will build it' - that just get the data out there and developers will do the work. This didn't happen.

"The Home Office defied us on crime and signed a contract to build police.uk - in direct breach of our edict on this," says Silva. "It's interesting because it had 3m views in the first six hours. It was the first of the open data releases that had real public consciousness breaking through. We've not built tools ourselves since."

Senior civil servants wanted to include this text in all government pronouncements on open data:

No government data shall be released unless its quality can be assured

Silva refused. His argument was that sunlight would drive improvements in the data itself. "They kept writing that in - I kept taking it out. I got into an argument with the civil servants about it. That would mean essentially no data ever released. That would have choked off the open data agenda from day one. They said if we release poor quality data it will embarass the civil service - but I believed the only way to improve that data would be to release it."

These were the commitments the government made in May 2010:

• Historic COINS spending data to be published online in June 2010.
• All new central government ICT contracts to be published online from July 2010.
• All new central government lender documents for contracts over £10,000 to be published on a single website from September 2010, with this information to be made available to the public free of charge.
• New items of central government spending over £25,000 to be published online from November 2010.
• All new central government contracts to be published in full from January 2011.
• Full information on all DFID international development projects over £500 to be published online from January 2011, including financial information and project documentation.
• New items of local government spending over £500 to be published on a council-by-council basis from January 2011.
• New local government contracts and tender documents for expenditure over £500 to be published in full from January 2011.
• Crime data to be published at a level that allows the public to see what is happening on their streets from January 2011.
• Names, grades, job titles and annual pay rates for most Senior Civil Servants with salaries above £150,000 to be published in June 2010.
• Names, grades, job titles and annual pay rates for most Senior Civil Servants and NDPB officials with salaries higher than the lowest permissible in Pay Band 1 of the Senior Civil Service pay scale to be published from September 2010.
• Organograms for central government departments and agencies that include all staff positions to be published in a common format from October 2010.

The full list is actually longer. In fact, the government has done most of what it said it would. The National Audit Office reported in April last year that 23 out of 25 key commitments had been achieved. However it also pointed out very few people actually look at the data the government produces.

Public spending 2011-2012 graphics and interactive Photograph: Guardian

And we have been critical of the way government releases spending data. Silva accepts those criticisms: "When it comes to the spending data we've come light years forward. But I think the big to-do on data is to go back and improve the quality of spending data. It's still too difficult for people to easily see where their money goes."

The world has changed now, and it's arguably the case that Silva is one of key people in making that happen. "It's amazing to look now and say it was only two and a bit years ago that we didn't have crime data on a street by street basis, we didn't have central and local government data released on a contract by contract basis. It's almost seen as a given now that all data's released."

"The test of policy should be: would it be possible for the next government to reverse what you've done in a term or two. No government would reverse this."