The Guardian attends Hack the Government 2013 with Rewired State

Last Saturday was the fifth national “Hack the Government Day”, organised by Rewired State. The idea is to bring together civic-minded technologists to create prototypes, visualise trends and develop ideas that make sense of the huge amount of government data that gets published. Three of us from the Guardian’s software development team went down to see what we could do to code a better country.

Much of what is produced by the government is brought together by data.gov.uk, which has over 9,000 data sets from all central government departments, as well as other public sector bodies and local authorities. This was the starting point for our hack.

Our first challenge was coming up with a good idea that could be produced within the space of a few hours. To do this, you really need to know what specific information is available – and with such a gigantic amount being published, just understanding what could be used is a challenge in itself.

Once we found some data we wanted to use, our next hurdle was to get this data into a format that our application could process. While the government’s open data efforts have led to a huge amount of data becoming available, there currently seems to be little consistency in the format in which it gets published. Our team, like others that day, spent much of our time trawling through spreadsheets and PDFs trying to find interesting data that could be used quickly.

A further difficulty comes in trying to compare datasets. For example, we ran into difficulties matching hospitals in a borough against spending data for the same region. For this to work, you need the datasets to contain matching IDs and cover the same time period. When looking at recent data, some of our ideal comparison datasets just weren’t available yet.

These are known problems and tools are being developed to help alleviate some of the difficulties. In our hack, we used MapIt to resolve postcodes to GSS IDs, which are widely-used within government datasets to identify geopolitical entities (as we found out when looking through spending reports).

Not straying too far from home, we ended up using data published by the Guardian’s own Datablog, which often contain tables and Google Docs spreadsheets that can easily be exported to a CSV, which we then translated into JSON for our application to use.

Despite our last-minute success in getting a working hack together, we failed to submit our idea in time to present (top tip for next time – follow the rules!). However, we were very interested to see how other people got on. Everyone was given three minutes to present their hacks.

Hacks ranged from fun (Whack-an-MP, based on the game Whack-a-Mole, where least-engaged MPs were easier to hit) to the more poignant (Hospi-Cats, which visualised how many hospitals are due to close or have closed via pictures of unhappy cats). An impressive and heavily data-crunched hack translated thousands of Hillsborough documents and created a search index. Our favourite hacks included Don’t Eat There which combined the Food Standard Agency, Foursquare and Twilio to notify the user to not eat at a restaurant if it is terrible. We also appreciated the Universirator, which linked university courses and starting salaries upon leaving – a useful guide for those going to university next year!

Lessons learnt which we will take away for next time:

• Submit your hack in time

• Research your data — know what’s available and how easy it is to process

• Don’t upgrade from Scala 2.9.2 to 2.10 ten minutes before you present

• Tweet the right hashtag

• The Guardian’s data store is really good for hacks

Thanks to Rewired State for organising the event – and for all the free food. We all had a lot of fun, and though we’re pretty sure our hack won’t irreparably damage the coalition and fix the economy, we’ll try again next year…