The State of UK Open Data 2024: From big bang open data to responsibly stewarding data with a purpose

The UK's open data journey, ignited in 2010, saw the public sector lead the charge, epitomised by Transport for London's (TfL) 2017 initiative, yielding £130 million in annual savings. Third-party developers thrived, leveraging TfL's data for innovative solutions, reshaping London's urban landscape. The Department for Environment, Food & Rural Affairs (DEFRA)'s 2015 open data surge birthed diverse applications, from flood risk assessments to archaeological discoveries. However this ‘big bang’ of open data was often done haphazardly and as a box-ticking exercise, so open data leaders shifted the paradigm from 'open it and they will come' to 'publish with a purpose'. The public sector responded with more powerful open data services such as DEFRA's Earth Observation Data Service, that remain as open as ever, but provide targeted around specific use cases.

 

On the other side, the private sector has come later to the party, and initially with less consistency, with open data rationales ranging from regulatory mandates to collaborative initiatives. Open Banking, despite its misleading moniker, epitomises this synergy and tension, spurred by government intervention demanding open data and secure data sharing. The ecosystem, fostered by open standards and APIs, burgeons with diverse offerings, from financial management to debt advice. Learning from this, initiatives like OpenActive, Open Energy and Stream, in the water sector, employ incentives to cultivate similar data ecosystems organically. Spearheaded by organisations like the ODI and Icebreaker One, these platforms facilitate open data and secure data sharing while fostering societal benefits. 

 

Recognizing the wealth of data held by private entities, visionary companies embrace collaboration as the cornerstone of innovation. Microsoft's Industry Data for Society Partnership, in partnership with ODI and GovLab, spearheads this movement, bridging the data divide globally. Through initiatives like the Planetary Computer, Microsoft champions open data infrastructure, fostering collaborations to tackle pressing issues like climate change and education. Guided by principles of openness, usability, and empowerment, these endeavours pave the way for responsible data sharing, propelling society towards a more data-inclusive future.

 

Currently, we are beginning to recognise the best stewards of data are often neither government nor private sector, but independent vehicles set up to steward data on behalf of society for a specific purpose related to social impact. These organisations, data institutions, can publish open data, share sensitive data responsibly, and maintain key infrastructure for open data such as standards, identifiers and policies. Although nascent in their conceptual understanding, many already exist in the environment and climate space, providing access to data about climatic trends, forest cover, extractive industries and more. These organisations, operating in more networked and decentralised ways may be the future of the open data, but their impact is only just beginning to be understood.