Getting The Most Out Of Open Data: Adaptive Intelligence For Better Social Outcomes

Earlier this summer, I participated in Convergence 2013 in Paris with Christian Frisch, cofounder and CTO of Data Publica. In our session, we discussed the evolution of open data programs looking at the changing goals of cities and other public sector organizations as well as the persistent challenges they face. 

The momentum for open government and open data has certainly grown over the past few years. The initial push for open data was to know exactly where the government was spending money. With the economic downturn of 2008, that was imperative — and still is. In the US, USAspending.gov and Recovery.gov published government budgets and allocation of spending from the US Recovery Act. In the UK, Where Does My Money Go? similarly publishes government spending by department and region. Yet open data does more than increase transparency and accountability. We’ve seen some great strides in civic innovation — new apps for finding out when the bus is coming or where the nearest toilets are. Open data generates business innovation as well — new lines of business for existing companies and new startup businesses in a wide range of industries from healthcare to real estate to financial services. 

But one of the persistent challenges for organizations is in how they can use the data they are collecting and publishing through their open data programs to improve operations and for long-term planning and policy decisions. In a recent Palo Alto city council meeting, the city CIO presented his IT strategy, including the progress made on transparency through their open data portal. One of the questions from the council was about the actual use of data. “How do you help us with data for effective decision-making?” The response was simply “not yet.” But the imperative is clear: Incorporate the data into operations and link open data with broader data governance and business intelligence initiatives. Fortunately, “not yet” doesn’t need to be that far off.  

My recent report, Getting The Most Out Of Open Data, discusses the role of data facilitators who clean and prepare data, as well as publish it in various formats to be consumed by the end user, either public or businesses in the private sector. Many flavors of these data facilitators have emerged to fill the gaps for cities (or other organizations). The city of Palo Alto uses the Junar Open Data Platform with API access to facilitate use of the data. Socrata, widely used by larger cities, provides open data infrastructure that offers performance management, GovStat.  

But the Holy Grail is not just to publish the data, or go back and look at the data to see how you did, but to really use the data in real time to make decisions that affect people’s lives. At an open house I attended earlier this summer at the IBM Research Labs in Dublin, demos of IBM’s Smarter Care solution provided a great example of incorporating data into processes in order to improve the outcomes. In one scenario, a newly single mother needs to find housing with convenient access to transportation networks that will allow her son to get to school and her to get to work. Fortunately, the social services agency uses Curam Social Services Platform with its Spatial Recommender. The client identifies the locations that she will need to get to easily across the city: her work, her child's school, her child's current foster parents' home (see below: panel 1). The system pulls data on available housing units across the city (panel 2) and can identify optimized housing options based on distance to work and school and taking into account access to public transportation (panel 3). While this scenario may seem simplistic, without access to housing and transportation data, the choice of social housing would be more difficult. This example illustrates how data can facilitate the process of providing social housing and improve the outcome for the client.

Successful data programs will consider not only publishing data, reviewing it for performance management, and using it to forecast and plan but will also prioritize using data in real time to improve social and business outcomes. That is what Forrester calls adaptive intelligence.