Open Data has enormous unfulfilled promise to change how governments work and to empower citizenship. As more governments and issue experts discover new potential in the public release of data, civil society groups still need clear guidelines and mechanisms for cooperation.
The Global Open Data Initiative (GODI) is an attempt to more clearly outline the institutions, organizations, and policies that make up the global open data community and to help move the community forward. GODI is a project led by civil society organizations (Fundar, Open Institute, Open Knowledge Foundation, Sunlight Foundation, and Web Foundation) to share principles and resources for governments and societies on how to best harness the opportunities created by opening government data.
In order to do be able to assess the needs of the community, GODI partners reached out to the key individuals and organizations that work with open data in their own country or region, and tried to learn more about their demands, goals, and challenges. We circulated a public survey and conducted interviews with open data experts and practitioners.
The survey received 39 responses from individuals working in 13 different countries. The respondents came from diverse professional backgrounds: research/education, business/consulting, and advocacy (around environmental issues, international aid giving, public procurement and public cultural institutions).
GODI partners also conducted interviews with 41 individuals from anti-corruption NGOs, development organizations, and academia. The interviewees represented 23 countries.
Below, the findings of the survey are organized into five thematic categories: open data standards, guidelines and definitions used by the respondents; global networks and the ways open data communities organize themselves; the respondents’ experiences interacting with governments about open data and advocating for government data; the challenges facing the open data community; and finally, the opportunities and the future potential for a global open data initiative.
II. Open Data Standards, Guidelines, and Definitions
The main question we asked respondents was which open data standards or guidelines they rely on in their work. There were a range of answers to this question, including several respondents who did not regularly rely on any standards or guidelines. Most of the respondents subscribe to the Open Knowledge Foundation or the Sunlight Foundation’s definitions and principles of open data.
While most interviewees agreed that the basic definition of open data is government proactively publishing data online, some respondents differed on the level of openness based on their vantage point. Some of the interviewees interacted with open government data through the process of creating tools and apps that run on government data. Others use open data as a way of tracking and judging the performance of government and public institutions.
Several people in developing countries had a varied interpretation of openness depending on the idea of how data leads to change—especially in countries where data is still not in machine readable formats. Others mentioned that the global discourse around open data tends to focus on formats for data publication and not data modeling standards, which will be increasingly important as data is to be shared across countries, regions and cities.
In terms of data formats, CSV, JSON and RDF were all mentioned several times. World Wide Web Consortium Standards were referenced several times, as were the Open Knowledge Foundation definitions and the five star data scheme. Creative Commons was the most frequently referenced data license.
Despite widespread agreement that standards were important, in practice the interviewees were not overly focused on them. “It’s a gradient, not a binary” was the response of one interviewee from the USA and a similar sentiment was reflected in the responses of others. A few gave the example of Excel files: while they would prefer open formats, for the moment, data in Excel was still better than PDF and at least they had the data.
We also found that open data is frequently perceived as a product of civil society organizations’ regular efforts on access to information rather than a timely and trustable resource provided by governments. In many countries, data is often collected through freedom of information (FOI) requests or through web scraping and then published in machine readable formats by CSOs themselves. In many cases, the only exceptions are highly aggregated budget data or population data at federal levels. Additionally, there is an increasing number of initiatives attempting to generate data independently of government involvement.
Several people in developing countries had a varied interpretation of openness depending on the idea of how data leads to change – practical openness is thus also seen as being contingent on the usability of data to those who are seeking to create change with it.
In some regions, such as Latin America, respondents and interviewees were often unaware that open data standards and guidelines existed, due in part to the limited availability of Spanish language resources. Translating existing resources, or drafting resources in other languages, will be an important and valuable step forward for the global open data community.
We were also interested in learning more about the terms or translations non-English speaking individuals and organizations use when referring to open data. Most of the individuals interviewed either use the English term open data or use a literal translation of the term in their native language. We found that participants did not see the translation of the term itself as a challenge; however, once translated, participants did not feel that there was widespread understanding of what open data actually meant within their countries. Many respondents noted that the term is dry and technical, which can impede efforts to evangelize and share knowledge about open data.
III. Open Data Networks and Communities
Global networks are extremely important for organizations to share knowledge and learn from each others’ experiences. When asked which global networks, email lists or communities are most helpful, many respondents identified the Sunlight Foundation’s international list and the various Open Knowledge Foundation lists and communities. A few other major international actors were mentioned as sources of information and community, including MySociety, the Open Government Partnership, the European Public Sector Information Platform, the World Bank, and the Open Data Institute.
Apart from these major international actors, many respondents stated that they used local lists and communities of journalists to gather news and information.
We found that many participants are eager for GODI to help connect the different strands of the open data movement and provide a place for people to come and find potential partners and collaborators. A few mentioned a need to connect those working on open data at the national level to the international conversation as well as to others working at the national level in different countries.
An interesting finding was that open data is still perceived as an important issue only by CSOs and funders that already work within transparency or accountability agendas. This is reflected by the difficulty they found communicating its importance to new funders, other CSOs and government officials.
IV. Advocating for Open Data
Given that a majority of the surveyed individuals interact somewhat regularly with government officials, we wanted to asses how knowledgeable these government officials were about open data and hear about the challenges and successes of advocacy efforts. Many respondents claimed that while a majority of government employees knew nothing or very little about open data, the specific individuals working on technology and open government are more familiar with the concept. As expected, knowledge of open data is typically isolated within relevant departments and branches of government.
As some respondents argued, governments have a lot to gain from making data openly available, which makes it even more important for the global community working to open data to communicate more effectively with elected officials. This also suggests that government-wide outreach and education will be a necessary step forward for the global open data community.
There was widespread agreement that achieving data disclosure policies required a combination of both legislative and persuasive tactics. Interestingly though, only a few respondents mentioned the use of guidelines or standards in their day to day advocacy. Going forward, developing advocacy guidelines and promoting the existing ones could be an effective method of strengthening ongoing reform efforts.
Open data for transparency and accountability is often met with resistance and suspicion. Several organizations noted that their ability to interact and engage with public officials diminishes notably when they are seeking politically sensitive datasets like company registers, budgets, or campaign finance information.
V. Challenges Facing the Global Open Data Community
We found an overall sense of optimism about the possibilities open data offers. However, it was also difficult for the participants to talk about successes without qualifying those successes with other parallel failures or remarking that even the successes were only small steps on a long road.
Unsurprisingly, the challenges faced in the last year by the majority of the organizations we heard from could be boiled down to: politics, access to data, data quality, and engagement.
There is a challenge of moving the open data agenda across the public sector generally in the absence of national policy and commitment. Organizations faced political resistance from governments unwilling to release data in the first place as well as efforts on the part of government to diminish the scope of FOI requests and the continued reliance on certain high-value datasets as a revenue stream. The lack of good freedom of information laws in many countries is inhibiting the development of open data.
Part of the problem is the low availability of credible impact studies that can prove the theory of change for open data.
There is some confusion between open data and big data with one of the challenges being to explain what steps are needed to open up datasets, especially with newer audience like commercial organizations. Further, open data is hindered by the inherent economics of selling data for money instead of promoting open data and innovation.
The problems, of course, do not end once data is made public. Data quality remains an issue. From an organization being delivered the state budget in the form of scanned images from the national Gazette to spelling mistakes and missing information, the quality of government data remains a challenge. This in turn impacts the ability of the organizations to inspire reuse, which is ultimately needed to convince governments that the release of data is worthwhile and beneficial.
Improving data standards is a key feature that emerged from a number of our interviews. Expanding on this point, one of our interviewees commented that data sets remain difficult to interrogate due to the highly technical, complex form they are presented in, which hampers any meaningful understanding of them. This was supported by another interviewee who lamented that data sets are often incomplete and old.
Some regions, such as the African continent, are historically known to be burdened by issues of poor infrastructure and connectivity. In this regard, for open data to make a lasting impact to a greater population of these regions, information needs to be presented in innovative ways for those that are without internet access–in a form that would ensure their uptake and allow them to interact with the data. With this in mind, traditional means of communicating data to remote areas of the continent seem more viable than newer more technical methods that would only be able to impact an inconsequential portion of the population.
Many other respondents argued that public education and outreach are a critical components of open data, as citizens must know about the existence of this resource before it gains optimum public value.
VI. Opportunities for the Global Open Data Initiative
There is a good deal of excitement about the opportunities presented by a global open data initiative. The most frequently mentioned opportunity was building a stronger global open data community and encouraging networking amongst civil society organizations operating in different countries.
Many individuals encouraged GODI to foster relationships between a wide array of organizations to share knowledge and information in an area that is emerging quite rapidly. A large civil society alliance could offer the clout necessary to push for national agendas in countries around open data and freedom of information, and it could also help the reform agenda by articulating an open data solution that fits into the domain of transparency and create a feedback loop for accountability.
Connecting national level advocated to both the international conversation as well as to other national level advocates would also be valuable. Such a strong network can also help avoiding duplicating efforts and might improve coordination. Additionally, GODI can help push the open data agenda beyond the small community currently working on the issue. Journalists, campaigners, and policy advocates stand to benefit immensely from open data, and GODI can help convince this multi-disciplinary audience.
Most respondents agreed that the open data community would benefit immensely from a more clearly defined evidence base and theory of change associated with open data. Some respondents suggested that gathering anecdotes or cases where open data has been successful would be extremely helpful. Importantly, respondents also noted that the evidence base for open data must be diverse in terms of geography and the achievements of open data. We need proof that open data can be valuable in a variety of country contexts and for a variety of reasons (economic development, accountable government, more effective public sector management, etc.).
GODI was encouraged to look into the questions of differences between developing and developed countries in regards to data and collaborative ways of creating the correct metric for a theory of change.
As noted above, governments have plenty to learn and lots to gain when it comes to open data. Many respondents noted that GODI can be valuable by helping conduct government-wide outreach and education, and also by providing advocacy support.
Many respondents noted that GODI will be most effective if it remains transparent and participative.
A. Full reports from each of the GODI partners:
B. List of countries represented:
- Bosnia and Herzegovina
- Czech Republic
- El Salvador
- Slovak Republic
- South Africa