I'm reminded of this recent post by James Fee, talking about geodata, but I think it applies to the general case:
"... The question was APIs or downloads...
Personally, I believe [data] is one of the best
ways for citizens to keep track of their government
(local to federal) ... APIs tend to deliver what
their “owners” want them to do. Raw data means
everyone has an opportunity to check each other’s
work. Of course, raw data can be manipulated as
well, but it is harder to obscure."
I couldn't agree more. APIs are great, but are not the key to open government, for two reasons:
1. They don't provide simple and easy access for non technical individuals into raw information.
APIs shouldn't exist for querying historical datasets if the dataset is not already available in a static format. Release the data, then build an API if there is demand (or the private sector doesn't do it, better, for you).
2. Historical data access is poorly served by APIs.
There is no such thing as a good 'general use' API[1]. API's are appropriate for specific service based transactions that involve some level of processing. Examples:
3. Bonus reason: government agencies suck at building APIs.
They're not good at determining what is genuinely high value to end users, they tend to prefer visible projects that can justify budget increases, over genuinely useful, but less easily communicated ones (cf. the US national highway system and pork barrel politics), and there is an entire industry of enterprise companies heavily invested in keeping it this way.
TL;DR Release the data, let users build the APIs. Everyone wins.
Bootnotes:
[1] I lie. That's exactly what publishing raw data at stable URLS on a website achieves.
The exciting thing about APIs isn't what they directly do for non-technical individuals.
Instead, a good API makes government into a platform for free (and paid) services to be built to deliver that data in innovative ways. Examples of this are starting to appear in places like Chicago which has opened up a lot of data access – for things including transit (bus tracking, etc.) and a lot more. Giving hackers platforms to innovate will definitely yield better results than just throwing gobs of data at the general public. (Never mind that not all raw data is created equal or that raw data also requires savvy people to distill.
It also means that it's potentially going to be easier for one unit of government to interact with (or at least query) another. That may be big as well.
Whenever you put the word 'government' and 'innovation' into the same sentence, you need to check your working. Government is, by and large, terrible at technical innovation. And innovation by mandate will, predictable, be a non-starter.
That said, there are areas where it is appropriate - see my earlier examples (tranit bus feeds are another one) - where the relevant government agency has found a niche that (a) only they can provide the data, and (b) it is a very easily defined problem that they're solving.
Finally, I really take issue with this statement:
> "giving hackers platforms to innovate will definitely yield better results than just throwing gobs of data at the general public."
Utter rubbish. The "general public" is not stupid. The general public includes huge numbers of people with the domain knowledge and wherewithal to analyse the data. To claim that open and transparent government is better served through an elite technical class, instead of directly to individuals is simply false.
Perhaps I'm misinterpreting your use of the term technical innovation, but weren't the Manhattan Project, NASA and many military projects innovation by mandate that were extremely successful?
It's quite one thing to form agencies with specific science and engeering goals (Manhatten Project, NASA) and populating them with experts and billions of dollars, quite another to expect the same results from other departments without that background.
And even then: the progress that SpaceX has made in the last few years is an almost text-book proof of my point. NASA provided SpaceX with a wealth of knowledge and experience that has been very poorly utilized (dollar for dollar) over the last 20 years. Within a very short time-frame we've suddenly seen innovation in the space sector like we haven't seen since the Apollo programme.
As for the military sector - given that maintaining a monopoly over the application of military force is (fundamentally) the primary function of government, it's not surprising that this is the one area of government that has understood it's place for a very long time: issuing contracts with stated operational aims and leaving the private sector to provide the innovation. Yes, the system is flawed - the F-35 programme is a bit of a disaster - but even then it can be at least partially blamed on government agencies interfering with the procurement process.
Yes, but it doesn't take lots of experts and billions of dollars to build an API, I would think.
And yes, we've seen tremendous innovation from SpaceX - but where would they be without that knowledge and experience from NASA? Just because NASA has been turned into a bureaucratic mess with funding that bounces around doesn't mean they haven't created a lot of value and new knowledge, most of which would never have been funded by the private sector. Government work isn't about having good returns on money spent, which strikes me as both a blessing and a curse.
I wonder how many current day innovations have sprung from the initial work of places like the DOE National Labs, and how many might in the future, for all that it's a government program.
> Yes, but it doesn't take lots of experts and billions of dollars to build an API, I would think.
Refer (3) in my original top post. This is something they would prove remarkably adept at achieving.
> ...but where would they be without that knowledge and experience from NASA?
Exactly! A thousand times so! Releasing their knowledge and experience that only a government funded entity could have amassed (c.f. raw data), has sling-shot a private enterprise capable of rapid innovation. This is precisely why I think that releasing the raw data is the most beneficial outcome.
Not really. No appointed official woke up one morning and thought "let's make an atom bomb". A bunch of scientists came to the govt and said, "we can make an atom bomb, we just need the funding".
I think this is more about changing the mindset of the typical workers at the federal agencies (hundreds of thousands of whom are not doing IT related tasks as their primary function). Those of us who work for these agencies as programmers & IT people have a big job to do.
Only recently have the govies started thinking about how their data can be useful to the general public. In the past everything had been stovepiped and guarded with peoples' live(lihood)s. Hopefully this makes them think about data integrity throughout the life of that data.
Think about how data used to be provided to the public before. A bunch of government folks had to collect data and make sense of it themselves and put it together in a report destined for congress. It's waaaaay different to just provide that raw data to the public. Rather, I think what we'll see is more sanitized data sets, after they've been internally analyzed and vetted (probably multiple times). Not exactly transparent.
But I hope one day, after many iterations of API building, we'll get to a point where the data truly is transparent.
> Only recently have the govies started thinking about how their data can be useful to the general public
The problem is that they can't anticipate how their data is most useful. Release and let the users decide.
> But I hope one day, after many iterations of API building, we'll get to a point where the data truly is transparent.
So, we should wait many years while, predictably, large budgets are blown on faltering "government innovation" projects, when we could just have the data and use it today?
LOL I think we both want the same thing. I'm just being realistic. I highly doubt you'll get all the data you really want, this year or next. Federal agencies have a hard time releasing data quickly. Too many checks and balances, too many people blocking, too many lawyers and unions, too many politicians.
I couldn't agree more. APIs are great, but are not the key to open government, for two reasons:
1. They don't provide simple and easy access for non technical individuals into raw information.
APIs shouldn't exist for querying historical datasets if the dataset is not already available in a static format. Release the data, then build an API if there is demand (or the private sector doesn't do it, better, for you).
2. Historical data access is poorly served by APIs.
There is no such thing as a good 'general use' API[1]. API's are appropriate for specific service based transactions that involve some level of processing. Examples:
3. Bonus reason: government agencies suck at building APIs.They're not good at determining what is genuinely high value to end users, they tend to prefer visible projects that can justify budget increases, over genuinely useful, but less easily communicated ones (cf. the US national highway system and pork barrel politics), and there is an entire industry of enterprise companies heavily invested in keeping it this way.
TL;DR Release the data, let users build the APIs. Everyone wins.
Bootnotes:
[1] I lie. That's exactly what publishing raw data at stable URLS on a website achieves.