UPDATE: Intro 29A passed the full City Council on Wednesday afternoon by a 48-0 vote.
The City Council Committee on Technology passed a landmark measure yesterday that would require all public data maintained by city agencies to be published in a single online portal and made accessible to the public through application programming interfaces (APIs) and other digital formats. The measure, Intro 29A, will go before the full City Council this afternoon.
Currently, New York City government data is available on a scattering of websites in a range of formats, and much important data remains difficult or impossible for outsiders to analyze or republish. The NYC OpenData portal currently hosts several hundred datasets handpicked by city agencies for use by software developers, including those participating in the BigApps competition. Yet many crucial city information systems, such as the Department of Buildings’ online records of construction permits and code violations, remain off-limits to outside reuse.
In a prelude to the committee vote, the bill’s primary sponsor, Councilmember Gale Brewer, said the measure “will make sure New York is at the forefront of any government transparency and technological revolution in terms of government openness.”
Brewer thanked her “Tech Posse” of government technology and good government groups that worked with her office on the bill: Common Cause, Reinvent Albany, OpenPlans, NYPIRG, Citizens Union, DoITT, and Andrew Rasiej of Personal Democracy.
Intro 29A arrives in the council with strong support from the Bloomberg administration. “The proposed Open Data bill represents the most aggressive commitment by any municipality in the country to systematically categorize and ‘unlock’ public data sets,” said Carole Post, commissioner of the New York City Department of Information Technology and Telecommunications (DoITT). “We were pleased to work with the City Council, open government advocates and agencies citywide to codify the Bloomberg administration’s historic transparency gains.”
Intro 29 would phase in over the next six years, starting with a technical manual to be released by DoITT this summer guiding city agencies on how to format and upload their data. Agencies would have until early next year to upload all public datasets currently on the internet to the new portal.By 2018, all public data would have to be posted. For the time being, DoITT is planning to power the new data portal with Socrata, the service that currently hosts NYC OpenData.
DoITT has agreed to annually publish a complete inventory of all public datasets maintained by each agency starting next year. The technology agency defines “public” data as any information that is already published on the internet or could be obtained via a Freedom of Information Law request, says agency spokesperson Nicholas Sbordone. The council itself, as well as comptroller, Public Advocate and borough presidents, would also be covered by the measure, according to Brewer’s office.
The information agency would get an assist from an online public feedback forum required by the Council’s measure. Explained Brewer, “Researchers, journalists, people who care about this issue will be able to say to the agency, ‘Why aren’t you putting X dataset up?’ Or, ‘X dataset has challenges we’d like to work with you to fix them.’”
Yet city agencies would have many opportunities to exclude information that is too sensitive or technologically difficult to include in the open data portal. For instance, information that reflects an “internal deliberative process of an agency” is exempt. And agencies can miss the 2018 deadline as long as they submit an explanation of why a particular dataset can’t be made available.
It’s also not yet clear who will be responsible at each agency. The Transparency Working Group, an ad hoc network of open-government advocates, recommended to the council that the city and each agency employ a chief data officer to ensure information is managed and shared. Without such a point person, said task force member Steven Romalewski, director of the CUNY Mapping Service at the Graduate Center of the City University of New York, “I don’t seee that coordination happening.”
Romalewski and others also have questions about how effective the public feedback forum will be in compelling agencies to share up-to-date, ready-to-use data. Currently on NYC OpenData, users can contact or leave feedback for the “owner” of each uploaded dataset, but reports of errors or questions that have been posted to the site since it was launched last year have often remained unanswered in the portal’s comments section. Right now, said Romalewski, “The comments go into a black hole.”
Open-data technologists and advocates say they now want to make sure DoITT and future administrations follow through. “Transparency about the execution part of this is here the rubber will meet the road,” noted Andrew Hoppin, former chief information officer of New York State Senate and now partner and co-founder at technology consulting firm New Amsterdam Ideas. Hoppin called the bill a “big step” for a city that he says is great at collecting data for reports but not universally good at sharing the raw data underneath those reports. Intro 29A would make sharing that data the rule, not the exception.
Added Hoppin, “It’s going to put a mandate out to every city agency to publish their data in a very comprehensive, rigorous, sustainable long-term manner.”