Stories

Council votes today on law to liberate (most of) New York City’s data

Intro 29A requires city agencies to make information available for use in websites and apps — but not until 2018

City Council member Gale Brewer (left), with Technology Committee chair Fernando Cabrera (right) at yesterday's vote on a measure opening city data to the public, expects a "technological revolution." Photo: Michael Keller

UPDATE: Intro 29A passed the full City Council on Wednesday afternoon by a 48-0 vote.

The City Council Committee on Technology passed a landmark measure yesterday that would require all public data maintained by city agencies to be published in a single online portal and made accessible to the public through application programming interfaces (APIs) and other digital formats. The measure, Intro 29A, will go before the full City Council this afternoon.

Currently, New York City government data is available on a scattering of websites in a range of formats, and much important data remains difficult or impossible for outsiders to analyze or republish. The NYC OpenData portal currently hosts several hundred datasets handpicked by city agencies for use by software developers, including those participating in the BigApps competition. Yet many crucial city information systems, such as the Department of Buildings’ online records of construction permits and code violations, remain off-limits to outside reuse.

In a prelude to the committee vote, the bill’s primary sponsor, Councilmember Gale Brewer, said the measure “will make sure New York is at the forefront of any government transparency and technological revolution in terms of government openness.”

Brewer thanked her “Tech Posse” of government technology and good government groups that worked with her office on the bill: Common Cause, Reinvent Albany, OpenPlans, NYPIRG, Citizens Union, DoITT, and Andrew Rasiej of Personal Democracy.

Intro 29A arrives in the council with strong support from the Bloomberg administration. “The proposed Open Data bill represents the most aggressive commitment by any municipality in the country to systematically categorize and ‘unlock’ public data sets,” said Carole Post, commissioner of the New York City Department of Information Technology and Telecommunications (DoITT). “We were pleased to work with the City Council, open government advocates and agencies citywide to codify the Bloomberg administration’s historic transparency gains.”

Intro 29 would phase in over the next six years, starting with a technical manual to be released by DoITT this summer guiding city agencies on how to format and upload their data. Agencies would have until early next year to upload all public datasets currently on the internet to the new portal. By 2018, all public data would have to be posted. For the time being, DoITT is planning to power the new data portal with Socrata, the service that currently hosts NYC OpenData.

DoITT has agreed to annually publish a complete inventory of all public datasets maintained by each agency starting next year. The technology agency defines “public” data as any information that is already published on the internet or could be obtained via a Freedom of Information Law request, says agency spokesperson Nicholas Sbordone. The council itself, as well as comptroller, Public Advocate and borough presidents, would also be covered by the measure, according to Brewer’s office.

The information agency would get an assist from an online public feedback forum required by the Council’s measure. Explained Brewer, “Researchers, journalists, people who care about this issue will be able to say to the agency, ‘Why aren’t you putting X dataset up?’ Or, ‘X dataset has challenges we’d like to work with you to fix them.’”

Yet city agencies would have many opportunities to exclude information that is too sensitive or technologically difficult to include in the open data portal. For instance, information that reflects an “internal deliberative process of an agency” is exempt. And agencies can miss the 2018 deadline as long as they submit an explanation of why a particular dataset can’t be made available.

It’s also not yet clear who will be responsible at each agency. The Transparency Working Group, an ad hoc network of open-government advocates, recommended to the council that the city and each agency employ a chief data officer to ensure information is managed and shared. Without such a point person, said task force member Steven Romalewski, director of the CUNY Mapping Service at the Graduate Center of the City University of New York, “I don’t seee that coordination happening.”

Romalewski and others also have questions about how effective the public feedback forum will be in compelling agencies to share up-to-date, ready-to-use data. Currently on NYC OpenData, users can contact or leave feedback for the “owner” of each uploaded dataset, but reports of errors or questions that have been posted to the site since it was launched last year have often remained unanswered in the portal’s comments section. Right now, said Romalewski, “The comments go into a black hole.”

Open-data technologists and advocates say they now want to make sure DoITT and future administrations follow through. “Transparency about the execution part of this is here the rubber will meet the road,” noted Andrew Hoppin, former chief information officer of New York State Senate and now partner and co-founder at technology consulting firm New Amsterdam Ideas. Hoppin called the bill a “big step” for a city that he says is great at collecting data for reports but not universally good at sharing the raw data underneath those reports. Intro 29A would make sharing that data the rule, not the exception.

Added Hoppin, “It’s going to put a mandate out to every city agency to publish their data in a very comprehensive, rigorous, sustainable long-term manner.”

Add a comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

3 Comments

  1. This bill is really impressive in scope and will certainly set NYC as a world leader in open data policies. I’m very excited to see this pass and an hopeful that it will be implemented as the bill describes.

    One small correction though: while the notion of “public” in this bill includes everything covered by Freedom of Information Law, the definition of “data” is a bit more narrow in scope. This bill certainly doesn’t include all digital information or information that could be digitized under FOIL.

  2. Thanks Philip. What would digital information that could be digitized under FOIL look like? Is that referring to a dataset’s metadata or more-so documents and text data that can’t be quantified in a spreadsheet?

  3. Intro 29A calls for regularly updated quantitative data to be put online in a single place. It does not require data sets that were created for a one-time study or analysis be posted. For instance, a database of an annual street tree census would have to be put in the central data site, but not a file derived from that created for a report on trees in Morningside Heights. The bigger point is that this bill empowers DOITT to push other agencies to put their data online. This is something DOITT currently uses the Big Apps competition to do.