What features do you want for your catalog of govt data?

Data is definitely getting sexy. Jonathan Gray of the Open Knowledge Foundation asks "What features should be included in a catalogue of open government data?" and points to a few other data repositories being built on the state and country level (like my own city of SF's CivicDB!). He also mentions the Sunlight Foundation's plan to build on and expand data.gov with a national data catalog that I had meant to write about a couple of weeks ago (go Sunlight!). So I'm putting this question to you, FGI's faithful readers. I'm sure you'll have a thing or three to add to the list of requirements for open data catalogs.

Here are a few suggestions for those building catalogues for (open) government data based on our experience developing CKAN:
  • Make the catalogue itself open!
  • Let others download the catalogue data in bulk (not just via an API)
  • Include information on how to get the data, and how it can be used
  • Make it versioned!
[originally tweeted by @EllnMllr. Go ahead and follow her!] Continue reading →

Continue Reading →

NARA’s draft 2009 strategic plan

The National Archives (NARA) has just released a draft of their 2009 strategic plan entitled, Preserving the Past to Protect the Future. It details NARA's goals for the next 10 years and offers an interesting window into the agency's focus. They are asking for public feedback by August 5, 2009 via e-mail (vision@nara.gov) or by fax (301) 837-0319. Especially read the section, "how will we know we have succeeded." Some of the benchmarks for success are achievable and admirable (i.e., "Within 30 days of the end of an administration, 100 percent of Presidential and Vice Presidential materials have been moved to NARA locations or NARA-approved facilities." and "By 2014, 100 percent of NARA records center holdings are stored in appropriate space.") but it's glaring that the first item of "success" is that "by 2016, 50 percent of agencies achieve passing scores for compliance with Federal records management policy." That seems like an extremely low benchmark to this non-archivist. Shouldn't ALL agencies already be compliant?

  • Goal 1: Our Nation’s Record Keeper
  • Goal 2: Preserve and Process
  • Goal 3: Electronic Records
  • Goal 4: Access
  • Goal 5: Civic Literacy
  • Goal 6: Infrastructure
...And the increasing demand by users for more online access to records has driven us to prioritize the processing of a backlog of records and to focus on increasing our digitization efforts. Our holdings in all media—paper, electronic, film, and so on—are constantly growing as the proliferation of Government records continues. At the same time, the public demand for access is increasing. NARA has a significant backlog of unprocessed holdings that are therefore not yet readily available to the public. New records are arriving faster than they can be processed. Clearly, we must refocus attention and resources on making as many records as possible accessible to the public. While we put great effort into readying records for public use, the expectation of easy online access to our holdings continues to grow. The American public expects information to be delivered almost instantly to their desktop with more and more information available via the Internet. However, we cannot provide online access to all our holdings in the next decade—the task is simply too big. Our focus must be on making our most requested holdings available online, and on providing researchers with online tools to help them in their work. Because of work done every day in NARA facilities across the nation, the public can examine the records that document the actions of Government officials, the entitlements of individuals, the events that make up American history, and in some instances, how those events have affected the rest of the world. With new social media applications (Web 2.0), we have the opportunity to reach the public with tools they already use. These tools allow us to operate more transparently and to interact with our users in new, more informal ways. Our primary response to the challenge of authentically preserving electronic records and providing access to these records in the future is the Electronic Records Archives (ERA). ERA will provide a comprehensive, systematic, and dynamic means of preserving and providing continuing access to authentic electronic records over time. When fully deployed, ERA will give us the means to preserve, process, and provide continuing access to Federal Government electronic records of archival value. Yet, at the same time, we must explore ways to provide economical storage and retrieval services for electronic records that remain under the legal control of the originating agencies. We must also continue to adapt our internal business processes and staff expertise to effectively use the ERA system and to ensure that it evolves over time to fit tomorrow’s technology. The Internet has introduced countless researchers to the holdings of the National Archives. This thirst for online access to information is exciting, as it brings more and more people to our virtual doors, yet the task of building an “archives without walls” is daunting. Implicit in this task is the ability for archivists and the public to virtually interact. Our strategies in this area will focus on developing dynamic partnerships to digitize and deliver our most popular holdings and create interactive tools to help researchers find the information they seek. By putting our country’s records literally at the public’s fingertips, we will be able to share our rich resources with more people than ever before.
[Thanks to John Wonderlich and the Open House Project for pointing this one out!] Continue reading →

Continue Reading →

National Data Catalog from Sunlight Labs

Our friends over at Sunlight Labs have announced a new project they call that "National Data Catalog" that will build on data.gov:

Today I’m happy to announce Sunlight Labs is stealing an idea from our government. Data.gov is an incredible concept, and the implementation of it has been remarkable. We’re going to steal that idea and make it better. Because of politics and scale there’s only so much the government is going to be able to do. There are legal hurdles and boundaries the government can’t cross that we can. For instance: there’s no legislative or judicial branch data inside Data.gov and while Data.gov links off to state data catalogs, entries aren’t in the same place or format as the rest of the catalog. Community documentation and collaboration are virtual impossibilities because of the regulations that impact the way Government interacts with people on the web. We think we can add value on top of things like Data.gov and the municipal data catalogs by autonomously bringing them into one system, manually curating and adding other data sources and providing features that, well, Government just can’t do. There’ll be community participation so that people can submit their own data sources, and we’ll also catalog non-commercial data that is derivative of government data like OpenSecrets. We’ll make it so that people can create their own documentation for much of the undocumented data that government puts out and link to external projects that work with the data being provided.
If you're interested in helping out on this effort, please join the National Data Catalog Google Group at http://groups.google.com/group/datacatalog?lnk=gcamv. Continue reading →

Continue Reading →

The Power of Versioning: Climate Change Bill

Our friends at Open Congress recently provided a concrete example of the benefit of being able to work with government provided data. In a July 1, 2009 blog posting titled See all the Last-Minute Changes to the Climate Change Bill blogger Donny Shaw notes:

We may never get the details of the back-room negotiating that took place leading up to the bill’s passage in the House on Friday, but with OpenCongress’s legislative versioning tool we can see exactly what was changed in the bill in the process and then start to figure out why. Just go to the text of the bill as passed by the House and select “Show Changes.” You can scan the entire bill and see, with color-coded text, exactly what was changed – red, stuck-out text denoting changed or removed sections in the bill, and green text denoting sections that were inserted or modified.
Donny spent about 30 minutes scanning through the bill's changes and documented what he found. What can you find? This sort of quick work at finding rush changes is only possible because copyright-free federal legislation is available to transparency organizations like OpenCongress to put into their change revision software. This gives regular citizens specialized access to legislation that was formerly only available to subscribers to expensive premium services. This is a good thing. The Government Printing Office's talks with the Library of Congress about bulk distribution of legislative data will only make things easier. Continue reading →

Continue Reading →

Eleven Great Sources of Government Data Sets to View in Google Earth

One great way to get your head around a large government dataset is to view it using Google Earth. I went on a hunt for the most interesting, striking and geography based government data sets currently available in the KML format used by Google Earth. There is a large gallery of tours and layers available from Google Earth's site, including some based on government data - but I wanted to look beyond them. Here are eleven data sources (in no particular order) that have KML files ready and waiting for you to download. For some of these you will need to read the instructions associated with the KML to understand what you are looking at and what special features are enabled. Some have multiple datasets within a single KML file -- others include animations. Often when you open them in Google Earth they will start out with either a helpful note or a built in graphical key.

Have a favorite KML formatted government data set I missed? Please share it in the comments. I found many of these by starting in Goggle's US Government Search and searching for Google Earth. Continue reading →

Continue Reading →

Archives

Powered by WordPress / Academica WordPress Theme by WPZOOM