Today, the White House Web site (whitehouse.gov) switched to the open source Drupal platform -- the same software running FGI! I'm glad they made the shift. It's one thing to talk about transparency the way the Obama administration has done, it's another to use tools imbued with openness and transparency in order to get to that goal.

White House opens Web site programming to public
By PHILIP ELLIOTT
The Associated Press
Saturday, October 24, 2009

[tip 'o the hat to Chris Messina!]

The Recovery Accountability and Transparency Board released an upgraded version of the Recovery.gov website on Monday, September 28. Recovery.gov is, per the website, "the U.S. government’s official website providing easy access to data related to Recovery Act spending and allows for the reporting of potential fraud, waste, and abuse." The site now has a zip code search for finding local Recovery Act awards, a Data Download section, and a new home page layout with more information upfront.

The reviews of the recent upgrade are out and can be summed up as "meh." The conclusion from interested bloggers seems to be that while a few improvements have been made around the edges, there is little new to shout about. Observers are waiting for the real show, the scheduled October 15 release of the first recipient contract data and October 30 release of the first recipient grant and loan data. From the blogs:

Meet the New Recovery.gov, "(mostly) the same as the old Recovery.gov", from OMB Watch Blog, September 28.

New Recovery.gov Goes Live, Key Data to be Released Later, from WSJ.com Washington Wire, September 28.

Grading the New Recovery.gov, a substantial review from Sunlight Labs, September 29.

Meanwhile, CRS librarians have updated their compilation of links to Recovery-related information on the web in this report available from OpenCRS.com: Authoritative Resources on the American Recovery and Reinvestment Act of 2009 (ARRA), updated September 10.

It is no surprise that plenty of people have been saving up their big announcements for the Gov 2.0 Summit. The OpenID Foundation, for example, used the conference venue to announce that they will be collaborating with the U.S. General Services Administration to test OpenID at several federal agencies, including the National Institutes of Health. For more information, see:

• OpenID press release, 9 September 2009
• OpenID and Open Government home page
• OpenID in Government FAQ
• Open Identity for Open Government Explained, from Identity Woman blog

Department of State Office of the Historian has just released the redesign of its site: www.history.state.gov. They've done a really nice job with the redesign including new and easier access to my favorite Foreign Relations of the United States. Users can now browse FRUS by themes like decolonization, instability in Latin America, US-China trade etc (though I'm surprised that there's no theme for Cuban Missile Crisis, Vietnam, SALT etc. Perhaps they'll add those additional themes). Users can also browse by country to find history of US diplomatic relations and links to other key publications like Department of State Background Notes, Department of State Country Information, CIA World Factbook, and Library of Congress Country Studies.

The new website boasts greater accessibility and searching within the Foreign Relations of the United States documentary series. It currently offers both textual and facsimile copies of Foreign Relations volumes from the Kennedy Administration through the Nixon-Ford administration. The Office plans to continue to digitize older volumes and eventually house all of the Foreign Relations volumes on its website. The website also contains updated sections on the history of the Department of State, biographies of notable diplomats, and an in-depth timeline of United States diplomatic milestones. The Office’s educational curriculum guides are also downloadable from the website. The Office hopes that through its enhanced presentation and organization, the new website will become the preeminent online resource for U.S. diplomatic history.
--Source: U.S. Department of State

[Thanks Resource Shelf!]

ProPublica.org has launched ChangeTracker, a new tool that watches pages on whitehouse.gov, recovery.gov and financialstability.gov, "so you don’t have to"!

When the White House adds or deletes anything— say a blog post, or executive order—ChangeTracker will let you know.

The latest changes are listed on the ChangeTracker website, or you can sign up to get alerts via their RSS Feed, Twitter, or email.

They also have a guide to show you how to make a tracker for your own website.

Why Are Federal Web Sites So Bad? Allan Holmes is prompted to ask that rhetorical question in Tech Insider (01/22/09) after reading Megan Mcardle's blog entry for The Atlantic, Whitehouse.gov gets a makeover.

There's more on the same subject By Alyssa Rosenberg at GovExec: Beefing Over IT (January 21, 2009).

Mcardle complains that although the new OMB looks much better than the old one, "that sleekness has been achieved by tucking even more of that unsightly information out of the way, where it won't mar the vista." Rosenberg wisely notes that cost is one problem -- there just isn't enough money to do everything. But she also says, "Beyond the issue of cost, different agencies require very different IT investments." In other words, different agencies have different missions and priorities.

In the context of long-term, free access to government information, it is important to mention that few if any have preservation or long-term access as a mission. Search for "preservation" in the GPO Access Act sometime and see how sanguine you feel about GPO replacing libraries for long-term access.

Rosenberg also sees the scale of the problem contributing to the difficulties users face:

That's not to say that government couldn't be a lot better. Follies like the Census Bureau's wasteful handheld contract or the failed Office of Personnel Management retirement calculator contract are a big drain on resources and bad for the government's reputation. Websites crash under strain. Websites are poorly designed (though that's often more a matter of aesthetics or IT) or poorly explained. But given the magnitude of the challenges, it's amazing government IT is in the state it's in.

I think it worth reading these and the comments they get just to get an idea of the different expectations that people bring to government web sites. Few of them address the Big Questions directly, but, in aggregate, they all do. Most people focus on "Why is it so hard to find the information I want?" and "Why is everything so complex (or ugly, or pretty but uninformative, or...)?

The Big Question, I think, is Who Should Be Responsible for What? Holmes in noting the hard-to-find problem ("[I]t is indeed vary difficult to find many government documents and the most sought after data agencies collect") refers to one of two papers that address the big question extensively, if not yet definitively:

• Brito, Jerry. Hack, Mash, and Peer: Crowdsourcing Government Transparency. Mercatus Center at George Mason University, October 24, 2007. (Also published in the Columbia Science & Technology Law Review Vol. IX, May 2008.)
• Robinson, David, Harlan Yu, William P. Zeller, and Edward W. Felten. Government Data and the Invisible Hand. SSRN, May 28, 2008. (Also in Yale Journal of Law & Technology, Vol. 11, 2008)

Brito says:

If government data is made available online in useful and flexible formats, citizens will be able to utilize modern Internet tools to shed light on government activities.... Today, however, the state of government’s online offerings is very sad indeed. Some nominally publicly available information is not online at all, and the data that is online is often not in useful formats. Government should be encouraged to release public information online in a structured, open, and searchable manner.

And Robinson takes this one step further:

Rather than struggling, as it currently does, to design sites that meet each end-user need, we argue that the executive branch should focus on creating a simple, reliable and publicly accessible infrastructure that exposes the underlying data.

Another way of saying this is what we at FGI have long advocated: Governments should see their primary role as instantiating the information in usable and re-usable formats, and announcing, releasing, and distributing that information. If agencies can also afford to create usable web sites, that's fine. But, if we can't rely on agencies to make their information findable and usable today, it is even less likely that we will be able to rely on all agencies always keeping everything we want online. At some point, we will either lose information or the information will be privatized or otherwise no longer available without fees. Agencies should make sure the information is available first and worry about their web sites second.

As the inauguration ceremony begins tomorrow, we can be assured that the Library of Congress and other partners in the End of Term Harvest project have captured much of the Bush administration's online presence. Many of these websites will be re-captured at later dates, providing an interesting look at how these websites will change over time, through different administrations.

On a related note, there will undoubtedly be changes in the coming days, weeks, months, that will eliminate some government agencies. We are trying to archive as many of these "dead" websites as possible in the CyberCemetery, to preserve them in their final form.

Please, if you know of a website that is disappearing, email or call me. I'm keeping my eyes and ears open, but there is a lot of content out there, and I welcome your help. After all, this information is for all of us!

Thanks, and I wish you all joy as we witness history tomorrow.

Hi to all you FGI readers! I'm thrilled to be this month's guest blogger.

As we all watch this historic presidential election unfold, there's another question going on in the back of our minds--how much of this online government information is going to change with the new administration, regardless of who's sworn in next January? As someone who works specifically with digital government collections, and whose primary job is capturing defunct government websites, this is of particular interest.

Most of you already know about the "Dot Gov Crawl" project that's been organized to address this issue. The project partners include the Library of Congress, the Internet Archive, the California Digital Library (CDL), the University of North Texas (UNT), and the U.S. Government Printing Office (GPO). We're working collaboratively to harvest and preserve government websites (primarily .gov and .mil domains), to form a snapshot of digital government information at the end of the current presidential administration.

The Internet Archive will be performing the comprehensive crawl, and Library of Congress is focusing on congressional materials. CDL and UNT will be performing in-depth harvests of specific government websites, gathering documents linked deep within the websites that may not be gathered in the Internet Archive crawl.

I encourage you to participate in the project. Communicate with the partner institution closest to you, and let them know if there are specific websites (or portions of websites) that are of particular interest to you.

At UNT, we're trying to focus on documents that support our regional interests, things that might be overlooked in the kinds of sweeping national topics that will be handled by the Internet Archive. We're requesting that librarians for the central United States send us things that you want captured--websites you use often, publications deep within websites that might not be captured in large crawls, topics of regional interest. Your requests will help us identify and prioritize the information that is preserved for future generations.

Please, submit any suggestions you have in the comments section below--I'll be monitoring them and adding them to our list. Thanks for your input!

Four U.S. States Partner with Google to Improve Public Access to Government Web Sites  

Last year we posted a note on ResourceShelf about the “2004 Presidential Term Web Harvest” containing more than 75 million .Gov and .Mil web pages, equal to about 6.5 terabytes of data. It's a project of NARA and The Internet Archive. The archived sites can be browsed or keyword searched.

Now available is the 109th Web Harvest.
What does it contain?
+ More than four million pages (42 GB) crawled and archived between 11/11/06 and 12/11/06
+ Browse by Members Name
+ Browse by Committee Name
+ Browse by Leadership
+ Browse by House or Senate Organizations

Go to: http://www.webharvest.gov/collections/

The harvest produced a public reference copy of the web sites for the purpose of continual availability to the public, and also produced a record copy to be retained in the holdings of NARA…Web sites included in the harvest were identified from information provided by the Web Systems Branch of the House Information Resources staff and by Senate webmasters in the Offices of the Secretary of the Senate and the Sergeant at Arms.

A bit more on ResourceShelf including a comment by Librarian of Congress, James Billington, about the average lifespan of a web site.