A few weeks ago, we posted a story about GAO selling exclusive access to GAO legislative histories to Thomson West (see "GAO *did* sell exclusive access to legislative history to Thomson West" and GAO subject for all GAO stories). This was a rich historical chunk of GAO information (20,597 legislative histories of most public laws from 1915-1995!!) and it was set to be locked up with T/W claiming exclusive rights and licensing access.

Well, not so fast. Carl Malamud, tireless hero of the public domain, got wind of the deal, and got the GAO to release 10 DVDs of legislative histories, containing 619,481 PDF files -- the pilot project scans they conducted. He has proposed a joint venture with the Internet Archive to scan the same materials with the same terms as Thomson West, give GAO one full copy of all their data AND put up the data online (presumably the Internet Archive) clearly marked as public domain material available for reuse without restriction. And what's more, Carl says, "If they say yes, we intend to ask Congress to earmark funds to pay the Internet Archive to scan this invaluable resource." !!

You can follow the paper trail on Carl's Federal Legislative History site. Below is the letter of unsolicited joint venture sent to GAO. Way to go Carl!!

A few weeks ago, Daniel had a great post, "GAO/Thomson-West Contract Raises Questions" in which he expanded on a Boing Boing post "Did the US gov't sell exclusive access to its legislative history to Thomson West?" and analyzed the Thompson-West contract with the GAO for digitizing 20,597 legislative histories of most public laws from 1915-1995. Today, Carl Malamud got an answer to his FOIA request to the GAO seeking access to the digitized images of those legislative histories. I'll let Carl tell it in his own words:

Well, the answer is now a definitive yes, that data has been sold down the river and is out to sea.
Public.Resource.Org sent in a FOIA request to GAO on this topic seeking access to the scanned data. Today's letter answering our FOIA request spells out the bad news. Turns out the GAO doesn't even get the data, they simply are given an account on Thomson's service. The rest of the government doesn't get access to this data, and the public is invited to stop by the GAO headquarters and pay 20 cents per page to copy paper.

This is one of those deals where the public domain got sold off ... GAO gets a bit of convenience by having their stuff scanned for them, but they gave up way more than they got in the deal, and the public (including government workers and public interest groups who need to consult this data) lost big-time.

Carl has put up his paper trail explaining the story. Here's the link to the Scribd group with the full paper trail on this issue, and here's the link to last week's response from the GAO.

This perfectly exemplifies the problems we see with government agencies entering into contracts with private companies to digitize public domain materials (see for example "NARA/TGN contract as a bad precedent"). We have no problem with government agencies contracting with private companies to digitize government information. The problem as we see it is that so many agencies seem ignorant of the fact that privatizing access to said digitized public domain information actually limits access in the long run.

Thanks to an alert from a dedicated but shy reader, our attention has been focused on a story on boing-boing titled Did the US gov't sell exclusive access to its legislative history to Thomson West? This story has links to documents relating to this deal requested by the redoubtable Carl Malamud. I took the time to read/skim through the contract documents and found this interesting section:

Taken from "Attachment A, Statement of Work" from the contract between Thomson West and GAO, posted at http://www.scribd.com/doc/2299358/Contract-Between-Thomson-West-and-GAO

Background: Since its inception in 1921, the US Government Accountability Office has compiled 20,597 legislative histories of most public laws from 1915-1995. These histories, spanning the 64th-104th Congresses, are currently being used onsite in the GAO headquarters Law Library in paper or microfiche format by GAO staff. On rare occasions other federal government employees are allowed onsite access to the paper or microfiche copies of these histories. Because of its historical and research value the legislative history collection shall be digitized to preserve the integrity of the files and improve the searchability of this valuable information resource.

Two years ago, GAO began a pilot project to convert a small number of GAO legislative histories from paper and microfiche formats to digital format. Since then 243 histories have been digitized using in-house resources and will be made accessible to GAO staff only through a web-based database on the GAO Intranet. The 243 histories consisting of 1,214,438 pages were randomly selected and include some of the largest histories in the collection. These histories shall also be re-scanned as part of this digitization contract.

This sounds like a major goldmine of information that really hasn't been shared with other parts of the government, let alone the public. It also sounds like GAO tried to do some of this work on its own but found it unviable. So left to itself, the information wouldn't contained in the paper files wouldn't be available to anybody. So I'm not surprised it went looking for a partner. But I am surprised and concerned that they went with a commercial partner when the GAO office is within driving distance of a number of major universities and when public-spirited organizations like the Internet Archive and Public Resource might have been happy to come up with a solution to provide this taxpayer-funded information at zero cost to the taxpayers and either zero or minimal costs to GAO. Conceivably there might have been some way for the Government Printing Office to incorporate this into GPO Access, although that certainly would have been at some cost to GAO unless Congress was willing to make an appropriation for this purpose. But any Congress that claims to be committed to strong public access should be willing.

Were alternatives to in-house digitization or wholesale privatization pursued? If not, why not?

Long time readers of FGI know that most government information is considered public domain and also subject to Freedom of Information Act requests. So what's to stop Carl, Internet Archive, or some other public minded group from exposing this rich trove of legislative histories to the public which were taxpayer funded to begin with? According to the GAO, plenty:

Taken from "Attachment A, Statement of Work" from the contract between Thomson West and GAO, posted at http://www.scribd.com/doc/2299358/Contract-Between-Thomson-West-and-GAO

FOIA Requirements: While GAO is not subject to the Freedom of Information Act (FOIA), GAO has regulations (4 CFR Part 81) that follow the spirit of FOIA. The paper or microfiche copies of the legislative histories (and possibly the PDF copies of the "GAO Materials" section) would be available for public inspection and copying. However, under GAO's public disclosure regulations, GAO charges a per page copy fee. Accordingly, any extensive copying would be expensive and the quality of the copies, for many of the histories would be poor.

I assume this was put into the contract to assure Thomson-West their investment would be secure from public-access zealots who have the idea that the American people should only be charged once instead of twice for government information. But the paragraph raises two important questions that I hope someone in Congress will ask GAO:

1) On what rational basis would you charge a per-page fee on the 1,214,438 pages that have already been digitized? Running a backup tape isn't the same as hand copying files. GAO should be directed to immediately release that database at zero cost unless they can carefully and believably document actual copying expenses including staff time. But a per page copy for PDF files isn't credible.

2) When GAO says "quality of the copies, for many of the histories would be poor", are they saying that the quality of copies would be poor just for the public or for Thomson-West as well? The first reading suggests a deliberate effort to sabotage no-fee public access, while the second reading suggests that Thomson-West customers will be paying a high price for lousy duplication. Neither option seems particularly fair.

I think I speak for all of us at FGI when I say that while digitization for greater access is a laudable goal, wholesale privatization without a careful, public examination of other, more citizen-friendly, alternatives is not acceptable. If you agree, please ask your Members of Congress to direct GAO to take a second look at this contract and facilitate no-fee access to this valuable set of legal materials.

I found this video on the Government Accountability Office website. It tells what GAO does through short news clips that appear in my total layman's opinion to be well within traditional fair use guidelines. As a video produced by the Government Accountability Office, an arm of the US Government, the video itself should be public domain.

Because of this analysis and because I think the GAO story is too good to stay locked up in a corner of its web site, I posted the video to YouTube, which is the embedded video here. For preservation purposes and just in case that YouTube doesn't agree with my analysis of the copyright status of the video, I also posted it to the Internet Archive, which is a fierce supporter of the public domain and fair use.

US auditor queries military Iraq casualty figures, Yahoo! News, Sep 7, 2007

An independent US government auditor on Friday cast doubt on US military statistics expected to show a huge dip in sectarian violence in Iraq under the current troop surge strategy.

Comptroller General David Walker said there was a "significant difference" of approach between the Government Accountability Office (GAO), which he heads, and Pentagon evaluations of violence in Iraq....

"We could not get comfortable with (the military's) methodology for determining what's sectarian versus nonsectarian violence," Walker told senators....

Walker was unable to go into further details, as the rest of the GAO's conclusions in the report on sectarian violence have been declared secret by the Pentagon, and urged senators to read the classified version of the study.

I've checked a few of my links to individual documents, topic search, and RSS feeds and they all worked, but you might want to verify your own.  Sometimes site re-designs can make links break badly.  They do have short and long versions of RSS feeds and they are available in RSS and ATOM format.  That is very nice! Thanks GAO!

GAO Launches Web Site Improvements To Enhance Visibility, Impact

WASHINGTON (August 20, 2007) -- The Government Accountability Office (GAO) today unveiled its newly redesigned Web site, www.gao.gov. The updated site makes the agency’s work easier to find, and it better explains what GAO is and what it does. New features on the homepage include a prominent dynamic display of GAO’s latest products, a streamlined “In the Spotlight” section, and a new “Key References” section with links for site visitors....

GAO anticipates further improvements to the site based on comments from its users and evolving technology tools. Furthermore, GAO products, including reports and testimonies, will continue to be posted daily, usually by early afternoon of the day they are released.