The National Security Archive at George Washington University announced its new blog today: Unredacted: The National Security Archive, Unedited and Uncensored. The announcement says:

The National Security Archive is pleased to open its virtual doors with a new behind-the-scenes blog, "Unredacted: The National Security Archive, Unedited and Uncensored,". The Archive's own experience with thousands of Freedom of Information Act and Mandatory Declassification Review requests provides a wealth of data and fundamental lessons that we hope to share with you.

"Unredacted" will highlight never before publicly seen government documents as part of our regular "Document Friday" series. The blog will feature commentary and analysis from our experts on current news stories, events, ongoing litigation and advocacy efforts, newly-released documents, and other hot topics. We will regularly highlight some of our top document collections -- including unpublished collections donated by top journalists and authors -- that are available to researchers and the public.

The new blog will also tell you more about the Archive's global activities, including reports from the field as Archive staff travel to document archives around the world, assist international courts and tribunals with human rights cases, support efforts to enact and implement freedom of information laws in other nations, and attend meetings and conferences with other NGO representatives and high-level government officials.

John Gilmore is an open software proponent, co-founder of the Electronic Frontier Foundation and perhaps most importantly an Archive-It partner (as an independent researcher). His Archive-It collections focus on open access to government information and policy as well as free and open source software.

John has been archiving sites related to wiretapping and the National Security Agency since 2007. Describing the reasons for creating this collection, John says:

"I'm trying to record and make searchable some documents related to the controversy over NSA wiretapping domestically without warrants, or with general warrants, which the Fourth Amendment outlaws. " 

This collection demonstrates how the recent change in administration has opened up further crawler access to the National Security Agency (NSA) website. Previously, most NSA web content was blocked to the Archive-It crawler (as well as other crawlers) using the robots.txt exclusion protocol. Looking at their old exclusion list, for example this one from 2008 you can just how much of their website was blocked from crawler access. (all the directories listed could not be accessed).

Since January 17, 2009 however crawlers have access to much more content.

At the Internet Archive, we have noticed similar changes in other .gov websites including www.whitehouse.gov (compare this version from 2006 to the current exclusion list).

Its exciting to know that moving forward John and other Archive-It partners will be able to collect more complete snapshots of government websites.

-Molly and Lori

From the AP article: Government guards papers from public eye

More than 1 million pages of historical government documents -- a stack taller than the U.S. Capitol -- have been removed from public view since the September 2001 terror attacks, according to records obtained by The Associated Press. Some of the papers are more than a century old. In some cases, entire file boxes were removed without significant review because the government's central record-keeping agency, the National Archives and Records Administration, did not have time for a more thorough audit. "We just felt we couldn't take the time and didn't always have the expertise," said Steve Tilley, who oversaw the program. Archives officials are still screening records, but the number of files pulled recently has declined dramatically, he said.

Update 1/9/2007 - I uploaded a scan of pages 390-399, plus title page and table of contents for book discussed below.

------------
One of the great things about living in a country that still has a significant commitment to government openness are publications which admit past mistakes on the government.

A case in point is the book The Role of Federal Military Forces in Domestic Disorders, 1945-1992 by Paul J. Scheips and published by the Army's Center of Military History's Army Historical Series.

While it is mostly profiles how the Army and National guard performed well during the civil rights movement, there is a chapter dealing with political surveillance (pages 390-399).

The book mentions the scale of the data collection and types of data gathered by the Army (p. 395):

The great danger of such data lay in the fact that "Army intelligence, in the name of preparedness and security, had developed a massive system for monitoring virtually all political protest in the United States." Military agents had even assembled private information about the finances, psychiatric records, and sex lives of individuals. The data collection was enormous, with "virtually every stateside unit" having its own set of files. Fourth US Army headquarters at Fort Sam Houston, Texas, for example, had the "equivalent of 100,000 file cards on "personalities of interest," and the III Corps at Fort Hood had computerized data on civilian political groups within its area. The size of the data banks demonstrated that the Army's domestic intelligence operations had been going on "in various degrees of intensity, since 1940," with roots extending back to World War I.

It also provides an evaluation of the surveillance in terms of effectiveness and it's harm to a democratic society (p.399) [Emphasis mine]:

Overall, the Army's venture into domestic surveillance generated a substantial backlash in the form of public and congressional criticism, while contributing little or nothing to the suppression of civil unrest. In some ways the Army's tendency to apply foreign intelligence-gathering methods to domestic situations reflected its earlier error of applying the standards of foreign war to the control of domestic disturbances. The basic problem was a mind-set in the intelligence community that saw conspiracy in protest and the threat of revolution in disorder. It was this way of thinking that led to an improper gathering and storage of a great quantity of information on American citizens that was seriously at odds with the tenets of a democratic society. In doing this, the Army failed to maintain a constitutional discipline over itself, and the civilian leadership was much too slow in calling it to task. It was a dark chapter in the Army's history of dealing with civil disturbances, which otherwise had been quite good during this troubled decade.

Remember, this isn't Tom Hayden or some loony lefty talking. This is the judgment of someone writing for the Army's own Center of Military History. This doesn't make it the official opinion of the Army, but they felt the book was worthwhile to publish.

Ironically, the rise of networked computing allows the construction of databases far more intrusive than the ones denounced in this book.

The book is also well worth reading for the constructive role that the Army played in desegregation. Overall, I think Mr. Scheips gives the Army good marks for their role in the 1960s, aside from their unlawful surveillance activities.

This book is a third in series. The title, author and SuDoc information for the set are:

1. The role of federal military forces in domestic disorders, 1789-1878 / 1988 Coakley, Robert W. D 114.19:R 64
2. The role of federal military forces in domestic disorders, 1877-1945 1997 Laurie, Clayton D. D 114.19:R 64/2
3. The role of federal military forces in domestic disorders, 1945-1992 / 2005 Scheips, Paul J. D 114.19:R 64/2/2005

As far as I can tell none of these books are on the Internet, but you can find them in many Federal Depository Libraries. Read them and then decide if we really want to grant the executive branch unlimited authority to watch our lives. We say no.