NARA Cannot Assure Complete Transfer of Bush Records, by Steven Aftergood, Secrecy News, January 5, 2009.

Steven has a link to “National Archives Oversight: Protecting Our Nation’s History for Future Generations,” hearing before the Senate Committee on Homeland Security and Governmental Affairs, May 14, 2008 and comments on the the integrity of the process of transferring the records to the National Archives.

Although the President is supposed to obtain the written views of the Archivist prior to any proposed destruction of non-permanent records, "the final disposal authority rests with the incumbent president… regardless of the Archivist’s views."

The New York Times reports today on the problems the National Archives faces in acquiring, organizing, managing, preserving and making available the records of the Bush White House.

• Bush Data Threatens to Overload Archives, By Robert Pear And Scott Shane, New York Times, December 27, 2008.

The National Archives has put into effect an emergency plan to handle electronic records from the Bush White House amid growing doubts about whether its new $144 million computer system can cope with the vast quantities of digital data it will receive when President Bush leaves office on Jan. 20.

Among the problems NARA faces? Volume: NARA anticipates getting 100 terabytes of data 50 times the what they got from the Clinton White House. This is the equivalent of five times the contents of all 20 million catalogued books in the Library of Congress.

Cooperation: "Millions of White House e-mail messages created from 2003 to 2005 appear to be missing and may not be recoverable. And in September 2007, the top lawyer at the National Archives wrote in a memorandum that he had 'made almost zero progress' planning the transition because the White House had ignored repeated requests for information about the volume and formats of electronic records." In addition, Vice-President Cheney's lawyers claimed in a court filing that neither NARA nor the court "may supervise the vice president or his office" for compliance with the Presidential Records Act.

Formats: NARA says that there are a large numbers of White House records created with proprietary commercial software.

Access: Paul Brachfeld, the archives' inspector general, said "The electronic records archives system may be able to take in a tremendous amount of e-mail and other records.... But just because you ingest the data does not mean that people can locate, identify, recover and use the records they need."

Bush E-Mails May Be Secret a Bit Longer, by R. Jeffrey Smith, Washington Post, December 21, 2008; A01.

Legal Battles, Technical Difficulties Delay Required Transfer to Archives...

The required transfer in four weeks of all of the Bush White House's electronic mail messages and documents to the National Archives has been imperiled by a combination of technical glitches, lawsuits and lagging computer forensic work, according to government officials, historians and lawyers.

...The risks that the transfer may be incomplete are also pointed up by a continuing legal battle between a coalition of historians and nonprofit groups over access to Vice President Cheney's records. The coalition is contesting the administration's assertion in federal court this month that he "alone may determine what constitutes vice presidential records or personal records" and "how his records will be created, maintained, managed, and disposed," without outside challenge or judicial review.

...The National Archives and Records Administration is supposed to help monitor the completeness of the historical record but has no enforcement powers over White House records management practices.

Following up on the report about NARA's Executive Office of the President (EOP) Electronic Records Archives (ERA) (http://freegovinfo.info/node/2178), here is a story that goes in to more detail, much of it quite scary.

• Bush's exit to put new e-records system to the test, By Mary Brandel, Computerworld, November 21, 2008.

Deb Logan, an analyst at Gartner Inc., says in the article that the onerous task will be sorting through the unclassified and unprocessed data that the Bush administration will leave behind. "It would be one thing if the stuff had to be moved seamlessly to a records repository, but it's just eight years of stuff," she says. "It will be nearly impossible to get it under control without a massive expenditure of human resources because the technology is not there."

This is scary when one considers that it took NARA over a year to process the 2 Terabytes of data from the Clinton administration and NARA expects 140 Terabytes of data from the Bush administration. Even given that NARA now has a system in place that did not exist for processing the Clinton data, there is a big difference between the dealing with the technology issues and the information management issues:

Logan says part of the blame lies with federal agencies themselves, pointing to a GAO survey that concluded federal agencies have failed across the board to fulfill their records management obligations, "not out of malice or neglect but out of the nature of the volume of electronic communications and the time frame in which they have to do it," she says. "Anyone who's putting an optimistic face on the job is not being realistic."

Optimism may be relevant from a technology point of view, Thibodeau acknowledges, but not from an information management point of view. "From my side of NARA, I don't deal with what's in the records, just whether we can get them into the system," he notes. "We allow the library staff to deal with the content."

...Logan says the problem of managing electronic records won't be resolved until the government agencies themselves do a better job of electronic records management, including classifying, de-duplicating and purging data through the use of systems such as archiving, records and policy management, content monitoring/filtering, and content analytics tools.

A GAO report says, "NARA believes that if it cannot ingest the Bush records in a way that supports search and retrieval immediately after the transition, it may not be able to effectively respond to requests from the Congress, the new administration, and the courts for these records—a critical agency mission." (The National Archives and Records Administration’s Fiscal Year 2008 Expenditure Plan, September 2008, GAO-08-1105.)

The National Coalition for History has a report on the meeting of the Public Interest Declassification Board (PIDB) in November where there were discussions of the declassification, archival processing, and release of the 9/11 Commission records and the National Archives and Records Administration's Electronic Records Archives (ERA):

• Electronic Records Archives & 9/11 Commission Records Update, National Coalition for History, November 26th, 2008.

Two interesting tidbits:

Ken Thibodeau, Director of NARA’s Electronic Records Archive (ERA), reported that four federal agencies are currently using ERA system and that all federal agencies should be using the ERA by January 2011.

Thibodeau also reported on the upcoming transfer of the electronic records from the Bush administration to the National Archives. NARA is developing a separate system, known as the Executive Office of the President (EOP) ERA, to ingest an estimated 100 terabytes of electronic records from the Bush administration.

For those of you following the saga of Anthony Clark and his attempt to get access to NARA records about Presidential Libraries (See: How hard is it to get NARA records about NARA? and Getting Documents from NARA about NARA - UPDATE), our friend Kate over at ArchivesNext.com has researched the issue and written a good article about the issues. Thanks Kate!

• NARA's Management of Presidential Library Records: Revelations and Results, by Kate Theimer, The Records Manager: The Newsletter of The Society of American Archivists Records Management Roundtable. Vol 3, No. 1 (November 2008), pp 24-25.

[Cross posted on LegalResearchPlus]

About one month ago, I posted an item about the difficulty of getting documents about NARA from NARA -- the entry was based on an article written by Anthony Clark (Why Is It So Hard to Get Documents from the National Archives About the National Archives?, History News Network, July 21, 2008).

Here is an update to this very interesting story. On the Archivists' Forum, there is a recent entry from Anthony Clark detailing the latest ups and (mostly) downs of this saga.

Clark writes:

"Some readers may know that I have had great difficulty accessing NARA's own records for my research into presidential libraries and NL, or NARA's Office of Presidential Libraries (see http://hnn.us/articles/52350.html) for more information). What you might not know is that in July NARA offered me a deal" - if I dropped all of my pending FOIA requests for NL's records, they would commit to systematically process all of NL's records - some 230 boxes - at a rate of nine boxes per month, until all boxes have been processed and made available. Just a few weeks later, not only did NARA "take back" part of that offer (while claiming it was never made), they have now reneged on it completely. I was so shocked by what NARA did today that I felt I had to make the list aware of what they had done. [Full details available on the Forum page.]"

As I hear more about this NARA-tive, I'll be sure to pass it along.

Anthony Clark, an independent researcher writing a book on the politics and history of presidential libraries, has written a provocative piece on access to National Archives and Records Administration (NARA) records about administration of Presidential Libraries:

• Why Is It So Hard to Get Documents from the National Archives About the National Archives?, By Anthony Clark, History News Network, July 21, 2008.

Clark claims that NARA is "improperly withholding its own records." He says that as part of NARA's job of overseeing the twelve presidential libraries, it has records that detail the development of the libraries through 1964, when NARA created the Office of Presidential Libraries (NL), but none of NL's records are available. NARA is calling these records "operational," which makes them available only through Freedom of Information Act (FOIA) requests.

Clark quotes Patrice McDermott, Director of OpenTheGovernment.org, as saying, "It is hard to understand how records that are old enough to have been destroyed if the records schedule had been followed can be considered 'operational.' Presidential libraries are an area of keen congressional and public interest and information about them held by NARA should be affirmatively disclosed to the greatest extent possible."

Clark's article has produced an extensive discussion and Comments, including the NARA Response by Gary M. Stern on July 24, 2008.

Kate at ArchivesNext has posted a thoughtful response after talking off the record to archives staff: Access to records of the National Archives, July 24th, 2008.

As we have seen through the conflict and problems of preserving White House e-mail, the law has not kept up with preservation of electronic messages.

A bill (H.R.5811, "The Electronic Message Preservation Act") moving through Congress would address the problems by adding a new Section 2911 to Title 44, Chapter 29. It would require the electronic capture, management, and preservation of electronic records, require that they be readily accessible for retrieval through electronic searches, and would establish mandatory minimum functional requirements for electronic records management systems to ensure compliance with the requirements.

The Bush administration is threatening a veto:

• White House Threatens To Veto House E-Mail Storage Bill, By Dan Friedman, CongressDaily, Jul 9, 2008 (subscription required, but freely available here).

The White House and officials at the National Archives and Records Administration (NARA) argue that the law gives NARA new responsibility and expands the agency's job from advice to oversight, but the sponsors of the bill say that it only affirms the National Archives' job of advising the White House on record-keeping.

The CongressDaily articles notes that:

A less-discussed but farther-reaching part of the bill updates the Federal Records Act to require federal agencies, also under standards set by the National Archives, to save all e-mail records electronically and create systems to allow electronic searches.

According to GAO and a committee report, most agencies now use "print and file" records systems for keeping e-mail, many of them spotty.

(See National Archives and Selected Agencies Need to Strengthen E-Mail Management, United States Government Accountability Office, GAO-08-742 June 13, 2008.)

A comment in the Committee Report (House Report 110-709, "Electronic Message Preservation Act" 110th Congress 2d Session, June 11, 2008) says:

To make federal agencies comply, I believe this legislation should include enforceable repercussion language. Ms. Patricia McDermott of OpenTheGovernment.org suggests this is the only way to make federal agencies comply with the Federal Records Act. Ms. McDermott states that she does not "think anyone has ever been prosecuted for destroying, much less failing to preserve federal records." Just ask former Clinton EPA Director Carol Browner. She supposedly oversaw the destruction of her computer files in violation of a judge's order requiring the agency to preserve its records.

It isn't often you see a discussion FOIA, FBI, NARA, and Records Retention Plan and Disposition Schedules in the popular press. This article describes the frustrations of one researcher when he discovered that records had been destroyed by the FBI.

• The Department of Forgetting: How an obscure FBI rule is ensuring the destruction of irreplaceable historical records, By Alex Heard, Slate, June 24, 2008.

The system's fundamentals make sense, I guess--very complicated sense--but to me the disturbing part comes at the end of the line. At some point 25 years after a case closes, a file that isn't marked "permanent" gets pulled and looked at by one or two people inside the FBI. There are no "knowledgeable representatives of the NARA" monitoring this crucial moment. If it's decided internally that the file isn't important, it's gone.

Michael Ravnitzky, an FOIA researcher based in the Washington, D.C., area, is no fan of the Records Retention Plan and likens it to an open-ended manual for strip-mining a priceless public record. "The FBI got a list of exceptional files given to them by historians, and they said, 'We'll keep that,' " he says. "We'll keep large files. Smaller files, we'll keep a sampling. Everything else gets tossed. That's what the plan is." Based on documents Ivan Greenberg obtained from the FBI, he estimates that 250 million pages were destroyed between 1986 and 1995.

National Archives Reticent About Broadening Mission, by Dan Friedman, CongressDaily, Jun. 2, 2008. [subscription required].
Update: NOW AVAILABLE from NextGov WITHOUT SUBSCRIPTION: http://www.nextgov.com/nextgov/ng_20080602_6498.php

The National Archives (NARA) is being put in an awkward position. Viewed as nonpartisan and professional, it is being tasked by Congress with new enforcement duties. Late last year, a new law passed that set up an "Office of Government Information Services" within the NARA to help set federal Freedom of Information Act policy. Another bill that is expected to pass the House would give NARA a new role requiring NARA to monitor White House e-mail archiving. This would change NARA's role from that of passively receiving records to actively monitoring and enforcing rules. National Archives Inspector General Paul Brachfeld said that "NARA traditionally has not viewed itself as an enforcement entity but rather one that focuses upon collegiality and relationships."

From the article:

Chafing at Bush administration secrecy, congressional Democrats are handing the National Archives and Records Administration new jobs promoting government transparency. Officials at the records agency appear to be balking at taking on unfunded mandates beyond their traditional role. If Congress wants the Archives to become open-government cops, archivists may prefer to remain librarians. "They have always had a narrow view of their mandate and have never been particularly inclined to seek any expansion," said Patrice McDermott of OpentheGovernment.org, a coalition of groups urging government transparency. "They see their mission as providing access to historical records. They see [overseeing] contemporaneous records as a shift."

(cross posted on legalresearchplus.com)

Earlier this month the National Archives and Records Administration released their Strategy for Digitizing Archival Materials for Public Access, 2007-2016. This is a follow-up to a draft policy released in September of last year.

A fair amount of the report discusses the use of partner organizations in the digitization effort. The draft relased in September was open to public comment, and NARA has posted their responses to those comments here.

(Thanks to the American Association of Law Libraries Washington Office and their monthly E-Bulletin)

[I found this interesting news on the wonderful blog: BeSpacific -Erika]

Press Release
(from archives.gov)

May 7, 2008

Washington, DC. . . On Tuesday, May 6, 2008, Archivist of the United States Allen Weinstein submitted a report, entitled The Founders Online, to the Committees on Appropriations of the U.S. Congress. This report is the National Archives response to concerns raised by the Committees that the complete papers of America’s Founding Fathers are not available online. The Founders Online is a plan for providing online access, within a reasonable timeframe, to researchers, students and the general public. The report is available electronically at the National Archives website: http://www.archives.gov/nhprc/publications.

In announcing the completion of the report, Professor Weinstein said, “We feel this plan would provide scholars and the public access to the best available versions of the complete papers; it would also protect the longstanding interests of the publishers and host organizations which along with the Federal government have invested great resources in the past four decades. Most importantly, it would build a monument to the Founders of our nation in their own words.”

The National Archives received suggestions from the editors of the papers of John Adams, Benjamin Franklin, Thomas Jefferson, James Madison, and George Washington, university publishers, and others in crafting a blueprint for providing access to the already completed print editions and the raw materials for the editions to come. If carried out, the plan ensures that interested readers worldwide can see the work in progress with the already complete editions accompanied by transcriptions of the papers yet to be published. To hasten the transition process, the National Historical Publications and Records Commission plans to invest $250,000 as a demonstration pilot project.

The plan outlines three basic steps that remain:

* Digitizing the existing 217 volumes and publishing the Papers on a single website to allow for research and inquiry across America’s Founding Era collections;

* Transcribing and otherwise preparing for publishing on the web the remaining papers (approximately 90,000 documents) and replacing these raw materials with authoritative annotated versions as these are completed; and

* Creating an independent oversight process to ensure that rigorous performance goals are established and met by the parties carrying out all aspects of the work.

Agencies not complying with record preservation policies, By Jill R. Aitoro, NextGov, April 24, 2008.

At the hearing, Linda Koontz, director of information management issues at the Government Accountability Office, released preliminary results from an ongoing GAO study of how four agencies managed e-mail and electronic records. ...Koontz said the agencies print and then file e-mails, but about half of senior officials were not following these procedures, and the e-mails for these officials were maintained in e-mail systems that lacked record-keeping capabilities, such as the ability to group the e-mails using a classification system.

The House is considering the Electronic Communications Preservation Act, which would strengthen policies for preservation of government records including White House e-mails.

Gary Stern, general counsel for NARA said that the legislation's potential cost to agencies could be "astronomical," and noted the bill's requirement that the National Archives would maintain authority over the White House's electronic records might be unconstitutional.

Patrice McDermott, director of OpentheGovernment.org, said:

"I understand the constitutional issues, and I don't have a good answer for that.... But one of the concerns is that there is no way to enforce accountability [of] records management in the White House. We understand it's a difficult dance [for NARA]. They're there at the invitation of the White House in many cases, but there needs to be some way for the outside community to hold the White House accountable."

Over the past week, there have been some good conversations about government contracts to digitize government information and the National Archives decision to not conduct a web harvest or snapshot at the end of the current Administration. There is good news and bad news.

The good news
The good news is that NARA's decision was not nearly as bad as it appeared to be when it was first announced in a memo on March 27, 2008, which was circulated only to Federal records officers (see: The National Archives Is Quietly Destroying Millions of Documents). In a thoughtful post on its web site (National Archives and Records Administration Web Harvest Background Information, April 15, 2008, NARA; pdf version available), NARA outlines in detail the reasons why it would not conduct an end of administration web snapshot or harvest of Executive Branch websites nor require agencies to do so. The reasons, I think, are sound and in keeping with NARA's commitment to preserving information of historical value.

In addition, the NARA memo of April 15 makes explicit the fact that its decision and memo of March 27 do not apply to Presidential records or to records of the Congress. It says that "NARA will continue to conduct a web harvest of Congressional web sites" and that NARA "will also receive a snapshot of the White House website" noting that "Unlike Federal agencies governed by the Federal Records Act, the White House is governed by the Presidential Records Act, under which all Presidential records are treated as permanent and transferred to NARA for preservation at a Presidential Library."

The NARA "Background Information" document is also, I think, worth reading for its clear description of the shortcomings of web harvests in general. I think it is very useful for us to be reminded of these shortcomings to the extent that we believe we can rely on them as an adequate form of preservation.

In more good news, the NARA/TGN contract is not as bad as it could have been. I mentioned this in my earlier post here (The NARA/TGN contract as a bad precedent) and similar comments have been made in the useful and interesting thread over at ArchivesNext (NARA latest digitization agreement: One archivist's perspective). Merrilee Proffitt, of RLG, says in a comment there that the NARA model for contracts with third parties "actually comes out looking pretty good" when compared to the criteria described in the RLG paper Good Terms - Improving Commercial-Noncommercial Partnerships for Mass Digitization (by Peter B. Kaufman and Jeff Ubois, D-Lib Magazine, November/December 2007, Volume 13 Number 11/12).

The bad news
The bad news, as James pointed out this morning, is that the GAO contract for digitizing is very bad indeed (GAO *did* sell exclusive access to legislative history to Thomson West). Quoting Carl Malamud, James notes that GAO gets access to the digitized data but does not get a copy of its own; the rest of the government doesn't even get access to the data. The public is left with the option of going to GAO headquarters and paying 20 cents per page to copy paper! As Carl says, "This is one of those deals where the public domain got sold off."

This morning there was more bad news. Kate at ArchivesNext reports that the Citizens for Responsibility and Ethics in Washington (CREW) has a new report Record Chaos: The Deplorable State of Electronic Record Keeping in the Federal Government, that concludes "that the federal government is severely mismanaging its electronic records." CREW also says that a House Committee proposal to amend federal record keeping laws "is anemic and fails to make the substantial changes necessary to bring the federal government into the 21st century."

And even the good news is tempered by the fact that we have less than we could and are a long way from an even an adequate system of permanent preservation of digital information or a long-term solution to digitizing non-digital information. We will have to hope that the White House will deliver a snapshot of the White House web site and that the snapshot will be accurate and complete. The behavior of the White House with regard to electronic records and email does not make us optimistic. The NARA/TGN deal is better than the GAO/Thomson deal, but still leaves much to be desired and, as pointed out even by defenders of the deal, it is unlikely that we will ever have free, open, networked access to the digital information that TGN digitizes. That means the real effect of the deal is to privatize the information.

Comment
For me, the biggest disappointment in these latest developments is that librarians and archivists seem to be too willing to accept "good enough" and not willing enough to argue harder for "better." There are lots of people who have good reason to argue for less access, more fees, less privacy, and more control of information, but librarians and archivists should not be among them. I believe that we should not spend time making the case for the private sector; it is fully capable of making its own case. We should spend our time fighting for free, full, open, public access, usability of information, and long term preservation.

The primary mission of private sector companies is to make money, not to serve the public. They may serve the public as a by-product of making money, but no for-profit company will go to its owners and say "we are going to do the best thing for public access" without the qualification "that will make us money." Unfortunately "making money" often conflicts with public access. Politicians (and some bureaucrats) will argue for greater control of government information; some will argue for secrecy of government information on the one hand and privacy-invading policies on the other. Most government agencies do not have information access or long term preservation of their information as a primary mission and the exceptions are notable (e.g. LOC, NARA).

In contrast, the primary mission of many libraries and archives is to provide free public open access with long term preservation and usability. While others may have some of those pieces as secondary goals, few if any have them all. For many libraries and archives these goals are not just their primary mission but their defining characteristic.

While digitization and digital preservation are neither easy nor inexpensive, that doesn't mean that we have to pay any and all costs for them. The digital era should be making it possible to provide better access without giving up free use and reuse, without giving up open access, without turning over control to those whose primary mission is something other than free, open, public access and long term preservation. But increasingly we see a combination of politics and economics leaving us with contracts that trump copyright and fair use, with "access" being negotiated at almost any cost (including loss of control), with DRM technologies that prohibit easy (or any) reuse, and with privacy protections being deprecated or even ignored. Even in the case of the NARA/TGN contract that is legally "better" than the GAO/Thomson contract, we are left with the effect of two-tiers of access and network access being essentially privatized and fee-based.

I believe that librarians and archivists should be pushing the boundaries and insisting for more and better, not accepting some benefits by negotiating away the big benefits we could be getting in the digital age. This is particularly important for government information that is in the public domain. If we can't make this work for public information that is not copyrighted, how will we be able to do so for information that is?

I'm not arguing for a perfect, ideal world that is impractical to achieve. I am suggesting that we should fight for everything we can get. We should celebrate when we make inroads with a contract (like NARA/TGN) that is better than the others (like GAO/Thomson) but we should do so by committing to doing better next time. We should not accept this as "good enough" -- because it is not and we can do better next time. In fact, every time we accept a less-than-perfect deal as "good enough," we make it a little harder to make a better deal next time. We lower the bar if we accept "good enough" and stop trying to achieve better. We should not take the time to convince ourselves or the public that this is as good as we can get; we should take that time to admit to the limitations and trade offs and to commit to doing better next time.

There is lots written these days about "the future of libraries" and "the role of libraries in the digital age" and many people openly wonder if there is a place for libraries at all. I think there are several places where libraries have a unique role to play in society and the areas of digitization and digital access and preservation are important ones.

We need to make the case for the public; for free, open, public access; for long-term preservation and usability; for public accountability in the control of information; for reader privacy. Librarians and archivists have a unique role in doing that. In doing so, we will face an uphill battle and trade offs, but we should never lose sight of our unique role in society. We should never cheapen our professions by making the case for less (there are plenty of people to do that). We should always make the case for more. We will not always succeed and we will have to make trade offs. But we should always do so in the context of staking out a territory that is different from the private sector and those who are willing to get less. We should stand up for rights that others are not willing to fight for. We must fight for it when there are so many forces aligned against free, open access.

I'd like to see us emulate Carl Malamud and CREW and Brewster Kahle more and do less of making excuses for TGN and Thomson.