This is yet another disturbing example of data loss documented by our friends at MuckRock. Evidently, a large amount of data from the Federal Energy Regulatory Commission (FERC) having to do with the infamous Enron Corporation has gone missing and even FERC staff do not know where it went.
How many examples will we need to post before libraries and archives get with the program and try and figure out ways to collect, archive, preserve and give access to born-digital information posted on .gov sites? And how does this particular example of data loss NOT happen again in the brave new world of open government data (aka H.R. 4174 the Foundations for Evidence-Based Policymaking Act of 2018 which was recently signed into law and described by Alex Howard)? If you’re as concerned as I am, you’ll contact FERC and request a copy of their data for your library.
Government investigations into California’s electricity shortage, ultimately determined to be caused by intentional market manipulations and capped retail electricity prices by the now infamous Enron Corporation, resulted in terabytes of information being collected by the Federal Energy Regulatory Commission. This included several extremely large databases, some of which had nearly 200 million rows of data, including Enron’s bidding and price processes, their trading and risk management systems, emails, audio recordings, and nearly 100,000 additional documents. That information has quietly disappeared, and not even its custodians seem to know why…
…While terabytes of information has disappeared, up to 4,516 documents remain available through a pair of predefined searches of FERC’s eLibrary. While FERC claims that they, not Lockheed Martin or CACI, do offer a trio of Enron datasets on CD, FERC has not responded to repeated requests for these datasets sent over the past two months.
Russ Kick of AltGov2 is again on the FOIA case. This time he’s analyzed the Department of Interior’s proposed changes to their FOIA regulations (and helpfully cobbled together the current regulations with DoI’s proposed changes). Comments on the proposed changes can be submitted electronically by JANUARY 28, 2019. Here’s the current regulations and the proposed rules changes posted to regulations.gov.
Here’s my own take on this. Department of Interior’s reason for updating its FOIA regulations is that they’ve had “Exponential increases in requests and litigation.” From Fiscal Year (FY) 2016 to FY 2018, incoming FOIA requests to the Department increased 30 percent (from 6,428 to over 8,350). So it would make sense that they’d want to update their regulations to deal with the exponential increase in requests. But instead of requesting more staff to deal with the increase in requests or look for ways to proactively release more records, their proposed changes:
- look to expand the definition and reasoning of “burdensome” requests that can be denied. “The bureau will not honor a request that requires an unreasonably burdensome search or requires the bureau to locate, review, redact, or arrange for inspection of a vast quantity of material.”
- place monthly limits on the number of requests per month from frequent requesters.
- Make it more confusing and difficult to submit requests (eg it’s unclear whether requests will be accepted via email in the new regs).
- No longer refer or forward mistakenly directed requests. “A request to a particular bureau or a particular bureau component (for example, a request addressed to a regional or field office) will be presumed to seek only records from that particular bureau or particular component and will not be forwarded to another bureau or component.”
- Change “time limits” to “time frames,” making the time to fulfill requests more squishy and undefined.
There’s probably more buried in this request. Check out Russ’ analysis at MuckRock. And PLEASE send comments to Department of Interior about their proposed changes. Comments on the proposed changes can be submitted electronically by JANUARY 28, 2019.
The Department of the Interior wants to drastically change how it deals with Freedom of Information Act requests. To do that, it had to make a proposal, published in the Federal Register, that the public can comment on for 30 days. In theory, it has to consider this input before finalizing any changes to its FOIA regulations.
That proposal was published on December 28th, 2018, which is 1) a Friday 2) in the middle of the week between Christmas and New Year’s Day 3) during a government shutdown. Any one of those is a tried-and-true way to slip something past the public, but all three simultaneously? That is the trifecta of bureaucratic underhandedness.
So, now that the holidays are behind us for another year, let’s take a look at Interior’s FOIA wishlist, which is primarily designed to hobble requesters and solidify the department’s power. (To make it easier to see the many changes Interior wants, I’ve created a redline version of their currently FOIA regulations, with added language in bold and deletions in strikethrough, which you can see here.)
Twitter and newspapers are buzzing with complaints about widespread problems with access to government information and data (see for example, Wall Street Journal (paywall 😐 ), ZDNet News, Pew Center, Washington Post, Scientific American, TheVerge, and FedScoop to name but a few).
Maybe when/if the government opens again, we should scrape the NIST and CSRC websites, put all those publications somewhere public. It’s worrying that *every single US cryptography standard* is now unavailable to practitioners.
— Matthew Green (@matthew_d_green) January 12, 2019
Matthew Green, a professor at Johns Hopkins, said “It’s worrying that every single US cryptography standard is now unavailable to practitioners.” He was responding to the fact that he could not get the documents he needed from the National Institute of Standards and Technology (NIST) or its branch, the Computer Security Resource Center (CSRC). The government shutdown is the direct cause of these problems.
Others who noticed the same problem started chiming in to the discussion Green started, noting that they couldn’t find the standards they needed in Google’s cache or the Wayback machine, either. Someone else suggested that “Such documents should be distributed to multiple free and public repositories” and said that “These documents are “Too important to have subject to a single point of failure.” Someone else said that he downloads personal copies of the documents he needs every month, but had missed one that he uses “somewhat often.” One lone voice wondered about “Federal Depository Libraries, of which I believe there is at least one in every state.” (James responded to that one, letting people know about the FDLP and End of Term crawl!)
There are at least two reasons why users cannot get the documents they need from government servers during the shutdown. In some cases, agencies have apparently shut off access to their documents. (This is the case for both NIST and CSRC.) In other cases, the security certificates of websites have expired — with no agency employees to renew them! — leaving whole websites either insecure or unavailable or both.
Regardless of who you (or your user communities) blame for the shutdown itself, this loss of access was entirely foreseeable and avoidable. It was foreseeable because it has happened before. It was avoidable because libraries can select, acquire, organize, and preserve these documents and provide access to them and services for them whether the government is open or shut-down.
Some libraries probably do have some of these documents. But too many libraries have chosen to adopt a new model of “services without collections.” GPO proudly promotes this model as “All or Mostly Online Federal Depository Libraries.” GPO itself is affected by this model. Almost 20% of the PURLs in CGP point to content on non-GPO government servers. So, even though GPO’s govinfo database and catalog of government publications (CGP) may still be up and running, during the shut-down GPO cannot ensure that all its “Permanent URLs” (PURLs) will work.
This no-collections-model means that libraries are too often choosing simply to point to collections over which they have no control — and we’ve known what happens “When we depend on pointing instead of collecting” for quite some time. When those collections go offline and users lose access, users begin to wonder why someone hasn’t foreseen this problem and put “all those publications somewhere public.”
The gap between what libraries could do to prevent the kind of loss of access the shutdown is causing and what they are doing is particularly notorious in the area of government information. Most federal government information is in the public domain and is available without technical or copyright restrictions or fees. There is nothing preventing libraries from building collections to support users except the will to do so.
Many library administrators are eager to proclaim that pointing to collections they do not control is the new role of libraries in the digital age. Those who promote this new model of services without collections then struggle to demonstrate the value of libraries to their user communities. This is difficult when those communities go directly to collections of information, bypassing libraries and, perhaps, wondering why libraries still exist at all.
This represents a failure by libraries to fulfill their role in society and in the digital information ecosystem.
When the shutdown ends, access will, presumably, be restored. In the wake of the many other problems caused by the shutdown (many of them immediate and even dangerous), this temporary loss of access to some government information may not seem pressing. But librarians should see this as another wake-up call. Hopefully, Depository Library Council’s recent recommendation regarding digital deposit will answer that call. Libraries should not focus on bemoaning the short-term problem. We should, instead, focus on making the next crisis impossible. We can do this by focusing on the long-term problems of digital collection development, preservation and access. The current crisis may be temporary, but when we rely only on the government to provide access to these important resources, access will remain vulnerable to the next crisis or misstep or conscious decision to cut off access. We need to recognize that government agencies do not always have the same priorities as our users.
Today, libraries cannot ensure long-term access to government information because they do not control it. But, if libraries select, acquire, organize, and preserve the government information that is vital to their user communities, then they can ensure long-term access to it. You will not have to persuade your users of the value of your library when you do what they value.
James A. Jacobs, University of California San Diego
James R. Jacobs, Stanford University
Steven Aftergood over at Federation of American Scientists’ Secrecy News project writes today that President Trump recently commented off-handedly that reports from the Department of Defense’ Inspector General should be private and not publicly accessible. It’s unclear if this off-the-cuff comment will lead to less public access to these important reports, but Aftergood notes that “secrecy in the Department of Defense has increased noticeably in the Trump Administration” but that the Pentagon still publishes a massive amount of information. What is clear is that this one small comment could have huge implications going forward from “For Official Use Only” markings to restrict access to information to perhaps an erosion of the FOIA process. This is certainly something to keep an eye on.
The recurring dispute over the appropriate degree of secrecy in the Department of Defense arose in a new form last week when President Trump said that certain audits and investigations that are performed by the DoD Inspector General should no longer be made public.
“We’re fighting wars, and they’re doing reports and releasing it to the public? Now, the public means the enemy,” the President said at a January 2 cabinet meeting. “The enemy reads those reports; they study every line of it. Those reports should be private reports. Let him do a report, but they should be private reports and be locked up.”
It is not clear what the President had in mind. Did he have reason to think that US military operations had been damaged by publication of Inspector General reports? Was he now directing the Secretary of Defense to classify such reports, regardless of their specific contents? Was he suggesting the need for a new exemption from the Freedom of Information Act to prevent their disclosure?
Or was this simply an expression of presidential pique with no practical consequence? Thus far, there has been no sign of any change to DoD publication policy in response to the President’s remarks.
[UPDATE 1/8/2019]: Here’s more context about the DoJ’s disinformation report on immigrants:
Case Closed: The Justice Department Won’t Stand Behind Its Report on Immigrants and Terrorism””. By Benjamin Wittes Monday, January 7, 2019.
Don’t look now, but the United States Department of Justice just came perilously close to admitting that it engaged in disinformation about immigrants and terrorism in a formal government report.
Also of interest in this post is the reference to an obscure law called the “Information Quality Act” which “requires federal government agencies to employ sound science in making regulations and disseminating information” (“What is the Information Quality Act (aka Data Quality Act?). JRJ
Well this is pretty messed up. The Justice Department is refusing to retract or correct a document that falsely links terrorism to immigration, according to the Washington Post. The Federal government is supposed to be a reputable publisher who is supposed to work for the American public. But when dubious, politicized reports based on misleading data are published, it calls into question the veracity of government published works.
This is the reason that information literacy — or as my buddy howard Rheingold calls it, “crap detection” (borrowed from Ernest Hemingway!) which 2 librarians at Dominican University helpfully turned into an acronym: Currency, Reliability, Authority and Purpose/Point of View! Unfortunately, a document like this would probably pass the information literacy standards set by the American Library Association. The reader must have a great amount of context in order to detect the crap in this report. Perhaps GPO will update the record for this document in its Catalog of Government Publications to include a note on the report’s dubious nature. yes, librarians need to teach crap detection, but they should also include context to metadata describing the information which is under their control.
The Justice Department has acknowledged errors and deficiencies in a controversial report issued a year ago that implied a link between terrorism in the United States and immigration, but — for the second and final time — officials have declined to retract or correct the document.