Home » post

Category Archives: post


More on NSA’s Google-like search engine to share data w agencies

Thanks Rachel for posting about the Intercept‘s new report about NSA’s search engine of harvested data. I thought readers would be interested in this DemocracyNow interview with Ryan Gallagher, the Intercept reporter who wrote this story. A particularly chilling part of the interview was when Gallagher described how the Intercept is now off-limits for Federal employees. Very creepy.

via NSA Creates Google-Like Search Engine to Help Other Agencies Access Collected Phone, Email Records | Democracy Now!.

FCC “Open Internet” Rule will hurt e-government

What will happen to government information on the web if we lose the little net neutrality we still have? Marvin Ammori of Slate’s Future Tense project, in partnership New America and Arizona State University, says that it will result in federal, city, and state government websites that run slowly and deliver errors.

Although Ammori doesn’t say so, I would guess that it will have another, second-order effect on government information. First, using the slow internet lane will make it more difficult for governments to deliver adequate e-government services. Second, Congress and local governing bodies will use this as an excuse, not to fund fast-lane access, but to reduce funding of government information delivery and e-government even further. Finally, the private sector will move in and offer better services and fast lane access, thus privatizing and commercializing government information access and delivery. Private companies will, of course, demand that they get all the government information they want for free and then they will charge the rest of us for access to this valuable, public resource.

On May 15, the FCC created a Notice of Proposed Rulemaking (Protecting and Promoting the Open Internet NPRM) that would permit cable and phone companies to create slow and fast lanes on the Internet. The FCC has received an unprecedented number of comments on this proposal (over a million), prompting the FCC to make the comments available to the public for analysis in XML format.

Docs Confirm NSA sharing data with 23 U.S. government agencies, via ICREACH search engine

original post @ LIS-GISIG

According to Classified documents obtained by The Intercept’ s Ryan Gallagher, “The National Security Agency is secretly providing data to nearly two dozen U.S. government agencies with a “Google-like” search engine built to share more than 850 billion records about phone calls, emails, cellphone locations, and internet chats.”

The documents provide the first definitive evidence that the NSA has for years made massive amounts of surveillance data directly accessible to domestic law enforcement agencies. Planning documents for ICREACH, as the search engine is called, cite the Federal Bureau of Investigation and the Drug Enforcement Administration as key participants.


NSA Doc: Sharing Communications Metadata

ICREACH contains information on the private communications of foreigners and, it appears, millions of records on American citizens who have not been accused of any wrongdoing. Details about its existence are contained in the archive of materials provided to The Intercept by NSA whistleblower Edward Snowden…

“The ICREACH team delivered the first-ever wholesale sharing of communications metadata within the U.S. Intelligence Community,” noted a top-secret memo dated December 2007. “This team began over two years ago with a basic concept compelled by the IC’s increasing need for communications metadata and NSA’s ability to collect, process and store vast amounts of communications metadata related to worldwide intelligence targets.”

The search tool was designed to be the largest system for internally sharing secret surveillance records in the United States, capable of handling two to five billion new records every day, including more than 30 different kinds of metadata on emails, phone calls, faxes, internet chats, and text messages, as well as location information collected from cellphones. Metadata reveals information about a communication—such as the “to” and “from” parts of an email, and the time and date it was sent, or the phone numbers someone called and when they called—but not the content of the message or audio of the call…

firstlook.org 2014-8-28 10 41 36


read more:  The Surveillance Engine: How the NSA Built Its Own Secret Google,

HT: democracynow.org

PACER Removes Court Case Documents

On August 10, 2014, The Administrative Office of the U.S. Courts, which runs PACER (Public Access to Court Electronic Records) abruptly announced that a bunch of cases would “no longer be available.” See below for a copy of the announcement and the cases covered.

PACER makes court papers available (for a fee) as well as court opinions (for free). It is not clear from the announcement, but it appears that the removal of these cases will include the removal of the opinions.

Some court opinons are available through CourtWeb (the “Online Federal Court Opinions Information System”) which “provides information on selected recent opinions of those judges of the United States Courts who elect to make information available on this site,” and from the websites of the individual courts.

As a joint “project” of the Government Printing Office and the Administrative Office of the United States Courts, FDsys has some United States Courts Opinions, but (apparently) none of the opinions in the list of withdrawn cases and none of the case files that PACER has. PACER includes: a Case Locator service (a national index for U.S. district, bankruptcy, and appellate courts), listings of all parties and participants including judges, attorneys and trustees, compilations of case related information such as cause of action, nature of suit and dollar demand, chronologies of dates of case events entered in the case record, A claims registry, A listing of new cases each day in all courts, Judgments or case status. (In the table below I have included the coverage of case opinions by FDsys.)

There have been attempts, including by Aaron Schwart, to free this information from the fee system (RECAP, Public.Resource.Org), but they have had limited success and have been met with vehement opposition (FBI Investigated Coder for Liberating Paywalled Court Records and Court Tells Users They Can’t Use RECAP).

Although PACER makes court opinions available for free, it is one of the agencies that is required (like NTIS) to recover costs. In 1988, the Judiciary sought appropriations from the U.S. Congress in order to provide electronic public access to court records but Congress did not provide the funds and instead directed the Judiciary to fund the initiative through user fees. As a result, the program relies exclusively on fee revenue (PACER FAQ). It has an elaborate free schedule for searching and obtaining court papers (Electronic Public Access Fee Schedule). The fees that PACER charges have been shown to far exceed its costs (PACER Federal Court Record Fees Exceed System Costs).

Most agencies are permitted by statute to charge fees for access to information. Even GPO is allowed to “charge reasonable fees” (44 USC sec 4102) to the public. (Although it is required to make its systems available to depository libraries without charge, we all know how well trying to charge the public for information available free at libraries worked out for GPO when it tried to charge for GPO Access back in the 1990s: Privatization of GPO, Defunding of FDsys, and the Future of the FDLP.) Still, the threat always remains that Congress will privatize or commercialize or monetize its information “assets” at any time. As long as we rely on GPO and government agencies to preserve and provide free access to government information, we are playing a risky game (When we depend on pointing instead of collecting). If we lose that game, it is our users who will pay the price (and the cost) and they will not thank us for failing to preserve government information and ensure free access to it.

The Annoucement:

On August 11, a change was made to the PACER architecture in preparation for the implementation of the next generation of the judiciary’s Case Management/Electronic Case Files (CM/ECF) system. NextGen CM/ECF replaces the older CM/ECF system and provides improvements for users, including a single sign-on for PACER and NextGen CM/ECF. As a result of these architectural changes, the locally developed legacy case management systems in the five courts listed below are now incompatible with PACER; therefore, the judiciary is no longer able to provide electronic access to the closed cases on those systems. The dockets and documents in these cases can be obtained directly from the relevant court. All open cases, as well as any new filings, will continue to be available on PACER.

court cases removed FDsys coverage
U.S. Court of Appeals for the 2nd Circuit Cases filed prior to January 1, 2010 2010-
U.S. Court of Appeals for the 7th Circuit

Cases filed prior to CM/ECF conversion 2005-
U.S. Court of Appeals for the 11th Circuit

Cases filed prior to January 1, 2010 2010-
U.S. Court of Appeals for the Federal Circuit

Cases filed prior to March 1, 2012 [none]
U.S. Bankruptcy Court for the Central District of California

Cases filed prior to May 1, 2001 2005-

Please contact the court directly to obtain copies of documents and dockets in the above cases. Contact information for each court is available on the Court Locator page.

Press coverage:

Why GPO’s proposed policy to allow Regionals to discard is a bad idea

The discussion surrounding ALA’s Government Documents Round Table (GODORT) response to a recent Government Printing Office (GPO) proposal to allow Regional Depositories in the Federal Depository Library Program (FDLP) to discard some government documents has been intense and impassioned. In the interest of brevity on the GOVDOC-L listserv, I decided to post a longer piece in response here that will provide more context, explanation and rationale for my listserv response as well as my perspective on the proposal and its impact on the FDLP. I agree with the recommendation of the GODORT letter that the community should take 4 steps before instituting any Regional library e-substitution policy. This is my personal perspective and should not be seen as me speaking for GODORT or other FDLP libraries. A copy of the final version of the GODORT letter may be found at GODORT’s ALA Connect site. Because a login seems to be needed on that site, I’ve also posted a copy of the letter here on FGI.

Thanks to Barbie Selby for her clear articulation of the rationale for accepting the GPO proposal to allow Regionals to discard some of their paper holdings. I think Barbie and I agree on the larger goals of the FDLP (preservation of and free access to FDLP materials) almost completely. Where we disagree is whether the GPO proposal will actually accomplish those goals.

Unfortunately, the GPO proposal (as written) allows and, I would argue, even encourages actions that will predictably result in bad outcomes — outcomes that are the opposite of our shared goals. I believe that we need a policy that is designed to accomplish our shared goals.

But let me respond to the points raised in support of the GPO’s proposed rule change and try to explain why I think this policy is fatally flawed as written and why the GODORT letter is appropriate and reasonable and why its recommendations are necessary if we want to accomplish our shared goals.

1) Should we implement the policy as written because it is “very limited” in scope at this time? No, because we actually do not know how limited (or extensive) the initial phase is. GPO’s proposal is limited to those items in FDsys with a digital signature, but we actually do not know how many titles or volumes that includes. For example, FDsys includes a collection of Congressional Hearings from 1985 to 2014 and, presumably, any paper copy hearings that have digitally-signed versions in FDsys and are older than 2007 are eligible for immediate discard. Has anyone actually checked to see how many volumes that includes? Tens? Hundreds? Thousands? This is just one collection in FDsys and, by my count there are at least two dozen collections that have digital signatures. It is difficult to characterize even the initial scope of the policy as “very limited” when we do not have an official count from GPO of how many volumes are eligible for discard. (Although GODORT didn’t ask for it, we would all be better able to judge the extent of the initial impact of the proposal if GPO provided some quantitative estimate of how many volumes are within scope of the policy today.)

2) Regardless of how many volumes are eligible for discard today, we must realize that the policy is not limited in scope: it will allow more and more discards over time. Presumably it is the goal of GPO to include (and digitally sign) more documents in FDsys as resources permit. Presumably, this is also exactly what Regionals want; if the policy were truly limited to a small number of volumes it would not meet the expressed need of Regionals for more “flexibility.” The discard policy is not limited by design; rather, it is designed to be expansive and expanding. It would be illogical to accept a policy that is designed to expand because it is small at the start.

3) Although the proposal acknowledges that FDLP should retain a “requisite number” of tangible copies, it does not define what that requisite number is and does not provide a means of determining what it is or of determining how many copies we already have. Without the data, it will be impossible to implement such a requirement or know if it is succeeding — or failing. The GODORT letter simply recommends that, before implementing a Regional e-substitution policy, we should get the data and research necessary to accomplish this goal we all share. The alternative is to try to accomplish the goal in the dark — without either data or research — and such an approach could only accomplish our goal by accident.

There is research on how many paper copies libraries should keep when they rely on digital surrogates in order to discard paper copies; but the existing research does not map well to government documents. The existing research examines academic journals, a body of literature that is inherently homogenous and assumes that paper copies will only be needed for preservation and not ongoing access. That is very different from the very heterogeneous govdocs literature — which encompasses every conceivable kind of publication and will be needed for both preservation and access. So, we need research that focuses on the special nature of government documents. I agree with Barbie when she says that the actual number we need may vary from title to title and that there probably is no one “magic number;” that is, in fact, precisely what the research on academic journals suggests. But the GODORT letter does not recommend finding or using a magic number; it just proposes that we do the research and gather the data so that we can make informed decisions before we start discarding. Without that, our only option is to discard assuming that there is a magic number, and we know what it is, and that we have that number of copies somewhere within the FDLP. We know none of those things and discarding without knowing would be irresponsible.

4) Although it is possible that the policy could result in good outcomes and might accomplish our shared goals, there is nothing in the policy to suggest that it will do so. In fact, the policy only has one new goal, which it states explicitly: “To allow regional depository libraries the option to discard certain tangible materials.” How can we accomplish all the good outcomes we want for such a policy if the policy does not even acknowledge them? Let’s be clear: this policy is not about enhancing access or preservation; the materials covered by the policy are already in FDsys and available online. The policy only does one thing: it reduces the number of paper copies in the FDLP. The policy is not designed to benefit the FDLP system or users of government information; it is designed to benefit a specific subset of FDLP libraries. Their gain of shelf space will do nothing for any other library or any user of government information.

5) If my view is accurate, why do some librarians find the policy attractive? I think it is a very hopeful policy that is based on good faith. We are a hopeful community and a community with shared goals. Surely, none of us would intentionally do anything that would obviously harm the FDLP or long-term, free public access to government information.

I think that those who support GPO’s e-substitution proposal as written and oppose the recommendation of the GODORT letter are putting their faith in the ability of the community to implement the policy successfully. Barbie says, for example, that geographic distribution would be ensured (although there is no mention of this in the policy and inadequate data available to ensure it). She says that regional librarians do not want to weed drastically (but this seems contradicted by the vehemence with which they are pursuing “flexibility” to discard at their discretion) and that regionals will want to use options other than discarding (which they can already do without this new policy). I do not question Barbie’s intentions or goals, indeed, I agree with them! As we evaluate a bureaucratic policy that, if implemented, could outlive us all, we should look not at our hopes and faith, but at the facts of the policy. The fact is that this policy allows 47 Regional libraries to make irreversible decisions; once they discard, they cannot un-discard. And the fact is that the design of the policy does nothing to further our goals. In fact, it is designed in a way that will make it easier for the FDLP to lose geographic distribution, easier to have fewer copies than we need, and will actually remove incentives to do more collaboration among libraries by providing discarding as an alternative to collaborative retention.

One hopeful aspect of the policy itself is that it puts GPO in the position of being able to slow down discards of paper: the Superintendent of Documents must approve of discards. And it requires Regionals to offer a publication to the selectives in their region and then to other depository libraries nationwide. This provides us the hope that GPO or selectives or other Regionals will, when necessary, preserve copies that a Regional is discarding, but it does not provide the data or research or other tools for either GPO or FDLP libraries to know when that is necessary. (Indeed, if we had those tools, the Regional would not make the mistake of withdrawing a needed title in the first place!) These, then, are hopeful procedures, not guarantees and, without the data needed, any decisions made with these procedures will be made in the dark. I also wonder: Will the first to ask for permission to discard a particular volume be more likely to get permission, and will the last to ask be less likely to get permission? If so, won’t that actually lead to libraries racing to be first to discard? Won’t that fuel a discarding frenzy?

In short, the policy sounds good only if you accept the good intentions and hopefulness of the community. But when you look at the details, you are left with questions. Can Regionals or GPO make those good decisions without the necessary information? What if our well-intentioned actions have consequences that we cannot understand or foresee because we lack the information to understand their consequences? The GODORT letter simply suggests that we gather the information we need to make the right decisions so that we actually can achieve our shared goals.

Even if you disagree with the above analysis, I want to make two final observations for your consideration.

A) Although this policy is explicitly about discarding paper copies of documents, it is not (only) about paper vs. digital. If libraries want to digitize their documents to make them more accessible (and they should!), they can do so today (and they should!). They can do so under existing GPO policies. No, this policy is about libraries that want to shift their responsibilities of providing preservation of and access to government documents to GPO. This is not an unintended consequence or an unknown by-product of the policy; it is its explicit goal as stated in its first sentence. Is this a wise choice? One of the statements in the policy that I would classify as “hopeful” is the assertion that FDsys “guarantees permanent preservation and access.” Everything about this policy (and indeed the twenty-year shift of preservation and access responsibilities from FDLP libraries to GPO) hinges on this. If we wish to believe in that “guarantee” we should, as the GODORT letter suggests, first insist that GPO prove this to us by having FDsys go through an audit and get certification as a Trusted Digital Repository. But, even with such a certification, we must also recognize that, regardless of GPO’s current actions and intentions, GPO can only do what Congress authorizes and enables with funding. Further, we have seen repeated examples in the last few years of the problems (technical, political, budgetary, bureaucratic) of relying only on the government to preserve and provide consistent, reliable, free access to government information. When we choose to not preserve documents and not provide access to them, we are literally choosing to not provide any value for our users. When we choose to hope that the federal government will be able (and willing!) to provide preservation and free access to our users for the long-term, we are making a very risky decision. These are bad and risky choices. This policy effectively opens the door to expand the scope of these bad and risky choices from born-digital documents to the entire corpus of two hundred years of FDLP collections. What will our users gain by us doubling-down on a failing policy? By choosing to not provide value to them? By choosing to replace a safety net of dozens of libraries with a single-point-of-failure federal digital system?

B) Although I would like to be as hopeful and optimistic as Barbie is, it is hard for me to justify that optimism. I do believe that government information librarians, even with a bad policy and a lack of necessary information, would do their best to use this policy responsibly. But I am worried that there is a countervailing pressure on government information librarians. We have all seen it on govdocs-l and heard it at ALA and at DLC — even if we are not anxious to name it and acknowledge it for what it is. Even Barbie points out this pressure when she says, “I very much fear that if regionals are not given some flexibility, such as this modest [sic] step by GPO, we will see more libraries drop regional status altogether – which would not be good for any of us.” And there you have it. No matter what government information librarians may want or what their intentions are, we have the threat that, if GPO does not allow Regionals to discard their documents and pass off their preservation responsibilities to GPO, they will drop their regional status even if it is bad for FDLP, bad for preservation, bad for access, and bad for users. Won’t this pressure affect the decisions every Regional makes if this proposal is approved as written? And won’t that lead to decisions that are good for the individual library making the decision (more shelf space! less responsibility!) but bad for FDLP and users everywhere?

Finally, I want to end on an optimistic note. If government information librarians unite around our existing shared goals, we can come up with better policies than this and better strategies than turning our responsibilities over to GPO. But we will have to acknowledge more than the “reality of our institutions” (i.e., what is good for my library) and realize that the FDLP is about shared responsibility. We know this; Barbie expresses it well, too. But perhaps we have not been good enough advocates for the FDLP, our collections, or our many current and future users. Perhaps we have been too willing to give in to pressure and do what our library managers want — even if it is not the right thing to do. What can we do? We can and should leverage our shared goals and the vocal expression of support (even among library managers!) for government information and use that to develop strategies that will benefit not just our individual libraries in isolation but the infrastructure of FDLP (GPO and libraries together). We should advocate policies that benefit us all — collectively as well as individually. We can build an infrastructure of shared digital responsibilities that can support a robust 21st century digital depository system. We can start by gathering the data necessary to make good decisions. And we can also take on the responsibility of sharing preservation and access of born-digital information with GPO. If we had done these two things earlier, we would have had the data to make good decisions and the regionals would have already had this born-digital information in their repositories and this bad policy would not even be necessary!

Now *that* would be real progress!