Home » Commentary » The State of FDsys and the Future of the FDLP

The State of FDsys and the Future of the FDLP

The recent report on the state of the Government Printing Office’s Federal Digital System (FDsys) should raise important questions for GPO and should be a wake up call for FDLP libraries. The report documents that the project is over budget, behind schedule, and lacks sufficient resources and planning to move forward successfully.

The report (Federal Digital System (FDsys) Independent Verification and Validation (IV&V) – Tenth Quarter Report on Risk Management, Issues, and Traceability Report Number 10­-05, “IV&V Risk Management, Issues, And Traceability Report,” January 14, 2010) was written by American Systems, under contract to the GPO Office of Inspector General and is attached to the following memo available from GPO:

The consultants were charged with assessing the state of FDsys implementation. Some of the findings of the report that will be of most interest to FDLP libraries are:

  • FDsys as it exists today “bears only partial resemblance of the system that was envisioned.”
  • The program is significantly over budget. The original cost estimate for the first phase of FDsys implementation was $16 million. Through August 2009, GPO had spent more than twice that (approximately $33.6 million) and, by the end of FY 2010, the total costs for FDsys contractor support will be approximately $42 million.
  • Even though the cost has more than doubled, the project is significantly behind schedule. “Release 1″ of FDsys was slated to be accomplished in three phases ending in the Fall of 2009, but only the first phase has been deployed — and even that phase is incomplete.
  • GPO now says that only 42 Collections will be migrated to FDsys instead of the originally-planned 55.
  • There is an on-going indexing problem within the FAST search product. The FDsys database was due to approach a critical threshold of 2.5 million records in December of 2009 and FAST will require changes to accomodate more records.
  • The majority of the work on FDsys after the release of the first phase of Release 1 has been centered on fixing problems and dealing with emerging issues. GPO is focusing more on fixing and upgrading a deployed system than on building the final system.
  • The FDsys Program has performed little to no analysis, planning, design, or development work for Release 2.
  • The “large number [25] of deployments [production builds] over a ten month period reflects the obvious fact that the originally deployed system contained numerous deficiencies.”
  • The lack of clear definition of the system and the lack of a detailed implementation plan prevent GPO from determining realistic cost estimates for future development and endanger the ability of GPO to develop and deploy the final system.
  • The consultants say that GPO does not have sufficient system engineering expertise to direct and oversee the development of FDsys and that this has resulted in a system with incomplete functionality, design problems, and numerous deficiencies. They recommend that GPO hire a senior system engineer and say that, without one, these problems will continue and future releases will likely take longer and cost more than anticipated. GPO management, however, completely disagrees with this recommendation.

There is more in the 38 page report, but the above gives you the gist of the problems.

The positive, the negative, and the risks.
There are some positive things. Much has been accomplished. Twenty-five of the most complex collections have been transferred from GPO Access to FDsys. The project managed to incorporate a significant design change during implementation to accomodate “Collections with numerous granules.” The project also was able to create a new capability to support public access to FDsys information via the Data.gov website.

But these accomplishments are overshadowed by the numerous problems that the report documents. Even the already-deployed system is apparently overwhelmed with problems. The report documents 232 problems that adversely affect the accomplishment of an operational or mission­-essential capability and notes that the many unresolved problems with the system create “a serious risk that the overall goals for FDsys … will take much longer and require significantly more funding to achieve.”

The failed “paradigm shift.”
Since 1993, GPO has been championing a “paradigm shift” in responsibilities in which GPO arrogated to itself the responsibility for both access and preservation of government information and diminished the role of FDLP libraries. (See, for example, the discussion on GPO’s draft regional libraries report and FGI comments.) We at FGI have been concerned about this shift from the beginning for a number of reasons. One of those reasons is the danger of entrusting all preservation and free access to any single organization. Any interruption or failure of that organization (financial, technological, political, etc.), could mean a catastrophic loss of access to government information for everyone.

We have been hopeful that interruptions would be small and short and failures would be in the future. But we have hoped to persuade the FDLP library community, including GPO, that it would be wiser and more prudent and more durable to build on the existing FDLP model of sharing responsibility for access and preservation across many institutions. These institutions with different infrastructures, governing bodies, technologies, and communities of users, would, we have argued, do a better job collectively than any one institution could do by itself.

We have feared the day when Congress would cut back GPO appropriations after we all were irreversibly dependent upon GPO — a day when it would be too late to create a new system of free, permanent, public access to government information. With the release of this report, we worry that that day may be closer than we had imagined.

The original design and specifications for FDsys were expansive and ambitious. That was a good thing. It would be wonderful if GPO could support FDsys and all of its almost three thousand system requirements and features. OAIS compliance, persistent naming, metadata management, and support for RSS are among the features we look forward to. And we hope for other features: maybe, someday, APIs and OAI-PMH support, for example. But what do we do if GPO does not have the resources or the expertise to fully develop FDsys? It is hard to read this new report without being concerned that this is exactly the reality we face today. We worry that this report is “the writing on the wall” that is telling us that “the paradigm shift” will not work and is not sustainable.

Many FDLP libraries (or at least the directors of those libraries, and, in many cases, the FDLP librarians as well) have been hoping for almost two decades that they could rely on GPO to provide the services that the FDLP libraries themselves used to provide. If this is proving to be a false hope what will happen next? Is it only a matter of time before Congress pulls the plug, or GPO throws in the towel, or the private sector raises a stink?

What are our options for the future?
Naturally, we hope GPO can continue to develop FDsys. It would best for access and preservation to have FDsys in place. But it would be better if FDsys was not our only resource for preservation and access. It would be better if we had more systems in place to complement FDsys. It would be better if we had a digital FDLP that shared responsibility for access and preservation.

What options do we face right now? The obvious status-quo next step is for GPO to get more resources. It needs more money and more expertise so that it can deal with existing problems and move forward faster and with better planning that will make it easier for it to succeed and do so in a reasonable time frame and on budget.

But FDLP needs a “Plan B” to deal with the real possibility of GPO not getting adequate resources to finish or maintain FDsys. What will happen if we don’t have a plan in place? We can imagine at least three generic scenarios: One, GPO will scale back and provide less access, or less secure preservation, or fewer collections, or some combination of those. Two, preservation and access will remain government-provided, but will become completely fee-based (somewhat like NTIS and STAT-USA). Three, the private sector will move in and demand, perhaps under OMB regulations, that GPO shouldn’t have undertaken this job in the first place and that the government shouldn’t provide a system that the private sector could provide. It would argue that raw information should be given to private sector companies who will produce their own preservation and access systems that will be fee-based. (We almost certainly will see a proposal to replace GPO’s single-entity model with a private-sector, fee-based, single-entity model. Ithaka is already laying the groundwork for such a proposal. [See: Ithaka report on the future of the FDLP.] To those of us at FGI, this seems the worst of both worlds: relying again on a single organization rather than a community of organizations and moving that model to a fee-based system.)

There is also the possibility that none of these will happen and we will simply lose access because no one will take responsibility.

A better option; a more durable future.
But there is one other possibility: a collaborative effort in which GPO deposits digital files with FDLP libraries and those libraries preserve those files and make them accessible. This would be a real digital depository system with shared, distributed responsibility. It would have many advantages but, in the context of the current report, it has one major advantage over the current system: it has no single-point of failure (which is what we have with the GPO, single-entity, paradigm-shift model).

Such a system will take planning and resources and will not be trivial to implement. But the time to start planning for such a system is now. It would be much worse to wait until FDsys is in technological or budgetary crisis. At that point it could be too late.

Print Friendly

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License.


4 Comments

  1. Jim,

    This was a very helpful overview of the current development status of FDsys.

    I would like to correct one factual error. You state that in our report “Documents for a Digital Democracy: A Model for the Federal Depository Library Program in the 21st Century,” Ithaka S+R is “laying the groundwork” to create a “private-sector, fee-based” preservation and access system. In fact, in our report, we write:

    * “The Program’s core mission of providing no fee “permanent public access” to government documents is just as important today as it ever has been” (page 10).

    * “The fundamental vision for government information held by stakeholders – permanent no fee public access – remains essentially unchanged in the transition to the digital age. Stakeholders told us again and again that they would like to see a world in which government information is seamlessly available to the range of potential users without charge at their point of need and is preserved appropriately for future generations. The digital transformation will only be successful if it conserves and enhances the ability to achieve this vision…To dramatically expand access and discovery and enable a wide range of innovative uses of government information, it must be made freely available in digital form” (page 34).

    * “The availability and accessibility of materials may range broadly. Some materials – especially the highest value digital collections such as those produced by Lexis and Readex – are only available through those libraries that pay fees to license them, but these collections are presented in highly tailored interfaces developed with a deep awareness of user needs. Libraries may or may not license tools at their own discretion, based on their evaluation of the utility of these tools and collections to their users relative to the availability of funding; they are in no way obliged to do so. These materials do not contribute to the goal of permanent public access as they are not available uniformly to the public. As such, they should not be considered as playing any role in the long-term preservation infrastructure for government information…Although these other collections serve a variety of critically important roles for users of government information, often making up for inadequacies in the FDLP, they remain fundamentally separate from the FDLP” (page 26).

    * “[T]here remains substantial value to be realized through the continuance of GPO’s traditional role in working to aggregate the publications of federal agencies into the Program” (page 43).

    * “To enable the opportunities associated with digital deposit and other service models, GPO should build upon its recent successes in releasing the Federal Register in XML form via free bulk download of all its digital holdings. To support such work, it may be useful for GPO to staff a developer services position, as has been recently suggested” (page 45).

    The framework that we outlined in our report emphasizes the importance of no-fee permanent public access, including a robust multi-entity model for preservation. Our report supports the vital roles of the GPO, the FDLP, FDsys, certified preservation archives, and digital deposit, as elements of a healthy ecosystem for free access and robust preservation. I look forward to participating in continuing dialogue on these critical issues.

    Kind regards,
    Roger

    Roger C. Schonfeld
    Manager of Research
    Ithaka S+R

  2. jajacobs says:

    Roger,

    Thank you for your comment. I would be very happy to hear from you that I have misunderstood the Ithaka S+R report. But, I don’t think I did. Let me explain. My understanding is that we are talking about two very different visions of the future of the FDLP.

    Two Different Visions.
    In one vision (the one we promote at FGI), FDLP libraries will be actively involved in building digital collections and providing services for those collections. They will collaboratively and collectively provide better preservation and better services to more communities of interest than any single-entity (or small number of entities) could provide on their own. Their strength and durability come from their diversity: diversity of technologies, diversity of funding sources, diversity of governing bodies, diversity of user communities, diversity of tools, diversity of interfaces, and diversity of collection content. This is a “both/and” vision of the future that is big enough to include big and small libraries, libraries of all collection sizes (from nothing to small to large), and even “dark” and “light” archives. Our vision is one that imagines a robust FDsys with the documents in it also in LOCKSS and big digital archives like Portico and HathiTrust. But our vision also imagines dozens, hundreds, maybe even a thousand FDLP libraries actively participating by having their own digital collections of Title-44 digital materials along with non-Title-44 digital materials.

    My understanding of the other vision of the future of FDLP (including the Ithaka S+R framework) is the one that GPO and others have been promoting since 1993: the “paradigm shift” in which FDLP libraries no longer have or control collections of government information. In the GPO-centric model, FDsys would be the sole, “official” repository of Title-44 materials. In this model, the role of FDLP libraries is diminished to providing services for those collections held and controlled by GPO. My understanding is that the Ithaka S+R report recommends only a slight variation of this model. Rather than libraries relying on GPO to preserve government information, libraries would pay Portico (or others) to preserve government information.

    Laying the groundwork for more of the same.
    This is what I meant when I referred to Ithaka “laying the groundwork” for a proposal for FDLP libraries to rely on a single organization rather than a community of organizations. And this is what I meant by a “fee-based” system: moving from preservation being provided by a government-funded agency (GPO) to a model that charges fees to member libraries for its services. To me, the Ithaka S+R framework is very close to the seventeen-year-old GPO vision that diminishes the role of FDLP libraries and greatly centralizes preservation.

    To be more precise and expand on my earlier statement, it would be more accurate to say that Ithaka S+R recommends a model for preservation (not access) that involves a “small number” of entities:

    GPO should develop formal partnerships with a small number of dedicated preservation entities — such as organizations like HathiTrust or Portico or individual libraries — to preserve a copy of its materials” (Documents for a Digital Democracy, page 44).

    In this recommendation, Ithaka S+R only changes the single-entity model to a “small number” of entities. In this model, most (or, perhaps, all) FDLP libraries would be left without collections to build or manage and the services that they could provide would be limited to those allowed by the small number of entities. Title-44 authorized FDLP library collections would essentially be replaced by new “partnerships.”

    This is an explicit recommendation for libraries to outsource preservation — and pay fees to do so. This is, as I understand it, the business model for Portico.

    It is not necessarily a bad thing for libraries to pool their resources by using organizations such as Portico and JSTOR and HathiTrust to perform complex, expensive tasks. But I believe that libraries should carefully examine the consequences of such a move and also examine alternatives. I believe that as more libraries take on the role of digital preservation and providing services for those digital collections it will become less complex and less expensive to do so. I believe that as more libraries actively participate in digital preservation, we will, collectively, have a better chance of successfully preserving more digital information than we will if we fob off that responsibility to a few big organizations. I worry that relying on a framework based on outsourcing and pay-for-service will promote competition among entities rather than cooperation. I wonder how much of a voice libraries and their communities-of-interest will have in setting policies of the big aggregators. I worry that a model like this is another “big deal” that we will regret later. I am wondering and speculating here, but I think we should ask these kinds of questions before continuing to accept the failing “paradigm shift” model.

    It is also important to understand that this part of the Ithaka S+R framework addresses only preservation, not access.
    Let me explain to those not familiar with your organization that ITHAKA is the parent organization to JSTOR, Portico, and Ithaka S+R (which wrote the report). Portico charges fees to libraries and publishers for its preservation services. Portico is, essentially, a fee-supported “dark archive.” Content in Portico only becomes available to users when a title is no longer available from its producer. Currently, for example, of the 10,861 journal titles in Portico, only 4 titles are available for access.

    Broader solutions.
    As I said above, I believe that it is possible for a future FDLP to include all kinds of organizations (including ones like Portico). At FGI, we believe that the role of FDLP libraries in the future of the FDLP should include active participation in building digital collections for preservation and access rather than simply relying on others to take on those roles. We believe that the old model of relying on one (or a small number) of others for these services is outdated and failing (as demonstrated by the current state of FDsys). We believe it is time to re-think that old model.

    Roger, I think that the Ithaka S+R framework could be broadened to be more inclusive and include more opportunities for FDLP libraries. If it already is that broad and I have misunderstood your proposals, if they are more expansive than I have portrayed them here, please let me know.

    Thanks again for your comment.

    Jim Jacobs

  3. jajacobs says:

    Roger,

    Thanks for continuing this exchange and filling in some of the gaps. I won’t be at DLC, but hope that you and my colleague James R. Jacobs will have a chance to continue this discussion in more detail.

    It is good to know that we do have a lot on which we can agree. To me, there seems to be in the community pretty general agreement: we all value long-term, free, public preservation of, access to, and services for government information. Everyone agrees that this will involve lots of different participants and stakeholders and that libraries can, or at least should, have a role.

    The big unresolved questions seem to be “who will pay?” and “what is the role of libraries?” The answers to those questions seem to depend on who your “constituents” are (“where you stand depends on where you sit”). The private sector has stock holders, GPO has JCP and Congress, non-profits have funding agencies, libraries have governing boards and user communities, etc. Library directors are driven by costs and budget cuts, library staff are driven by decisions made by their management, and so forth. Even within groups, the dynamics are complex and sometimes ruthless. I have spoken to librarians who want a bigger role, but either assume they won’t get it or are afraid to ask.

    Just to be clear: we at FGI are unpaid volunteers who advocate “Free Government Information.” That’s free as in no-fee and free as in free from constraints and controls on re-use. It isn’t clear to me what the role of Ithaka S+R is. Is it a disinterested research group, or a market research group for ITHAKA, or an advocacy group, or something else?

    Obviously, FDLP is facing a time of change and change is hard. Tough financial times make change all the more difficult. Prudent library directors are, no doubt, looking for places to cut the budget. No one wants to unnecessarily duplicate services when there is not enough money to get everything done.

    With all that as context, I think your last comment raises more questions than it answers.

    For example, “Putting aside the merits of any particular vision for the future…” doesn’t seem like a very good way to plan for the future. Don’t we need to examine what we should do before we decide how to do it? Don’t we need to lead and not follow trends or base our future on what some thought would work (the “paradigm shift”) almost 20 years ago?

    I’m confused by your response that the report “did not assume” that “libraries would need to pay for these preservation services” and that “there could be a variety of models, including GPO funding.” As a practical matter, how can you, on the one hand, effectively reject a collaborative model of FDLP libraries having digital collections apparently because of a lack of adequate funding, and then, on the other hand, not specify who will pay for the more centralized model? How can you base a future on the hope that GPO will be able to afford to pay Portico when it cannot currently afford to build FDsys?

    It is nice to hear that Ithaka S+R “did not assume … that a dark model would be pursued” by Portico if it participated. But that raises questions. Won’t there be a big cost for Portico to develop a “light” archive? How do you know that Portico will or can do that? Or, would Portico keep costs low by having a minimal user interface? (I didn’t notice, in the list of interviews conducted, that you interviewed anyone from Portico. This again prompts me to wonder about the relationship between Ithaka S+R and Portico and whether the report speaks for Portico or, if not, what justifies the report’s assumptions.)

    I wonder if, in developing the framework for the FDLP, you examined how economies of scale might work for different models (e.g., pooling resources into a few archives, versus building collaborative tools that many libraries could use).

    I wonder if there are incentives that could be used to encourage library directors to support FDLP. I would guess that library directors were eager to support FDLP when it increased their volume count. If it is reasonable to assume that GPO could fund centralized archives like Portico, isn’t it also reasonable to consider the possibility of GPO facilitating the building of digital infrastructures in libraries that chose to participate in a digital FDLP? Couldn’t GPO and others with technical experience (Portico, HathiTrust, OCLC, LoC) help libraries build the expertise they needed for digital infrastructures that would support institutional repositories, next-generation catalogs, local digital archives, scanning projects, and other digital library projects, as well as FDLP? Wouldn’t such things provide new incentives for libraries to support FDLP membership? With a bit of vision, couldn’t FDLP become a leader in building digital libraries rather than an orphan of the print era?

    FDLP libraries have faced other technological changes that, at the time, seemed expensive and difficult. Think about the history of microfilm, microfiche, floppy disks, CDs, DVDs and the first public service workstations. In each case there were ample reasons for libraries to prefer to avoid the new technology; (they were too heavy, too obscure, too difficult to use, too difficult to provide service for, too expensive, etc. In the 1970s, it took almost seven years to get microfiche into the program!). If you gave the libraries a choice over those changes, I would guess that many would have said something like, “Putting aside the merits of any particular vision for the future, the reality is that adequate funding is required to carry out these activities. If someone else will do this for us, we have a long list of things we’d do instead.”

    The difference between then and now is that GPO has promised libraries that they will not have to take on digital-library responsibilities and many libraries have “bought in” to that model. Now that that promise is not looking as reliable as it once did, we need a new plan. The fact that many librarians and library directors are hoping that someone else will do this for them does not make that hope right or that plan any different from what we have been doing. Basing a “framework” for the future only a survey of stakeholders at a time like this will not necessarily lead to the best solutions.

    If we think about the future of the digital FDLP and digital libraries in general, there is a lot to be said, even financially, for a more collaborative approach. For example:

    1. There is too much digital information to preserve without lots of players cooperating. Just the decision of what to save needs many voices or we will lose valuable information.

    2. Lots of copies keep stuff safer than single, backed-up copies.

    3. As I mentioned in my last comment (see above), having lots of collaborators will be more durable because of the various kinds of diversity that those collaborators will bring in aggregate.

    4. I think we are faced with a “chicken and egg” situation: If we permit producers to give us any digital formats that they find easy to produce, we will be forced to have expensive “ingest” functions to deal with hard-to-preserve files. As a consequence, it will be hard and expensive for any organization to be a certified repository. If, on the other hand, libraries stake out a role and lead, and if we have lots of libraries/archives/repositories demanding of producers easily-ingestable, easily-preservable digital materials, the process will be less expensive and less difficult for all. I believe Portico is already doing this to some extent and that is good. Why not look to that effort paying off in reduced costs of being a certified repository? Why not encourage other repositories to share the load?

    Roger, I think we do agree on the outcome we want and we agree that libraries will have a role in building that outcome. But I’m not sure we agree on what is the best role for libraries to play. The Ithaka report seems to me to advocate something very similar to the old, GPO, centralized-repository model. The role of libraries in that model is more like that of a travel agent who can help users but who controls nothing. Travel agents aren’t doing too well, while private sector information providers like LexisNexis are still in business because they get actual digital content so that they can add value and provide services. Which is the better, more sustainable model for libraries? I think the central role of libraries in society is shifting control of information from producers to communities.

    Thank you again for contributing your comments to this thread and helping us all understand the “Documents for a Digital Democracy” report better.

    Jim Jacobs

  4. Jim,

    Thank you for your clarification.

    Ithaka S+R developed our framework based on a broad set of interviews with diverse members of the government information community. Let me say a little bit more about what we learned and how that connects to the framework.

    We heard a fairly consistent story across numerous federal depository library directors, at the largest research libraries and at smaller institutions as well, that they are presently not in a position to invest resources in building local collections of digital government documents. Consequently, our report was responding to and addressing that input. There is a concern that sufficient resources might not be made available for local initiatives to acquire, maintain, preserve, and provide access to digital documents. Putting aside the merits of any particular vision for the future, the reality is that adequate funding is required to carry out these activities. If these resources are available, it would be great if libraries do this work simultaneously with the development of centralized options. Having both a small number of central efforts and local efforts will offer an additional form of diversity; the value of coordination with central approaches and the value of a range of approaches that a diverse ecosystem offers. If you are right that the complexity and cost of locally managing digital collections will fall radically over time, we will be in a position to benefit from this combination of local and central approaches.

    Given our shared objective to ensure no-fee permanent public access and shared concern about relying on GPO alone for preservation and access services, we stressed the importance of other options. For preservation, we recommended that GPO build “formal partnerships” with preservation archives. We called attention to two preservation archives (HathiTrust and Portico) that were then moving through the CRL certification process, but any certified preservation archives interested in this role should be considered. We did not assume (as you state) that libraries would need to pay for these preservation services or that a dark model would be pursued; there could be a variety of models, including GPO funding, to ensure that certified third-party preservation is ensured for these most important digital materials.

    An effort by GPO to build formal relationships with one or more certified preservation archives should not replace a broader vibrant ecosystem. For this reason, we recommended that: “A multiplicity of discovery and access environments should be allowed, and indeed encouraged, to flourish. Libraries, partner organizations, commercial entities, and others, all can have a role” (page 45). I would like to see libraries play as large a role as possible in this environment, and I look forward to seeing the services that libraries will continue to develop to serve the needs of 21st Century users of government information.

    If you will be at the DLC meeting next week, perhaps we could continue the conversation at greater length in person, or otherwise by phone in the coming weeks.

    Best,
    Roger

Post a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Archives

%d bloggers like this: