Toward estimating file sizes of online federal documents

July 21, 2006 / 2 Comments on Toward estimating file sizes of online federal documents

Earlier this year, Daniel estimated the average size of an online federal document as between 5MB and 10MB. Libraries investigating digital deposit and provision of permanent public access to these resources need to estimate the cost of storage for these documents.

For the past week, I've played around in an entirely nonrandom sample of online docs to try to get an accurate estimate. Although I'm not close to a reliable estimate, I'd still like to share what I've done...

The process:

  1. grabbed all (1,234) MARC records with 856 fields from DDM2 for the GPO Timestamp range 2006 06 01 - 2006 06 30
  2. used wget to retrieve all URLs listed in those 856 fields
  3. slapped the wget logs into a vaguely useful excel spreadsheet (thanks to liberal regexp-ing in jEdit)

The basic results:

TOTAL URLS 1342
TOTAL SIZE MB 2004.7
AVG SIZE KB 1530

'Course, these numbers don't mean much against a little scrutiny. The 856 field often points to table of contents pages (when it points to the document at all...), and that single page is all that gets counted in this simple investigation.

PDF files might offer a better estimate than HTML files. Although publishers can split up documents into multiple PDF files and have a "Table of Contents" PDF file point to these multiple resources composing a single bibliographic unit, this doesn't appear to be too common. When 856 fields point to PDF files, they tend to be self-sufficient, whole bibliographic units. So here are the numbers for pdf files retrieved using the 856 fields:

FILETYPE pdf
TOTAL URLS 815
TOTAL SIZE MB 1961
AVG SIZE KB 2464
STD DEV SIZE KB 7605
MAX SIZE KB 148902

In a true demonstration of futility, I looked at 124 of the HTML files (of the 525 in the June 2006 DDM2 sample) that are stopping points for the 856 pointers. Most of these totally-non-random-sample HTML pages to not constitute the entire document described in the MARC record. I developed various wget capture strategies for 84 of these online documents, and the average size of the "cluster" of files captured per 856 pointer was 8.17 MB (median: 3.19 MB, std dev: 13.09 MB).

In a vaguely related exercise, I grabbed the various files composing Foreign Relations of the United States, vols. E-1, E-5, and E-7. Sure, they're outliers w/r/t size, but I thought I'd mention them anyway...

VOL MB FILES
E-1 318 880
E-5 143 687
E-7 618 892

CONCLUSION:

I don't have one yet. At the end of the week, though, 5-10 MB seems like a pretty good estimate to me.

Continue reading →

Continue Reading →

Part 11: Nonlawyer’s Journey through Title 44 – Introducing the Sales Program

July 20, 2006 / 1 Comment on Part 11: Nonlawyer’s Journey through Title 44 – Introducing the Sales Program

This post, all earlier postings in this series, and my "not a lawyer" disclaimer can be found at http://freegovinfo.info/title44 or through our library under Nonlawyer's Journey through Title 44. After a few weeks hiatus, I'm ready to tackle another major section of Title 44 that affects the dissemination of government information and Federal Depository Libraries. I'm speaking of 44 USC chapter 17: Distribution and sale of public documents. While this is a section of law that many depository librarians, myself included, wish didn't exist, it does and so needs to be examined. In addition, it was first proposed with good intentions. In an age of paper, the Sales Program helped provide some of the funding needed to make publications freely available to the public through Federal Depository Libraries. In a mostly electronic age I believe that the existence of the sales program is a direct threat to free access to government information, at least to access at times and places of a citizen's choosing. But I'll more to say about the interactions current and possibly future between the depository program and the sales program after we've concluded our examination of chapter 17. Unless people ask questions either through comments or by e-mailing me at dnlcornwall AT alaska.net We begin our consideration of chapter 17 with Sec. 1701:

TITLE 44--PUBLIC PRINTING AND DOCUMENTS CHAPTER 17--DISTRIBUTION AND SALE OF PUBLIC DOCUMENTS Sec. 1701. Publications for public distribution to be distributed by the Public Printer; mailing lists Money appropriated by any Act may not be used for services in an executive department or other Government establishment at the District of Columbia, in the work of addressing, wrapping, mailing, or otherwise dispatching a publication for public distribution, except maps, weather reports, and weather cards issued by them or for the purchase of material or supplies to be used in this work. The Public Printer shall perform this work at the Government Printing Office. The head of an executive department, independent office, and establishment of the Government at the District of Columbia, shall furnish from time to time to the Public Printer mailing lists, in convenient form, and changes in them, or penalty mail slips, for use in the public distribution of publications issued by the department or establishment. The Public Printer may furnish copies of a publication only in accordance with law or the instruction of the head of the department or establishment issuing the publication. This section does not apply to orders, instructions, directions, notices, or circulars of information printed for and issued by an executive department or other Government establishment or to the distribution of public documents by Senators or Members of the House of Representatives or to the Senate Service Department, House of Representatives Publications Distribution Service, and document rooms of the Senate or House of Representatives. (Pub. L. 90-620, Oct. 22, 1968, 82 Stat. 1278.) Historical and Revision Notes Based on 44 U.S. Code, 1964 ed., Sec. 95 (Aug. 23, 1912, ch. 350, Sec. 8, 37 Stat. 414; July 2, 1954, ch. 455, title I, Sec. 101, 68 Stat. 397). ``House of Representatives Publications Distribution Service'' is substituted for ``House Folding Room'' because of the change of name under authority of Public Law 88-652. The term ``executive department, independent office, and establishment of the Government'' is substituted for ``executive department and other Government establishment'' for uniformity. Section Referred to in Other Sections This section is referred to in section 1722 of this title.
As near as I can read this section, it seems to keep maintenance of mailing lists of items with the agencies, while insuring that agency money won't be used in mailing publications to customers. But if a publication is being distributed within the government only, then the agency picks up the distribution tab. If someone has some better insight, please offer it in the comments. Next time we will look at 44 USC 1702, which spells out some of the sales duties of the Superintendent of Documents, the same official who is in charge of the Federal Depository Library Program. [Update - July 25, 2006] I was mistaken about the Sales Program EVER subsidizing the depository program. Here is the the real story straight for Superintendent of Documents Judy Russell:
----- Original Message ----- From: "Russell, Judith C. (SuDoc)" To: "'Discussion of Government Document Issues'" Cc: Sent: Tuesday, July 25, 2006 5:05 AM Subject: RE: New Discussions at FGI: July 24, 2006 In his most recent analysis of Title 44, Dan Cornwall states: "In an age of paper, the Sales Program helped provide some of the funding needed to make publications freely available to the public through Federal Depository Libraries." The GPO Sales Program is by statute a self-sustaining program. Revenue from the Sales Program has never been used to fund the Federal Depository Library Program (FDLP), which receives its funding from annual Congressional appropriations. By providing a means for individuals and organizations to purchase copies of Federal government publications, the GPO Sales Program complements free public access through tangible depository collections and free online access. It will continue to serve the public by providing an alternative to using tangible Federal publications in libraries or downloading/printing copies from the Internet. Judy Russell Judith C. Russell (jrussell@gpo.gov) Managing Director, Information Dissemination (Superintendent of Documents) U.S. Government Printing Office Phone: 202-512-0571 Fax: 202-512-1434
Continue reading →

Continue Reading →

Federal Depository Libraries – GovDocs for Jocks

July 15, 2006 / 2 Comments on Federal Depository Libraries – GovDocs for Jocks

I apologize to all the govdocs jockies who are a little underwhelmed by my 101 approach to all things document-y, but I've been enjoying all the new stuff I've learned. Today I'm talking about Federal Depository Libraries. If you went to library school you may have had a class in GovDocs where you learned about FDLs and their complicated and mysterious cataloging and classification system and met very odd people who made these systems their lives. By the end of the GovDocs class, most people were terrified but a few were hooked. I can remember almost every GovDocs librarian I've ever met and I don't think this is true for any other type of librarian I've encountered. So, here are some things you need to know. Other GovDocs fanboys and fangirls, please add more information in the comments. This page has an intro to the FDL program for further reading.

  • The US has 1250 Federal Dpository Libraries nationwide. You can find the one nearest you by going to this page. Want to print a list for your state? Go to this page.
  • The broad purpose of these libraries is to "provide local, no-fee access to Government information in an impartial environment with professional assistance"
  • ANYONE can go into a FDL, even if the collection is part of a larger private library that you would not otherwise have access to. This is true, for example, at the Elmer Holmes Bobst Library where they have a very strict visitor's policy and you have to read all the way to the bottom of this page until you see the words "Government Document U.S. depository (Bobst 6th floor) open to general public"
  • The purpose of these repositories is changing in the face of the shift to more and more information being created and disseminated digitally. The FDLP was created and mostly maintained as an avenue to get print publications from the Government Printing Office to the public who they serve. The GPOs role is shrinking
Continue reading →

Continue Reading →

Best titles ever!

July 13, 2006 / Leave a comment

Hi all. Spurred by the recent posts on govdoc-l, I've made a page to collect all of those strange/odd/intriguing/inspiring government documents. Check it out and let us know if you have a favorite that you'd like to add to the list. Continue reading →

Continue Reading →

Best titles ever!

July 13, 2006 / 14 Comments on Best titles ever!

Below are some government document titles that are interesting/strange/intriguing. We set this up after several emails on govdoc-l. Click on any cover for a larger image. All images hosted by Flickr. We've created a WorldCat List for the titles below when we could find records. Some of them aren't in WorldCat or GPO's Catalog of Monthly Publications. Please send your suggestions for this list to admin AT freegovinfo DOT info. If you want to explore the possible future barriers to the enjoyment of titles like these, please explore our issues page.

  • M16A1 rifle : operation and preventive maintenance. DA Pam 750-30. While the title is not particularly funny, the fact that this is a comic book (and a heavily mysogynistic one at that) distributed to all US military personnel in Viet Nam makes this a good title to highlight here. Find in a library
  • After the Cold War: Living with Lower Defense Spending. (1992) Office of Technology Assessment. [Y 3. T 22/2:2 C 67/SUMM. This one seems funny in retrospect!
  • ALONE: Facts about Mental Health and Mental Illness was revised to: YOU ARE NOT ALONE: Facts about Mental Health and Mental Illness [HE 20.8102:Al7/985] "One of my favorite titles is from the National Institute of Mental Health, 1985. The document had a printing error in the title and was ordered destroyed by the Superintendent of Documents. "This error seriously detracts from the point that the publisher was trying to make." Find "Alone" in a library Find "You are not alone" in a library
  • An Annotated list of Literature references on carpets and rugs 1940 to 1963 [A77.17:C22]. (June 1965, Department of Agriculture) An Excerpt: There is no magic instrument endowed with a power to give its bearer a complete single factual answer to many of the questions asked by consumers today on the selection and care of carpets and rugs (so the Ag Department needs to tell us, obviously) Find in a library
  • Bat Bombers. By C.V. Glines. Air Force magazine, October 1990 Vol. 73, No. 10. "After the bats set fire to a hangar and a general's car, the Army Air Forces had seen enough of the experiment."
  • The Bear Essentials (Irregular) [HE 20.3057:]
  • Child Maltreatment: A "What to Do" Guide for Professionals Who Work With Children - Published by Canada's National Clearinghouse on Family Violence. Contributed by Amy West. Find in a library
  • Citizenship and the Strawberry Jam: FDA's Life Protection Series (1967, for FDA and US dept of health, education and welfare)
  • Cooking up solutions: cleaning up with lasagna [EP 1.2:C77] Find in a library
  • Dante Home Reading Course. [I 16.12: 18] - Photo contributed by Christine Gray.
  • Design of Clean Rooms - A Classified List of Selected References 1955-1964
  • Distinguishing Bolts from Screws. US Customs and Border Protection, March 2006.
  • Disposition of Useless Papers in American Consular Offices August 22, 1921. US Serial Set 67th Congress 1st Session Senate Report no 271 [Y 1.1/2:Serial]. This title was found and tweeted about by Ezra Klein and Aaron Swartz on 1/3/2011. Find in a depository library
  • Do you know oatmeal? [A 1.26:6] - Photo contributed by Christine Gray
  • Don't be a "sugar daddy" to moonshiners! The True story of moonshine. [T 22. 2: M77]. Find in a library.
  • The Doubtful Identity of Fungus No.517, by C. Audrey Richard, Pathologist. U.S. Department of Agriculture, Forest Service, Forest Products Laboratory, in cooperation with the University of Wisconsin, Madison, Wisconsin, [no date]; Published in Proceedings of the American Wood Preservers' Association, 1937.
    Find in a library
  • Elder abuse, neglect, and exploitation : are we doing enough? [Y 4.J 89/2:S.HRG.108-445] Find in a library
  • Employment of homosexuals and other sex perverts in government (1950)
  • Epic of Gilgamesh: Clinical Practice Guidelines for Post-Deployment Health Evaluation and Management (Flash movie). Department of Veterans Affairs / US Department of Defense. Originally posted on FGI blog. (Added 12/30/06)
  • Everything you always wanted to know about shipping high-level nuclear wastes [E 1.16: 0003] Find in a library - Photo contributed by Christine Gray.
  • Fakability of the air force preference inventory - Find in a library
  • Familiarity and recognition of nonsense shapes
  • Fastener Quality Act : report of the Committee on Commerce, Science, and Transportation on S. 795. Find in a library.
  • Fatalgrams. U.S. Dept of Labor, Mine Safety and Health Administration. [L 38.15:] Find in a library / Get the T-Shirt (Not affilated with FGI) - Photo contributed by Christine Gray. As were photos for:
  • Fertilizers in a national emergency [A 1.6: 966] Find in a library - Photo contributed by Christine Gray
  • The golden age of bathing. [I 29.2: B32/4] Find in a library - Photo contributed by Christine Gray.
  • Gobbledygook has gotta go. Bureau of Land Management [I 53.2: G53] Find in a Library.
  • Government versus homosexuals. by United States. Congress. Senate. Committee on Naval Affairs. (1975) Find in a library
  • Health effects of pesticide use on children [Y 4.L 11/4:S.hrg.101-569]
  • Hills Bros. coffee can chronology : field guide by Steve Lanford; Robin Mills; Chuck Adkins; United States. Bureau of Land Management. Fairbanks District Office.
  • Identifying sheep killed by bears [A 13.79:INT-313]. U.S. Dept. of Agriculture, Forest Service, Intermountain Forest and Range Experiment Station, 1981. - Photo contributed by Christine Gray. B says:
    It is an oldie, but still brings a chuckle when I think of it. In 1981, I began as a clerk in the Gov Pubs unit at ******** University. On my first day, I was to open a new box of materials and process it all the way through to get some sense of the work flow. "Identifying Sheep Killed by Bears" was the very first publication I pulled from the box. It kind of set the tone for my time in the department.
    Find in a library
  • The impact of computer aliens along the Mexican and Canadian borders. [Y 3.W 52:2 C 73/PT.2]. This is the cover title of part 2 of this 3 part set of hearings, published in 1968-69. David says:
    This is my favorite typographical error on a Federal publication. The publication deals with the impact of nonresident alien workers who travel daily to jobs in U.S. border cities.
  • Impregnation of concrete pipe. [Prepared] for the Water Quality Office, Environmental Protection Agency., [EP 2.10:11024 EQE 06/71]. Find in a library. James S. says:
    The title alone is pretty funny in a puerile kind of way, but the cover picture is downright creepy in a Freud kind of way.
    Regardless, we should thank the brave young servicemen who were willing to sacrifice themselves for love of science and love of country.
  • Index of blank forms, [D 101.22:310-2/]. Amy says:
    It has been discontinued, although not for the reason you might think: "Merged with: Index of administrative publications; Index of blank forms; Index of doctrinal training organizational publications; Index of technical publications; Index of supply catalogs and supply manuals, excluding types 7, 8, and 9, to form: Consolidated index of army publications and blank forms."
    Find in a library - Photo contributed by Christine Gray.
  • It's always been dumb--now it's illegal too! anabolic steroids pose serious health risks to users (Poster). Washington, D.C. : Demand Reduction Section, U.S. Drug Enforcement Administration, [1991?] Find in a library (WorldCat)
  • Kill or get killed : riot control techniques, manhandling, and close combat, for police and the military. U.S. Marine Corps [D 214.9/6:12-80] with illustrations! Find in a library (James notes that it was such a hot title that someone actually published it privately -- in the same year! Check out the OCLC records: 25298518 (US doc) 3328029 (paladin press). It looks like they were published at the same time, or at least in the same year. Daniel was able to obtain a PDF copy of a 1991 reprint of the Marines publication (Fleet Marine Force Reference Publication (FMFRP) 12-80, Kill or Get Killed,) and found it bore this note: Copyright 1976 Paladin Press. Reprinted with permission by Paladin Press, P.O. Box 1307, Boulder, CO 80306. Web site: http://www paladin-press .com In addition the author is listed as a retired officer, so it looks like the private pub came first!) - Cover photo provided by Kathy Amen.
  • Know your 8-inch Howitzer. Dept. of the Army, Headquarters, US Army Armament, Munitions and Chemical Command], 1984- [D 101.22/12:750-6] Find in a library - Photo contributed by Christine Gray.
  • Methods of reduction of psychological stress due to radiation, AFPTRC-TN (series) [D301.36/7: 57-19].
  • Lawn Mowers: Noise and Cost of Abatement (EPA 1974)
  • Let potatoes fight. [Y3.F73/2: WGC/ w49]
  • Let's use TV! [ED 1.302: T23] Find in a library - Photo contributed by Christine Gray.
  • L.U.S.T.LINE Subtitle is: A report on Federal and state programs to control leaking underground storage tanks. A publication of the New England Interstate Water Pollution Control Commission, funded with support from EPA's Office of Underground Storage Tanks. Link to the publication's online archives and index David says: My favorite article title from L.U.S.TLINE (from #41, June 2002): "Looking for Leaks in All the Wrong Places: A Short Story With an Epiphany," by Marcel Moreau
  • Manual for Medical Examination of Aliens. Washington, D.C. : U.S. Dept. of Health, Education, and Welfare, Public Health Service. [FS 2.6:Al 4/956]. Find in a library
  • Meat is ammunition, play fair with it! [PR 32.5002:M 46/2] Monthly catalog, 1945, p. 648. A wartime exhortation to U.S. consumers to adhere to meat rationing limits.
  • Methods of reduction of psychological stress due to radiation
  • National community fallout shelter design competition (1965)
  • National Money Laundering Strategy [T 1.67/2:]. I've heard there's corruption in government, but this is ridiculous!
  • Out of the vapors: a social and architectural history of bathhouse row [I 29.2:B 32/3 ] Find in a library - Photo contributed by Christine Gray.
  • PMS Blue Book. HE 20.4001/2: PMS in this case stands for Program Management System. Find in a library - Photo contributed by Christine Gray.
  • Patents in space. [Y4.J89/1:99-16] Find in a library - Photo contributed by Christine Gray.
  • Plane Clothes: Lack of Anonymity at the Federal Air Marshal Service Compromises Aviation and National Security. What a painful pun! Here's the PDF from fas.org because the link to text at gpoaccess didn't work (as of Aug 8, 2006)! Please contact us if you click the link and it works. (11/24/08 update: link to text STILL doesn't work but there's a PDF available of the committee print).
  • Preservative treatment of poles (USDA 1911)
  • Psychological Aspects of Equipment Design (1949). (USAF document. Attached is a page from the document "knobshapes." Photo contributed by Sarae Montgomery.
  • Reducing Americans' Vulnerability to Ecstasy Act of 2002 : hearing before the Subcommittee on Crime, Terrorism, and Homeland Security of the Committee on the Judiciary, House of Representatives, One Hundred Seventh Congress, second session on H.R. 5519, October 10, 2002 [Y 4.J 89/1:107/109]. James says: wikipedia points out the ironic acronym of the title, "RAVE act"! Who ways our representatives aren't hip to the groove?! Find in a library
  • Relation of Barium to the Loco-weed disease
  • Some effects of modification of information about a previous response upon the acquisition of two lever positioning habits. USAF
  • Space handbook : a war fighter's guide to space / prepared by Michael J. Muolo ; compiled by Richard A. Hand ; edited by Richard A. Hand, Bonnie Houchen, Lou Larson. [D 301.26/6-8:SP 1/2]. Find in an FDLP library. Document also available online and as a PDF from FGI.
  • "Sprocket Man" (PDF). U.S. Consumer Product Safety Commission (AKA CPSC Comics!) Find in a library
  • Status Report of U.S. Treasury-Owned Gold [T 63.131:] The person who submitted this says:
    this one has a perfectly boring title but I have nicknamed it "Gold-hoards of the T-men", which sounds much more exciting!
  • State-of-the-art dummy selection. U.S. Dept. of Transportation, National Highway Traffic Safety Administration. TD 8.2:D 89/5/v.1 Find in a library. Janette says:
    I guess the other volumes were classified "top secret".
    Photo contributed by Christine Gray.
  • Step into action! A guidebook for the above-knee amputee. US Dept of Health, Education and Welfare. [FS 2.6/2:ST 4/964] Find in a library - Photo contributed by Christine Gray
  • This is Ann ... she drinks blood! prepared and distributed by Army Orientation Course, Special Service Division Army Service Forces, War Dept. Unattributed illustrations by Theodor Seuss Geisel (Dr. Seuss). Find in a library (WorldCat).
    This is Ann . . . she drinks blood!
    • Publisher: Washington, D.C. : Special Service Division, Army Service Forces, War Dept. : U.S. G.P.O., 1943.
  • Trends in Money Laundering [T 1.67:] and National Money Laundering Strategy [T 1.67/2:] - Photo contributed by Christine Gray.
  • The use of a sentence completion test in measuring attitudes toward superiors and subordinates
  • Virtual Cement and Concrete Testing Laboratory Consortium, Annual Report. [C 13.58/4:]. Lori says: "Virtual cement and concrete? A virtual testing laboratory? A virtual consortium? Who knows?
    Find in a library
  • Vitamin A in war and peace. [A 1.59:2] - Photo contributed by Christine Gray
  • Wake up America: a national sleep alert. [Y3.2: SL2/SL2] Find in a library
  • A winning combination: wild horses and prison inmates. [I 53.2: H78/10] Find in a library - Photo contributed by Christine Gray.
  • WRA: A Story of Human Conservation. (Hosted at the Internet Archive!) U.S. Dept. of the Interior, War Relocation Authority, 1946. (I 59.2:W 19/3:). Jim says:
    This is the final report of the agency which oversaw the World War II internment camps for US citizens of Japanese and other suspect descents. The title is the very definition of Orwellian, and I often use it to help drive home the point that government documents are mirrors of their time.
    Find in a library
  • (Canadian govt document) "Who Are the Zombie Masters and What Do They Want?" (PDF). A 1994 discussion paper prepared for the Premier's Council on Health, Well-Being and Social Justice. Find in a library
  • Young scientists look at drugs (1968 for FDA/HEW)
  • Your personal affairs Your personal affairs / Armed Forces Information and Education, Department of Defense. [D 2.14:PA-6] 1963
Now that you've had a taste of the kinds of information your government produces, why not explore the issues that may prevent full access to government information in the future?a href= the ironic acronym of the title, http://www.flickr.com/photos/10325651@N00/198050294/ahttp://static.flickr.com/64/198463301_7de3b72beb_s.jpgimg src=img src=http://www.flickr.com/photos/alaskanlibrarian/209582945/in/pool-besttitlesever/ Continue reading →

Continue Reading →

Post navigation

← Older posts Newer posts →

Archives

Powered by WordPress / Academica WordPress Theme by WPZOOM