The Securities and Exchange Commission announced a new service that allows you to "instantly compare what 500 of of the largest American companies are paying their top executives."

• Chairman Cox Unveils New Internet Tool With Instant Comparisons of Executive Pay (press release) Washington, D.C., Dec. 21, 2007

The announcement says that The Executive Compensation Reader is "available today on the SEC's Web site at http://www.sec.gov/xbrl."

When you go to that web site, you'll find "Interactive Financial Reports", "Executive Compensation", and (coming soon) "Financial Explorer" and "Investment Management Risk Return Summary".

But...A big, boldface disclaimer is on the bottom of the page:

These applications are on servers not controlled by the SEC. The SEC does not endorse these sites, their sponsors, or any policy, activity, product, data, or service they offer. These sites may be removed from the Internet with little or no warning. The purpose of the data reported is to test XBRL technology. The data is largely unaudited and unreviewed. It is not an official SEC filing. Use these applications at your own risk. Do not rely on the reported data, or documents rendered from the data, to make investment decisions.

The Interactive Financial Reports is a link to "http://216.241.101.197/viewer" which is run by viawest.net which describes itself as a "provider of colocation, managed hosting and business continuity solutions ViaWest has become a production partner to thousands of companies. We can design, implement and support your critical data and infrastructure – allowing you to focus on your core business."

The Executive Compensation link points to "http://216.12.130.224/" which is run by servervault.com which is also a hosting company. It's home page says it provides "secure, robust IT infrastructure management and hosting services, specifically engineered to help federal agencies and commercial entities achieve the most stringent security posture as well as comply with a wide range of IT security regulations, including FISMA, DITSCAP, Sarbanes-Oxley and HIPAA."

Both sites sport an SEC logo and a numeric url that obscures the fact that you are looking at a .net or .com site, not a .gov site.

Is this a temporary situation and will these sites become .gov? is the SEC simply outsourcing the hosting? Or will these sites become commercial after a trial period with the companies or the SEC charging for services?

At minimum, this kind of outsourcing obscures the authenticity of and provenance of the information and makes the job of GPO, libraries, web-archivists, and citizens more difficult.

Recently the Office of Management and Budget (OMB) announced the launching USASpending.gov, which is a relaunch of www.FederalSpending.gov, to provide citizens with easy access to government contract, grant, and other award data (See New OMB Federal Spending Database). But now, Secrecy News says that several defense intelligence agencies will withhold unclassified information about their contracts from USASpending.gov.

• Intel Agencies to Withhold Contract Info from Public Database by Steven Aftergood, Secrecy News, December 18, 2007.

Several defense intelligence agencies will withhold unclassified information about their contracts from a new public database of government spending.

The new database at USAspending.gov is intended to provide increased transparency regarding most government contracts.

But when it comes to intelligence spending, there will actually be a net loss of public information because categories of intelligence contracting data that were previously disclosed will now be withheld.

Secrecy News has made available a PDF of a A U.S. Army Field Manual that explains the rudiments of map reading. Distribution of the manual is restricted, and it has not been approved for public release. Maps and Legends Secrecy News, December 28, 2007.

• MAP READING AND LAND NAVIGATION. FM 3-25.26 January 2005 HEADQUARTERS DEPARTMENT OF THE ARMY Washington, DC, 30 August 2006

A new study by the Pew Internet and American Life Project and the University of Illinois at Urbana-Champaign says that "members of Gen Y are the leading users of libraries for help solving problems and in more general patronage.... Furthermore, it is young adults who are the most likely to say they will use libraries in the future when they encounter problems: 40% of Gen Y said they would do that, compared with 20% of those above age 30 who say they would go to a library." 

• Information searches that solve problems: How people use the internet, libraries, and government agencies when they need help (December 30, 2007) Leigh Estabrook, Evans Witt, and Lee Rainie. (full report. PDF)
• Information Searches That Solve Problems (Press Release)
• Young Adults Heavy Library Users by Anick Jesdanun, Associated Press December 30, 2007.

Thanks to Docuticker for pointing out this new Congressional Research Report on Federal data mining efforts:

Data Mining and Homeland Security: An Overview (PDF; 231 KB)
RL31798
Source: Congressional Research Service (via Federation of American Scientists)

Aside from cataloging currently known datamining efforts by the federal government, the report identifies four areas of concern:

As with other aspects of data mining, while technological capabilities are important, there are other implementation and oversight issues that can influence the success of a project’s outcome. One issue is data quality, which refers to the accuracy and completeness of the data being analyzed. A second issue is the interoperability of the data mining software and databases being used by different agencies. A third issue is mission creep, or the use of data for purposes other than for which the data were originally collected. A fourth issue is privacy. Questions that may be considered include the degree to which government agencies should use and mix commercial data with government data, whether data sources are being used for purposes other than those for which they were originally designed, and possible application of the Privacy Act to these initiatives.

I've heard people say that data mining by the government is no big deal since advertisers and other corporate interests do it all the time in efforts to focus marketing and improve profits. If we don't have privacy from corporate types, why should the government worry us? Because ratty data used by a marketer might result in a bald man getting shampoo ads, but when the government relies on ratty data for law enforcement, innocent people can get jailed or harrassed.

Hopefully Congress will be more vigilant on this issue.

In a govdoc-l posting on December 21, 2007, Jan Goldsmith informed the depository community that the Federal Depository LIbrary at UCLA celebrated it's 75th anniversary. Her post states how they celebrated:

The UCLA Library celebrates its 75th anniversary as a member of the Federal Depository Library Program this year.  We were designated a depository in 1932 by Senator Samuel Shortridge. Over the course of these past years, we have provided free access to federal government information for the University community and beyond.  In honor of our anniversary, Kris Kasianovitz and Jan Goldsmith, with the professional assistance of Ellen Watanabe and Dawn Setzer, mounted an exhibit in the Young Research Library lobby.  Gary Strong signed letters notifying all legislators whose district we serve, including our Senators and Congress representatives.  For more information regarding 75 years of collection and services of the Federal Government Documents at the UCLA Library visit our Anniversary web page, at: http://www.library.ucla.edu/libraries/yrl/depository_exhibit.html

Let's hear it for UCLA for providing three quarters of a century of public service to the people of Los Angeles!

Back in November, Citizens for Responsibility and Ethics in Washington, along with Project on Government Oversight, Public Citizen, Electronic Frontier Foundation and the Sunlight Foundation launched a new website,  GovernmentDocs.org, which will house governmetn documents obtained through Freedom of Information Act (FOIA) requests. From the press release:

The database will house Freedom of Information Act (FOIA) responses, and other government documents, from a number of organizations, that can be browsed, searched and reviewed. It is the only one of its kind.

Traditionally, government watchdog groups have either posted FOIA documents on their websites as unsearchable PDFs, or statically highlighted several pages within a document to bolster their findings. This has historically limited the public's access to FOIA documents, and minimizes the opportunities for use by researchers, journalists and citizen reviewers for further research and disclosures. Governmentdocs.org changes that:

• Each and every document goes through an optical character recognition (OCR) process, so that the text of each document is entirely searchable.
• A powerful search engine provides full-text searches and hit highlighting.
• Citizen reviewers can add information to each document page and highlight important findings, allowing for more robust and targeted searches.
• Every page of every document has its own unique URL so that documents can be linked, shared, or posted onto websites.
• The database is a coalition effort, so all of the organizations' documents will be housed on governmentdocs.org and searches will work across collections.

Still No Directory of Federal Websites, E-Gov Act Ignored.   By Coby Logen, .gov Watch. November 05. 2007

How many government websites are there? How many HHS or DOJ sites are there? You and I have no way to know. American taxpayers cannot even know how many public websites their government is funding. By law, we should—but the system is broken.

The E-Government Act of 2002 set a deadline of two years to develop a "public domain directory of public Federal Government websites" (Section 207(f)(3)). But this directory still does not exist 5 years later.

As we've noted here before (Is your search engine finding the government information you need?), the problem of relying on commercial search engines to find government information is that a lot of government information on the web goes un-indexed by those search engines.

• Most fed data is un-Googleable By Jason Miller FCW (December 17, 2007). "After five years, a major E-Gov Act provision goes unmet because of search problems."

Sen. Joseph Lieberman (I-Conn.), chairman of the Homeland Security and Governmental Affairs Committee, says "There are more than 2,000 federal government Web sites not included in commercial search engine results. Is it accidental, or is there a policy, or it is just laziness? I would like to know why" and "Agencies do not let commercial search engines index their sites."

I wonder if that is true? I wonder if there is any document librarian who can answer that question or point to which sites are not indexed?

It is probably closer to the truth to say, along with John Needham, Google's manager for public-sector content partnerships, that government  "databases" are being missed by web crawlers and that  "Agencies are concerned more about how information is presented than if users are finding it."  In other words, agencies would probably like to have their information indexed, but haven't figured out how to do so, or don't have the budgets to do what is necessary.  It probably isn't "laziness" but lack of funds and other resources; it probably is sometimes "accidental" in that some may not know what to do.  It is probably sometimes even "policy" -- but probably less often.

But, one big problem is that we don't really know the scope of the problem or the cause.  FDLP librarians should be pushing GPO, researchers, and library schools to research these issues so we have answers.

Carl Malamud says his motivation is to make the workings of the government more accessible at no cost and that "This is society's operating system."

• Documents of Library in Boston to Go on Web. By John Markoff, New York Times (December 27, 2007).

A digital library partnership, including two nonprofit organizations and the Boston Public Library, is preparing to begin making digital copies of the library's paper-based government documents collection, which will then be made available on the Internet.

The project, which will take two years and require the hand scanning of millions of pages of government hearings and related publications, will cost an estimated $6 million, according to the project's sponsors.

Boston Public Library librarians said they planned to begin by digitizing the House Committee on Un-American Activities hearings from the 1950s, which is regularly sought after by its patrons.

The project is being undertaken by Public.Resource.Org, a nonprofit group seeking to open public access to government records, and the Internet Archive, a San Francisco-based digital library.

The project is the brainchild of founders of the two organizations, Carl Malamud and Brewster Kahle...