As we’ve noted here before (Is your search engine finding the government information you need?), the problem of relying on commercial search engines to find government information is that a lot of government information on the web goes un-indexed by those search engines.
- Most fed data is un-Googleable By Jason Miller FCW (December 17, 2007). "After five years, a major E-Gov Act provision goes unmet because of search problems."
Sen. Joseph Lieberman (I-Conn.), chairman of the Homeland Security and Governmental Affairs Committee, says "There are more than 2,000 federal government Web sites not included in commercial search engine results. Is it accidental, or is there a policy, or it is just laziness? I would like to know why" and "Agencies do not let commercial search engines index their sites."
I wonder if that is true? I wonder if there is any document librarian who can answer that question or point to which sites are not indexed?
It is probably closer to the truth to say, along with John Needham, Google’s manager for public-sector content partnerships, that government "databases" are being missed by web crawlers and that "Agencies are concerned more about how information is presented than if users are finding it." In other words, agencies would probably like to have their information indexed, but haven’t figured out how to do so, or don’t have the budgets to do what is necessary. It probably isn’t "laziness" but lack of funds and other resources; it probably is sometimes "accidental" in that some may not know what to do. It is probably sometimes even "policy" — but probably less often.
But, one big problem is that we don’t really know the scope of the problem or the cause. FDLP librarians should be pushing GPO, researchers, and library schools to research these issues so we have answers.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.