Finding Information on the Web

The "Invisible Web"

Search engines do not cover everything that is available on the Web. Some Web sources are available by subscription or purchase only, and search engine spiders, or any non-paying Web surfer, are blocked from accessing the contents. Wesleyan Library's list of Indexes and Databases are examples of research sources available only by subscription or purchase. But there are other freely available resources on the Web which are difficult or impossible for search engines to cover. 

A search engine's spiders often cannot index graphics, CGI scripts, PDF files, and information in online databases. Databases are a particular problem for search engines, since they create "pages" dymanically on demand for an individual user, and thus are not available for a search engine's spider to scan. But since these databases are so versatile at providing customized content for users, more sites are putting content into online databases rather than on static pages. Thus, a growing portion of the Web cannot be indexed by standard search engines. This includes contents of phone directories, dictionaries, manipulatable maps, online catalogs, and much more. There are, however, directories which help you find online databases and other sources for finding information that a search engine could not retrieve. Typically, the directories allow you to search or browse for a general topic to find a list of databases likely to contain specific information.

Suppose, for example, you want to find statistics on the population of Middletown, CT, organized by income, education, and race. Entering the terms "middletown connecticut income education race" into a search engine likely would not find anything useful. But if you go to the InvisibleWeb site and search for the general category "population statistics" or "census data", you will be pointed to the State of the Cities Data Systems database or the US Census Bureau site, where you can find a database which you can use to create a table listing the data you want. 

Directories to the Invisible Web. Search or browse by general category to find resources for specific topics (e.g. "population statistics" to find databases which include statistics on income, education, and race of Middletown residents): 

• InvisibleWeb - An annotated directory of over 10,000 databases, archives, and other research sources whose contents are difficult or impossible to find with traditional search engines. Along with describing the contents of a site, annotations include whether a site requires registration or a fee to use. 
• InvisibleWeb.Net - Bills itself as "a directory of some of the best resources the Invisible Web has to offer." Annotated index of links to searchable online databases covering a wide variety of academic and nonacademic topics. From the authors of the book The Invisible Web: Finding Hidden Internet Resources Search Engines Can't See. 
• Direct Search - Lists, with some brief annotations, of online databases, compiled by an academic librarian and intended for academic researchers. 
• Complete Planet - Claims to be "the complete source for search engines and databases;" large index of resources, but it includes many "visible web" sites along with databases and other resources for invisible information.
• Internets - Large collection of searchable online databases on a variety of subjects, both academic and nonacademic. 
• A collection of special search engines - Long annotated list of links to subject specific search engines and searchable databases. 
• IncyWincy - IncyWincy bills itself as an Invisible Web search engine, but it is more of a subject directory that focuses on including invisible Web links. Very large index, but most sites are visible; however, it does highlight databases and other invisible sites included in the index. 
• Langenberg.com - Small but useful collection of searchable databases. 
• Refdesk.com - In addition to databases, Refdesk.com includes sites for news and current events, and an extensive list of sites of all types for finding information on a wide range of topics. 

Other Avenues to the Invisible Web: 

• Search Engines: include a term such as "database" or "directory" along with a general description of your topic (such as "census and database" for census statistics on the residents of Middletown). This will retrieve databases that include the term "database" on their static or search pages. 
• Subject Directories: select a directory for the appropriate subject and browse or search the directory to find any "invisible" sites it lists. 

Internet Archive - Another part of the invisible Web is the former Web, i.e. pages that are no longer available. Once a page is taken down, or even altered, its former content is not just inaccessible, it's gone. But the Internet Archive has been copying Web pages since 1996 and maintaining those copies in its database. If you know of a page that used to exist but that is no longer available, search the Internet Archive for it.

Next: Listservs and Newsgroups     

Contents - Internet - Searching - Search Engines - Subject Directories
Invisible Web - Listservs and Newsgroups - Evaluating - Citing