Dr. Edward Owusu-Ansah's Homepage

Only half a decade ago a chaotic Web made searching a virtual impossibility for the less informed. Even the savviest searchers had no easy task. Knowledge of specific addresses (uniform resource locators or URLs) was a prerequisite. Gophers were the first attempt to organize the contents of the Internet. They were server based and arranged in a menu format. Veronica came along with the capacity to index all of "gopherspace." Jughead followed Veronica. But the rapid development of the World Wide Web called for an organization that allowed the use of hyperlinks and permitted easy to use and highly integrative full-text searching with graphical browsers. Search engines were developed in response to this need. The first to emerge was WebCrawler in 1994. Lycos, Infoseek, OpenText (defunct in 1998), Alta Vista and Excite followed in 1995, HotBot in 1996 and Northern Light in 1997.

With the debut of the search engines the searcher's task became easier. Perusing the web and indexing pages and sites for retrieval, these indexing systems made the search process more convenient. There is still a long way to go towards perfection. The search engines, meta search engines and subject directories of today, though our best resources for searching the vast and ever growing content of the Web, are still plagued with imperfections and inconsistencies, and none offers the searcher the ability to search the total content of the World Wide Web. Therefore, knowing and understanding them provides the best chance for the most efficient and successful search. What are they, what do they do and do best, which of them should the searcher use and when? These are critical questions to ponder if the goal is to do research on the Web and do so effectively. They are part of the focus of this chapter. The search options and strategies necessary for effective searching are also discussed later, as well as the evaluative processes for determining the quality of information found on the Web.

Search Engines

Search engines are huge databases containing Web page files assembled automatically by machine. There are two main types: individual search engines and meta search engines. In popular usage the individual search engines have usurped the "search engine" name. These search engines use "spiders" to crawl through sites and pages on the World Wide Web (WWW). The spiders identify new sites to be added as well as any changes in those already covered by the search engine. Words found on the various web sites are indexed. URLs may also be submitted by the creator of a Web page for inclusion in a search engine database.

Using a search engine involves submitting keywords and phrases the searcher expects the engine to identify and match as it scans its index of sites. This is not a search of the entire Web but rather of a portion of it, captured by a specific indexing system. Yet due to the depth of coverage, search engines are the best suited for finding sites and pages containing unique keywords, phrases, quotes and information found in the full-text of Web pages. Retrieval may be very large since search engines index word by word. For thorough and comprehensive results the searcher will need to master the use of more than one engine. No single one covers the whole Web and no two search engines are of the same size, speed or content. A growing tendency of search engines to provide subject directories on their home pages is gradually blurring the distinction between search engines and subject directories, as directories in turn provide search engine capabilities.
Here are the names and web addresses of some popular current search engines:

AltaVista

Excite

Google

HotBot

Lycos

Northern Light

Web Crawler

For a brief description of all the major search engines go to searchenginewatch.com's listing of the major search engines.

Meta Search Engines

Meta search engines search the databases of several individual search engines, doing so simultaneously with the aid of a single interface. They thus provide a fast way to determine which individual engines can retrieve the best results for a particular search. They return only the top list of items found in the individual databases, therefore having a high degree of relevancy. Meta search engines are best suited for a quick overview on a subject or unique term, and are ideal for simple searches. However, the user should bear in mind that few of these are truly effective. Submitted queries may not be transferred to the individual search engines in a manner that would allow a retrieval of all the records possible when the engines are searched individually.

As these tools continue to improve, there may be better results in the future. For now, individual search engines appear to be the most effective Web search tools, though the ability to query several search engines simultaneously still appears an attractive alternative in some specific instances, as those mentioned here. That attraction reinforces the practical value of meta search engines. Here are the names and web addresses of some popular meta search engines (with the individual search engines they search included in parenthesis):

Dogpile
(Alta Vista, Excite, Go2.com, Infoseek (Go.com), Lycos, PlanetSearch, Thunderstone, WebCrawler, What U Seek, Yahoo!)

MetaCrawler
(Alta Vista, Excite, Infoseek (Go.com), LookSmart, Lycos, Mining Co. (About.com), Thunderstone, WebCrawler, Yahoo!)

ProFusion
(Alta Vista, Excite, GoTo, Infoseek(Go.com), Lycos, Magellan, Snap (now NBCi) , Webcrawler, Yahoo!)

SavvySearch
(Alta Vista, Direct Hit, Excite, HotBot, Infoseek(Go.com), Lycos, WebCrawler, Google, Thunderstone)

For a virtually complete list of meta search engines go to: MetaSearchEngineDirectory.Com.

Subject Directories

Subject directories are created and maintained by human editors, not electronic spiders. They involve editorial review and selection of sites for inclusion in the directory, usually with annotations. Directories tend to index only homepages or the top level pages of a site, thus lacking the detailed in-depth approach of search engine. There is a growing tendency for some subject directories to transform themselves into portals for added commercial benefits. Portals are directories that act as gateways to the Web, listing not only popular subject areas, but also offering additional service such as current news, stock quotes, travel information, maps, advertisements, etc.

Since directories are usually organized into subject categories and sub categories that can be browsed hierarchically, the searcher can use this arrangement to his/her advantage by browsing through from the general to the more specific. The searcher should however bear in mind that dead links are often a problem with directories, and the strength of directories lies in their treatment of popular topics, organizations, commercial sites and products. Subject directories may be popular or scientific. Popular directories are the products of commercial organizations. They provide information on popular subjects (travel, software, sports, etc.). The most popular general subject directory is Yahoo!. Other good alternatives are Open Project Directory and Metaplus. Scientific libraries, universities, and professional organizations compile scientific subject directories. Some examples are BUBL, Argus Clearinghouse, Galaxy, Galileo, and Study Web. Subject directories may also be general or specialized. General subject directories include sites on any popular or scientific top, while specialized directories include sites relevant to one particular field. Most of the directories mentioned so far belong to the general category. Some examples of specialized subject directories are Galen II for health care, NetEc for the field of Economics, and Psych Web for Psychology.

Library Gateways

These are collections of links that have been reviewed and recommended by subject specialists to support research. They point to high quality sites and specialized databases created by researchers, experts, professional associations and organizations, government agencies and businesses with a deep interest in a particular field, who accumulate and collect subject links to that field. Having been reviewed and evaluated by experts, they provide the best source for high quality information. Examples of library gateways are:

	Academic Information
	AlphaSearch
	Argus Clearinghouse
	Digital Librarian
	Infomine
	Internet Public Library
	Librarians' Index to the Internet
	New Athenaeum
	WWW Virtual Library

Web Subject Guides

These are online subject guides created and maintained by individual libraries. Subject specialists within the library, who evaluate the sites and create links to them on the library's Web site, to support research, assemble them. Unlike commercial sites these are more targeted, and the conditions for inclusion may be very specific, most sites being highly academic or professional. Such are the "Internet resources" listed under individual "class resources" provided as part of the College of Staten Island Library's "Library Web Guides" links and available through the CSI Library's homepage (under "MATERIALS"). Students are encouraged to consult these for subject oriented Internet research in the listed subject areas.

Owusu-Ansah home|| Introduction|| Search Engines|| Meta Search Engines|| Subject Directories|| Library Gateways|| Web Subject Guides||

Send comments to: Dr. Owusu-Ansah Page updated: July, 2009