The Spider's Apprentice, Search Engine FAQ

How To Use Web Search Engines

How to get the most from *search engines like *AltaVista, *Infoseek, *Excite, *Webcrawler, *Lycos, HotBot, *Open Text and the *Yahoo Directory.

(starred links are Hyper-Live connections to our Zoom-Inform demo, which provides in-depth information about companies, products and technical terms)

Page 2-- Spidap's Basic Search Engine FAQ

1. Why do I need a search engine?

For the same reason that you need a card catalogue in a library. There is lots of great and useful information in a library, but it's physically impossible to examine all the books personally. Not even the most indefatigable web-surfer could hyperlink to all the documents in the aptly named World Wide Web. There are millions of pages and billions of words on the Web. And every minute of the day, folks are posting more.

The search engines and directories help you sift through all those billions of 1's and 0's to find the specific information you need.

2. If it's impossible to examine all the documents on the Web, how do the search engines do it?

They use software programs known as robots, spiders or crawlers. A robot is a piece of software that automatically follows hyperlinks from one document to the next around the Web. When a robot discovers a new site, it sends information back to its main site to be indexed. Because Web documents are one of the least static forms of publishing (i.e., they change a lot ), robots also update previously catalogued sites. How quickly and comprehensively they carry out these tasks varies from one search engine to the next.

3. Which search engine is "the biggest"?

They'd all like you to believe they're either "the biggest" or, if they can't possibly claim that, "the best." AltaVista seems to generally be regarded as the biggest. It's currently claiming to have indexed 30 million pages. HotBot claims to have indexed 36 million (although we're not totally convinced, since they're still not getting some of the sites that show up on Alta Vista). Both Infoseek and Excite are now claiming over 50 million pages.

Anyway, bigger is not necessarily better. Do you really want to wade through hundreds of "hits" for every query?? If not, click here to have a look at our own search technology, which helps you get useful information.

4. What's the difference between a Web directory like Yahoo and a Web search engine like Lycos?

There is less difference now than there used to be, because many search engines are building subject catalogues to help you search. But think of a Web directory as a subject catalogue--something like the subject catalogue in your local library. Yahoo attempts to organize Web by dividing it into topics and subtopics. Some examples include: Arts, Science, Health, Business, News, Entertainment. If you're looking for information on the Web that fits neatly into an obvious subject or category, go first to Yahoo (see link below) or one of the other Web directories.

Think of a Web search engine as an index that enables you to seek out specific words and phrases. With the search engine's help, you can locate individual appearances of such words in documents all over the Web.

This can be both a blessing and a curse--but it's more commonly the latter! You might get way too many hits. Or you might discover that your keyword has meanings you didn't anticipate. Or you might get no hits at all.

Try searching Lycos for info about the rock group "The Who," and you'll see the problem. Most search engines ignore words like "a," "an," and "the." Some, but not all, of the big search engines can handle phrases of more than one word. Some can handle proper names. Some can't.

Knowing which search engine does what can be a huge time-saver.

5. Okay, so which search engine does what?

In brief, here's a quick run-down of some well-known general topic search engines. More comprehensive details are available by clicking on Search Engine Details.

Yahoo is a actually a directory--a subject index. It's an attempt to catalogue and organize the entire Web. Search on a subject or topic. If Yahoo can't give you a hit, it automatically sends your query to its partner AltaVista for a search through that index. The Yahoo site includes all sorts of other functionality, including excellent coverage of business-related sites.

Excite is now a concept-based search engine and a Web-directory site. This means it will try to figure out what you mean...instead of just searching on exactly what you say. This gives Excite a great deal of flexibility. Like Infoseek and Yahoo, Excite now includes the option of a "Personal Page," where you can define all the subjects that you are most interesting in searching.

AltaVista is a powerful keyword search engine. It allows you to search on phrases of more than one word if you put the phrase into quotations marks, i.e., "health care facilities" or "Bill Gates." Alta Vista has lots of search-refining options. It's big and fast, and very popular among dedicated web-surfers.

Lycos is big, thorough and comprehensive. Its main flaw is a limited ability to refine a search--you can't search on phrases. Like Excite, Lycos is gradually becoming more like Yahoo, acting as a Web Directory as well as a search engine. Its link to RoadMap (generates maps based on addresses) is very cool!

Open Text is user-friendly, flexible and versatile, but not as extensive as some of the other search engines. In fact, its index seems to be pretty small, and new URLs take a long time to show up. Open Text has recently begun charging for "preferred listings," meaning that a company can pay a lot of money to assure it comes up early in a list of "hits."

Infoseek now uses a new search technology, known as Ultraseek, which is bigger, faster and more accurate than its previous engine.

Webcrawler has just been replaced by Excite as AOL's search engine, which raises questions about how long it'll stay around.

HotBot (formerly Inktomi) is fast and has a cool, counter-culture look.

6. How Can I Find Out More Details About These Search Engines?

For a more detailed ratings and analysis, click here to jump directly to Spidap's final page. To learn how to search engines work, click here. And to go directly to our Web Search Wizard and quickly find what you want, click here.

7. Number One Frequently Asked Question From Our Spidap Users:

How can I get my site to come up on the search engines??

Our reply: If you post a page on the Web, sooner or later the search engines will find it and index it. So far, our own tests have shown that AltaVista does this faster than anybody else, although this could change if some of the other search engines live up to their PR.

However, if you want to see your page a little sooner, each of the popular Web search engines provides a way for you to add your own URL. Sometimes this is obvious on their initial page; other times you have to drill down into the site to find out where and how to do this. But it can always be done.

To facilitate this process, we recommend that you use the latest version of HTML, which allows for the use of meta tags in the "head" of your document. With meta tags you can provide your own keywords for the search engine to index you with; and you can also provide a description of your site that the search engines can use to show to Web surfers or researchers when your site comes up as a hit.

You can, of course, also submit your site to various amateur and professional sites who will then submit it themselves. Some of these charge a hefty fee for doing so. We're not sure why anybody would pay several hundred dollars for something they can do perfectly well themselves, though. Publicizing your Web site on various newsgroups and Internet mailing lists is also a good idea, although you should be careful not to violate "netiquette," especially if your site is a commercial one.