It takes more than a browser to search the Web. It takes smarts, the right tools, and tips for tracking down leads, no matter where they take you.
It was a cold November morning. A fog weighed on the city like a bad hangover. I was working hard on a case of cigarettes and not much else when he walked in the door. A real mess, from his cheap business suit all the way to the 386 notebook. I could tell he was in trouble.
"I don't know what to do . . . I've looked absolutely everywhere and I can't find anything!" he blurted.
"It all started last week. My boss called me in and asked me to do some research on Megasoft, our main competitor. So I went online and started dowsing . . ."
"Browsing," I corrected. But the kid had a point even if he didn't know it. Trying to find out about Megasoft by browsing is like looking for water in the Mojave using a forked stick.
"Yeah, whatever. Anyhow, I spent all night clicking on links, and all I found was a bunch of silly home pages and a lot of advertisements."
He was tangled in the Web and tangled bad. I had to show him how to get around, and quick. Fortunately, he came to the right place. I'm a Web searcher. I find things. That's my job.
I tried to break the bad news to him gently. "Listen, kid," I said. "Browsing is a waste of time. If you want to find anything, you gotta search the Web, not browse it."
Looking at him, I realised the poor sap didn't know a Web directory from a search engine, let alone how to use metasearch tools, offline browsers, and personal Web assistants. He had a lot to learn.
"Pull up a chair, and I'll tell you the smart way to search the Web." I lit a cigarette, leaned back, and started in.
Web directories, like Yahoo's and Magellan's, are like card catalogues: they file everything in categories such as Sports, Entertainment, Computers . . . you get the picture. Once you find your topic in a Web directory, you've got a handful of key sites to start with. And that might be all you need. Most directories include brief descriptions of each site and lead you to a site's home page, but not to specific pages within a site.
In short, your average Web directory is a no-nonsense tool - it gets the job done, and quick. When you're looking for information on a general topic, a directory is the best place to start, particularly if you already know your way around it.
But Web directories cover only a small fraction of the pages available on the World Wide Web. That's where search engines like those found on AltaVista, HotBot, and Lycos come in. You tell the engine what you're interested in, and it finds all the Web pages it knows about that match. To keep their records current, the search engines use programs called spiders or bots that follow links from page to page, recording all or part of the contents of each page as they go, eventually casing much of the Web.
Because no human intervention is required, search engines can cover much more of the Web than directories can. But you need to know how to use them if you don't want to waste your time sifting through thousands of possibilities.
Stake out the subject
If you're looking for general information, start with a Web directory. I use Yahoo, because it's got a lot of material and I'm familiar with the way its categories are organised - but you might prefer Magellan, or maybe the A2Z directory in Lycos.
On the other hand, if you're looking for a specific concept or a phrase, like Maltese Falcon, or something that's not easily categorised, like bulletproof vests, the Web directories won't be much help. You've got to use a search engine.
You'll likely need more than one, since the same query gives different results from engine to engine. I like to start with AltaVista, since it covers a lot of the Web, and then try Infoseek Ultra (http://ultra.infoseek.com) - a new version of Infoseek - because it gives results right away.
With search engines, take your time and do the job right. If you give some thought to your queries, you'll get better results. Try entering a handful of words or a phrase related to your topic, like San Francisco hotel if you're looking for a place to flop when you're in the City by the Bay.
The more specific you can be, the better. Don't worry about redundancy - synonyms can help narrow the field of your search. Leave out nonessential words like prepositions and articles (of, to, and, the, and so on) - most search engines ignore them anyway.
Say you're looking for the inside skinny on the Whitewater scandal.
Don't enter the query whitewater - you'll find yourself swamped with ads for whitewater rafting expeditions. The query Whitewater Clinton Bill Hillary real estate gets you what you want.
If a search site returns no hits or too few, your query may be too narrow - or it may use the wrong terms. Try another query with fewer words, or one with different, less specific words. Covert surveillance takes you nowhere? Try spy or stakeout.
If the search engine returns hundreds or thousands of hits, your query is probably too broad. If you don't find what you want in the first two or three pages of results, stop. Try again with more specific words or more restrictive query options.
Don't be afraid to try different search engines. Often the same query on other engines will turn up completely different hits.
Once you've tapped out the directories and search engines, there's a whole lineup of Web pages and software tools that can help sniff out what you need. They fall into three groups: metasearch tools, which submit the same query simultaneously to several engines; search assistants, which help you manage searches more efficiently; and offline browsers, which check Web pages and download them if anything's new.
Tired of pounding the pavement? Instead of sending a query repeatedly to different sites, use a metasearch tool - a Web site or a program that submits your query to several engines simultaneously, then rounds up the most likely suspects on a single page.
One of the best Web-based search consolidators is SavvySearch. With SavvySearch, you enter your query and specify search options, such as how many hits to return from each search engine and how to display the results.
Once you've entered your query, SavvySearch submits it to three search engines that contain the kind of information you told it you're looking for. Then it collects all the results on one page. At the bottom of the page is a clever toolbar that allows you to submit the same query to different sets of search engines. It's a quick and convenient way to run searches on many engines. Best of all, it's free.
On the downside, SavvySearch limits you to 50 hits per search engine, and it can often be slowed by network traffic. For comprehensive searching, you may want a PC-based assistant, such as FastFind. An alternative to SavvySearch and similar sites is a metasearch program, such as Bitsafe Computer Services' Arf (www.execpc.com/bitsafe/arf). Arf is a stripped-down, inexpensive ($US20) utility for automating searches and consolidating the results. Its straightforward interface lets you specify a query, a maximum number of hits (up to 10,000), and a search engine (you're limited to AltaVista, DejaNews, Infoseek, Lycos, and WebCrawler). Click on the Start button, and Arf returns a simple, unsorted list of hits; in addition, it downloads the pages that each hit links to. To perform more than one query simultaneously, you can run several copies of Arf at once.
For a few dollars more, the ForeFront Group's $US70 WebSeeker (www.ffg.com) offers many additional options, such as sorting the list of hits alphabetically. WebSeeker queries as many as 20 Web and Usenet search engines simultaneously and compiles a list of hits that you can sort or search. WebSeeker can be somewhat slow, but it has a substantial amount of querying power to assist the Internet sleuth.
A metasearch tool can save you the trouble of visiting every search site yourself, but you still face the problem of sorting through too many (or not enough) hits. The solution is search assistants - programs that enhance, augment, and help you keep track of your searches. Some of these tools stake out the Net for you, running the same queries again and again and notifying you only when something new turns up.
Between you, me, and the rats in the wall, the world's full of these search assistants, and most of them aren't worth a damn. Each takes some time to learn, so choose with care. I'll clue you in to one of the easiest, and two of the most powerful.
One of the dandiest search assistants is the $US35 More Like This, from Knowledge Discovery (www.morelikethis.com). It's a simple toolbar that floats on top of your browser. You use it to run sophisticated queries quickly and easily: just click on the More Like Query button, enter the word you're looking for, and click OK.
This submits the query to the search engine you choose and opens the results page in your browser. Better yet, you don't need to learn any search syntax.
This translates your query into something the search engine can use. You can also click on the More Like Title button to submit a query based on the title of the Web page you're currently looking at (much like the More Like This option offered by the unrelated Excite search engine). If the page's title is "Surveillance Tools", for example, More Like This may take you to other sites where you can find information on snoop merchandise. You can also copy text from any application to the Clipboard, and click on More Like Clipboard to submit that text as a query.
More Like This also offers special concept searches, which automatically amplify your queries with related words to help zero in on relevant sites.
If you enter the word bogart and select a concept search, More Like This may augment your query with words such as humphrey, maltese, bacall, and so on.
More Like This is a minimalist tool that stays out of the way until you need it. For a heavier-caliber weapon, try Symantec's $US49.95 Internet FastFind (www.symantec.com), a collection of several useful utilities for finding and managing Internet information. As a metasearch tool, FastFind collects all the results on a single page, which you can view in your Web browser, sorting them by relevance or by Web site.
Other utilities from FastFind's main menu can find files and download them from FTP sites, notify you when Web sites change, and search the Web for updates to your system's drivers and installed programs.
Quarterdeck's new $US49.95 all-in-one search tool, WebCompass 2.0 (www.quarterdeck.com), is in the same league. In fact, WebCompass has more features than this town has crooked cops. The beta version I checked out looks pretty good. WebCompass 2.0 gives you folders for organising your searches by topic and for managing the Web pages you're interested in. It monitors those pages for changes, and its agents run your queries for you at scheduled times, notifying you if anything's new.
Once you've found the pages you're interested in, you'll probably want to keep an eye on them. But only a sucker would visit every one of those bookmarks every day, looking for changes to each site. That's where offline browsers come in.
Offline browsers download and sort entire Web sites, including their links, or just certain pages. You can also use them like search assistants to run queries again and again. Just set a standing query (such as the weather in Malta), and that's it.
Some offline readers let you read news or other announcements; others function as research tools. Like search assistants, some are more trouble than they're worth.
If you're short on scratch, you'll love FreeLoader 2.0 (www.freeloader.com). It's a free program that integrates with Netscape Navigator or Internet Explorer to download sites and topics. It lets you schedule the downloads so you can avoid the Net's peak hours.
In return, you have to put up with the ad banners alongside its toolbar, but that's a small price to pay.
In the same vein, but offering less flexibility, is PointCast (www.pointcast.com), which delivers news on your favourite topics, using a special Windows screen saver to show you the headlines. While you can choose what kind of news you want to see (business, political, financial, sports) and how frequently you want that news updated, you're limited to the menu of news choices that PointCast provides. It's mainly just a nice way to scan the headlines, not a tool for finding specific information quickly. PointCast is free.
FirstFloor's $US24.95 Smart Bookmarks 2.02 (www.firstfloor.com) is billed as an enhancement to your browser's bookmarking feature; however, it's really a separate program that not only manages bookmarks but also downloads Web pages for offline browsing. It takes a little time to learn how to use it, but the effort pays off. Once you've created a bookmark for a site or a query that you're interested in, Smart Bookmarks monitors the site for changes.
When there is one, the program tells you - and you can tell it to download that page and the pages it links to.
If you need a more powerful offline browsing utility, try Open Market's $US29.95 OM-Express (www.openmarket.com). Among these utilities, this offline browser is the easiest to use, and it comes with a simple, clean interface and a straightforward quick-start guide. OM-Express shows Web pages' original URLs in your browser even when they reside on your system.
Want to test out the latest agent technology free? Empirical Media's WiseWire is a Web-based agent that learns your preferences as you use it, presenting increasingly well-honed selections each time you return. However, it's slow and awkward to use. It's an interesting glimpse of the future of online searching, but it's not an effective search tool yet.
If you're serious about online research, take a look at Folio Retriever 2.1 ($US39.95, www.folio.com), an offline program that builds on the Folio Views infobase format, dumping the contents of downloaded sites into fully searchable infobases that you can annotate and share with others. Retriever includes a fully functional version of Folio Views 3.1, though it's limited to importing HTML pages.
FolioRetriever's power lies in letting you handle the data once you've downloaded it.
The best info - for a price
IBM's infoSage (www.infosage.ibm.com) is my secret weapon for staying on top of my business. Twice a day, infoSage delivers a customised selection of business news, stock quotes, and other information to an e-mail box or personal Web page. To set up or modify your infoSage profile, which specifies what topics and stories you get, you must use IBM's quirky and awkward Windows software, but a browser and e-mail are all you need to read the news.
Subscriptions are $US24.95 per month; the first month is free. There are additional pay-per-view charges for specific types of information, such as Standard & Poor's company profiles.
If it's financial data you're after, you can hardly do better than Quote.com (at - no surprise - www.quote.com). Quote.com has an abundance of free stock quotes and financial information for the savvy investor; you only need to register at the site to gain access to a wealth of information. For yet more wealth, you can pay subscription fees (starting at $US9.95 per month), and Quote.com will throw in additional news stories, customisable stock charts, a portfolio tracking service, and historical stock data.
To get real depth on a company, an industry, a legal matter - almost anything on record - the best source is Knight-Ridder's venerable Lexis-Nexis service (www.lexis-nexis.com). Lexis, a huge archive of laws, cases, and public records, has been around since 1973. Its companion service, Nexis, provides access to the full-text archives of thousands of newspapers and magazines, a news clipping service, and more.
The catch? Lexis-Nexis costs more than a big night on the town.
Subscriptions are $US100 or more per month, with additional fees for using many databases. But if your business needs access to this kind of information, it's well worth the expense.
For general background information on almost any topic, Encyclopedia Britannica's Britannica Online (www.eb.com) provides well-researched, well-written articles that include full bibliographies - a handy feature if your next stop is the local public library.
Individual subscriptions aren't cheap: it'll cost you $US14.95 per month, or $US150 per year plus a one-time $US25 registration fee, but that still beats the two-ton book set handily.
Bookmark your queries
Bookmark the first page of query results so you can return to it. Instead of typing Dashiell Hammett whenever you're looking for some detective trivia, just go back to the saved query, and the search engine will rerun it - possibly with new, updated results.
Okay, so you found a likely Web page - but it's about 50 screens long. Never fear - in Internet Explorer or Netscape Navigator, just type -F and enter a word you're looking for. The browser will jump to the first appearance of that word on the current page.
If you're tired of typing out Boolean operators, you can use symbols instead: & (ampersand) instead of AND, | (the pipe character) instead of OR, ! (exclamation point) in place of NOT, and (tilde) for NEAR. For example, type bogart bacall instead of bogart NEAR bacall.
Looking for images?
AltaVista can search for text in an HTML tag. The query image :comet.jpg will return any page referencing a file called "comet.jpg" in an image tag - and, with a name like that, it's a good bet that file will be a picture of a comet.
Use plain English
Excite is designed to handle conversational sentences well, so queries like "Where is a good Italian restaurant in New York City?" or "learn how to speak Chinese" can be surprisingly effective.
Find quote sources
To find sources for short quotations, such as, "That which does not destroy me makes me stronger," just enter the quote into Excite. It's all right if you don't get the quotation exactly right - there's a good chance that the results pages will tell you where it came from and give you the correct wording as well.
Eliminate irrelevant words
While Lycos doesn't support Boolean searches, you can use the minus sign (-) to refine your searches. Words with the minus sign are less likely to appear in the list of query results. For example, enter the query business -monkey if you want to find out about business, but not monkey business.
Find whole words
Lycos treats an entry as a substring as well as a complete word. If you enter the word graph, Lycos will search for graphs, graphics, and graphite. To limit Lycos to the exact word you entered, put a period (.) at the end of the word.
Find specific information
In addition to Web and Usenet searching, Infoseek offers options that let you search for company information, e-mail addresses, recent news, and Frequently Asked Question files. Just select the directory you want to search from the drop-down list on the main query page.
Keep your caps on
To search for proper names, capitalise them when entering your query: Hammer, not hammer.