How Search Engines Work

It all starts with Crawling

There are various definitions of deep web. We'll focus on a couple that are relevant to your research work.

Many people think when you enter a search in a search engine, like Google or Yahoo, the search goes out on the entire internet and looks for your terms.

Not so.

When you use a search engine you're really just searching that search engine's database of web sites. To build their databases, all search engines send out little programs called bots or spiders. These programs search, or crawl, the internet looking for web sites to add to their databases.

Spiders can easily crawl "static" web pages, like this tutorial site. Static pages basically means you don't have to perform a search to get to the information -- it's right there on the page.

But there's a second kind of information on the web. Some web pages access information stored in databases. The information on these web pages is created when you interact with the web page, such as entering information in a search box. These web pages are called dynamic or database-driven web pages.

Next >>