Robot is a program that runs automatically without human intervention. Typically, a robot is endowed with some artificial intelligence so that it can react to different situations it may encounter. In WWW terms, search engine spider (robot) is a program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It's called a spider because it crawls over the Web. Because most Web pages contain links to other pages, a spider can start almost anywhere. As soon as it sees a link to another page, it goes off and fetches it. This is useful for indexing and updating web sites. Large search engines, like Google, Yahoo have many spiders working in parallel.
Answered by
A.R.Karthick
, an ibibo Master,
at
11:43 AM on February 08, 2008