How Do Search Engines Work – Web Crawlers
It is the search engines that finally bring your website to the notice of the prospective customers. Hence it is better to know how these search engines actually work and how they present information to the customer initiating a search.
There are basically two types of search engines. The first is by robots called crawlers or spiders.
Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A ‘spider’ is an automated program that is run by the search engine system. Spider visits a web site, read the content on the actual site, the site’s Meta tags and also follow the links that the site connects. The spider then returns all that information back to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well. Some spiders will only index a certain number of pages on your site, so don’t create a site with 500 pages!
The spider will periodically return to the sites to check for any information that has changed. The frequency with which this happens is determined by the moderators of the search engine.
A spider is almost like a book where it contains the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may index up to a million pages a day.
Example: Excite, Lycos, AltaVista and Google.
When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.
One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to other pages in the Web. By checking how pages link to each other, an engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page.
Thanks for providing the very nice information.Very useful.
Has anyone used this site before. It looks great and simple to use.
After Google’s recent screw ups with privacy and Buzz I now find myself cautious with any new service they offer. Sooooo, will they share the filenames of our transfers with everyone on your contact list? There has to be a privacy concern somewhere in this
.-= Excel Tutorials´s last blog ..Showing or Hiding a data table or chart legend =-.
This topics is a very good and nice topics. I will get more information in this article. U visit my site i think u get more information in our site.
Here I found some good points . I searched this topic and found out your blog and conclude that most people will agree with your blog.
I have heard about spider method. But the search engine algorithm is new to me. I am looking for more explanation about it. I hope you can also provide it. Thank you.
I wanted to ask what affect pinging websites to the search engines has on the rate at which the search engines ‘spider’ your website. Does it have any affect at the speed at which they will spider your website?
Thanks
Jason
Interesting to know how do search engine works. I think this useful information can help many newbie developing their sites.
Good informative article
Does anyone still use Excite and Lycos?
I think Google and Bing get most of the search traffic these days
“One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page” important thing to get right on any site but a difficut task!
neil@one way links´s last [type] ..Standard Membership Free!
Hmm, didn’t know too much about this topic till I read this article. Well I’ll bookmark this and put it to use hopefully. Though on a less serious side note I’m not sure why they refer to them as “spiders” as my arachnophobia kicks in after the mere mention of such a word.
Search engine marketing, or SEM, is a form of Internet marketing that seeks to promote websites by increasing their visibility in search engine result pages (SERPs) through the use of search engine optimization, paid placement, contextual advertising, and paid inclusion.The New York Times restricts the definition to ‘the practice of buying paid search listings.
Thez @ flintstones costume´s last [type] ..Flintstones Outfits
I commonly call search engine spider as exchanged backlinks and this is one of the easiest way gathering links for one site.
Googles privacy issues concern me as well
I wondered how SEO actually worked! Quite as you would expect really i guess.
This was interesting to read, spiders, the internet Guru’s of the web. Well if you have the potential site, the spiders will come along and help bring you to the top that’s for sure.
dan@Titanium Cookware´s last [type] ..Raisin Roll Delight
it give a lot of information
Crawlers are definitely one of the key players in search engines.- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- BlogCatalog.com
Web Traffic Generation´s last [type] ..February 2012
Blogroll