Winning the search engine game


So I’ve built a website for my firm, why can’t I find it on the search engines?

A: Most search engines try to produce an index the web by following links from pages they visit, so if another popular page links to yours the search engines will pick you up eventually. Eventually, however, can mean eight months or longer.

You can accelerate this process by submitting your page to the search engines. This puts you on a waiting list for the search engine to visit or ‘spider’ your site to index it. Although modern search engines are attempting to find ways of dealing with this backlog, it can still take between one and six months to become indexed.

It’s important to check and resubmit your site to each search engine frequently once you are listed, as your appearance can slip and disappear altogether in time. The necessity for re-submission changes from engine to engine and from month to month but generally ranges between weekly and every other month. However, you should not resubmit your site if it is ranking well on an engine as this may cause its ranking to fall.

The whole process is speeded up and simplified even further by using a ‘paid inclusion’ option. Currently Inktomi is the only European search engine to offer this service. This option provides search results to over 125 search engines worldwide.

Q: It all sounds a bit unscientific. Why is it so much effort to get listed?

A: Some of the problem stems from the exponential speed at which the net is expanding – with billions of pages out there on the web the search engines are finding it harder, and therefore slower, to index promptly.

This is further frustrated by bulk automated submission software and ‘spammers’ who put a significant load on the search engines’ ‘add URL’ systems.

The latter causes most engines to operate an ‘anti-spam’ policy, which ignores pages submitted in a fashion that looks like spamming. As each engine has a subtly different policy, one has to tread the fine line of optimising your site to do well but not ‘spamming’ any engine, a hard thing if you don’t know what they’re looking for.

To avoid spamming the best thing is to find out what each engine looks out for and to optimise your site without going overboard.

Some search engines such as HotBot look for meta tags – part of the HTML code at the top of the page – while others such as Google look for key words contained within your page’s text.

Some look for link popularity – links to your site from other important and relevant high traffic sites. Without knowing who looks for what, search engine optimisation can be a hit or miss game, and if you don’t optimise your website for each and every engine’s unique methods, you’ll miss more than you hit.

Q: What’s the difference between a directory and a search engine?

A: A directory is a human-compiled listing of websites, organised by category, while a search engine is a computer-compiled listing of websites that automatically gives each page a relevancy score for the various keywords it incorporates.

With a search engine, your site has the opportunity of being indexed (but not necessarily ranked well) no matter what type of material it contains. Directories index only the most pertinent sites within its categories, as the sites are personally reviewed by the category’s editor and not just by a piece of software.

Q: How does a search engine work?

A:There are three distinct processes that search engines go through when reading a webpage.

Firstly, a ‘spider’ – an automated programme – visits web pages and gathers the information contained within them.

The ‘spider’ will read the source code of a page read its meta information and will follow links on that page to retrieve more information from different pages. ‘Spiders’ should return to sites regularly although with the massive volume of pages being submitted these days we have seen a marked decline in the regularity of re-spidering.

‘Spidered’ information is placed in the index of the search engine and is available to the end user, who is searching for your page.

The lead in times between ‘spidering’ your pages and their availability in the index vary for different search engines. This can be from as little as 2 weeks to as long as 6 months.

Finally the search engine goes through is the ranking algorithm. This strips the source code off the page, and taking other factors into account, such as other links to the URL, scores the page against various keywords. This determines which pages and in what order the engine returns results against a search.

In association with

Related reading