Linking Structure & Strategy: Let the Spiders Crawl

One of the primary ways that search engines find site pages is by following the links from your home page to other inside site pages (this process is also known as “spidering” or “crawling”). In order to allow search engines to effectively crawl your site and locate your inside pages, it isimportant to ensure that your menu structure does not present search engines with any barriers that interfere with their ability to follow internal links. Complicated JavaScript or other dynamic menu features can sometimes get in the way of effective crawling. As far as search engines are concerned, when it comes to finding links to your site pages, the simpler the better.

One tool that you can use to “see what the search engines see” is to use a search engine spider simulator, such as the one linked to below. If you cannot see all of the links to your site pages after performing a simulated crawl, then you will need to take a look at your code to assess what may be getting in the way of effective search engine crawling.

Search Engine Spider Simulator:
www.searchengineworld.com/misc/tools.htm (NOTE: select “Sim Spider” from the Tools list)

A great way to build a search engine-friendly road map for search engine spiders to follow and index site pages is to create Site Map with simple text links to all of the pages on your site. A Site Map also has the added usability benefit of providing visitors with an overall snapshot of all site pages.

Link Popularity: The Importance of Inbound Links
Another way that search engines find your site pages is by following links to your site from other external sites (provided that those sites themselves are listed in a given engine). Having such links to your site not only provides search engines with additional opportunity to find your pages, but also provides increased visibility for your site by putting it in front of visitors on another site.

Many top search engines, such as Google, will factor in the number of sites linking to yours in determining its results for a particular search query. This is known as “link popularity”. One way to think about link popularity is that each external link to your site counts as a “vote” for your site. So, the more links you have pointing at you the better, right? Well, not necessarily. Because search engines also know how to count the link popularity of the sites linking to yours,a single link from a popular site will weigh more heavily than many links from obscure unpopular sites. When it comes to getting links, quality over quantity is the way to go.

All things being equal, of two sites with content of comparable relevance and structure, the sitewith higher link popularity will get more prominent placement in search engine results.

Who’s Linking At You?
If you want to research pages that currently link to your site, Google supports a neat little trickthat lets you do just that. If you go to google.com and type in your URL proceeded by “link:” (without the quotes – e.g. link:www.your-url.com) the results will give you a list of all pages listed within google that link to your site. You can also use this method to research sites that linking to organizations similar to yours – a site that links to organizations like yours, or even adirect competitor, may be a good lead to follow-up with in requesting a link of your own.