Using a ROBOTS meta tag to ensure that search engines index the correct pages within a site.
Being selective in the pages returned by a search engine will reduce 'noise' in the search results list, and enable end users to more easily locate key pages. It is recommended no follow attribute is added to specific links, and the ROBOTs meta tag is used to prevent search engines indexing nominated pages.
No follow
Most websites include a number of contextual links - for example FontSize - in the header or footer of the page. There is no value in search engines following these links and attempting to index the associated content. It is therefore recommended that a nofollow attribute is added to these links in the stylesheet.
ROBOTS meta tag
From a performance point of view, it is good practice to ensure that pages containing only dynamic content - such as the site map or search results page are not indexed by search engines. If an XML site map has been submitted to Google, the SEO benefits may be diluted if bots generate their own list of content from the search results page. It is therefore recommended that bots be commanded to not follow links on the search results page via a ROBOTS Meta Tag:
1. When configuring the meta tag use the Authors to Complete population option.
2. Add the 'nofollow' value to pages containing content which may be indexed elsewhere. Examples may include the search results page, pages containing only an asset browser, pages containing only a page lister.
Note: the following values may be added to the tag:
NOINDEX instructs search engine robots to not index the page
NOFOLLOW instructs web crawlers not to follow links in the page content.
NOARCHIVE instructs a (supporting) search engine not to store a cached copy of the page.
NOODP instructs a (supporting) search engine to not use the DMOZ (the open Directory) description for your site.
NOSNIPPET instructs Google not to show a snippet (description) under your search engine listing, it will also not show a cached link in the search results.
This approach may be taken a step further, and the ROBOTS meta tag may be used to request that low level pages not be indexed. It is sometimes sufficient to have only a landing page appearing in search engine results in regard to a specific search phrase. The end user will then be presented with options to navigate to more detailed related content. In this instance the ROBOTS meta tag could be applied to all pages below the landing page.
Note that the ROBOTS tag merely contains a request that content not be indexed. Most browsers will honour this request, however some may not. Ultimately the only way to ensure that content is not indexed is by restricting access via page permissions.
