Resolving duplicate content issues

Feb 16, 2009 10:54 GMT  ·  By

Rivals Microsoft, Google and Yahoo have inked a partnership designed to help search engines deal with duplicate content. The Redmond company indicated that detecting and consolidating duplicate pages is one of the most problematic areas when it comes down to the website indexing process. Given the context in which a webpage has multiple URLs pointing to it, search engines have problems identifying the canonical URL. And in fact, search engines by themselves cannot filter the clutter of duplicate URLs, and Microsoft, Google and Yahoo are now turning to webmasters to help them improve relevance.

“Live Search has partnered with Google and Yahoo to support a new tag attribute that will help webmasters identify the single authoritative (or canonical) URL for a given page. The link tag defines a relationship between a document and an external resource. In this case, that resource is the canonical URL. The following is an example of the new link tag attribute for canonicalization: <link rel="canonical" href="http://mysite.com" _fcksavedurl="http://mysite.com"/>,” revealed Nathan Buggia, Live Search Webmaster Team.

Buggia added that Microsoft estimated that the command would be implemented in the near future, and that it would help resolve the vast majority of duplicate content problems. However, this will only happen if webmasters perform domain canonicalization and normalization of URL parameters. For websites where admins ignore the best practices for normalizing their URLs, search engines will continue to come across duplicate content issues.

“This tag will be interpreted as a hint by Live Search, not as a command,” Buggia stated. “We'll evaluate this in the context of all the other information we know about the website and try and make the best determination of the canonical URL. This will help us handle any potential implementation errors or abuse of this tag. You can use relative or absolute URLs in the “href” attribute of the link tag. The page and the URL in the “href” attribute must be on the same domain. For example, if the page is found on “http://mysite.com/default.aspx”, and the ”href” attribute in the link tag points to “http://mysite2.com”, the tag will be invalid and ignored.”