Softpedia
 

NEWS CATEGORIES:



NEWS ARCHIVE >>
SOFTPEDIA REVIEWS >>
MEET THE EDITORS >>
Home > News > Webmaster > Tips and Tricks

January 10th, 2008, 17:35 GMT · By Catalin Bocanu

Methods to Block Spiders

SHARE:

Adjust text size:


Google Logo
Enlarge picture
There are situations when you do not want that certain URLs of your website to be indexed by search engines
robots. Fortunately, solutions exist if you need to remove outdated content
like entire web pages, individual images and more.

The general methods for blocking search engines robots use robots.txt files, specific meta tags definitions or a .htaccess file. If you want to remove web content from Google or to prevent the indexing of a website or parts of it, the most recommended options would be the creation of a custom robots.txt file or the implementation of robots meta tags into the HTML code of your pages. In order to block the Google bots for further indexing actions, the file robots.txt, which must be placed in the root of your domain, will have to contain the next two lines:

CODE
User-agent: Googlebot
Disallow: /

The directive specified in the robots.txt file disallow the entire website indexing by the Google bot. In a similar manner you can specify only a directory of your website. Or, as an alternative, you can insert appropriate robots META tags into the HEAD of your HTML pages:

CODE



The meta tag considered in the example tells to Google robots not to index the current web page and also not to follow the links existing on that page. After you define a robots.txt file or the meta tag that blocks the indexing actions of Google bots, go to the Google Webmaster Tools website and select the desired content you want to remove from Google index. In a similar way you can also remove a cached copy of a Google search result.

It is good to know that certain robots does not respect the directives defined by robots.txt or META tags. In order to create a better protection for your website content privacy it is recommended to use .htaccess files to block spiders.

TELL US WHAT YOU THINK:

2,108 hits · Link to this article · Print article · Send to friend · Subscribe to news

MUST-READ RELATED ARTICLES:


Search Engine Spider Simulators

Adding a Search Engine Feature to a Website

Break Down Your Internet Competitors with SEO SpyGlass

Management of A Free Custom Search Engine

The Efficient Usage of Google Webmaster Tools

READER COMMENTS:



No user comments yet.
Be the first to express your opinion!
Copyright © 2001-2012 Softpedia. Contact/Tip us at

WindowsGamesDriversMacLinuxScriptsMobileHandheldNews

SUBMIT PROGRAM   |   ADVERTISE   |   GET HELP   |   SEND US FEEDBACK   |   RSS FEEDS   |   UPDATE YOUR SOFTWARE   |   ROMANIAN FORUM