Many web developers are using different tips to improve their site rankings but, when it comes to Googlebot, they're all afraid that
a wrong crawling process can affect the number of visitors.
Vanessa Fox, a Google employee, described the way that Googlebot works, to help webmasters develop their websites.
So, if you have your site down for maintenance, you're probably afraid that Googlebot will index your page as a down for maintenance page. "You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later," Vanessa said.
If you're asking yourself what's more useful between using meta robots tag and a robots.txt file, Vanessa Fox offers the answer: "Googlebot obeys either, but meta tags apply to single pages only. If you have a number of pages you want to exclude from crawling, you can structure your site in such a way that you can easily use a robots.txt file to block those pages (for instance, put the pages into a single directory)."
As a conclusion, if you have more questions about how Googlebot works, you should read all the documentation provided by Google and, if you didn't find the answer, ask help from the company's employees.
Find us on Google+
