How to force Google to read my robots.txt file
In the following article we show how to force Google to read the robots.txt file on a web page immediately.
When developing a new website is good practice to configure the robots.txt file to not index the website until not have real content.
The robots.txt file is a text file located in the root of our website, which contains a series of directives that search engine robots should read to know what parts can be indexed in a website.
For example if we have a directory that we do not want Google to index, then we add the directive:
If we want Google not index any part of our website at the development stage, we add the directive:
In a development environment these actions are performed to prevent Google from indexing test content in the search engine, since we are developing the website and there is still no real content.
However, once we have the website ready with the actual content and we want Google to index the page, we have to remove the policy indicated above and refresh the robots.txt file.
The problem is when we notice that Google updates the robots.txt file every x days (usually between 3 days and 1 week), but we want our updated website on Google immediately.
In Google Webmaster Tools, Google application where it should be your website to control certain points of interaction of a website with Google, there is the option to test the robots.txt file that has Google in their system and the date of the last time has processed, so you can see if the file has Google is updated with your latest changes.
You can see this option robots.txt testing tool in the next page: www.google.com/webmasters/tools/robots-testing-tool
If Google has not updated the robots.txt file in their system and you need to force Google to read the updated robots.txt file, you must go to the option Fetch as Google page: www.google.com/webmasters/tools/googlebot-fetch
In this page you must include "robots.txt" inside the text box and press the button "Get and Process".
This option will download the robots.txt file updated and Google will refresh the file with the latest version immediately, and we will get it Google will begin scanning our website and place in its index.