From many of we beginner users of wordpress we do not know how much a robot or bot of a search engine is value for. They crawl through our content and store the information on search engine server so that user can access our site. But there can be harm also if robots.txt is not configured correctly they might access our wp-admin where the access log information is stored. so here we will learn how to Configuring ROBOTS.txt.
first of all
What Is Robots.txt?
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Here’s a basic “robots.txt”:
User-agent: * Disallow: /
Now To prevent Google from indexing a bunch of useless pages on your site, you can use a robot.txt file to block Google from those parts of your site.
Use a basic text editor, like notepad.exe, and create a new file called robots.txt. Paste the following into the file.