Configuring ROBOTS.txt for WordPress

From  many of we beginner users of wordpress we do not know how much a robot or bot of a search engine is value for. They crawl through our content and store the information on search engine server so that user can access our site. But there can be harm also if robots.txt is not configured correctly they might access our wp-admin where the access log information is stored. so here we will learn how to Configuring ROBOTS.txt.

first of all

What Is Robots.txt?


robots_txt_visual

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Here’s a basic “robots.txt”:

User-agent: *
Disallow: /

Now To prevent Google from indexing a bunch of useless pages on your site, you can use a robot.txt file to block Google from those parts of your site.

google-robot

Use a basic text editor, like notepad.exe, and create a new file called robots.txt. Paste the following into the file.

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
Allow: /wp-content/uploads/

User-agent: Googlebot
Disallow: /*.pdf$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Disallow: /*?*
Disallow: /wp-*
Disallow: */feed/
Disallow: */trackback/
Disallow: /cgi-bin/
Disallow: /go/
Allow: /wp-content/uploads/


Related Posts with Thumbnails

15 Responses

      • I always make sure that robots.txt is the first thing I do after I setup my wordpress blog. It is the best way to make your site likeable to Google.
        .-= NPXP´s last blog ..5 Tips For New Bloggers =-.

    • @chris you are welcome at Techmadly.com

      no this does not affect SEO and even if you are writing your robots.txt in smart way you can even get rid of spam which SEO consider very strongly.
      like you have a post and that is also defined in any other category then robots will take it as two post having same content and consider you spamming. but if you bar category listing by robots.txt you wll have a postive effect.

Leave a Reply

Your email address will not be published. Required fields are marked *