From many of we beginner users of wordpress we do not know how much a robot or bot of a search engine is value for. They crawl through our content and store the information on search engine server so that user can access our site. But there can be harm also if robots.txt is not configured correctly they might access our wp-admin where the access log information is stored. so here we will learn how to Configuring ROBOTS.txt.
first of all
What Is Robots.txt?
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Here’s a basic “robots.txt”:
User-agent: * Disallow: /
Now To prevent Google from indexing a bunch of useless pages on your site, you can use a robot.txt file to block Google from those parts of your site.
Use a basic text editor, like notepad.exe, and create a new file called robots.txt. Paste the following into the file.
User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
Allow: /wp-content/uploads/
User-agent: Googlebot
Disallow: /*.pdf$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Disallow: /*?*
Disallow: /wp-*
Disallow: */feed/
Disallow: */trackback/
Disallow: /cgi-bin/
Disallow: /go/
Allow: /wp-content/uploads/
No related posts.
such post help new users like us for new technologies.
thank you
.-= archit´s last blog ..Zephr’09|Aviation Fest at IIT Bombay =-.
@archit ya this post is written for the new bloggers who do all ting but forget to configure their robots.txt pages i think this is a necessary step.thanks for comment
I always make sure that robots.txt is the first thing I do after I setup my wordpress blog. It is the best way to make your site likeable to Google.
.-= NPXP´s last blog ..5 Tips For New Bloggers =-.
nice article am looking for.
updating my flock to get your updates
great info about robots.txt many bloggers really dont know the exact importance of robots.txt thanks for sharing dude.
wanna link exchange ?
.-= Devilslab-Your guide to stay updated´s last blog ..Best Mobile Web Browsers | Download Latest and Best Mobile Web Browsers =-.
@devilslab you are welcome . and thanks for adding value to post.
[...] 10.write a Robots.txt file for your blog: we have already discussed how important a robots.txt file is for your blog so after generating sitemap by google sitemap you must create a good robots.txt. post related to creating Robots.txt may help you. [...]
You need to use “No Self Pings” plugin buddy or else your trackbacks from your own site would be looking ugly in the comments section.
.-= Suneel´s last blog ..WordPress Acquires After The Deadline Contextual Spelling and Grammar Checking Plugin =-.
@suneel thanks for the suggestion
i will install the plugin you suggested .
like when you start typing in something and everything that could related to that comes down..
hi,
thanks for that post but i have a question.
does it affect seo in a negative way?
thanks
@chris you are welcome at Techmadly.com
no this does not affect SEO and even if you are writing your robots.txt in smart way you can even get rid of spam which SEO consider very strongly.
like you have a post and that is also defined in any other category then robots will take it as two post having same content and consider you spamming. but if you bar category listing by robots.txt you wll have a postive effect.
Thanks for this tips man
thank you. I was searching for such information and I found your instruction useful.
this is great tips, thanks for you