Wednesday , 1 October 2014
Home » blogging » How to edit Robots.txt in WordPress – A simle better robots.txt file for WP

How to edit Robots.txt in WordPress – A simle better robots.txt file for WP

Since we already discussed on how we can edit robots.txt in Blogger, I would also like to discuss about how we can edit the robots.txt in wordpress. Well, before Discussing the same, let us see what is Robots.txt file ?

What is Robots.txt files and why do you need a robots.txt file in your blog ?


Robots.txt is a file which the search engines look for, everytime they crawl your webpage.  Robots.txt files tells the search engines which pages to index and which pages not to index. This will help the search engines to neglect the 404 error pages and other pages that you don’t want to be in search engines. Officially as per google, Robots.txt is defined as

robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)

The main advantags of using a robots.txt file are ;-

  1.  You can avoid indexing 404 and other error pages
  2. You can control the pages getting indexed in search engine
  3. You can avoid the Duplicate content issues
  4. Your blog will be SEO friendly
And the disadvantages of using a robots.txt file is that
  1. If done wrong, It may block your site from getting indexed in search engine

How to add & edit Robots.txt in WordPress ? – A simple yet better robots.txt file for WP

WordPress is highly search engine optimized and search engine friendly and already has a good built-in robots.txt file. But since WordPress is a professional choice and looks a bit confusing, You may find it hard to edit the robots.txt in WordPress.

In case of WordPress, It’s rather simple. It automatically creates a robots.txt and since it is auto generated, you may won’t be able to locate it on your server files. The virtual file will be something like this

User-agent: * 
Disallow: /wp-admin/ 
Disallow: /wp-includes/

But to modify it Just install this plugin called WP Robots Txt to your WP blog. After Installation, Go to settings -> privacy settings to find your robots.txt file. Edit robots.txt in wordpress

Then add this to your robot txt file

User-agent: * 
Disallow: /wp-admin
Disallow: /wp-includes 
Disallow: /wp-content/plugins 
Disallow: /wp-content/cache 
Disallow: /wp-content/themes 
Disallow: /wp-includes/js 
Disallow: /trackback 
Disallow: /category/*/* 
Disallow: */trackback 

This will disallow the user agents like google search, Google image, mobile, adsense etc in entering the pages like your theme files, hosted files etc and will allow them to crawl the rest of your website completely. Alternatively, If you are getting too much 404 errors in your sitemaps or if you want to block a specific section from getting indexed in search engine, you may add a line “disallow:” followed by the path of the category you would like to block. for eg: To block search engines from including your comments, you should add the following lines to your robots.txt file

Disallow: /comments/

Likewise, you may block any contents by adding them in robots.txt files. This can be very useful if your site creates 404 errors in any sections, If your site has duplicate content issue, if any plugins goes wrong etc.

I had tried my best to make this article the simplest, and I believe that this had been useful for you. In case you are not sure how to do it, I advice you not to do it and you may contact me for any help on this topic ! The comments are open and would love to hear your feedbacks and quarries.

About Rajeel

I'm Rajeel, a teen blogger from India who loves computers, sports, internet and all other kind of tech stuffs. This blog thing is one of my favourite hobby and the one that eats up much of my time. But Really, I'm loving it :) Find me in G+

6 comments

  1. I really liked all the tips and totally agree with you on all the points. thank you for sharing a nice piece of functionality!

    I am new to internet world and recently set up a blog. Google was not showing up this website of mine lately. After so much struggle, I landed at this site and I have now changed the privacy settings. Lol..
    Thanks in advance. I just hope that my site gets indexed NOW!

  2. Awesome! I really enjoyed this reading.. and Im thinking about trying this out! Thanks again and I look forward to your response.

  3. Thank you for the fantastic and most generous content. once again you over deliver for your readers. I do enjoy reading your blog posts and take inspiration from them

  4. Absolutely right information you are given on this blog the every website need a robot.txt file. great information

  5. Great post and definitely which I recognize.That is such a simple truth.I definitely give away alot more information than I do trying to sell to my audience though.It should be able to communicate its core ideas clearly to the broadest audience possible as well.

  6. Hi, Rajeel. I really love this post. Love the way you write your posts. The information in this article is really unique and useful for me. After reading this article, I think I have some ideas for myself. I do follow your articles recently. Thanks for sharing this post. Hope to read more interesting information from you. Have a nice day.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

CommentLuv badge

Scroll To Top