Since we already discussed on how we can edit robots.txt in Blogger, I would also like to discuss about how we can edit the robots.txt in wordpress. Well, before Discussing the same, let us see what is Robots.txt file ?
What is Robots.txt files and why do you need a robots.txt file in your blog ?
Robots.txt is a file which the search engines look for, everytime they crawl your webpage. Robots.txt files tells the search engines which pages to index and which pages not to index. This will help the search engines to neglect the 404 error pages and other pages that you don’t want to be in search engines. Officially as per google, Robots.txt is defined as
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)
The main advantags of using a robots.txt file are ;-
- You can avoid indexing 404 and other error pages
- You can control the pages getting indexed in search engine
- You can avoid the Duplicate content issues
- Your blog will be SEO friendly
- If done wrong, It may block your site from getting indexed in search engine
How to add & edit Robots.txt in WordPress ? – A simple yet better robots.txt file for WP
WordPress is highly search engine optimized and search engine friendly and already has a good built-in robots.txt file. But since WordPress is a professional choice and looks a bit confusing, You may find it hard to edit the robots.txt in WordPress.
In case of WordPress, It’s rather simple. It automatically creates a robots.txt and since it is auto generated, you may won’t be able to locate it on your server files. The virtual file will be something like this
But to modify it Just install this plugin called WP Robots Txt to your WP blog. After Installation, Go to settings -> privacy settings to find your robots.txt file.
Then add this to your robot txt file
This will disallow the user agents like google search, Google image, mobile, adsense etc in entering the pages like your theme files, hosted files etc and will allow them to crawl the rest of your website completely. Alternatively, If you are getting too much 404 errors in your sitemaps or if you want to block a specific section from getting indexed in search engine, you may add a line “disallow:” followed by the path of the category you would like to block. for eg: To block search engines from including your comments, you should add the following lines to your robots.txt file
Likewise, you may block any contents by adding them in robots.txt files. This can be very useful if your site creates 404 errors in any sections, If your site has duplicate content issue, if any plugins goes wrong etc.
I had tried my best to make this article the simplest, and I believe that this had been useful for you. In case you are not sure how to do it, I advice you not to do it and you may contact me for any help on this topic ! The comments are open and would love to hear your feedbacks and quarries.