Robots Txt File Do you think you need one? I didn’t until I saw what Google was indexing on my site! Have you ever had a look to see what Google has indexed for you? A quick way to do that is shown in this image !
When I looked there were links to my post great! But there were also links to things like my WordPress Login and various other WordPress core files! That has to be a security issue I’m sure. So I started to research this and found, fortunately a very simple answer! I needed a Robots Txt File.
A Robots Txt File tells the search engines exactly what you want them to look at and index on your site and maybe more importantly what you don’t want them to look at and index!
There is another issue here too and that is duplicate content. Whilst Google doesn’t exactly penalize your site for having the same content in multiple places it is advisable to limit the amount of times the same content appears.
For instance one post could be seen on your home page, then in the related category or categories if you found it was relevant to more than one, then in the archives, then in the tags and that is just off the top of my head. So that one post could appear half a dozen times on Googles index! There are many thoughts out there that think this will dilute your page rank!
As Google understands blog type sites now more so than it did, it will make an attempt to show the most relevant occurrence on your site. There is a great story which illustrates the need for the robots txt file much better than I can here.
So after much research and looking at examples online I came up with this robots txt file.
I used this plug in and simply pasted the text in to it. That was it, now I will see if my site improves in the rankings or if there are any problems associated with what I chose to include.
My aim was to get rid of the indexed links that should not be there like my wp-admin login and to limit the duplicated content on the indexes.
I don’t want you to take my word though, please do a bit of research yourselves, I will provide a few starting points below. Feel free to use the robots txt file that I have as a start, edit it as you see fit. Please make sure that you fully understand exactly what the file will do though before you implement it.
You may come to different conclusions than I have so please do let me know your views in the comments section I will answer each one.
Related links for robots txt file.
No related posts.
No comments yet.