Comments on: How to Add A Robots.txt to your site http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/ Strategies to help you build your niche store empire Wed, 07 Jan 2009 21:13:25 +0000 http://wordpress.org/?v=2.5.1 By: Shelley http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-1266 Shelley Sun, 22 Jun 2008 03:45:19 +0000 http://www.nichestorestrategies.com/?p=10#comment-1266 Hi Rochelle, Thanks for your reassurance. That's what I did. I went and checked which pages it couldn't view and they were the ones I had disallowed. Just gave me a bit of a scare at first. :) Thanks again for your help. Shelley Hi Rochelle,

Thanks for your reassurance.
That’s what I did. I went and checked which pages it couldn’t view and they were the ones I had disallowed.
Just gave me a bit of a scare at first. :)

Thanks again for your help.

Shelley

]]> By: Rochelle http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-1241 Rochelle Sat, 21 Jun 2008 13:08:40 +0000 http://www.nichestorestrategies.com/?p=10#comment-1241 Shelley, I'm so sorry for not responding sooner to your question. It was an oversight on my part. This message is ok. If you created a robots.txt file using my instructions then there are five places in your site you are telling robots to ignore and not index: Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /cont/ Disallow: /themes/ Disallow: /scripts/ The message you saw is letting you know that there are places that are disallowed. It's letting you know about this in case you didn't mean to disallow any pages, or in case you disallowed a location you actually want indexed. Once I did exactly that. I accidently disallowed my entire site (not good for indexing!). So, if you see this message it is a good idea to verify that you are only disallowing pages that you don't want indexed. If you find that you accidently disallowed locations that you DO want indexed, just remove the 'Disallow: /whatever file or folder is disallowed/' from your robots.txt. Rochelle Shelley,

I’m so sorry for not responding sooner to your question. It was an oversight on my part.

This message is ok. If you created a robots.txt file using my instructions then there are five places in your site you are telling robots to ignore and not index:

Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /cont/
Disallow: /themes/
Disallow: /scripts/

The message you saw is letting you know that there are places that are disallowed. It’s letting you know about this in case you didn’t mean to disallow any pages, or in case you disallowed a location you actually want indexed.

Once I did exactly that. I accidently disallowed my entire site (not good for indexing!). So, if you see this message it is a good idea to verify that you are only disallowing pages that you don’t want indexed. If you find that you accidently disallowed locations that you DO want indexed, just remove the ‘Disallow: /whatever file or folder is disallowed/’ from your robots.txt.

Rochelle

]]> By: Shelley http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-1105 Shelley Thu, 12 Jun 2008 07:30:45 +0000 http://www.nichestorestrategies.com/?p=10#comment-1105 Hi Rochelle, When working through your checklist, I completed the task to disallow the safety on ebay page from my robots.txt file. I've just gone back into Google Webmaster tools and there is a warning on the site: "URLs roboted out When we tested a sample of the URLs from your Sitemap, we found that the site's robots.txt file was blocking access to some of the URLs. If you don't intend to block some of the URLs contained in the Sitemap, please use our robots.txt analysis tool to verify that the URLs you submitted in your Sitemap are accessible by Googlebot. All accessible URLs will still be submitted." Can you please advise if this is normal? If so, will it affect any indexing that may occur? Thanks Shelley Hi Rochelle,

When working through your checklist, I completed the task to disallow the safety on ebay page from my robots.txt file.

I’ve just gone back into Google Webmaster tools and there is a warning on the site:

“URLs roboted out
When we tested a sample of the URLs from your Sitemap, we found that the site’s robots.txt file was blocking access to some of the URLs. If you don’t intend to block some of the URLs contained in the Sitemap, please use our robots.txt analysis tool to verify that the URLs you submitted in your Sitemap are accessible by Googlebot. All accessible URLs will still be submitted.”

Can you please advise if this is normal? If so, will it affect any indexing that may occur?

Thanks

Shelley

]]> By: Rochelle http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-438 Rochelle Mon, 21 Apr 2008 01:01:02 +0000 http://www.nichestorestrategies.com/?p=10#comment-438 Joette, My sincere apologies to you. I had some HTML codes in the robots-txt.txt file that I have now removed. Thank you for bringing this to my attention. Rochelle Joette,

My sincere apologies to you. I had some HTML codes in the robots-txt.txt file that I have now removed. Thank you for bringing this to my attention.

Rochelle

]]> By: Joette http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-437 Joette Sun, 20 Apr 2008 16:47:23 +0000 http://www.nichestorestrategies.com/?p=10#comment-437 Checked my robot txt in google tools and for many of the lines the result was "Syntax not understood" line six title was "Accepted but correct syntax includes a colon" Is this normal, I followed the instructions. Thanks for any help, I love-love-love this site, Joette Checked my robot txt in google tools and for many of the lines the result was “Syntax not understood” line six title was “Accepted but correct syntax includes a colon”
Is this normal, I followed the instructions. Thanks for any help, I love-love-love this site, Joette

]]> By: How to Add your Dynamic Sitemap to your site’s robots.txt | Niche Store Strategies http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-142 How to Add your Dynamic Sitemap to your site’s robots.txt | Niche Store Strategies Mon, 25 Feb 2008 17:23:13 +0000 http://www.nichestorestrategies.com/?p=10#comment-142 [...] to your robots.txt file (click here if you don’t know how to do this or see How To Add a robots.txt if you don’t yet have a [...] [...] to your robots.txt file (click here if you don’t know how to do this or see How To Add a robots.txt if you don’t yet have a [...]

]]> By: How I Create a New Niche Store | Niche Store Strategies http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-117 How I Create a New Niche Store | Niche Store Strategies Sun, 24 Feb 2008 21:50:07 +0000 http://www.nichestorestrategies.com/?p=10#comment-117 [...] Eighteen Add robots.txt to [...] [...] Eighteen Add robots.txt to [...]

]]> By: How to Add a Dynamic Sitemap to your Site | Niche Store Strategies http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-58 How to Add a Dynamic Sitemap to your Site | Niche Store Strategies Mon, 11 Feb 2008 16:42:09 +0000 http://www.nichestorestrategies.com/?p=10#comment-58 [...] Seven Add the dynamic sitemap to your robots.txt (click here if you don’t yet have a [...] [...] Seven Add the dynamic sitemap to your robots.txt (click here if you don’t yet have a [...]

]]> By: How to Add robots.txt Meta Tags to Your Site | Niche Store Strategies http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-49 How to Add robots.txt Meta Tags to Your Site | Niche Store Strategies Sun, 10 Feb 2008 18:28:20 +0000 http://www.nichestorestrategies.com/?p=10#comment-49 [...] it is a good idea to add a meta tag that helps search engine spiders find your robots.txt file (click here for instructions on how to add a robots.txt to your site). Doing this may also help your site get [...] [...] it is a good idea to add a meta tag that helps search engine spiders find your robots.txt file (click here for instructions on how to add a robots.txt to your site). Doing this may also help your site get [...]

]]> By: How to Add a Dynamic Sitemap to your Site | Niche Store Strategies http://www.nichestorestrategies.com/how-to-add-a-robotstxt-to-your-site/#comment-16 How to Add a Dynamic Sitemap to your Site | Niche Store Strategies Mon, 04 Feb 2008 17:02:50 +0000 http://www.nichestorestrategies.com/?p=10#comment-16 [...] Three Help search engines index your site quicly by adding your dynamic sitemap to your robots.txt (click here if you don’t yet have a [...] [...] Three Help search engines index your site quicly by adding your dynamic sitemap to your robots.txt (click here if you don’t yet have a [...]

]]>