I think this may have been covered, but I can't find it in a search. How do I prevent some boards from being crawled by search engines (Google) ? Can I make it so that the forum itself will be crawled but so that the actual boards will not? In other words, I know that I can make it so that the actual thread can't be viewed unless someone is a member, but I don't want any part of a private thread to show up in ANY search engine. I have one "Public" board and the rest are private. It would be fine for the Public board to be crawled (in fact I want it to be) but I absolutely do NOT want the Private boards to be crawled. Any help would be greatly appreciated! TIA,

Coleen

Interesting. After doing some searching on Google, I found (I think - I HOPE) what I was looking for.

In order to disallow search engines from crawling your website or pages on your website, I knew you needed some kind of a "robots.txt" file, but wasn't sure what went in it or how to specify to only disallow certain pages. I want to have the Main Forum crawled, and two "public" boards, but don't want any of my Private boards to be crawled. So to do that you can put your robots.txt file directly in the root of your website (in this case I guess since my forum is a sub-directory of my home page I would have to put the robots.txt file in the wwwroot of my website...can anyone confirm that?) any way here is a link to a good resource on the robots.txt file: Introduction to "robots.txt" 

The main thing, I "think" is to set it up to allow Google to search my main website but disallow all the private sub directories. i.e., in my text file I have the following text:

User-agent: *
Disallow: /Forums/yaf_topics2_Newly-Bereaved.aspx
User-agent: Googlebot
Allow: /aheartbreakingchoice.com
http://aheartbreakingchoice.com/Forums/ 

So hopefully this will work. Has anyone else used robots.txt to prevent (or allow) their forums to be crawled? Have you had any issues?

Thanks!

Coleen

I use this method, but I disallow everything, works well. Not sure about being selective.

Thanks Thantis I hope this works like I think it will! 🙂

I've noticed that a section that only I have permission to see is somehow being crawled as well. Always has confused me.

After doing some searching on Google, I found (I think - I HOPE) what I was looking for.

---