We do not want to close the whole forum from indexing for better SEO, but we'd like to get rid of all these problems when different URLs generated automatically by the forum engine point to the same page. For the above example, the main topic page is /Fo​rum​/ya​f_p​ost​st2​45_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d--​non​-re​pro​duc​ibl​e.a​spx, and the other links just show the same contents though they mean different things for the users.

In fact, what we need is to allow the Google bot to open the root forum directory (/Forum/) and index all topics by visiting the tree of links it can find starting from the forum home.

After analyzing WMT error reports and crawling the links on the forum manually, I've modified our robots.txt file the following way to solve the problem (the corresponding excerpt):

It seems, this should allow Googlebot to find really all topic pages and exclude all topic duplicates and such unwanted pages like yaf_profile31.aspx or ya​f_p​ost​sm5​68f​ind​las​tpo​st_bla-bla-bla.a​spx.

Am I right? I admit that I might overlook something. Or is there any other solution of our problem?

We do not want to change any YAF source files as we will need to do that every time when we upgrade to a new version.