YAFLogo

tecman
  • tecman
  • 100% (Exalted)
  • YAF All-the-Time Topic Starter
11 years ago
We're receiving notifications about duplicate meta descriptions and page titles in Google's WMT. The error report looks like this:

Pages with duplicate meta descriptions:

Edit box not closed even after clear method (non-reproducible): I am getting a few occasions while e

/Fo​rum​/ya​f_p​ost​sm5​68_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d.a​spx

/Fo​rum​/ya​f_p​ost​sm5​68f​ind​las​tpo​st_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d.a​spx

/Fo​rum​/ya​f_p​ost​sm5​70_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d--​non​-re​pro​duc​ibl​e.a​spx

/Fo​rum​/ya​f_p​ost​st2​45_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d--​non​-re​pro​duc​ibl​e.a​spx

We do not want to close the whole forum from indexing for better SEO, but we'd like to get rid of all these problems when different URLs generated automatically by the forum engine point to the same page. For the above example, the main topic page is /Fo​rum​/ya​f_p​ost​st2​45_​Edi​t-b​ox-​not​-cl​ose​d-e​ven​-af​ter​-cl​ear​-me​tho​d--​non​-re​pro​duc​ibl​e.a​spx, and the other links just show the same contents though they mean different things for the users.

In fact, what we need is to allow the Google bot to open the root forum directory (/Forum/) and index all topics by visiting the tree of links it can find starting from the forum home.

After analyzing WMT error reports and crawling the links on the forum manually, I've modified our robots.txt file the following way to solve the problem (the corresponding excerpt):

User-agent: Googlebot

Allow: /Forum/$

Allow: /Forum/yaf_topics

Allow: /Forum/yaf_postst*_*.aspx$

Disallow: /Forum/yaf_postst*findunread_*.aspx$

Disallow: /Forum/yaf_postst*findlastpost_*.aspx$

Disallow: /Forum/

It seems, this should allow Googlebot to find really all topic pages and exclude all topic duplicates and such unwanted pages like yaf_profile31.aspx or ya​f_p​ost​sm5​68f​ind​las​tpo​st_bla-bla-bla.a​spx.

Am I right? I admit that I might overlook something. Or is there any other solution of our problem?

Sponsor
bbobb
  • bbobb
  • 100% (Exalted)
  • YAF Developer
11 years ago
Links to the pages can be transformed to 'nofollow' .
tecman
  • tecman
  • 100% (Exalted)
  • YAF All-the-Time Topic Starter
11 years ago
What do you mean? Do we need to modify the forum engine?

We do not want to change any YAF source files as we will need to do that every time when we upgrade to a new version.