YAFLogo

JP
  • JP
  • 100% (Exalted)
  • YAF Leader Topic Starter
13 years ago
SInce you can now see some of the "Search bots" in the active users list by checking the "Show Crawlers in Active Lists" it would be nice to know where the criterias for bots are defined so you can add to them...

Since some bots use several IP ranges, it would be nice to be able to define which bots are which, since some bots still show up as "guests".... Since MSNBOT and Googlebot now show up for some IP address ranges, the principle works, but they are far from complete...

I have some MS addresses (Bing) shown as MSNBOT, but other MS Bing crawler addresses are shown as guests...

Would be nice if it was simply editable by a "webmaster", kinda like an XML file or in a DB table...

We who have search bots visits daily usually know where they come from, regarding IP ranges...


He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
Sponsor
bbobb
  • bbobb
  • 100% (Exalted)
  • YAF Developer
13 years ago
My hosting provider have soft which gathers general info like this for me.

The bots are only detected if they say - we're bots! Some bots use anonymous visits on purpose to gather an extra info. Are you sure that the visits are of any use for SEO? Ithink that NO. Maybe someone has more info on this...

JP
  • JP
  • 100% (Exalted)
  • YAF Leader Topic Starter
13 years ago
Well if 20 "guests" from the same IP range visit you, and you see the IP:s come from Yahoo, Yandex, MS, Google or whatever - you know they are not "human guests".... And usually they all use Mozilla 0.0 as their "browser"...

Google, Yahoo, MS and likes each have several identified IP class C ranges reserved for this to gather info... Maybe some yet-to-be-identified too...

As a sport, I sometimes do a search on newly posted items in the forum when I see a Googlebot or such on the site, and I usually get a first page hit when using a forum topic subject or parts thereof on the identified bot provider's search engine ...

Since I use Google Analytics, I can at least follow the Googlebot visits in time too...

As I said, it would be nice to be able to configure recognition of what you know are bots/crawlers/non-humans... 😄


He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
bbobb
  • bbobb
  • 100% (Exalted)
  • YAF Developer
13 years ago
Thanks, for the info.:)

Technically it's not a problem at all to make a cached table as an option, I'm personally too busy to make it now. Maybe it's can be tagged as a feature request. Else it can become a feature which no one is requested for.