Does anyone have a reliable way to filter search bot activity in the Domino web server log (domlog.nsf)? Or to re-phrase the question, how can I see just requests from genuine browser clients?I would like to know how much activity on our web site is from humans, and how much is from search spiders.
I have tried using information in the ‘useragent’ field, with mixed success. For example, a new view using the following selection formula is somewhat accurate:
browsers := “msie” : “safari” : “firefox” : “jakarta” : “blackberry” : “opera” : “gecko”;
others := “winnt” : “win98”;
SELECT @Contains(@LowerCase(useragent); browsers : others)
However, I still think that some activity is getting through, as the complementary view:
SELECT !@Contains(@LowerCase(useragent); browsers : others)
still contains some activity that does not look like search spiders.
Any tips appreciated.
thanks,
p