BrowscapXmlParser uses too much resource per request, here's my fix

I just upgraded to version 3.80 and run a setup where I have my own webapi running on top of nopcommerce.

After the upgrade, I noticed the GC running very frequently within seconds on subsequent identical calls that dont involve much of the system.

A quick cpu profile showed that 25% of my request processing time was spent in BrowscapXmlParser's regex function.

public bool IsCrawler(string userAgent)
{
    return _crawlerUserAgentsRegexp.Any(p => Regex.IsMatch(userAgent, p));
}

To fix, I changed it to populate on load a thread safe cache object that can be checked per request.  This eliminated this method from even being registered by the cpu profiler as worth of investgation which means it's likely around or below a 1/10th a percent of the request processing.  

A significant improvement from 25%. This also fixed the frequent GC executions I was seeing.

https://github.com/nopSolutions/nopCommerce/pull/2181/commits/e6acb0215a2208ccc04ffe2d81dec7738a1ffc6b

It might be useful to get acquainted with this topic

Updated PR : https://github.com/nopSolutions/nopCommerce/pull/2181/commits/8ea07bbfebb13d7e09f87c9724fe02eb3edcff55

Thanks. But cookies are not a way to go (just to detect an search engine). Please also note that search engines ignore cookies (don't use and send them)

A search engine's user agent will be stored in the cache for future requests.  The cookie is for anything that supports it like a browser.

Pseudo logic of my change is

If ua in cacheobject return

If cookie exists return

Get browscap result

If search engine, store in cacheobject

Write cookie

Return


This prevents browscap getting called for the same search engine making requests to the box.

Cookie enabled clients are also prevented.