I was having an issue with multiple customers being created due to my use of UpTimeRobot.
The line:
crawlerItems = XDocument.Load(sr).Root?.Elements("browscapitem").ToList();
returns a zero length list (because there are no elements in the crawlerOnly file. However, the next line checks for null:
if (crawlerItems == null)
Because crawlerItems is a zero length List, the full browscap.xml file is never loaded.
In a related item, I tried deleting the browscap.crawleronly.xml file, which should cause the system to pull all the crawler items from browscap.xml and place them in browscap.crawlersonly. However, the line:
crawlerItems = XDocument.Load(sr).Root?.Elements("browscapitem")
//only crawlers
.Where(IsBrowscapItemIsCrawler).ToList();
doesn't load any items. Am I missing something? Based on this, all crawler bots may be generating a large amount of blank customers in the customer table because the system isn't classifying them as a bot correctly.