builtin@search_engine_record.com hammering database

Hi, we are currently running v4.0 (eagerly awaiting 4.3) and our database has seen a massive spike in usage. I increased the DTU not long ago, but it has now started averaging 100% load again.

Looking at the logs, all of the errors are related to "Execution Timeout Expired." and from builtin@search_engine_record.com.

these are trying to access URL's like:
https://myurl.com/engine?specs=2%2C22%2C23%2C27%2C29%2C31%2C33%2C34%2C36

Is there anything I can do to halt this DDOS?

networkfusion wrote:

If its a google bot, then you can instruct them to exclude them from crawling using rel=nofollow on the links generated for the filter. You can also configure and instruct google crawlers using Google Search console.

If there are other bots, the best thing is to find their IP address from logs and block them to access your site.

Thanks. Since I use cloudflare, I have blocked a couple of crawlers that were causing most of the exceptions and considered to be "spam". Unfortunately that has not solved the DB issues, so I will work towards getting 4.3 working ASAP and raise a new topic if DB performance is still an issue...

networkfusion wrote:

That's good.

I have the same problem... a lot of records in LOG for the same activity from bot
How did you solve the problem?

V4.3 you can set the Clear Log routine in System\Scheduled tasks is one way of just routinely clearing the log based on a time period. Although it is not going to stop bots from crawling your sight, Or running your customerId values out into the outer galaxy. After many years of upgrading to the next nop version your customerIds grow exponentially by guest records created by bots.

It is a long time ago, but it was mainly looking at the inner exception code to find out the exact crawler and then assessing it to see if it was from a market the store does not care about (e.g. China). I then added suspect ones to my firewall rules on cloudflare (since I use that for free dns and ssl purposes).

I also used a custom script to delete all guest accounts without cart items in the database, since the scheduled task for doing it kept timing out (which was also eating DB resources). Once that was done, things became relatively stable again.

I actually have my task disabled and do a similar routine, forces one to review the logs prior to removing. But the task is there if someone wants to just use as is. I to place certain IP's on restriction to the server.