Extremely high database load after upgrading to 3.8 - did i do something wrong? Need help ASAP

Good to see there is some progress :)

But sukesh:

Do you see the same stored procedure in your project as Microsoft does? The first one in the following post:
https://www.nopcommerce.com/boards/t/44286/extremely-high-database-load-after-upgrading-to-38-did-i-do-something-wrong-need-help-asap.aspx?p=3#175911

If it worked OK with 3.70, then you may want to DIFF it against the one with 3.80 to see what changed.

a.m. wrote:

I could try that out. How?

[email protected] wrote:
This stored procedure was changed in version 3.80. Please see this commit

a.m. wrote:
Have you had a chance to test it? Any changes?

Hello.

I'm not sure how to change the current procedure.

I sent nopcommerce a ticket on premium support.

I will update this thread when there is progress.

This has been very time consuming, and i have a update.

The reason for the problems we have been having is actually google bots spamming me 3 times per second. Why? No idea.

I started logging SearchProducts with customername, and 3 times per second the user builtin@search_engine_record.com keeps searching for products(categorylist).

After i started returning an empty result list, without doing any queries my database went from 100% cpu usage to 3% cpu usage.

Now, the categories are not cached, which is a problem in itself. There is probably a good reason for it, but it hurts. We have 5 stores which have quite a lot of traffic, and i don't know how blackfriday will go. A simple loadtest from azure with 20 concurrent users for 3 minute on 1 category is VERY heavy load for the database. After 1 minute a single navigation takes 10-15 seconds. We have a S3 database for every nopcommerce solution.

Hi Lintho,

Did you ever figure out a solution for this issue? We are facing the exact same problem (also on Azure), with the exact same query.

Thanks,
Shawn

I will give you a answer when im on my computer. Short version: i ended up not returning products if the request is from a crawler and if the request contains specs in the url. You can do a insert log in productsearch method with the current request url to see if the problem is because of bots indexing.

However, nopcommerce should really handle all the requests coming from the crawlers.

Are u using web master tools? I found out that google had indexed about 2 million specs urls.