High Load and timeouts starting at 30s

Hi,

We were the first company to develop a live, fully responsive, multi-store nop site, running 13 stores in total from one application (nop 3.00) - football clubs shops.

We used to have a lot of problems when the server was under load, so we changed servers, along with an upgrade.

After still having issues, we moved to 4.00 when it was released, along with another server change/upgrade, but we still had issues, specifically when we had kit launches, and the load was on the heavy side (to be honest, we're only talking about 300-500 customers at a time, so not really what I would class as high).

Anyway, we again changed servers, to what we have now:

Dual E5-2620 v3 CPU’s
Dual 1TB SSD in raid 1
64GB ECC ram
Windows Server 2019
IIS 10
MSSQL, with web licence
Plesk
1Gb/s port

We're still experiencing timeouts under load, so thought we'd run some tests on basic installs of both 4.00 & 4.30 - no products, no categories, so really are basic installs!

We've done these tests on very basic installs as our production sites use some bespoke code/plugins, so wanted to see what the load would be live on clean installs.

Here are the results for 4.00.


Timeouts starting at 30s.




Here are the results for 4.30.


Timeouts starting at 30s.




During the tests, for 4.00, nop.web uses around 40%, although, on our production sites, we've seen it as high as 100%!

By the way, we also run other sites, using 3.7, 3.8 and 3.90. One is another multi-store (3.70), and we have a 3.90 with heavy code changes - these all run fine under a 500 load, and all have bespoke code changes/custom themes.

We're at our wits end with this now and would appreciate anyone chiming in, that may have had similar issues?

Regards,
Craig

p.s. We've tried all sorts, including changing the times for scheduled tasks, stopping scheduled tasks completely, but only seem to get a very small improvement, and nothing worthwhile.

We also tried the MSSQL Provider V2 plugin, but had multiple CGI / FASTCGI processes, which loaded up the server.

Interesting analysis and report.

1. What kind of improvements were done earlier to handle the load test issues.

2. Do you have similar results that ran on Visual Studio Load Testing ?  

3. Are the images too heavy ? or it has just the default data ?

4. Do you have data collected with the default data that gets when installed fresh ?

What do you think on the below questions ?
Is that data issue ? Is that a NopCommerce Software Issue ? Rendering Issue ? Database Issue ?

kmsrao wrote:

Hi, and thanks for the reply!

The two tests for 4.00 & 4.30 are fresh installs, without the default data - so empty basically! I'm just waiting for my tech guy to run a test on a fresh, 4.30 with the default installed products / categories, so will report back once I've got that data.

Personally, I think it's a Nop issue, but I'd love to see if others have the same issue, or can test their own?

We're also going to run full tests of 4.30 on Linux & MySQL to see if the issue may lie with IIS / .net etc

I've not run any visual studio tests on my local machine, mainly because it's not the fastest as it is lol

We've got another kit launch this week, plus a new club launch on Monday, and has to go right!! Otherwise, we're placed into a very very difficult position with the client of course.

At the moment, or 4.00 falls flat with anything close to 500 concurrent, and grinds to a halt near enough at 300... not good!

The database is fine, we're running SQL Server with a Web Licence, so we don't have the limits associated with SQL Express.

Okay, so after testing a basic, 4.30 version, along with the base categories/products that install, pushing 500 concurrent, we get zero timeouts.

That changes once we start to push past 1000, and 2000 where we start to see timeouts from 30s and the test gets aborted because it reached the error threshold!!!

But the server loads doesn't go above 35%?!

Response Times
Average  : 5103 ms
Min/Max: 147 / 10650 ms

Response Counts
Success: 1353
Timeout: 729
400/500: 0 / 899
Network: 0

Bandwidth
Sent: 615.31 KB
Received: 53.46 MB

Redirects
Valid: 0
Invalid: 0

Thats awesome.

As mentioned in the 4.3 Release data, The improved cache management and getting rid of entity framework did the magic.

Given the data , looks like scale out option might work for you instead of scale up.  you may want to explore that option.

Just to add, While doing a normal deployment testing of working of the website, I did see Linux seems to be a bit faster than the windows counter part.

But that is just one data point. may or may not be entirely true. But you can check that option as well.

Okay, so installed Redis, and we're still hitting a wall at 30s with 1000 clients...

Someone must have an answer for this - anyone from nop able to help????

This is a basic, simple install of nopCommerce 4.30, with just the products/categories from the install process. No custom code, no plugins (apart from the ones installed at the time of installing nop).

Did you get any answers from NopCommerce Team ?

Have you read the Performance guide?

It more of a general context and not on exact numbers.

The concern here is that even with the default data that comes with nopcommerce installation, the response time is that it is timeingout after 1000 clients.

Is that the limit that we need to live with and scale out after that ?