Nop 3.8 - 3.3 GigaByte Memory usage!

Wondering if this is normal :)

Using nop 3.8
When starting the site - the memory usage is around 500MB - not too bad considering we are using EF.

But as soon as I click on the administration link - where the dashboard will eventually show up - IIS starts gobbling memory at an alarming rate. I grows from 500 MB all the way to 2.9 GB before the admin page even shows up.
At the point the admin page is up but the "Latest Order" widget is not yet populated.

So I wait a little longer - finally the "Latest Order' widget is populated - and now the memory usage is 3.3GB!!!

Of course the admin pages are very slow to open. The DB contains roughly 130,000 orders in there.

I know entity framework is not made for a real production site - but 130,000 orders is reasonable - I would have thought it could handle that.

My previous site was doing the same action - loading the same exact orders with an old php site using stored procedure and it took less than 1 second to load. But now - with nop it takes 1 mn at least - and the memory usage is of course insane.

Any comments?
Is this really normal?

Frederic

Hi Frederic,

1. Is it possible for you to debug the application? If yes, please set breakpoint into "OrderList" method of \Nop.Admin\Controllers\OrderController.cs file. What is value of "PageSize" property of "command" paraneter when you open the dashboard?
2. Collapse this widget and reload the dashboard page. Is it still slow?

Sorry for the delayed response it has been a crazy last couple of days.

Also thank you for your very clear instruction.

I did as you mentioned and the page size is 5. Which is what I see on the screen so I guess that makes sense.

When I click on the administration link on top of the page -
It takes 30 second for the page to appear - but it is still busy and is not responsive.
It takes an additional 60 second for the admin dashboard page to become responsive. The dashboard widget with orders was the one still showing it is loading during that time.
In total it is around 90 seconds for the admin dashboard to become usable.

I also took a screenshot of the memory usage when I nopCommerce is done loading the admin page.
It is at 3,437MB!!!

I wanted to uploaded here so that you guys can see it but I could not find a way to uploaded the picture without first putting it on the web.

Frederic

FredBell wrote:
1. Do you mean widget with graph (year, month week)?
2. Does it work fine when you collapse it?

The widget I am referring to is the Latest Orders widget at the bottom. This one takes forever to load.
Collapsing it made no difference.

I have started to do some profiling of the code.
Again - I have 125,000 orders and 75,000 customer in the DB.
But only 200 products.

I started the app - opened the administration dashboard, open the order list page (showing 100 order per page) - went back to the administration dashboard page and now closed the app and the profiler.

Doing this simple 3 steps I see the following in the profiler:

1 - OrderController.OrderList account for 86,000+ calls
1 - a - OrderService.SearchOrders calling pagedlist is half of it. (48,000+)
1 - b - CurrencyServices.GetCurrencyById is the other half. (37,000+)

Going deeper into the code I can see the caching does not really work for some items) currency and stores - does not work for sure)
Due to AUtoFac misconfiguration or misunderstanding in nopcommerce. The LifeTimeScope setting on some caching component renders the cache useless (in a way it actually hurts becuase the developers clearly thought it works).

I am looking at where is the best place to fix the code - if it can be fixed - I certainly hope I can. Otherwsie I am not sure what I will do yet.
I think I'll look into setting store cache and currency cache as in memory cache. I do not see why it is done at the life time scope at the moment. Maybe there are reasons I do not know yet. Please tell me if it is ok to change these 2 caches.

I have a lot more profilling to do. This information is only the beginning of my investigation.
But I thought you would be interested in hearing the first results from running the profiler on the code base.

Profilling code is complex and takes time - so I hope this effort will be successful.

Any advice would be appreciated.

Frederic

Thanks a lot for detailed reply. What browser are you using?

And one more question. Is it possible to share your solution with source code? As we still cannot reproduce this issue

I am using chrome in general.
For the performance profiling I used ie10 and edge.

I did not see a performance difference between all these 3 browsers.

The changes I suggested earlier have made a difference.
I reduced the load time of the dash board from up to 90 second down to 45 seconds.
This is simply by fixing 2 cache issues in the dependency registrar / container builder.

But it is still not as good as it should be. Ideally we should be under less than 10 seconds. Even that would still be slow but it would be more acceptable to my employees.

The biggest issue I saw was with Order controller and search order list - specially pagedlist. But I have not yet had a time to look into it.
Based on profiler result I see a few major issues left:

In adminstration Home Controller 2 piece of codes are killing the system:

           model.NumberOfOrders = _orderService.SearchOrders(
                pageIndex: 0,
                pageSize: 1).TotalCount;

            model.NumberOfCustomers = _customerService.GetAllCustomers(
                customerRoleIds: new [] { _customerService.GetCustomerRoleBySystemName(SystemCustomerRoleNames.Registered).Id },
                pageIndex: 0,
                pageSize: 1).TotalCount;

I do not know why yet - but if this code relies on entity framework to get the value then we are dead.
Becuase that would mean all data would need to be returned before we can do a simple count.
But of course that cannot be the case - right? Because that would crazy and be a million time slower than a simple DB to get 1 integer value back.

  

As for sharing - of course I will.
Is there a way for me to check in the code back into GIT?

Frederic

Alright - new update.

I am not sure how to tell you that - but yes the code was insane.
Anyhow - there is not much to say. I know the product is free so I will not complain.
In fact I will do the opposite.
I really like what you have done with nopCommerce. I spent the last 3 weeks looking at the code and I think it has real potential.

As for the craziness in the code, I am sure the person who wrote it works for free. Nonetheless, you should talk to them - there is no excuse for not knowing basic computer science stuff like that.


I fixed the issues - the page loads much faster now. From 90 second down to 3 seconds. But there is still one big issue left.

What I managed to fix today regarding the administration home page:
The 2 caches that were not setup right.
And the 2 insane DB calls that retrieved the entire DB tables with 100,000+ records just to compute a count :)
Funny - I think this is funny now...

Unfortunately there is still one major issue with the page - the search order and the paged list. But I don't have time today left to fix that. It is already 4:30 AM for me.

You said you could not reproduce the issue.
The code is the code - if you test it with data you have to see the issue. I downloaded that code from your site. So I am hoping we are using the same code base. It is version 3.8.

Here is what I suggest: simply load a DB with at least 100,000 orders - 50,000 customers and probably 1000 products and start using nopCommerce in that configuration.
This should give the team a better sense of where the product is in terms of real life performance.

Again - I think it is a great product with real potential. I like it and I thank you for it.
But if you do not change your data access layer design you have no chance to compete. Using an ORM will never work for a professional system.
If you redesign your data layer you should be at least 100 times faster than you are now.
Gee I just spent 5 hours on your admin page and I changed it from 90 sec to 3 sec. Imagine if the rest was fixed as well.

I know you do not know me - but I have been doing this for 22 years now - working as a consultant fixing shitty project that other people wrote. Corporation call me when their project is virtually dead. And you know what, the vast majority of the time it was becuase some young kid working in that corporation decided to use an ORM. And after month of work - the company could not fix it. Solution was simple - get rid of it and use the DB and the set language it contains for what it is really good at.

Anyhow - you will do whatever you want - and that is cool. We are just trying to help you.
Hopefully you understand.

As for the code chnages and all the fixes. Let me know what you want.
I am happy to share with the community of course.

Have a wonderful day and thank you for your help before.

Frederic

Hi Frederic,

FredBell wrote:
These two methods cannot be the case because they do not load all records into memory. They load only one record and then use "TotalCount" property. Please see "pageSize" property (we pass 1)

FredBell wrote:
Please share it using any file sharing service, Google Drive, etc.