4.3 .NET Core Very Slow to return View

We are currently migrating from NopCommerce 3.8 to 4.3 and therefore also migrating from ASP.NET MVC to .NET Core. We have a custom plugin for our Category and Search pages. When I try to navigate to a page, all of the heavy logic completes in a reasonable amount of time, but there is still an additional 2 -3 minutes of seemingly idle time before the page is displayed. I don't quite understand what is happening during this huge window of time, but when I set a breakpoint at the end of the .cshtml view and step over, after the long wait, I am taken to the Nop.Services.KeepAliveMiddleware.cs file line 49:

await _next(context); (199,000 ms elapsed)

I've tried using Diagnostic Tools to see what is going on, but it just shows nothing happening during this window of time. Any ideas why it takes so long to return the view..?

Do you have the source code for your custom plugin?  You'll need to set breakpoints within the plugin itself to see what's taking so long.  The view is just waiting on the plugin for model data and a breakpoint in the cshtml isn't going to give you insight into what's actually going on.  In the cshtml you'll be able to see the data returned, but not what took so long to yield that data.

If you're using SQL Server, you can use its Profiler to see queries against the database in realtime, but without the plugin source code you won't be able to fix them.  My guess is some runaway recursion on your categories or inefficient search methods are going on, and you see the Middlware error but it's actually a database timeout that's happening.

Also, a lot has changed between 3.8 and 4.3, has the custom plugin been updated accordingly?  They did away with Entity Framework in favor of LinqToDB in 4.3, and it could be a simple matter that your plugin isn't working at all.

Do you have logging enabled?

The custom plugin is our own and is what I am mainly working on migrating. It is pretty much finished and working fine without errors outside of this performance issue. Everything comes back fine as expected when running locally it just takes some time. Until now I figured it was  just because I am running locally and that is always slow, but after deploying to an App Service it just throws a Gateway timeout error every time despite the rest of the site running quickly and smoothly.

Yes I have set breakpoints in there as well. All of the logic in there to retrieve the categories and everything else completes in a reasonable amount of time, then it returns the view and then my breakpoints in there all get hit right after. Then after that is when it sits idle for a long time. I have put breakpoints in all of the main methods that are getting processed including the View Components that get triggered after the fact (previously they were partial views in 3.8) and all of this logic completes in a reasonable amount of time.

I have also viewed the queries made to the database using the Diagnostic Tools Events, none are made during this idle window.

Logging is enabled. I have logs in nearly every heavy processing logic and can see the same behavior from the log time stamps as well. All of the main logic is completed in a reasonable amount of time, yet the page takes 2 - 3 minutes to return and display afterwards.

After all of this, none of my breakpoints get hit during the 2-3 minute idle window which is why I can't figure out what's going on.

Can you temporarily disable/uninstall your plugin to see how categories and search function in a base installation?  That could help rule out database/server issues.

When trying to load the default NopCommerce Category page I keep getting this error:

InvalidOperationException: The following sections have been defined but have not been rendered by the page at '/Views/Shared/_ColumnsTwo.cshtml': 'Breadcrumb'. To ignore an unrendered section call IgnoreSection("sectionName").

I've tried modifying _ColumnsTwo.cshtml to both ignore (as suggested) and render the Breadcrumb section, but it keeps throwing the same error.

Ah ok, so category breadcrumbs are generated recursively so it does sound like it's going to be related to an SQL timeout or something in your Category table went awry during the upgrade.  

Set a breakpoint in the Category service and watch the while loop for GetCategoryBreadCrumbAsync:
https://github.com/nopSolutions/nopCommerce/blob/bed6c7afd0d8fe2ae486870a4098b0e18a8eff68/src/Libraries/Nop.Services/Catalog/CategoryService.cs#L769

If it returns in a reasonable amount of time then figure out what occurs next.

I have verified that there is no SQL timeout or issue with any of the C# category logic. I narrowed the issue down to the line in the view's layout that calls RenderBody() and that the issue disappears if I disable Specification Attribute Filters. It seems that there are so many filters due to so many results on larger categories, I guess its too much for it to render.

I find it odd that this was not an issue in 3.8, but now it is in 4.3. The only solution I can think of for this is to dynamically load in only spec filters that the user clicks on.

If you continue to upgrade to 4.40 perhaps the specification attribute performance increases:
https://github.com/nopSolutions/nopCommerce/issues/276

To re-iterate, it isn't an issue of the spec attribute logic, the C# code to retrieve them and also the part in the view to loop through and generate HTML all run quickly. (We also don't use default NopCommerce for retrieving them anyway, all data is retrieved from the Solr search engine which is much faster than database calls)

The issue is with the view engine itself being slow to render so many elements. The only other issue I could find that is quite similar to mine is this:

https://github.com/aspnet/Mvc/issues/7704

It seems that their issue was due to Kestrel though, so not so sure how helpful that is since we're running on IIS, but it seems to suggest something at the core WebServer/View Engine level just can't keep up with rendering thousands of elements on the page.

I'm having a meeting with my team this afternoon to discuss potential solutions, but I'm thinking its probably going to come to changing to loading the filters after initial page load through AJAX.