[nopCommerce 4.60.6] Administration->Catalog->Products never completes

Further description:
I have a product catalog with 7037 products (3092 active/non-deleted)
Note this issue is not in 4.60.5, at least I cannot reproduce it on my test system, running on the same server as the main site.

Steps to reproduce the problem:
Simple, go to Administration, Catalog, Products. This gear in top right corner is turning and the query never finishes (I waited up to 15 minutes and during that time CPU on the server is spiking over 95% - until the IIS process is restarted). I debugged the issue and saw that a huge number of tasks is started (don't have an exact number, but in one Debug->Break the task list have ~2500 entries)

Steps I took to analyze the issue:
I deduced that the issue was in PrepareToGridAsync (in method PrepareProductListModelAsync in ProductModelFactory line ~795)

I started to remove all the async calls and set some default values:
productModel.SeName = "n/a";
(productModel.PictureThumbnailUrl, _) = ("images\default-avatar.jpg", (string)null);
productModel.ProductTypeName = ProductType.SimpleProduct.ToString();
if (product.ProductType == ProductType.SimpleProduct && product.ManageInventoryMethod == ManageInventoryMethod.ManageStock)
{
productModel.StockQuantityStr = "42";
}

and then, one at a time, I reset to original code, and it is the following line that is causing the issue
(productModel.PictureThumbnailUrl, _) = await _pictureService.GetPictureUrlAsync(defaultProductPicture, 75);

If I limit the search, by entering an item name, and re-searching, then thumbnails are generated in the images\thumbs folder and the return is very quick. 40 to 100+ products tested.

I see that this is similar to https://www.nopcommerce.com/en/boards/topic/62578/product-list-page-in-admin-area-doesnt-show-any-products/page/2 - but the resolution there is vague (what API?)

So, any idea what could be going on here?
Or what else I could try?

This works fine on a clean 4.60.6 installation.
What's your AdminAreaSettings.DefaultGridPageSize?

Are your images stored in DB or file system?
How did you do original image import?  (using nopC Import, or some other?)

(I suspect the "API" they refer to is the nopCommerce API which is  a separate paid product)

Hey guys,

thanks for your questions

AdminAreaSettings.DefaultGridPageSize is set to 100
Images are stored in db, but thumbnails obviously in the file system. Created originally directly in nopCommerce product pages, but also some integration with an ERP system which also stores the images in the db - although that is minimal, most images are added in nopCommerce.

Note - this site has been live since 1.40 of nopCommerce - and there is nothing new here, except an upgrade to 4.60.x (from 4.20)

If it's happening in GetPictureUrlAsync, then maybe it's blocking there due to file create / access.  Can you disable any virus scanning temporality and test again?

Note that in GetPictureUrlAsync it does "block" when writing files:

//the named mutex helps to avoid creating the same files in different threads,
//and does not decrease performance significantly, because the code is blocked only for the specific file.
//you should be very careful, mutexes cannot be used in with the await operation
//we can't use semaphore here, because it produces PlatformNotSupportedException exception on UNIX based systems
using var mutex = new Mutex(false, thumbFileName);\
...

Thanks all for this. I now know what is happening and will be contacting the nopCommerce team directly with a couple of things.

The issue - for me - was a single image (in PictureBinary) for a single product (which happens to be on the first page of the product list).

That image is a TIFF image, which is *not* supported anymore by nopCommerce, since SkiaSharp doesn't support TIFF images.
The image is ~46Mb (46420516 bytes in the db) - and I've conclusively verified that

        public virtual async Task<PictureBinary> GetPictureBinaryByPictureIdAsync(int pictureId)
        {
            return await _pictureBinaryRepository.Table
                .FirstOrDefaultAsync(pb => pb.PictureId == pictureId);
        }

*never* returns for this image and CPU spikes to > 95%. Deleting that image manually from the DB (product_picture_mapping, PictureBinary and Picture) resolves the issue for me

Just a note - nopCommerce suggested that this would be better served as a custom plugin - since they feel the current solution works fine (it doesn't hehe), so I created one.

For reference: https://github.com/biggik/Nop.Plugin.Services.BucketPictureService

Thanks for contributing.  Although I do have question / concern about your solution because you said:
a) "Note this issue is not in 4.60.5, at least I cannot reproduce it on my test system"
b) "The issue - for me - was a single image ... That image is a TIFF image..."
Yet your solution (two-level hierarchy for folders) seems to deal with the Windows File System performance issues when there are many files in a single folder.

You are right

If I may, I would like to strike out the following original conclusion
      Note this issue is not in 4.60.5

since that is most likely not correct - the image was added around the time I switched.

I would also like to add that I have not experienced the CPU issue since I switched to the BucketPictureService - it behaves much better on my system