Site breaks when loading big categories

I have a few main categories that have 40 k and even 50k items. When some one tries to go into the root directory it might work 3/10 times but other times it just breakd:

IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..

https://gyazo.com/77853364d66cc689a1992d2ecfedeba6

How to cope with this?

Well, I see "NopAjaxFilter..." in your error message, so I suggest you test it without using the 3rd party plugin(s).  Or, open a support ticket with the plugin vendor.

Yeah disabling it the site actually works way better !