Serious IIS Application Pool recycling bug/design flaw

If you've got a NopCommerce 4.0 store in production right now you may be periodically encountering the following error for no apparent reason -

An error occurred while starting the application.

If you have stdoutLogEnabled="true" in your web.config file you'll find an error like this one -

Application startup exception: System.Exception: Plugin 'Must be assigned to customer role'. Access to the path 'C:\inetpub\wwwroot\Plugins\bin\Nop.Plugin.DiscountRules.CustomerRoles.dll' is denied.

If you've encountered this error like I have, you probably thought it was some temporary glitch and simply restarted your website and forgot about it.

However this is not some temporary glitch, this is something that will occur EVERY TIME your IIS Application pool is recycled.

When the application pool is recycled IIS does something called an "Overlapped recycle" where it loads the application in a fresh application pool and then kills the old application pool when the new one is loaded. It does this so that users browsing you site don't have to wait for the application to load.

Why is this a problem? Well Nopcommerce does something a little weird during where it copies all plugin bin files into a shadow directory at /Plugins/bin. Because the original application pool is still running it has a lock on the files inside the shadow directory, so the new application pool is unable to delete/overwrite them.

This is a serious problem with NopCommerce which every user is going to encounter.

As a temporary fix I recommend disabling overlapped recycle though the application pool advanced settings -



With overlapped recycle disabled it means that when your application pool is recycled your customers are going to have to wait for the site to reload. You can disable recycling completely but this is not recommended. The default setting is to have a recycle occur every 1740 minutes, but what I've done is change my recycling settings to have the recycle occur at a specific time of the day when there are not likely to be many people using it (5am).

Thank you for this, I was just debugging when this post came in. Great timing.

What is the value of your "UseUnsafeLoadAssembly" parameter in your /appsettings.json file?

it's set to True

I also faced the same error when installing a plugin (hosted on Azure app service). Please have a look at this serious problem.

Your problem is unrelated [email protected]. Use stdout log to find out what your error was and quit hijacking my thread.

Andrei, have you had the time to look at this? Any more information I can provide?

Hi Tim,

We don't have any ideas now. How often does it happen? After each pool restart? What hosting are you using?

After every pool restart. I'm hosting on a VPS.

No Timmit I think I also faced the same problems.
My website is hosted on an Azure app so I don't know when the IIS pool is recycled. But sometime it automatically showed the same error and I have to restart the App Service to make it work again. (I will try to set the stdout log but have to wait some days)

I also saw that the Nop Partner demo site has the same problem here?
http://nopcommerce.on-everleap.com/

https://imgur.com/a/sglpn

Today the problem occur again on my site (using Azure app). Here is the stdout log :

Application startup exception: System.Exception: Plugin 'Seven Spikes Core'. Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied.
Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied.
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
 ---> System.Exception: Plugin 'Seven Spikes Core'. Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied. ---> System.IO.FileLoadException: Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied. ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
   --- End of inner exception stack trace ---
   at System.Reflection.AssemblyName.nGetFileInformation(String s)
   at System.Reflection.AssemblyName.GetAssemblyName(String assemblyFile)
   at Nop.Core.Plugins.PluginManager.PerformFileDeploy(FileInfo plug, ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 201
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 417
   --- End of inner exception stack trace ---
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 453
   --- End of inner exception stack trace ---
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 464
   at Nop.Core.Infrastructure.NopEngine.Initialize(IServiceCollection services) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Infrastructure\NopEngine.cs:line 154
   at Nop.Web.Framework.Infrastructure.Extensions.ServiceCollectionExtensions.ConfigureApplicationServices(IServiceCollection services, IConfigurationRoot configuration) in D:\projects\Main_Repository\shop\shop_heykorean\Presentation\Nop.Web.Framework\Infrastructure\Extensions\ServiceCollectionExtensions.cs:line 49
   at Nop.Web.Startup.ConfigureServices(IServiceCollection services) in D:\projects\Main_Repository\shop\shop_heykorean\Presentation\Nop.Web\Startup.cs:line 44
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
System.Exception: Plugin 'Seven Spikes Core'. Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied.
Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied.
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
 ---> System.Exception: Plugin 'Seven Spikes Core'. Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied. ---> System.IO.FileLoadException: Could not load file or assembly 'SevenSpikes.Nop.Plugins.Core.dll' or one of its dependencies. Access is denied. ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
   --- End of inner exception stack trace ---
   at System.Reflection.AssemblyName.nGetFileInformation(String s)
   at System.Reflection.AssemblyName.GetAssemblyName(String assemblyFile)
   at Nop.Core.Plugins.PluginManager.PerformFileDeploy(FileInfo plug, ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 201
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 417
   --- End of inner exception stack trace ---
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 453
   --- End of inner exception stack trace ---
   at Nop.Core.Plugins.PluginManager.Initialize(ApplicationPartManager applicationPartManager, NopConfig config) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Plugins\PluginManager.cs:line 464
   at Nop.Core.Infrastructure.NopEngine.Initialize(IServiceCollection services) in D:\projects\Main_Repository\shop\shop_heykorean\Libraries\Nop.Core\Infrastructure\NopEngine.cs:line 154
   at Nop.Web.Framework.Infrastructure.Extensions.ServiceCollectionExtensions.ConfigureApplicationServices(IServiceCollection services, IConfigurationRoot configuration) in D:\projects\Main_Repository\shop\shop_heykorean\Presentation\Nop.Web.Framework\Infrastructure\Extensions\ServiceCollectionExtensions.cs:line 49
   at Nop.Web.Startup.ConfigureServices(IServiceCollection services) in D:\projects\Main_Repository\shop\shop_heykorean\Presentation\Nop.Web\Startup.cs:line 44
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
Hosting environment: Production
critContent root path: D:\home\site\wwwroot
: Microsoft.AspNetCore.Hosting.Internal.WebHost[11]
      Hosting startup assembly exception
System.InvalidOperationException: Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   --- End of inner exception stack trace ---

Hello Andrei,
could you please look at this and suggest me some temporary solution for now ? Can I copy the plugin dll files manually to /Plugins/bin and disable shadow copy in PluginManager ?