Keep Alive and other Tasks not running (403)

Hello everyone.  

I have reviewed every forum post on the topic of Keep Alive and other Automated Tasks not running and it seems this has been a persistent issue for many years that has never really been properly addressed or resolved with a scattering of users either changing the store URL or modifying the source code themselves to work around it which is not really the best solution unless from the developer of the platform.  Sadly it is my turn to experience this issue and we require assistance as I am out of ideas.  Willing to Screen-share the server if anyone has an idea or needs to see the configuration (pm me)

We are working on a new customer NOP store with this issue being the primary problem keeping us from going live with this site.  I am the Web Host that is providing the environment and we are working in conjunction with a nopCommerce Developer to bring this site to life.  Developer has spent many hours on this as have I so we are now reaching out to the Forum.  Here is what I have

Windows 2016 Server
NopCommerce 4.50.3
URL equivalent:  nopstage.customerstore.com  (obscured real name of site)
HTTPS enabled:  Yes
Store URL:  https://nopstage.customerstore.com/

Symptom:  None of the Automated Tasks are running but most will run manually except for the Keep Alive task which gives us a 403 error.

Steps taken so far with no success:
1) many variations of the Store URL with and without https as well as with and without the trailing "/".  Also tried with and without SSL enabled
2) removed all Request Filters and other URL Rewrite rules in IIS that could be passing the response to a 403 error
3) Since it is in development currently, we allowed FULL permissions to the entire domain for all users.
4) entered in a hard-coded HOSTS file record for the IP to Domain
5) Verified the website is accessible from the server's browser and is ping-able.
6) tried white-listing the server and website IP address.
7)  verified that going to https://nopstage.customerstore.com/keepalive shows me "I am alive!" in the browser (obscured actual domain name)
8) verified that going to https://nopstage.customerstore.com/keepalive/index displays a blank page.
9)  NOTE that everything else works fine with the site so I am not concerned with any server prerequisites not being met as if that were the case, nothing would work.

Not sure if this will help as many have posted similar log results but here is the log entry for the Keep-Alive error we are experiencing  (I changed store name and actual domain name)

Short message
The "Keep alive" scheduled task failed with the "One or more errors occurred. (Response status code does not indicate success: 403 (Forbidden: Access is denied.).)" error (Task type: "Nop.Services.Common.KeepAliveTask, Nop.Services". Store name: "Customer Store". Task run address: "https://nopstage.customerstore.com/scheduletask/runtask").

Full message
System.AggregateException: One or more errors occurred. (Response status code does not indicate success: 403 (Forbidden: Access is denied.).)
---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden: Access is denied.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at Nop.Services.Common.StoreHttpClient.KeepAliveAsync()
   at Nop.Services.Common.KeepAliveTask.ExecuteAsync()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Nop.Services.ScheduleTasks.ScheduleTaskRunner.ExecuteTask(ScheduleTask scheduleTask)
   at Nop.Services.ScheduleTasks.ScheduleTaskRunner.<>c__DisplayClass8_0.<ExecuteAsync>b__0()
   at Nop.Core.Caching.MemoryCacheManager.PerformActionWithLock(String key, TimeSpan expirationTime, Action action)
   at Nop.Services.ScheduleTasks.ScheduleTaskRunner.ExecuteAsync(ScheduleTask scheduleTask, Boolean forceRun, Boolean throwException, Boolean ensureRunOncePerPeriod)


Any help with some new ideas would be most welcome.  As I mentioned, I am available to screenshare if you PM me.

- Mark

Ensure your server can find the host nopstage.customerstore.com and it has an IP address which is correct
i.e. run the cmd command on the server and ping nopstage.customerstore.com

Done ... that was my #5 on the "steps taken so far".

5) Verified the website is accessible from the server's browser and is ping-able.

Also, we input a server HOSTS file entry to ensure that subdomain points to the correct IP even though that was not really necessary (a just-in-case measure)

Can you think of anything perhaps server-side that might be preventing ASP.NET/nopCommerce tasks from running?  I am not infallible but I am fairly confident I have tried all the suggestions from other posts on this topic but I could have missed something.

Can you please ensure that the store within your nopCommerce is properly configured? For example, it must have an URL with HTTPS if you have switched on SSL and the hosts value should have something. If that is okay, then please check with stdlogs once.

Best regards,
Atul

RE: "Keep Alive and other Automated Tasks not running "
Are any of your tasks running OK?

nopAdvance.com wrote:

----- -----
Atul,
I mentioned the configuration attempts in my #1) of my "Steps taken so far with no success:" in my original post:  "1) many variations of the Store URL with and without https as well as with and without the trailing "/".  Also tried with and without SSL enabled

Configuration is good as far as I can tell.  This is why I tried just about every variation of the Store URL.

The stdout logs, at least to me, are not very helpful as that just list the issue and tell me it is a 403 error but offers no clue as to the root cause.  Was there something specific you would like me to share with you on the logs?  

New York wrote:

----- -----
New York,

None of the tasks are running automatically but most like cache, sendmail, etc will run manually.  The only one that gives me a hard on screen error on manual run is the KeepAliveTask so that is the one I am focusing on as it is the most verbose in the logs (other others give no detail in the logs as they are just automation errors and all the same).

What do you see if you enter  yourdomain.com/keepalive in your browser address bar?
https://demo.nopcommerce.com/keepalive

New York wrote:

----- -----
New York,

This was referenced in #7 and 8 from my original post:
7)  verified that going to https://nopstage.customerstore.com/keepalive shows me "I am alive!" in the browser (obscured actual domain name)
8) verified that going to https://nopstage.customerstore.com/keepalive/index displays a blank page.

Grasping at straws territory now :(
System does a HTTP post to the addresses for Schedule Tasks
You are getting 403 error
The 403 Forbidden error is an HTTP status code which means that accessing the page or resource you were trying to reach is absolutely forbidden for some reason.
Maybe something in the web.config customHeaders
Assuming there is no defaults values that will be activated when removing the current settings - I dont know
But as a test you could try removing all