I have checked the post in the link. what i understood is to create robots.custom.txt file and placed in the root folder.
I am confused why i should create robots.custom.txt when there is already robots.txt file according to this link in our website : www.cleanbeautykw.com/robots.txt
I cannot find this file in the server!!! Can you please have look to our robots.txt file and see if it okay or need update and advise me how to edit it?
I have checked the post in the link. what i understood is to create robots.custom.txt file and placed in the root folder.
I am confused why i should create robots.custom.txt when there is already robots.txt file according to this link in our website : www.cleanbeautykw.com/robots.txt
I cannot find this file in the server!!! Can you please have look to our robots.txt file and see if it okay or need update and advise me how to edit it?
I have checked the post in the link. what i understood is to create robots.custom.txt file and placed in the root folder.
I am confused why i should create robots.custom.txt when there is already robots.txt file according to this link in our website : www.cleanbeautykw.com/robots.txt
I cannot find this file in the server!!! Can you please have look to our robots.txt file and see if it okay or need update and advise me how to edit it?
Thank you
As you are using nopCommerce4.6.x and at this version, the robot.txt is moved to the wwwroot, previously it was in the root directory. nopCommerce4.6.x is by default disallowed some paths from the bot. You can check it from the General Setting(https://admin-demo.nopcommerce.com/Admin/Setting/GeneralCommon ). The robots.custom.txt will only be needed when you want to disallow more paths and set other custom rules. If you are developer then can check the Nop.Web==>Factories==>CommonModelFactory==>PrepareRobotsTextFileAsync and Nop.Web==>Areas==>Admin==>Factories==>PrepareRobotsTxtSettingsModelAsync methods.
robots.custom.txt is only needed when storeowner want to set their custom rules