How to edit robots.txt

Hi,

I'm new here, therefore if I'm posting something that already exists, please, let me know.

I've added a sitemap.xml on my website a few days ago and I realized through Google Search Console that robots.txt has been blocking many webpages and I have no idea how to access the file and edit it.



I'm afraid of customers trying to go to "Cart" page or checkout and be blocked.

Could someone help me please?

Thanks a million!

Regards,

otaviodias wrote:

Hi,

Robots.txt does not block access of the pages to visitors. It just blocks them to the crawlers. So your customers can surely access Cart or Checkout pages. Don't worry about that.

Also, robots.txt is generated by application. And simplest way to modify it is to upload the custom robots.txt in root web folder so the one generated by application will be overwritten by the one you uploaded.

Hope it helps. Appreciate Upvote if it helps.

Thanks,

Krunal

Hi,

Thanks a million for your answer,

I get you, but, it's another point, I know even how to edit a robots.txt and where I find "roots folder", but I don't have access, who is my domain is Nop-commerce

If you do not have access then probably you can not change that. :(

Sorry, I don't understand, I am not able to access it, but nop-commerce who is my host is able to access, wasn't their responsibility to do it?

otaviodias wrote:

Do you have FTP access?
If so create a txt file named "robots.additions.txt".
Add additional robots.txt directives: https://moz.com/learn/seo/robotstxt
Save and upload the robots.additions.txt file to the root directory of you website.
Most importantly, check your work. A mistake could issues with crawlability and indexability. Now in regards to the existing (hardcoded) directives you see in the robots.txt file. Those are ok. The cart doesn't need to be and shouldn't be indexed. You don't want a search engine returning a search result for an empty cart right? Only pages of value...
https://support.google.com/webmasters/answer/6062598?hl=en&ref_topic=6061961

You most likely could ask your host to add the file to the root directory as well.