How to edit robots.txt

7 months ago
Hi,

I'm new here, therefore if I'm posting something that already exists, please, let me know.

I've added a sitemap.xml on my website a few days ago and I realized through Google Search Console that robots.txt has been blocking many webpages and I have no idea how to access the file and edit it.



I'm afraid of customers trying to go to "Cart" page or checkout and be blocked.

Could someone help me please?

Thanks a million!

Regards,
7 months ago
otaviodias wrote:
Hi,

I'm new here, therefore if I'm posting something that already exists, please, let me know.

I've added a sitemap.xml on my website a few days ago and I realized through Google Search Console that robots.txt has been blocking many webpages and I have no idea how to access the file and edit it.



I'm afraid of customers trying to go to "Cart" page or checkout and be blocked.

Could someone help me please?

Thanks a million!

Regards,


Hi,

Robots.txt does not block access of the pages to visitors. It just blocks them to the crawlers. So your customers can surely access Cart or Checkout pages. Don't worry about that.

Also, robots.txt is generated by application. And simplest way to modify it is to upload the custom robots.txt in root web folder so the one generated by application will be overwritten by the one you uploaded.

Hope it helps. Appreciate Upvote if it helps.

Thanks,

Krunal
7 months ago
Hi,

Thanks a million for your answer,

I get you, but, it's another point, I know even how to edit a robots.txt and where I find "roots folder", but I don't have access, who is my domain is Nop-commerce
7 months ago
If you do not have access then probably you can not change that. :(
7 months ago
Sorry, I don't understand, I am not able to access it, but nop-commerce who is my host is able to access, wasn't their responsibility to do it?
3 months ago
otaviodias wrote:
Sorry, I don't understand, I am not able to access it, but nop-commerce who is my host is able to access, wasn't their responsibility to do it?


Do you have FTP access?
If so create a txt file named "robots.additions.txt".
Add additional robots.txt directives: https://moz.com/learn/seo/robotstxt
Save and upload the robots.additions.txt file to the root directory of you website.
Most importantly, check your work. A mistake could issues with crawlability and indexability. Now in regards to the existing (hardcoded) directives you see in the robots.txt file. Those are ok. The cart doesn't need to be and shouldn't be indexed. You don't want a search engine returning a search result for an empty cart right? Only pages of value...
https://support.google.com/webmasters/answer/6062598?hl=en&ref_topic=6061961

You most likely could ask your host to add the file to the root directory as well.