I am hoping someone knows the anwser to this, I understand the robots text file is now auto generated, which is great news, but I want to disallow both "/productemailafriend" and "/producttags" in the robots text file, as this site has 23,000 products and is going to be a nightmare to sort out, I have gone into the CommonController and added both of them into the list, I build the solution and then open the text file in the browser but they still do not show up in the robots.txt file, am I missing something? do I need to do something else?.
My apologies after a bit more investigation I found that I had edited the wrong site, which is why I could not see it.
1. add "/" before "ru/...." in file because https://webmaster.yandex.ru/robots.xm write error for parameters without "/" 2. add " Crawl-delay" parameter
.... changeset 68cb714bb2a3 (it contains some description on how to use it)
Hi. ... in some description "... 1. Create and add robots.additions.txt to the root of the site. " but in source "... 848 string robotsFile = System.IO.Path.Combine(_webHelper.MapPath("~/"), "robots.custom.txt"); " ... for certain reasons it's time to correct it. Best Regards.
To override robots.txt (fixes /.well-known/* apple app lookup by google bot) 1. Open your http://yoursite.extension/robots.txt 2. Copy all the contents. 3. Create a file called robots.custom.txt* 5. Paste the contents from robots.txt to robots.custom.txt 6. Add Disallow: /.well-known/* to robots.custom.txt...save. 7. Copy robots.custom.txt back up to the root of the website. 8. Open http://yoursite.yourextension/robots.txt NOT robots.custom.txt ...verify contents.
*NOTE: There is a file in the root of the website called robots.additions.txt (this file does not work).
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.