Adding disallowed items to Robots Text file via (commonController)

Hi Guys

I am hoping someone knows the anwser to this, I understand the robots text file is now auto generated, which is great news, but I want to disallow both "/productemailafriend" and "/producttags" in the robots text file, as this site has 23,000 products and is going to be a nightmare to sort out, I have gone into the CommonController and added both of them into the list, I build the solution and then open the text file in the browser but they still do not show up in the robots.txt file, am I missing something? do I need to do something else?.

My apologies after a bit more investigation I found that I had edited the wrong site, which is why I could not see it.

And yes I now feel very stupid.

Hi,

I also would like to see this as a default setting in NOP and a file to change it. My code is here:

public ActionResult RobotsTextFile()
        {
...
            //Load robots.txt additions to end of file.
            //Or even load complete file regardless of this file
            string pathToFiles = Server.MapPath("~/Content/");
            string robotsFile = System.IO.Path.Combine(pathToFiles, "Robots.txt");
            if (System.IO.File.Exists(robotsFile))
            {
                string robotsFileContent = System.IO.File.ReadAllText(robotsFile);
                sb.Append(robotsFileContent);
            }
...
            Response.ContentType = "text/plain";
            Response.Write(sb.ToString());
            return null;
        }

}

I hope it will become a work item and will be integrated so I do not have to fix it on every upgrade.

J.

Quantis001 wrote:
Sure, thanks. Please vote here

how to edit  robots.txt file   in  nopcommerce 3.4  ?

I need:

1.  add  "/"  before   "ru/...."  in file because https://webmaster.yandex.ru/robots.xm write error for parameters without "/"
2.  add " Crawl-delay"  parameter  


please help

Finally done. Please see changeset 68cb714bb2a3 (it contains some description on how to use it)

a.m. wrote:
Hi.
... in some description "...
1. Create and add robots.additions.txt to the root of the site. "
but in source "...
848               string robotsFile = System.IO.Path.Combine(_webHelper.MapPath("~/"), "robots.custom.txt"); "
... for certain reasons it's time to correct it.
Best Regards.

To override robots.txt (fixes /.well-known/* apple app lookup by google bot)
1. Open your http://yoursite.extension/robots.txt
2. Copy all the contents.
3. Create a file called robots.custom.txt*
5. Paste the contents from robots.txt to robots.custom.txt
6. Add Disallow: /.well-known/* to robots.custom.txt...save.
7. Copy robots.custom.txt back up to the root of the website.
8. Open http://yoursite.yourextension/robots.txt NOT robots.custom.txt ...verify contents.

*NOTE: There is a file in the root of the website called robots.additions.txt (this file does not work).