I hope I'm writing in the right session, I noticed a big problem that our site that runs with nopcommerce generates, the problem is that it generates product TAGs and creates links that are indexed on Google with all the relevant problems.
It's from an AJAX call (to dynamically load products matching tag). The 'fetchUrl' response does return HTML, so it would not normally be blocked by Google robot / web crawler.
I believe the format would be /tag/products?tagId=*
(FYI, Technically speaking... The fetchUrl is set up in ViewData here \Views\Catalog\ProductsByTag.cshtml to be used in here \Views\Catalog\_CatalogProducts.cshtml ;)
It's from an AJAX call (to dynamically load products matching tag). The 'fetchUrl' response does return HTML, so it would not normally be blocked by Google robot / web crawler.
I believe the format would be /tag/products?tagId=*
(FYI, Technically speaking... The fetchUrl is set up in VieewData here \Views\Catalog\ProductsByTag.cshtml to be used in here \Views\Catalog\_CatalogProducts.cshtml ;)
Thanks for your reply, I'll try to set the site not to generate the link with the TAG.
However, it didn't yield the desired outcome. The problem lies in the fact that the page at https://www.illuminoshop.com/tag/products?tagId=21900 is being called from a partial view named "_ProductsInGridOrLines". Adding the code directly into this partial view had no effect.
Upon further investigation, I was able to pinpoint the source of the page generation. It stems from the endpoint defined in the code:
Hi Please don't comment out or remove the route. It will cause issues There are a few things here 1. If you want the tag but don't want tag pages to be indexed by Google or search engines then add Html.AddHeadCustomParts("<meta name=\"robots\" content=\"noindex\">"); code on top of the ProductsByTag.cshtml page. (I still don't see it being applied on the website link you have provided. If you want this implemented you need to add it on the active theme's view page) 2. If you don't want to show the tags at the left panel of the catalog page then go to /Admin/Setting/Catalog and set the Number of product tags (cloud) value to 0 under the Tag section 3. If you don't want products associated with the tags then remove the product and tags mapping from the product page at the admin. 4. If you want to keep the mapping of the tag and products but don't want it to be displayed at the product detail page then you can comment out or remove the @await Html.PartialAsync("_ProductTags", Model.ProductTags) from the ProductTemplate.Simple.cshtml view file. 5. If you don't want the product tags at all then you can simply remove all the tags from the tag page at the admin
The problem of generating links from TAG has not been completely resolved, the site continues to generate links from TAGs, for example: https://www.illuminoshop.com/drink https://www.illuminoshop.com/regolabile
...
The problem lies in the fact that the page at https://www.illuminoshop.com/tag/products?tagId=21900 is being called from a partial view named "_ProductsInGridOrLines".
They are not the same thing. The latter should be handled by the above suggested " robots.txt disallow the path ... /tag/products?tagId=*" The former should be handled by above mentioned " ... noindex meta on the tag page ... code on top of the ProductsByTag.cshtml page"
In either case, don't expect Google to see the changes immediately. Even if you request a recrawl of a robots.txt file, it could take several hours for changes to be seen.