Noindex and Nofollow

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.
1 year ago
hi friends,
i got nopCommerce 4.4 with many products on it, today google crawled it but index many linked from plugins or login page,
what's best solution for setting Noindex or Nofollow for these url ?
thanks
1 year ago
Add robots.custom.txt to your root folder of nopCommerce project and add disallow paths  which need to no follow
Or you can add your disallow paths to the source code of nopcommerce here

Helpful link 1.https://www.nopcommerce.com/en/boards/topic/73298/how-to-edit-robotstxt
2.https://medium.com/@esty_c/robots-txt-in-asp-net-core-36007684ea02
1 year ago
Thanks you Rashed Khan,

1- as i checked the Search Console i got many link from Login, Register, Attachment, Review and ...
i write a sample Robots.txt and send the original link from search console, thanks to check it's true or not ? (the attachment url is for image, i want to index it for google image)
2- as i search, i found that there is robot.txt and robot.Custom.txt and robot.addition.txt ,
so the question is i should use which of them and what's use of every one ?
3- the last question is that with disallowing on robot.txt, how can we use follow tag on this robot.txt ? (i means i want Noindex with Follow)

Thank you
Hamid

Indexed Link:
https://farshstore.com/register?returnUrl=/contactus-2
https://farshstore.com/register?returnUrl=/shipping-delivery
https://farshstore.com/register?returnUrl=/conditions-of-use
https://farshstore.com/register?returnUrl=/
https://farshstore.com/register?returnUrl=/?attachment_id%3D14953
https://farshstore.com/productreviews/49
https://farshstore.com/productreviews/61
https://farshstore.com/productreviews/62
https://farshstore.com/login?returnUrl=/
https://farshstore.com/login?returnUrl=/filterSearch
https://farshstore.com/?attachment_id=14575
https://farshstore.com/?attachment_id=14552
https://farshstore.com/?attachment_id=14455

My robot.txt for that url:
User-agent: *
Disallow: /productreviews/
Disallow: /register?returnUrl=/
Disallow: /login?returnUrl=/
Disallow: /product-tag/
Disallow: /?attachment_id=*

User-agent: Googlebot-Image
Allow: /?attachment_id=*

Sitemap: https://farshstore.com/sitemap.xml
1 year ago
h.jalalishad wrote:

1- as i checked the Search Console i got many link from Login, Register, Attachment, Review and ...
i write a sample Robots.txt and send the original link from search console, thanks to check it's true or not ? (the attachment url is for image, i want to index it for google image)

you can check those link for better idea
-https://developers.google.com/search/docs/advanced/robots/create-robots-txt
-https://moz.com/learn/seo/robotstxt
h.jalalishad wrote:

2- as i search, i found that there is robot.txt and robot.Custom.txt and robot.addition.txt ,
so the question is i should use which of them and what's use of every one ?

where do you find robot.addition.txt? nopCommerce use robot.custom.txt

h.jalalishad wrote:

3- the last question is that with disallowing on robot.txt, how can we use follow tag on this robot.txt ? (i means i want Noindex with Follow)

Noindex: /example-page-1/
reference https://www.deepcrawl.com/blog/best-practice/noindex-disallow-nofollow/
1 year ago
Thank you Rashed for your time,

as you ask i find the robot.addition.txt here:
https://www.nopcommerce.com/en/boards/topic/40973/add-noindex-tag-in-a-specific-category

as i checked the referenced, in robots.txt just Disallow and Allow will be work as function, so how can control the follow/Nofollow and Noindexing function for any page or directory? (against about html Tag, cause i haven't access to site developer and I'm site manager)

i see that there is a Google Tools for testing robots.txt here:
https://www.google.com/webmasters/tools/robots-testing-tool
but when i got there with owner account, it show that I haven't any property! (i verify my account owner with Domain option on DNS setting)

in the DeepCrawl knowledge, it write "Google doesn’t officially support robots.txt noindex, and you shouldn’t rely on it because although it works today, it may not do so tomorrow"
so what's google doing now?
should i Noindexing with disallow my private folder or customer info page ?

Thanks
Hamid
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.