Google has started indexing pages even they are disallow in robots.txt.
Especially "changetaxtype" causes problems.
We have activated the changetax type so that our customers can displaying prices with or without VAT.
When google starts crawling the pages without VAT, the prices no longer match to google shopping ads.
The Google merchant center sees a different price on the "page witout VAT" page and use this price at our shopping ads, which is not allowed.
These ads must include VAT.
If we in the google merchant center turn off automatic price correction, the ads will be deactivated "invalid" because the price does not match the crawled paged.
Picture from google search console:
Example url where the price is incl. VAT:
https://www.elsag.dk/changetaxtype/0?returnUrl=%2Fcee-stikprop-16a-3pol-n-j-400v-ip44
Example url where the price is excl. VAT:
https://www.elsag.dk/changetaxtype/10?returnUrl=%2Fcee-stikprop-16a-3pol-n-j-400v-ip44
Does anyone have an idea how to fix this ?
It has not been a problem in the past because google accepted that it should not crawl Disallow url.
In the guide at google it is written they will not guarantee that disallow url will not be crawled.
Google refer to "NO index" which can not be used in this case.
Br.
Jens