I have recently upgraded to V4.0 from 3.7, however since then my DB allocated DTU usage is constantly at 100% (it is running on Azure). I have doubled the DTU amount from 5 to 10, and although better it is still suffering 100% usage most of the time. I am also getting a lot of Execution Timeout Expired errors in the logs relating to builtin@search_engine_record.com trying to access urls like https://www.myewebsite.com/service-items?specs=1%2C25 (which seems an illegal url parameter anyway).
I have changed:
<a href="@spec.FilterUrl">to
<a href="@spec.FilterUrl" rel="nofollow">in _FilterSpecsBox.cshtml but this doesnt seem to have helped much...
I have also seen a fix for 3.8 which recommends setting the following in Web.Config:
<rewrite>
<rules>
<rule name="SpecsSpiderCrawl" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{QUERY_STRING}" pattern="specs=.*" />
<add input="{HTTP_USER_AGENT}" pattern="googlebot|bingbot|Baiduspider|Yandex|YoudaoBot|sogou|NaverBot|moget|ichiro" />
</conditions>
<action type="Redirect" url="http://{HTTP_HOST}/{R:0}" appendQueryString="false" redirectType="Permanent" />
</rule>
</rules>
</rewrite>
However as 4.0 is running on ASP.Net Core, this doesnt seem to work and according to https://docs.microsoft.com/en-us/aspnet/core/fundamentals/url-rewriting?view=aspnetcore-2.1&tabs=aspnetcore2x it is not done in web.config anymore...
Does anyone have an example to fix the problem?
P.S. Azure is also recommending some changes to indexes to increase performance, but I have resisted as I dont want to screw it up.