Fig 1: Look at AWS cloud watch. CPU usages, web traffic significantly reduced.
Fig 2: Server response time incredibly improved - Newrelic.
You can write a simple trigger to catch unwanted bots ip and block it.
Create table to track all search engine ips
CREATE TABLE [dbo].[x_SearchEngine](
[Id] [int] IDENTITY(1,1) NOT NULL,
[LastIpAddress] [nvarchar](50) NULL,
[CreatedOnUtc] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Create trigger to insert ips into previously created table
CREATE TRIGGER [dbo].[x_searchengine]
SET NOCOUNT ON;
IF EXISTS(SELECT 1 FROM inserted where Id = 42693958)
insert into x_SearchEngine
select LastIpAddress, GETUTCDATE() from inserted where id = 42693958
Wait few hours and run following query and find bots who is killing your server
select lastipaddress, count(*)
group by lastipaddress
order by count(*) desc
Find country from ip using iplocation.net
and see if the country/ip is relevant for your business otherwise block it.
You can also visit my Quora post for more details