Improve Send Mass Email

a.m. wrote:
Why is that?  How is it different from using an ESP?

I asked a similar question a few minutes ago.  Here is my scenario and why a 'MailChimp' solution is not always feasible.

I have modified nopCommerce to allow a client who has purchased leads from a third party to upload those leads into the system, generate a customer(and everything that goes with it), a product recommendation, and finally a 'Confirmation/Welcome/Registration' type email.

These leads come in large batches and typically the customer(of each lead) is expecting some sort of feedback/product recommendation right away.

It is not really feasible for an admin to have to export all new customers and upload them (in realtime) to MailChimp just to generate a simple email.

I totally understand that this is not in the scope of most store owners needs and is likely not a high priority for the nop team.  However, I believe it is a worthwhile endeavor to try and come up with a solution.

To me the most logical place to look at is when a Scheduled task is running.  Is it possible to pause/delay the same scheduled task if it is already running?  I have already added some customizations to deal with the number of tries to send the email and subsequent logs.  The hard part is eliminating duplicate tasks running.

I look forward to any comments suggestions anyone has and I will post anything I come up with.

t

SWW wrote:
I don't know why, but I know that it happens. The same situation was with nopCommerce official site when I sent emails using [email protected]. Some ISP just marked my emails as SPAM and added the email ([email protected] in this example) to a list of email which are used to send spam. I've also read that it happens with other sites. I had to personally contact some of them (such as AT&T) and ask to remove this email ([email protected]) from their blacklist. That's why I'm using MailChimp now. It's really good and useful (takes about 1 minutes to send out all emails, reports, etc)

"I don't know why, but I know that it happens."

Where you using BCC field for addressing?  I noticed in the code that you could fill your entire customer list inside of this field.  That almost always triggers a SPAM flag.

You have to send 1 email to each recipient.

joebloe wrote:

I'm not familiar with mailchimp but with the Solution I recommended earlier in the thread, they have an API to interface with. It is a simple matter to send email addresses via the api when for instance a customer registers or when a list is uploaded to Nop. I'd be surprised if mailchimp didn't have a similar API.

Email marketing suits have a much richer feature set than Nop and are likely to stay ahead of Nop in this respect. Nop is after all an eCommerce platform not a fully featured email marketing tool.

HTH

Dave

I completely agree with you Dave.  For my specific situation I haven't found (yet) MailChimp to support the 'real time' email support my client is looking for.  For regular email campaigns that are not as time sensitive my client fully expects to use MailChimp and is OK with that.

For anyone else who is interested, here is what I did to solve large batches of new customers imported into nopcommerce.  Note: when I say large I am talking about 10k to 20k at a time.

The thing to keep in mind as others have echoed is that sending email is not as straight forward as it seems.  It is very easy to get blacklisted and email delivery companies are better suited to handle all the quirks of this business.  I would strongly recommend reading up on best practices on how to avoid being blacklisted.  I incorporated some of my findings into my examples below.

1. I added a 'WasSent' flag to QueuedEmail
2. I added a message setting for maximum number of tries(it is hard coded at 3).
3. I added a SentEmail log. This is optional and for my situation necessary because of auditing needs by my client.
4. I added another catch block that logs SmtpException status codes. This will give you good insight into what SMTP servers are reporting back to you when or if they reject your email.  I only log these once the maximum number of attempts have been reached.
5. I added an IQueryable method for pulling the QueuedEmails into memory for sending. My method looks like so:

        public IQueryable<QueuedEmail> GetReadyToSendEmails()
        {
            var now = DateTime.Now;
            var query = from qe in _queuedEmailRepository.Table
                        where !qe.WasSent && now >=
                        (EntityFunctions.AddMinutes(qe.CreationDate, _messageSettings.ResendInterval * qe.SentTries))      
                        select qe;

            return query;
        }

6. You can see that I added a resend interval.  I have this set at 15 minutes.  In other words, if my sent tries are set to 4, and the first attempt bounces, then the email will be sent at 15 min, 30 min, 60 min, and finally deleted.
7. Finally, in the Execute portion of Sending Queued Emails I set a timestamp of NOW+45sec before the foreach loop begins, pull only a set number of records (500-1000 or however many you think you can send in 45 seconds), and check the time before continuing in the loop.  If the foreach loop is greater than 45 seconds, then break.  This gives you a 15 second buffer before the SendEmail task executes again.

Because of the WasSent flag we only pull what made it in our 60 second send window.  Couple this with a DeleteSentEmails task that goes off every 60 seconds as well and we pretty much maintain the data integrity we need to ensure duplicate emails are not sent, and that a reasonable resend strategy is in place.

Lastly, I haven't been able to test this for very long so I am sure some unexpected problems will pop up but I can at least get a better idea about what might be the problem and continue to use nopcommerce for larger scale customer registration.

Hope this helpful.

joebloe wrote:
No

joebloe wrote:
Could you please share your changes?

BTW, QueuedEmail already has SentOnUtc property. It's nullable. You can use it in order to know whether an email was already sent. So there's no any need to add a new WasSent flag.

Ok.  Here is what I have.  Items 1 & 2 are pretty self-explanatory or optional as Andrei noted in this previous post.

NOTE: I changed a few things in Nop.Services.Messages.  The only reason was just personal preference and some naming conventions.  The basic ideas are still the same.

#3- Sent Email Log
Id,
CustomerId,
EmailTemplateTypeId,
SentTries,
DateSent

Explanation: Pretty straight forward.  Because my client required auditing of every email sent I decided to set up a separate log that was as flat and basic as possible while still giving me access to just about everything else through a separate join.  EmailTemplateId is an enum value that I added to categorize the various message templates.  I customized the email template create/update page to include Html and text versions of the email (good practice and a known anti-blacklist tactic).  I split the page in 2 columns and put the list of tokens applicable to the EmailTemplateTypeId selected in that area.  I used an ajax call to load the tokens and a little jQuery to load each selected token into the HTML or Text version of the email template at the cursor point. e.g. When you click a token it inserts itself into where the cursor/caret is.

#4 - Additional Error handling

        public void SendQueuedEmail()
        {
            var stopTime = DateTime.Now.AddSeconds(45);//probably switch this to a setting so it can be adjusted when needed
            var maxTries = _messageSettings.MaxNumberOfSendTries;
            var queuedEmails = _queuedEmailService.GetReadyToSendEmails().ToList();
            foreach (var queuedEmail in queuedEmails)
            {
                var now = DateTime.Now;
                if(now >= stopTime)
                    break;
                var emailAccount = queuedEmail.EmailAccount;
                if (queuedEmail.SentTries > maxTries)
                {
                    //log it
                    _logger.EmailMaxTries(string.Format("The maximum number of email tries has been met for '{0}'", queuedEmail.To),                       
                        queuedEmail.CustomerId);
                    //delete it
                    _queuedEmailService.DeleteQueuedEmail(queuedEmail);
                }
                else
                    SendEmail(emailAccount, queuedEmail);
            }
        }

This is similar to the Execute method of QueuedMessagesSendTask.  I put most of my 'non-CRUD' methods into Manager(This one is EmailManager) classes. Just a preference.  The only real difference is setting a 'timer' to send as many emails as possible before the SendQueuedEmails task executues again.

The addressing is the same for SendEmail. I made SendMail a private helper. Here are the differences.

private void SendEmail(EmailAccount emailAccount, QueuedEmail queuedEmail)
{
//addressing section same as nop
var isSuccessful = true;

            #region Send Message

            using (var smtpClient = new SmtpClient())
            {
                try
                {
                    smtpClient.UseDefaultCredentials = emailAccount.UseDefaultCredentials;
                    smtpClient.Host = emailAccount.Host;
                    smtpClient.Port = emailAccount.Port;
                    smtpClient.EnableSsl = emailAccount.EnableSsl;

                    smtpClient.Credentials = emailAccount.UseDefaultCredentials
                                                 ? CredentialCache.DefaultNetworkCredentials
                                                 : new NetworkCredential(emailAccount.Username, emailAccount.Password);

                    smtpClient.Send(message);
                }
                catch (SmtpException exc)
                {
                    queuedEmail.SentTries = queuedEmail.SentTries + 1;
                    if (queuedEmail.SentTries > _messageSettings.MaxNumberOfSendTries)
                    {
                        //log only the last attempt to get status code
                        _logger.Email(string.Format("Error sending email: '{0}' - Status Code: {1} - Message: {2}",
                                                    queuedEmail.To, exc.StatusCode, exc.Message),
                                      queuedEmail.CustomerId, exc.InnerException);
                    }
                    //keep trying
                    _queuedEmailService.UpdateQueuedEmail(queuedEmail);
                    isSuccessful = false;
                }
                catch (Exception exc)
                {
                    queuedEmail.SentTries = queuedEmail.SentTries + 1;
                    //log all general exceptions
                    _logger.Email(string.Format("General Exception when sending Email: '{0}' - Inner Exception: {1} - Message: {2}",
                                        queuedEmail.To, exc.InnerException, exc.Message), queuedEmail.CustomerId, exc.InnerException);
                                                  
                    //keep trying
                    _queuedEmailService.UpdateQueuedEmail(queuedEmail);
                    isSuccessful = false;
                }

                #endregion

                if (!isSuccessful)
                    return;

                queuedEmail.WasSent = true;
                queuedEmail.SentTries = queuedEmail.SentTries + 1;
                _queuedEmailService.UpdateQueuedEmail(queuedEmail);

                //log it
                var sentEmailLog = new SentEmailLog
                                       {
                                           CustomerId = queuedEmail.CustomerId,
                                           EmailTemplateTypeId = queuedEmail.EmailTemplateTypeId,
                                           SentTries = queuedEmail.SentTries,
                                           DateSent = DateTime.Now
                                       };
                _queuedEmailService.InsertSentEmailLog(sentEmailLog);
            }

Not a lot of difference here sans additional catch block and SentEmailLog.

#5- The method to pull the QueuedEmails - I'll post this again.

        public IQueryable<QueuedEmail> GetReadyToSendEmails()
        {
            var now = DateTime.Now;
            var query = from qe in _queuedEmailRepository.Table
                        where !qe.WasSent && now >=
                        (EntityFunctions.AddMinutes(qe.CreationDate, _messageSettings.ResendInterval * qe.SentTries))      
                        select qe;

            return query;
        }

EntityFunctions is part of System.Data.Objects and allows you to use linq with dates.

#6 My QueuedEmail object
EmailAccountId,
CustomerId,
EmailTemplateTypeId,
From,
FromName,
To,
ToName,
CC,
Bcc,
Subject,
HtmlBody,
TextBody,
CreationDate,
SentTries,
WasSent

NOTE: Admittedly I changed up quite a bit.  I believe I also changed how campaigns work and some of this may be reflected in QueuedEmail entity.  My client wanted custom reporting tied to campaigns. So what I did was create a saved report table that contains every conceivable property that makes up a customer graph. The report gets saved and is assigned to a campaign.  By adding a 1-click or Export to CSV button, the client can either queue up the customer emails or export them to a CSV list for usage with MailChimp, ConstantContact, or any other email provider.  He just didn't know which one he wanted to use so I skipped the plugin/api route.

Now that I look at how long this is, I hope I haven't confused anyone.  I guess I really did change a lot and would be happy to share/discuss with anyone who has a question.  Hope this helps.

t

Thanks

joebloe wrote:
Shouldn't highlighted text be DateTime.UtcNow?