ESP IP Reputation Score Comparision
Email deliverability is one of the most important aspects of any successful newsletter. We tested and compared the email deliverability for 12 of the top ESPs. I thought the community would be interested in the results.
Below are the ESPs we tested.
- MailerLite
- EmailOctopus (Amazon SES)
- EmailOctopus (Managed)
- Sendfox
- ConvertKit
- MailChimp
- Aweber
- Button Down
- SendInBlue
- Substack
- Revue
- GetResponse
Our tests indicate MailerLite, EmailOctopus, and MailChimp are the best performers, with Substack and Sendinblue being the worst performers.
Here is the full blog with the full analysis: https://newslettercrew.com/esp-deliverability-comparison/
Check it out and see how well your ESP does in deliverability. Is it what you were expecting for your ESP?
Trending on Indie Hackers
Hi Yaro, this is interesting but it's hard to evaluate your conclusions without understanding your testing methodology. Can you elaborate on how you did these tests and calculated your scores?
Also, I was informed by @td_evans of EmailOctopus that IP reputation matters a lot less and that your domain engagement is the more important factor. This is what he said in response to my inquiry about EO's deliverability:
"Interesting to see you mention deliverability. In my experience deliverability is a misunderstood concept a lot of the time. Any service, even one completely riddled with spammers, will have high deliverability. The term Deliverability means simply – are the emails delivered? We need to consider that hard bounces will bounce irrespective of platform, as they are invalid emails, and that while soft bounces can be triggered admittedly by poor quality IP's or content, very few mainstream services (such as Gmail/Hotmail) use IP blocklists anymore. So the service you use doesn't really make that much difference to whether an email is delivered. Now, in terms of inbox placement, that can be impacted but there is now a trend to domain engagement above and beyond anything."
I'm still trying to understand these concepts, so any additional info is welcome.
Yep so within Google you have both an IP and domain reputation, this can be viewed by setting up Google postmaster tools.
Domain reputation (most readily impacted by engagement) these days is the factor with the greatest impact on inbox placement. IP has a much smaller impact and for a correctly configured domain with SPF/DKIM in place and a high reputation the choice of ESP shouldn't make that much difference.
Thanks Tom, your insight is much appreciated! Do you know what constitutes "engagement"? Is it simply opening the email? Or does gmail track actions such as time read, links clicked, or other factors?
So – all this is, based on my experience, but others might have conflicting experiences. Gmail are famously secretive about the algorithms.
The best indication to get into the Inbox (over Promotions) is likely in replying. But clicks and opens are positive actions which we believe all are tracked.
If you're experiencing issues with emails going to promotions/spam I've even seen good success from simply sending a campaign to 10 friends (all based on Gmail), calling them up and asking them to move it to the inbox.
Interesting stuff. Is there a way for ESPs to monitor the percentage of emails going to promotions/spam? I'd love to know that as a user.
ESPs can track and report on delivery of an email but it's a very resources intensive task and a strain on architecture so most don't do it as a user-facing feature.
Tracking inbox placement is actually very complex and not very accurate for any single method used, like using GlockApps or other similar tools. That's because the same email can land into spam or inbox for different subscribers at the same provider based on how they interacted with sender emails before, so it's highly nuanced an relative.
I just published this post on IH, check out the diagram
https://www.indiehackers.com/post/10-tips-for-better-email-deliverability-0f73b1fbee
Well yes. ESPs get to know what percentage of their newsletters are going in spam.
Plus there are multiple tools which can predict inbox placements.
There's no way for an ESP to know if a particular campaign went into spam. We don't get that kinda feedback from the inbox providers.
Only way is through seedlists and end-user panel data.
Google don't share that data, however most ESPs (including us) will either have a seed list which has a recurring email sent out or have set-up a customer panel to gauge major delivery issues. It's not perfect though.
You can set-up your own seedlist using something like this https://glockapps.com/tutorials/how-to-test-inbox-placement-and-spam-score/
Got it, thanks!
Thanks, super helpful!
Hi and sorry to join the conversation late. I'm the writer of the article.
Coming back to the question well I do think the headline "ESP IP Reputation Comparison" is a better option but yes IP reputation does matter many times.
Delivery does depends on multiple things like Spam Filters, Domain Reputation, IP Reputation and how the individual user interacts with the mail.
That said IP play a important role along with others and I'll try to change the headline to something which conveys my message.
Thanks for writing the article. I'm still learning how all this works, so every bit of info is helpful!
I just posted tips for email deliverability, you may find it helpful:
https://www.indiehackers.com/post/10-tips-for-better-email-deliverability-0f73b1fbee
@stevenkkim
Thanks a lot for taking interest in our article. Now coming back to your question and our process. So the first thing we did was to choose some publications aka newsletters on each of the mentioned services.
Next, we tracked the email sender IP for a period of 30days and used the provided IP to find the sender trust score and delivery rating. As we already had the IPs with us, we used a reputable tool called SenderScore by Validity. This gave us the highs and lows of the reputation score. Then all we did was average the max and min value to come to a more accurate number which we have shared in the article.
One key point to note that we have randomly picked the newsletters which show the efficiency of the tool(ESP) but as we are on limited resources we weren't able to collect a huge dataset but the data shared is very well calculated for best and accurate results.
Thanks for the additional info Yaro. So then I think the title of this post should really be something more like "ESP IP Reputation Score Comparision" – which may have some impact on deliverability but not the same thing – is that fair?
Yeah that's a good point. I just changed!
This comment was deleted 5 years ago.
Both deliverability and inbox placement are complex and factors go well beyond IP reputation. It's a puzzle with pieces that all matter, see this diagram:
https://images.bigmailer.io/wp-content/uploads/2019/08/email-deliverability-factors.png
The 2 flaws with the article and methodology that I see are:
The most accurate way to test would be with the same sender, same content, same time of day, same day of week, etc. - same everything but the email platform.
The most challenging thing about deliverability IMHO is that each mailbox provider has its own rules and logic and so the composition of an email list has a big impact on deliverability. Approach to solving deliverability issues can vary from simple volume throttling to improving engagement (harder) and everything in between.
Image diagram source: https://www.bigmailer.io/blog/improve-email-deliverability-best-practices/
P.S. Deliverability != inbox placement, emails in Spam folder are "delivered".
Hi Webbie,
Happy to answer your questions.
Most users we tested where on a shared IP so it gives us a better understanding on how well these ESPs are managing the IPs.
Yes you are absolutely right, most ESPs have maintain multiple tiers which is used based upon the customer. I chose the newsletters with small to medium amount of subscribers cause those are the ones who mostly read our content. Plus if an ESP is managing good IP reputation on smaller customers then they surely are doing great job with bigger customers.
Thanks for responding @Ankit_Kaknut
Thanks for the info! 👍
Hi @yaroslawbagriy - If you do this again, I'd be interested in seeing where Constant Contact lands on this list. I am a partner with them.
Thanks for the list! There were a few providers I could add to my IH Toolkit.
I thought this article was really useful in comparing the different ESPs if you are using one of them and having trouble, or if you are looking for a new ESP.
I use Aweber, and haven't seen any problems with deliverablity that I could tell. Good confirmation.
Thank you for the sharing your tests!