I was reviewing some stats for and was curious about what email service top apps use; being particularly interested in transactional email services. Is there a way I can go through my mailbox, check product emails and check the service for each? Sounds like a fun experiment to give a shot. Mailintel Connecting my mailbox With my recent , connecting to my mailbox wasn’t difficult. I am using , a JS IMAP client. experiments with IMAP emailjs-imap-client ImapClient = ( ).default ;( { { confOption = { : { : process.env.EMAIL, : process.env.PASS }, : } (process.env.PORT === ){ confOption.useSecureTransport = } imap = ImapClient(process.env.HOST, process.env.PORT, confOption) imap.connect() } (e) { .log(e) } })() const require 'emailjs-imap-client' async ( ) function try const auth user pass logLevel 'error' if '993' true const new await catch console My environmental variables look something like this: EMAIL= PASS= PORT= HOST= "myemail@yahoo.com" "mypassword" 993 "imap.mail.yahoo.com" If you are connecting to a Yahoo mail, you will need to . Using your real email password won’t work. generate an app password Gmail is a little more complex. To start with, you need to your account. You can then go with any of the options: enable IMAP access . But don’t forget to turn this off once you are done with the experiment. If you still can’t connect, you may need to . Allow less secure apps allow app access to your account (more secured option). Create an app password Checking product emails Getting product emails is a tricky one. How do you identify product emails from regular emails? How do you differentiate marketing from transactional emails? This can be done but the ideas I came up with were not worth the effort. In the end, I decided to pull all the emails instead. (This was also easier because I have a Yahoo mail dedicated to product signups, subscriptions and newsletters). ImapClient = ( ).default ;( { { imap.connect() box = imap.selectMailbox( ) start = +box.exists - ( ) { messages = imap.listMessages( , , [ , ]) (! .isArray(messages)) { } start += (start >= +box.exists) { } } } (e) { .log(e) } })() const require 'emailjs-imap-client' async ( ) function try // …connection code await const await 'INBOX' // Reading 5k mails at once can choke the process so let's chunk into 50 mails per request. We are also assuming there are > 5k emails in the mailbox let 5000 while true const await 'INBOX' ` : ` ${start + } 1 ${start + } 50 'uid' 'body[]' if Array return // Do stuff with email here 50 if break catch console Checking the service provider How do I know the email service used for the mail? I checked a couple of email headers and noticed a couple of places the provider details can be extracted from. Here is what an email header looks like: At first glance at some headers, the header looks the most straight forward. Here are a couple of message IDs from 3 different mail headers. Message-Id Message-ID: <010001703764a668-4ebb47b9-4454-4081-afc4-e6448cd22897-000000@email.amazonses.com> Message-ID: <ZCA0TmwnS-SqPHh2_-gciw@ismtpd0039p1iad2.sendgrid.net> Message-ID: < > 22.BD.09488.F97644E5@ar.mta1vrest.cc.prd.sparkpost By looking at the host part of the , it's easy to know the service provider. But not so fast. Looking at more headers, I noticed some have a host that is different from the sender value in the header. Below are examples from Bitbucket and Letsencrypt. (Some parts truncated for brevity). message-id message-ids received Message-ID: Received: by filter0225p1iad2.sendgrid.net with ~ < > pr-~@bitbucket.org Message-Id: Received: from mail132-5.atl131.mandrillapp.com ~ < > ~.expiry@letsencrypt.org The headers seem the most accurate place to get the provider but it poses its challenges. The headers are structured in different ways depending on how the mail is sent. After reviewing a couple of headers in Yahoo mail, I was able to find a pattern to match the provider. (Not 100% accurate). received received For more than one received header, match of the second header. by * received If that doesn’t exist or does not have a host address or there is just one received header, then match the of the first header. (EHLO *) received Now we can rewrite our script to use headers. To be able to easily extract this, I will be bringing in . received Mailparser ImapClient = ( ).default simpleParser = ( ).simpleParser PULL = ;( { sites = {} { confOption = { : { : process.env.EMAIL, : process.env.PASS }, : } (process.env.PORT === ) { confOption.useSecureTransport = } imap = ImapClient(process.env.HOST, process.env.PORT, confOption) imap.connect() box = imap.selectMailbox( ) exists = +box.exists (exists <= PULL) { .log( ) } start = exists - PULL ( ) { messages = imap.listMessages( , , [ , ]) (! .isArray(messages)) { } ( message messages) { mail = simpleParser(message[ ]) headers = mail.headers.get( ) esp (headers.length > ) { header = headers[ ] match = header.match( ) (match && match.length > && match[ ].indexOf( ) !== ) { esp = match[ ].split( ).slice( ).join( ) } } (!esp) { header = .isArray(headers) ? headers[ ] : headers match = header.match( ) (match && match.length > && match[ ].indexOf( ) !== ) { esp = match[ ].split( ).slice( ).join( ) } } (!esp) { } (sites[esp]) { sites[esp]++ } { sites[esp] = } } start += (start >= exists) { } } imap.logout() imap.close() } (e) { .log(e) } .log(sites) })() const require 'emailjs-imap-client' const require 'mailparser' const 10000 async ( ) function const try const auth user pass logLevel 'error' if '993' true const new await const await 'INBOX' const if console 'You specified a pull number more than or equal to the number of emails in the mailbox' return let while true const await 'INBOX' ` : ` ${start + } 1 ${start + } 50 'uid' 'body[]' if Array return for const of const await 'body[]' const 'received' let if 1 const 1 const /by ([^\s]*)/ if 1 1 '.' -1 1 '.' -2 '.' if const Array 0 const /EHLO ([^)]*)/ if 1 1 '.' -1 1 '.' -2 '.' if // Todo: Track failed matches continue if else 1 50 if break await await catch console console Results This was the breakdown after stripping the results to the top ones. Few notes: , and are from . As you know, Mailchimp provides marketing email service. Mandrillapp.com is the transactional email service for Mailchimp. It used to be a standalone service until it became deeply integrated into Mailchimp. rsgsv.net mcsv.net mcdlv.net Mailchimp is for . spmta.com Sparkpost is for , currently owned by Adobe. Provides marketing email service. marketo.org marketo.com Interesting to see intercom there. It's amazing the number of products that use . Intercom and belong to , obviously. Mailgun also recently acquired . Mailjet provides both marketing and transactional email service. It will take more deep-diving into the headers (or content) to figure if the mail was sent as a marketing or transactional email. mailgun.net mailgun.org Mailgun Mailjet , also like Mailjet, offers both marketing and transactional email service. It’s the most used from my experiment but it’s hard to know what fraction of that is marketing and what other is transactional. PS: they were recently acquired by Twilio. Sendgrid Conclusion I merged the same providers for a more accurate chart. This is only a fun experiment and doesn’t show the true market share of the services. Since the data is based on the product emails in just my mailbox, it is flawed by selection bias. However, it’s something you can run on your mailbox for the fun of it. You can also extend it to see what service your favourite product/app uses. Remember, you may need to look at the full headers of some of your emails to come up with the best way to know the service provider. (Originally published ) here
