Parse multipart/related emails

I’m trying to parse emails and convert tables within them into pandas dataframes. Since some of the emails are multipart, I took some code from this answer.

The following code works fine but it breaks with multipart/related emails (no tables are found).

HOST = 'imap.gmail.com'
m = imaplib.IMAP4_SSL(HOST, 993)
m.login(USERNAME, PASSWORD)
m.select('Inbox')

result, data = m.uid('search', None, "UNSEEN", '(FROM "xxx@xxx.xxx")')
print(result)
if result == 'OK':
      for num in data[0].split()[:]:
            result, data = m.uid('fetch', num, '(RFC822)')
            if result == 'OK':
                  email_message = email.message_from_bytes(data[0][1])
                  b = email_message 
                  body = ""

                  print(b.is_multipart())
                  if b.is_multipart():
                      for part in b.walk():
                          ctype = part.get_content_type()
                          cdispo = str(part.get('Content-Disposition'))

                          # skip any text/plain (txt) attachments
                          if ctype == 'text/plain' and 'attachment' not in cdispo:
                              body = part.get_payload(decode=True)  # decode
                              break
                  else:
                      body = b.get_payload(decode=True)
                  soup = BeautifulSoup(body)
                  table = soup.find_all('table')
                  df = pd.read_html(str(table))[0]
                  display(df)

Here’s the header of one of the multipart/related emails:

Received-SPF: pass (google.com: domain of 0100017dfe181d85-07cbd269-a94a-4b30-8d52-1e4e48a34639-000000@amazonses.xxxxxx.com designates 54.240.11.40 as permitted sender) client-ip=54.240.11.40;
Content-Type: multipart/related; type="text/html"; boundary="--_NmP-f890ebfb5c0d8a34-Part_1"
From: xxxxxx <noreply@xxxxxx.com>
To: xxxxxx@gmail.com
Subject: Watchlist Summary for Mon, December 27, 2021 (Futures)
Message-ID: <0100017dfe181d85-07cbd269-a94a-4b30-8d52-1e4e48a34639-000000@email.amazonses.com>
Date: Mon, 27 Dec 2021 22:52:13 +0000
MIME-Version: 1.0
Feedback-ID: 1.us-east-1.xy6STr9N8VtfY9IEmltVU/dtudHWlVMH37XgJn5/ROY=:AmazonSES

----_NmP-f890ebfb5c0d8a34-Part_1
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html lang=3D"en"><head><meta charset=3D"UTF-8"><meta http-e=
quiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8"><meta http-equ=
iv=3D"X-UA-Compatible" content=3D"IE=3Dedge"><meta name=3D"viewport" conten=
t=3D"width=3Ddevice-width, initial-scale=3D1.0"><!-- So that mobile webkit =
will display zoomed in--><meta name=3D"format-detection" content=3D"telepho=
ne=3Dno"><!-- disable auto telephone linking in iOS--><title></title><style=
 type=3D"text/css">}
.ad p {
  margin-top: 4px;
}
</style><style type=3D"text/css">#data-table,
.data-table { max-width:100%; min-width:100%; width:100%; border-collapse:c=
ollapse; }
#data-table th,
#data-table td,
.data-table th,
.data-table td { color:#000000; border-collapse:collapse; padding:4px; whit=
e-space:nowrap; border:1px solid #D8D8D8; }
#data-table .body tr:nth-of-type(odd),
.data-table .body tr:nth-of-type(odd) { background-color:#f3f3f3; }
#data-table table tbody .spacer td,
.data-table table tbody .spacer td { border:none; }

.preHeaderHide { display:none !important; mso-hide:all !important; }
/* Outlook link fix */
#outlook a { padding:0; }
/* Resets: see reset.css for details */
.ReadMsgBody { width:100%; background-color:#ebebeb; }
/* Hotmail background and line height fixes */
.ExternalClass { width:100%; background-color:#ebebeb; }
.ExternalClass, .ExternalClass p, .ExternalClass span, .ExternalClass font,=
 .ExternalClass td, .ExternalClass div { line-height:100%; }

Any ideas? Thanks

Answer

you want to parse text/html parts

you should check for content type == ‘text/html’

Advertisement

Answer