HTTP endpoint POST error decoding image/gif in body

Anvil Q&A
1

Hey all,

I’m trying to receive a HTTP post request from Mailgun in Anvil using the http_endpoint, but I keep having problems with decoding the Media object, using get_bytes().

The error message:

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xa2 in position 12622: invalid start byte
at /usr/local/lib/python3.7/json/init.py,line 343
called from [API, line 17](javascript:void(0))
called from /downlink/anvil/_server.py, line 1292

I’ve tried all imaginable encodings to get it to work, but the problems seems to be a gif which is included in the body of the post request, which can’t be decoded from the media object.

Does anyone have any idea on how to solve this?

My server code:

import anvil.email
import anvil.users
import anvil.tables as tables
import anvil.tables.query as q
from anvil.tables import app_tables
import anvil.server
import json

# MAILGUN INBOUND MAIL API ENDPOINT
# https://<redacted>.com/_/api/mailgun/mail_inbound
@anvil.server.http_endpoint('/mailgun/mail_inbound',methods=['POST'])
def mail_in(**params):
  print(str(anvil.server.request.headers))
  bb = anvil.server.request.body.get_bytes()

  data = json.loads(bb)
  print("DATA: \n" + str(data))
  
  bb = anvil.server.request.body.get_bytes()
  data = json.loads(bb)

  return f"Successfully tested with params {params}"

The POST Request header:

{‘host’: ‘api:3000’, ‘user-agent’: ‘mailgun/treq-16.12.0’, ‘content-type’: ‘multipart/form-data; boundary=bd909c40-242b-44ea-b004-3288335ab6b8’, ‘cookie’: None, ‘content-length’: ‘15574’, ‘accept’: ‘/’, ‘x-forwarded-for’: ‘54.144.55.146’, ‘accept-encoding’: ‘gzip’, ‘x-real-ip’: ‘54.144.55.146’, ‘anvil-host’: ‘.com’}

The POST request body:

–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Content-Type”

multipart/mixed; boundary=“------------020601070403020003080006”
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Date”

Fri, 26 Apr 2013 11:50:29 -0700
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“From”

Bob bob@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“In-Reply-To”

517AC78B.5060404@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Message-Id”

517ACC75.5010709@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Mime-Version”

1.0
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Received”

by luna.mailgun.net with SMTP mgrt 8788212249833; Fri, 26 Apr 2013 18:50:30 +0000
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Received”

from [10.20.76.69] (Unknown [50.56.129.169]) by mxa.mailgun.org with ESMTP id 517acc75.4b341f0-worker2; Fri, 26 Apr 2013 18:50:29 -0000 (UTC)
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“References”

517AC78B.5060404@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Sender”

bob@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“Subject”

Re: Sample POST request
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“To”

Alice alice@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“User-Agent”

Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“X-Mailgun-Variables”

{“my_var_1”: “Mailgun Variable #1”, “my-var-2”: “awesome”}
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“attachment-count”

2
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“body-html”

Hi Alice,

This is Bob. 

I also attached a file.

Thanks,
Bob

On 04/26/2013 11:29 AM, Alice wrote:
Hi Bob,

This is Alice. How are you doing?

Thanks,
Alice

–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“body-plain”

Hi Alice,

This is Bob.

I also attached a file.

Thanks,
Bob

On 04/26/2013 11:29 AM, Alice wrote:

Hi Bob,

This is Alice. How are you doing?

Thanks,
Alice

–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“content-id-map”

{“part1.04060802.06030207@mail.cashflowcoding.com”: “attachment-1”}
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“from”

Bob bob@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“message-headers”

[[“Received”, “by luna.mailgun.net with SMTP mgrt 8788212249833; Fri, 26 Apr 2013 18:50:30 +0000”], [“Received”, “from [10.20.76.69] (Unknown [50.56.129.169]) by mxa.mailgun.org with ESMTP id 517acc75.4b341f0-worker2; Fri, 26 Apr 2013 18:50:29 -0000 (UTC)”], [“Message-Id”, “517ACC75.5010709@mail.cashflowcoding.com”], [“Date”, “Fri, 26 Apr 2013 11:50:29 -0700”], [“From”, “Bob bob@mail.cashflowcoding.com”], [“User-Agent”, “Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4”], [“Mime-Version”, “1.0”], [“To”, “Alice alice@mail.cashflowcoding.com”], [“Subject”, “Re: Sample POST request”], [“References”, “517AC78B.5060404@mail.cashflowcoding.com”], [“In-Reply-To”, “517AC78B.5060404@mail.cashflowcoding.com”], [“X-Mailgun-Variables”, “{"my_var_1": "Mailgun Variable #1", "my-var-2": "awesome"}”], [“Content-Type”, “multipart/mixed; boundary="------------020601070403020003080006"”], [“Sender”, “bob@mail.cashflowcoding.com”]]
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“recipient”

monica@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“sender”

chandler@mail.cashflowcoding.com
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“signature”

0f42395537d8f7bc31e85e530849b7a5a9b2f00222bc41b9cf76d617983808eb
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“stripped-html”

Hi Alice,

This is Bob. 

I also attached a file.

Thanks,
Bob

On 04/26/2013 11:29 AM, Alice wrote:

--6d16ff46-6470-4ce0-958e-c79e7a6fe2bf Content-Disposition: form-data; name="stripped-signature"

Thanks,
Bob
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“stripped-text”

Hi Alice,

This is Bob.

I also attached a file.
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“subject”

Re: Sample POST request
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“timestamp”

1616061118
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“token”

dfe085487e0c484d0733c153f2a1cb781147fcaffddd7cab91
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“attachment-1”; filename=“crabby.gif”
Content-Type: image/gif
Content-Length: 2785

GIF89a!  ¢ ÿÈÈÿ êÿÿÿ !ÿ NETSCAPE2.0 !ù
 , !  hHºÜþP6£¤nÔ¥mgŸ Û e5–‘‹r-!ÊL˜¯˜ßç‹¿6í7!¾^¥ÄC• $
z©¤Æ¨{Óx—[.åê|ŽÎ$ •»Â)¯%óÛ†æå÷ÓÆsn€E~yƒ‚‚ !ù
 , !  dHºÜþP6£¤nÔ¥mgŸ Û e5–’‹r-d¾"MÌ/> ‚Âߧ\Ð% ÐJ@ãDa2¢†œèÍÉl:©7¶2¸b)]årD^I¨˜xMi-[
Çd·ÜºìÆ_öõ~‚ !ù È  , 
HJ"Ú+ÊI«½8ëÍ{ !ù  , !  GHºÜþ0ÊI«½ø0ˆÊG @¶œaªcih7.ùÂò¬Èy–À ïV€qƒjM@f“ó¼ð„ÀÝkX›u¡Þð’ !ù  , !  gHºÜþŒ6£ª g-9åƒ6 6ÞEš šµ¬+Ê,Á¯«î)Ó›o2ÌõZ:ӍGDŠx(gf¨–êV²Z¦#dm’Î¥ •RÇ¢eìÛ¶ç·7%€ws‚† !ù  , !  gHºÜþŒ6£ª g-9åƒ6 6ÞEš šµ¬+Ê,Á¯«î)Ó›o2ÌõZ:ӍGDŠx(gf¨–êV²Z¦#dm’Î¥ •RÇ¢eìÛ¶ç·7ç‰w€…‚s !ù  , !  hHºÜþp4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~ƒ‚‡ !ù  , !  hHºÜþp4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌG¬J‚xy†ƒ~ !ù 2  , !  hHºÜþo4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~ƒ‚‡y !ù
 , !  `Hª³þ°Á7꼤Ê;@ßãy ÁŠg
.]«‚@Û,:;±<×7–ïÕÓí6’bj’4"aÍå+³t14X• s6:€°8éÅd…UÐyÕ5m-lΛŒO¾™<~jßëÿ+ !ù
 , !  gHºÜþp, e%×Þ‘ãîÊ H"è”ØxÉR‚ Äᛶ–-âÝË7oÂj7“ZÀ¸ìì˜Cä
ê,.›Ž–ò¼’ƒ+‘(›¹á¦)4Vz¿Úʃc¯åÉì–‹¯0€wx‚†x !ù
 , !  hHºÜþo4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~ƒ‚‡y !ù  , !  hHºÜþo4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~‚†ƒy !ù  , !  hHºÜþp4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~ƒ‚‡ !ù  , !  hHºÜþp4 µn\¥+é ÈÀ5 Á’&Ú¦–K‚BòëðÀ·YÍ×Cíp? ‘bB“¡_ÊØäp¤Ë‰t«Ôx™[lf€U’Î,9©
‰ˆ\wÛ£1r瞐úËÌW”€x~‚†ƒ !ù  , !  gHºÜþŒ6£ª g-9åƒ6 6ÞEš šµ¬+Ê,Á¯«î)Ó›o2ÌõZ:ӍGDŠx(gf*¨–êV²Z¦#dm’Î¥ •RÇ¢eìÛ¶ç·7%€ws‚† !ù d  , !  hHºÜþP6£¤nÔ¥mgŸ Û e5–’‹r-!ÊL˜¯˜ßç‹¿6í7!¾^¥ÄC• $
z©¤Æ¨{Óx—[.åê|ŽÎ$ •»Â)¯%óÛ†æå÷ÓÆsn€E~yƒ‚‚ !ù
 , !  dHºÜþP6£¤nÔ¥mgŸ Û e5–‘‹r-d¾"MÌ/> ‚Âߧ\Ð% ÐJ@ãDa2¢†œèÍÉl:©7¶2¸b)]årD^I¨˜xMi-[
Çd·ÜºìÆ_öõ~‚ !ù 2  ,    HºÜþ0ÊIëBaË»ÿ`X% !ù
 , !  JHº:üp¹È†¥ÔÎ FǏ’rÁ”`”çV­&ªÂ\«¬ü¶§LÛ¸œ.Óm&^+´%Äž«Ñ[4XIu˜êz¿à°xL.›Ë !ù
 , !  hHºÜþP6£¤nÔ¥mgŸ Û e5–’‹r-!ÊL˜¯˜ßç‹¿6í7!¾^¥ÄC• $
z©¤Æ¨{Óx—[.åê|ŽÎ$ •»Â)¯%óÛ†æå÷ÓÆsn€E~yƒ‚‚ !ù
 , !  bHº:üp¹È†¥ÔÎ FǏ’rÁ”`”çV­&ªÂ\«¬ü¶§LÛ¸œ.Óm&^+´%Äž«Ñ[4XIu()^<€°éV.ÁªK“
%·èv„ŒEÊç×¼cŸ¿‡ô€| !ù
 , !  hHºÜþP6£¤nÔ¥mgŸ Û e5–'‹r-!ÊL˜¯˜ßç‹¿6í7!¾^¥ÄC• $
z©¤Æ¨{Óx—[.åê|ŽÎ$ •»Â)¯%óÛ†æå÷ÓÆsn€E~yƒ‚‚ !ù
 , !  bHº:üp¹È†¥ÔÎ FǏ’rÁ”`”çV­&ªÂ\«¬ü¶§LÛ¸œ.Óm&^+´%Äž«Ñ[4XIu()^<€°éV.ÁªK“
%·èv„ŒEÊç×¼cŸ¿‡ô€| ;
–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf
Content-Disposition: form-data; name=“attachment-2”; filename=“attached_файл.txt”
Content-Type: text/plain
Content-Length: 32

This is the content of the file

–6d16ff46-6470-4ce0-958e-c79e7a6fe2bf–

2

The problem stems from the POST type Mailgun is using, which is multipart/form-data, json.loads() seems to have some problems with parsing that. I’ve now found the workaround of not having Mailgun forward my received emails to the app, but instead I’m using their store and notify routing option. This way the notify POST request is sent with content-type “application/x-www-form-urlencoded” and without attachments, which makes it a LOT nicer to work with :slight_smile: