Encoding error trying to load a .csv file

Anvil Q&A
1

I have this code in a button on a form. My table is form enabled. I’ve verified at the command line that my file is plain text/utf-8 encoded. I have that file loaded as an asset. This is throwing a unicodeencode error on line 54 (bold)

Here’s my button code:

def button_1_click(self, **event_args):
url = anvil.server.get_app_origin() + “/_/theme/MARSYS.csv”
file_contents = URLMedia(url).get_bytes()
# Convert bytes to a string.
csv_string = str(file_contents, “utf-8”)
# Create a list of lines split on \n
line_list = csv_string.split(’\n’)
for line in line_list:

    flist = line.split(",")
    newRow = app_tables.marsys.add_row(industry="Maritime")
    newRow["top_level"]="Fleet"
    newRow["category"]=flist[2],
    newRow["sub_category"]=flist[3],
    newRow["vendor"]=flist[4],
    newRow["int_serial"]=flist[5],
    newRow["int_eth"]=flist[6],
    newRow["int_usb"]=flist[7],
    newRow["modbus"]=flist[8],
    newRow["tech_vendor"]=flist[9],
    newRow["tech_product"]=flist[10],
    newRow["tech_version"]=flist[11],
    newRow["notes"]=flist[12],
    newRow["marsys_key"]=flist[14],
    newRow["tech_key"]=flist[15]

https://anvil.works/build#clone:4CIBA67BEIWN2PVJ=RYPS5DDUBXE6SIFGFFNSBXWD

2

This should be faster, because it doesn’t need to create the row and modify it many times:

dic = {
  'industry': 'Maritime',
  'top_level': 'Fleet',
  'category': flist[2],
  'sub_category': flist[3],
  'vendor': flist[4],
  'int_serial': flist[5],
  'int_eth': flist[6],
  'int_usb': flist[7],
  'modbus': flist[8],
  'tech_vendor': flist[9],
  'tech_product': flist[10],
  'tech_version': flist[11],
  'notes': flist[12],
  'marsys_key': flist[14],
  'tech_key': flist[15],
}
app_tables.marsys.add_row(**dict)
3

Ah, makes sense. However, after making this change, the original post/error is still an obstacle, it’s not liking this line: csv_string = str(file_contents, “utf-8”)

4

Sorry, I was confused by the “8 hours” and “into a table” in the title and I thought it was a performance problem.
I changed the title so it will attract the right type of answers.

1 Like
5

thanks… I just keep hitting these obstacles that cost me days

6

How does this error occur on a server module when I’ve passed it a plain text/utf-8 encoded .csv file:!

TypeError: decoding to str: need a bytes-like object, StreamingMedia found at [CHServerModule_one, line 23](javascript:void(0)) called from [load_tables_util, line 22](javascript:void(0))

Just to make sure my input file is not a culprit here, I checked it at the command line (on a Mac):
marsys.csv: text/plain; charset=utf-8

7

If you move that code to a server-side it works…

On the client side: I think the problem in this is using the URLMedia - in the docs and in the auto-complete it comes with a warning:

Create a Media object representing the data at a specific URL. Caution: Getting data from URLs directly in your code will often fail for security reasons, or fail to handle binary data.

I think this is the issue that you faced… I wouldn’t use an asset here - I would use the data_table media object…

Using the media object in the datatable worked fine on the client (if you make your tables client readable/writable):

file_contents = app_tables.files.get(filetype='MARSYS')['file'].get_bytes()
csv_string = file_contents.decode()

But I would definitely put this kind of function on the server - it will be quicker and more secure.

@anvil.server.callable
def load_MARSYS():
    file_contents = app_tables.files.get(filetype='MARSYS')['file'].get_bytes()
    csv_string = file_contents.decode()
   
    line_list = csv_string.split('\n')
    for line in line_list:
      flist = line.split(',')
      d = {
        'industry': 'Maritime',
        'top_level': 'Fleet',
        'category': flist[2],
        'sub_category': flist[3],
        'vendor': flist[4],
        'int_serial': flist[5],
        'int_eth': flist[6],
        'int_usb': flist[7],
        'modbus': flist[8],
        'tech_vendor': flist[9],
        'tech_product': flist[10],
        'tech_version': flist[11],
        'notes': flist[12],
        'marsys_key': flist[14],
        'tech_key': flist[15],
      }
      app_tables.marsys.add_row(**d)

Edit: Answer above probably superseded by this thread:

Please post the most simple example loading a .csv into a table - #14 by stucork