Import csv or excel too slow

Anvil Q&A
1

Hello, so, I was building an app that receives a csv or a excel file, and put the data into the app tables. I was using this topic: Anvil Docs | CSV and Excel import to help me. But it is actually too slow. The files i need to import have tons of lines, (more than 30 thousand) and it takes like 50 minutes to execute the proccess. So, I wanted to know if there is any way that can make it faster. Thank you all :slight_smile:

2

Welcome to the forum @Louis

You can try this make a list of all rows, and insert whole list into
Simple Object column in the table.

# Add these lines to the top of your Server Module
import pandas as pd
import anvil.media

@anvil.server.callable
def store_data(file):
  #new code
  my_list = []

  with anvil.media.TempFile(file) as file_name:
    if file.content_type == 'text/csv':
      df = pd.read_csv(file_name)
    else:
      df = pd.read_excel(file_name)
    for d in df.to_dict(orient="records"):
      # d is now a dict of {columnname -> value} for this row
     #app_tables.your_table_name_here.add_row(**d)

     #new code
     my_list.append(d)

  #new code
  app_tables.my_table.add_row(id="1", list_value = my_list)

image

3

It would be usefully to put print statements into your code to see where the bottleneck is.

Ie how long does it take to load the spreadsheet into the data frame etc

4

Hey tony! Thanks for your time! But for some reason, when I do this to my code, I’ve got this error: anvil.server.UplinkDisconnectedError: Uplink disconnected
Even though I used anvil.server.wait_forever()

5

Utils — Anvil Extras documentation has some toys to help make that timing exercise easier.

6

I actually use this, so I don’t have to deal with pandas and anvil.media: