Anvil app for analyzing large datasets

Anvil Q&A
#1

I’m looking to use Anvil for making an app for running a machine learning model on a user’s data. I’d like to let users load their data with a file loader and then store it in the app’s data table, but I’m running into the issue where writing large data sets to a data table takes too long.

Now, I think I can work around this by just using two file loaders. One to load the file and then select the columns to use in the model. This info gets stored in global variables. Then use a second file loader which takes the same file and uses the selected columns to run the model. But this just seems really clunky and could be confusing so I’d rather find a way to load the data once and then just work with it from there.

I’m wondering if anyone else has experience with something like this or if anyone knows whether this is feasible?

#2

Yes, I (and others) have experience doing something like this and, yes, it’s feasible.

#3

Thanks! Can you explain a bit more? How can I enable the storing of users’ large datasets in an app without it taking so long to load and be stored in a data table?

#4

Do you have any further advice?

#5

You might want to have a look at this previous discussion:

#7

Is the client writable view any faster than writing to a regular data table? My issue isn’t so much that I’m running into a time-out, my files are csv format, so they are actually quite small. Only a few thousand KB. But they contain tens of thousands of rows, so my issuye is more that it’s not acceptable to make a user wait so long just to upload their data file. Is there any way to just store a file on an app to be accessed by two different server calls?

#8

I figured this out. The solution was actually much simpler than I thought. You can simply store the ‘file’ object from the file loader as an attribute. Code example:

def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)
    self.working_file = None
    
  def file_loader_1_change(self, file, **event_args):
    """This method is called when a new file is loaded into this FileLoader"""
    #this stores the data in the anvil data table
    self.working_file = file
1 Like