File Upload Size Limits using data tables

[dconnell]

Are there practical limits to file size when I’m uploading files to data tables?

[campopianoa]

Please see the pricing page:

Pricing Plans

And some time limit info:

Is the server timeout limit depends on what plan you have?

Yes, sort of! On all plans, the timeout for a Server Function call (ie anvil.server.call()) is 30 seconds. However, if you want to perform a longer-running task, you can use Background Tasks. For those, Free plans are currently limited to 30 seconds, but paid plans can run for hours if necessary.

And some RAM info:

How much RAM / memory do apps have access to?

Thanks for the question @ananth.krishnamoorth . Server runtimes on Anvil shared-hosting plans (Individual and Dedicated) have a 1GB burst limit on memory. It depends on what kind of processing you are planning to do, but I would say it likely would not work out under those plans. If you were on a Dedicated plan we could of course select a larger amount of storage space for your needs. May I ask what you are trying to do with the 1GB csv file? Perhaps there is a way you could avoid processing th…

[dconnell]

I’m considering allowing people to upload log files so I can automate analysis, some of these files could be substantially larger than 1GB.

I am familiar with server background tasks, but the dilemma I found while testing is this:

1. In client code a file_lo* ader is used to select (let’s say) five files totaling 5GB
2. file_loader calls an anvil.server.task, passing the five files
3. anvil.server.task calls a server.background_task, passing the five files
4. anvil.server.background_task processes the file files

I think I’m missing something because this is timing out at step 2

[p.colbert]

Would it help to compress the files, e.g., with zip? A shorter file takes less time to transfer, so should be less likely to time out.

[dconnell]

well, yes it does help to compress files, however, for example I have one .zip file that has 5 log files - the size of the .zip file is 20GB

[campopianoa]

Hmm, you may have to consider processing your data in smaller chunks and/or rethinking where those files are being uploaded to (i.e., perhaps you could transfer them directly to Google drive).

[dconnell]

Agreed… in handling data this large I need to think carefully about minimizing copy or movement

[p.colbert]

@dconnell

minimizing copy or movement

That may also mean placing the processing power (and code) on the same machine as the data. Otherwise, you end up copying it again.