How much access does an APP have to memory? For example, can I open a 1GB csv file using pandas on Anvil Server side code?
Hi @ananth.krishnamoorth, it’s a very interesting question.
The best way to know the answer is to actually open it and you will see the result.
Can’t, I am on Free Plan (ModuleNotFoundError: No module named ‘pandas’)
Thanks for the question @ananth.krishnamoorth . Server runtimes on Anvil shared-hosting plans (Individual and Dedicated) have a 1GB burst limit on memory. It depends on what kind of processing you are planning to do, but I would say it likely would not work out under those plans. If you were on a Dedicated plan we could of course select a larger amount of storage space for your needs.
May I ask what you are trying to do with the 1GB csv file? Perhaps there is a way you could avoid processing the whole thing and save on some memory.
I have about 9GB data sitting on AWS S3. I connect my jupyter notebook on AWS (16GB RAM, 2 cores) and run the pandas code, which takes about 15 minutes to complete. Maybe I should do the processing somewhere else and bring only aggregated data into Anvil?
That would certainly make sense. Anvil’s Uplink will probably be a good way to share the aggregated data as needed - if you already have a Jupyter notebook running, connecting it to your app with the Uplink should be straightforward. Here’s a video about the process which might help.
I have tried the notebook uplink earlier, it works well. Thank for helping me understand the mode of use.
1 Like
Great, I’m glad it is working for you! If you think my replies helped solve the issue you can mark it with the green ‘tick’ below the text. That way other people with a similar problem can find what has helped.