Hi there! I’m having an issue with the time it takes to load my data into my app. For reasons I shall not delve into here, I’ve decided that the best way is to keep a .json file online, either in a data table as a media object, or in app_files using the Google API.
The .json file is 3,8MB, and is taking minimum 10.2 seconds to load. Over many hours of exhaustive (not frustrating! Coding is too much fun for that) checking, I’ve found the fastest way of doing it is:
file = app_files.folder.get(userFileID)
file_string = str(file.get_bytes(),'utf-8')
local_list = json.loads(file_string)
(I should have mentioned that the above is using the Google API - storing and retrieving the exact same file from data tables takes about 10 seconds LONGER!!!)
I cannot use json.load(file) because ‘json.load’ is not supported - only json.loads and json.dumps.
Everything else takes longer. Data table rows - longer. Data table file - longer. Text files - longer.
I need the list of dictionaries locally as there are many complex calculations that I need the app to perform on the fly and depending on the whims of the user, and I cannot rely on the resultant server calls this would entail - it would slow down the user experience unacceptably.
Is there a faster way of doing it? Fun challenge right?! Thanks in advance for the help! (I’ve dreamt of installing a service worker that saves the .json or JSON locally, but am far too green for that I think. Also…maybe downloading the file into memory first, then reading bytes. I have the feeling the app is reading bytes over the internet which might be slowing things down…)
Bruce
As @Tony.Nguyen kindly suggests, Anvil can store JSON directly using the “simple object” column type in the DataTables service. Give that a whirl to see if this helps.
1 Like
Good evening to both @Tony.Nguyen and @alcampopiano!
Thanks for the speedy response. Unfortunately that’s one of the first things I tried…the simple object column seemed MADE for my purposes. However, simply loading my data into a list of dicts (remember - 3,8MB of it!) takes between 21-22 seconds, transfer (either string or JSON) another 9 or so. Straight reading from the data table in the client shortens it by a second or two to about 28-29 seconds total time.
Those were early days. Since then, I’ve worried at the ‘problem’ like the proverbial dog with slipper…ending on Google as my quickest load time. I reckon 7-8 seconds for 3-4MB of json isn’t the worst time…but surely it CAN be better right? That’s why I was thinking of service workers (seemingly the new app cache of the web) storing things locally for quicker load times, then quickly comparing db’s with a few bits of data…
Hmm, I’m not really sure, but others may have clever ideas. You could read the forum posts on “optimizing performance” and related topics as there are helpful ideas there that may fit your use case.
You could try persistent server modules (business plan or above) but they do not always guarantee that global variables will stick around forever.
Other than that, you could also give a short description of your app and how users interact with it in case there are clever design tricks you could use to alleviate the long loading time (background tasks, loading chunks, etc).
If you have non-sensitive data, you could share a clone of your app (a simple stripped down version that is easy to run) and others here could take a closer look, if possible.
An edit/update: Thanks to both your ideas, I decided to clean up my data tables of previously tried and now unnecessary columns. Voila! OK, not VOILA VOILA, but Voila. Loading time now comparable to Google API, plus/minus a second. Which makes things a lot simpler. So thanks gentlemen!
@alcampopiano, I will definitely keep the above in mind and think about cloning a pared down version in here…