What I’m trying to do:
In my app (a simulation model game) I need to manipulate a huge (3841 by 10982) numpy matrix. The array would fit (in the Hobby plan and above) into a data table. Can I set this data table up programmatically, to avoid typing in 10982 column headings by hand? I would assume accessing cells in a db internal to Anvil is the fastest way to go.
What I’ve tried and what’s not working:
Keep the matrix in an external (to Anvil) db. Since I need to access elements thousands of times for each simulated timestep, this is way to slow.
Keep the matrix in a file
2.1 within Anvil: Apparently, Anvil data files are meant to be static, ie not to be written to
2.2 external to Anvil: Could get the ftp connection to work, but did not manage to access the file from my Anvil server module. But even if I got to work, using ftp is probably too slow.
This suggests keeping the matrix in memory, in numpy format, while those timesteps are being applied, possibly in a Background Task.
To persist the matrix, you might consider a SimpleObject column. It can store lists of lists of numbers, so it should be possible to convert to/from that format.
An alternative might be a Media column. It stores binary data, and a numpy matrix (the array elements, at least) can be considered one contiguous binary blob. This avoids conversion to/from text format, so loading & storing should be much faster than a SimpleObject column.
In the same table row, you could use additional table columns to record the matrix dimensions, if needed.
If you don’t need to search (via app_tables.tablename.search filters) inside the data, use a Media column. Simple object columns that get too large can adversely affect Postgres’ indexing and search speeds (in at least one app that I know of).