I suspect that the high memory usage comes from parsing the file. During parsing, you have two copies of the data: one in the original format, and one in the final format (ndarray
).
If that’s so, then it may be best to do that not in Server code, but in Uplink code instead. Since your Uplink code runs on your own PC, you can more easily have more memory resources.
Your Uplink code can save the result to a binary (Media) column, in a database table, for access by Server code.
If the Server still runs out of memory when loading the binary data as a single ndarray
, then you might chop it up. Conveniently, a 2x2 array divides neatly into 4 well-defined chunks, which could be stored in 4 Media rows.