Apologies for the frustration you’re dealing with here! That error message is annoyingly unclear, but it does usually does mean your server code has run out of memory and been killed. (This is true even when the message doesn’t explicitly mention memory - we’re working to make this inference more accurate but unfortunately when a process runs out of memory it gets killed at quite a low level so we’re left reading the tea leaves trying to work out why it died suddenly.) The way that your code has been responding to reducing memory usage strongly suggests you’re on to the right path, and just having trouble fitting under the limit.
Both Personal and Professional plans (at the time of writing) have a memory limit of 1GB for an instance of your code (higher limits are available on the Business plan); this is why you didn’t see an improvement with an upgrade. Your options are, roughly:
To continue optimising, as you have been, to ensure that the computation you require can happen in under 1GB of RAM. I’d recommend trying to store as much data as possible on disk rather than in memory (for example, saving your data to disk, then using a streaming XML parser). Something to be aware of is that all BlobMedia objects are stored in RAM (again, that’s at the time of writing; we have intentions to be smarter about large BlobMedia objects), so handling them can blow out your memory usage rather easily.
To move the RAM-intensive operations to an Uplink process running somewhere with more RAM headroom
To move to a Business plan (I’m aware it’s a jump), where we can allocate considerably more memory to your process.
So @masikh this sounds like your next option is to abandon keeping the xml that you get from the zipped url as a string in memory.
You should be writing it (uncompressed, as an xml text file) to the server modules local disk as a TempFile, then using the file handler to pass to iterparse(). This should allow it to stream from disk instead of reading/holding the entire xml string in memory.
I may be stating the obvious here, but this is when I work offline.
Once I have established what problem I need to solve, in this case memory usage optimization, I work in PyCharm, where I have debugger and profiler, and I am much more productive. When I have a working version, I push to Anvil, run and, if it fails, keep working on it offline.
It is difficult to work blindly, without a clear target. Even now that Meredydd put a number on the memory limit, the target is still fuzzy. But it’s clear that you need to decrease the memory usage, and if you are still working on it online, I hope you are not, I strongly recommend to work on things like @ianb suggestions offline.
Thanks for your inputs again. I had to take a few days break for my own sanity.
In the end I was able to resolve the issue by breaking up the xml within python (as a text file) into the multiple tables it would produce then sending those components one at a time into my sql db. A stored procedure then parsed it into the tables.
I’m still finding it super strange that exiting this loop:
for event, element in iterparse(io.StringIO(xml_content), events=("start", "end")):
which was simply streaming the xml one node at a time, caused the RAM to spike.
It’s a mystery I won’t resolve but in the mean time I’m able to proceed again.
This is (or was in your previous code) wrapping the entire string from xml_content into a file handler, then streaming it to the xml parser. The entire xml file is still in memory in the string that StringIO is wrapping.
It would use less memory if you wrote out the contents of that entire string as a temporary, on-disk file in the server module, cleared the string from memory, then handed the file handler for that on-disk information over to iterparse() using pythons normal builtin open() style file handler.
I had an inkling that was the problem. However the file was too large for Anvil to not allow me to use BlobMedia (which I believe loads it all into RAM). I may not have thought through the process sufficiently.
When the parser started the profiler showed me usage was <700MB. This would creep up to <850MB RAM over the course of streaming the xml then suddenly at loop exit crash the whole thing.
The only this else I have is that having built systems like this before (back in python 3.3, so a while ago), when working correctly, python was only using 100-200 megs of memory to process 1GB to 3.5GB xml files. To do this they had to be moved to disk to work though.
I only remember because it is not something I would normally do unless memory hogging is a concern.