Saving big data frame from pandas in client code

[chicho_knwe1]

Hello,

I have a data frame from pandas and it is more than 20 000 rows. Whenever I try to do that I get this:

How do I create a media object from pandas dataframe? I tried doing it, but for some reason, it did not work correctly.

[stucork]

1. create a BytesIO object
2. convert the dataframe to an excel file or csv where the first argument is the BytesIO object
3. create a BlobMedia object from the BytesIO object, calling its getvalue() method

An example with excel file

Download excel file

How about something like this: import io import pandas as pd @anvil.server.callable def export_to_excel(data, columns): df = pd.DataFrame(data, columns=columns) content = io.BytesIO() df.to_excel(content, index=False) content.seek(0, 0) return BlobMedia(content=content.read(), content_type="application/vnd.ms-excel") and on the client side, you might have this in a click hander for a download button: def download_button_click(self, **event_args): result = anvil.serve…

With csv file

    content = io.BytesIO()
    df.to_csv(content)
    blob = anvil.BlobMedia(content_type='text/csv', content=content.getvalue())
    return blob

[chicho_knwe1]

Hello, I have a further qustion. Now I am able to save the data in an BlobMedia object, but when I pass that variable to another server function I get the same error. How can I fix that?

[p.colbert]

Hmm… I wonder how well your data will compress? I.e., compress it before transmission, so it’s small, and re-constitute it on the receiving end?

[chicho_knwe1]

I have tried the pickle package. Howerver, whenever I try to return a pickled object I get this exception:

[p.colbert]

As above, when you have an object of type bytes, you can wrap in an io.BytesIO() object. This will let you make a BlobMedia object out of it, which can be transmitted.

The resulting object might not be any smaller, though, which is why I suggested compressing (not pickling) your data.

[chicho_knwe1]

You mean using something like zlib?

[p.colbert]

Yes.

I’m not sure what sort of compression is available in the Anvil Client. Because it is running in a browser, not a full PC, the Client has a subset of the Python Standard Library.

See Data Compression and Archiving — Python 3.7.10 documentation for a list of several different modules you can try. If one of them isn’t available, or isn’t fully implemented, try another.

Once you find one that works for you, you can wrap the resulting binary data in a MediaObject (as above), and send it. On the server, you’d reverse the process (MediaObject → bytes → decompressor → decompressed data), with the corresponding decompressor from the standard Python library.

I hope that makes sense.

[chicho_knwe1]

Yes, it makes sense. I will try it.

[chicho_knwe1]

Solution:

I have tested everything and nothing worked. The thing is that I have changed the user experience(UI) and that is why I needed to save this data in the client code, but before the change it was working fine. So I tested it the old way and I got the same error.
It says that the error is at MatchForm, line 68. However it was in my server file. The red line is where the exception was thrown and it was because I was reading the whole csv file, which not only has 25 000 rows, but it also has many columns and when it was trying to send it as a parameter to the background function it was way too big. All I had to do was add the two green lines, which basically take only the columns that I need and remove any empty lines. As for saving the variable in the client code, I just save the file that was uploaded in a local variable and pass it whenever the button “Parse” is pressed.