What I’m trying to do:
I can download an entire (or filtered) data table from a repeating_panel in the client quite nicely and quickly, but am struggling to download a subset of the same data (i.e. specific columns not all columns) without resorting to VERY slow and convoluted operations (see below).
The following client code works, but for the “lite” download (i.e. just 4 specific columns out of about 30) it has to create a temporary table which unsurprisingly takes several minutes for a 10,000 row long CSV file, compared with several seconds for the “All columns” version to download directly from memory/client:
from datetime import datetime, timezone
from anvil.tables import app_tables
from anvil import *
self.repeating_panel_1.items = app_table.mytable.search()
def download_csv(self, **event_args):
"""This method is called when the button is clicked"""
lite = event_args['sender'] is self.button_csv_lite
if not lite: # download all data from repeating_panel
csv_file = self.repeating_panel_1.items.to_csv()
if lite: # download selected columns from repeating_panel only
app_tables.my_temp_table.delete_all_rows()
columns = ["A", "B", "C", "D"]
for row in self.repeating_panel_1.items:
app_tables.lite.add_row(**{k: row[k] for k in columns})
csv_file = app_tables.my_temp_table.to_csv()
csv_file = anvil.BlobMedia('text/csv', csv_file.get_bytes(), name="my_data.csv")
anvil.media.download(csv_file)
What I’ve tried and what’s not working:
-
The above is working, just VERY slow and requires a whole table to be built with client-server traffic, needlessly, just to request all that data back again for the client to use the
.to_csvmethod. -
I’ve tried manually feeding just the desired columns as a list of dicts into
repeating_panel.items, but if I’m understanding right, theto_csv()method can only be used for a table or for a search. When I manually set the data of arepeating_panelusingself.repeating_panel_1.items = <list of dicts>it is no longer possible to extract usingto_csv()… I get an error something like “Cannot use to_csv on list”. -
I’ve tried constructing a list of dicts, passing to a server function to use Pandas or CSV to create a file or a fileIO object, then convert into an Anvil Media object and return that to the client for download. Successful but not surprisingly so slow that I need to set up a background task to avoid timeouts and then write extra client code to wait for completion before downloading from the client. Apart from being very convoluted and requiring extra processing and spin-up time for the background task, it’s just intuitively very inefficient sending to and from the server if all the data already exists in the client.
The frustrating thing is that I can’t seem to access the lazy search iterator or to_csv method - it’s an all or nothing deal apparently?
Any help to complete this seemingly trivial task in the client, simply and quickly, would be greatly appreciated.
PS As a workaround I’ve been running a really simple script on my local machine… this shows how trivial the actual task should be, but the complications arise from navigating Anvil’s implementation of to_csv I think.
from pathlib import Path
dir_path = Path(r"C:\Users\peter\Downloads")
files = [x for x in dir_path.glob('my_data*.csv') if "LITE" not in x.name]
for file in files:
df = pd.read_csv(file, encoding="utf-8")
columns = ["A", "B", "C", "D"]
df[columns].to_csv(
new_name := str(file).replace(".csv", "_LITE.csv"),
encoding = 'utf-8',
index = False,
)
print(f'Created: {new_name}')