Make file load faster or asynchronously in background?

Anvil Q&A
1

I’ve written a small game where players make a word list. When they switch from the game_form to the results_form, the program displays their words, marking those that don’t appear in a (static) master word list:

player_list = [item if item in self.big_word_list else f"{item}(?)" for item in player_list]

The ‘big word list’ is on the server, loading when the results_form is created:

self.big_word_list = set(anvil.server.call_s('read_dictionary_file'))

(I change it to a set to speed up searching.) The server call takes several seconds because the big word list dictionary file has hundreds of thousands of words.

Is there a way to either speed the list’s loading, or have it happen in the background? I learned a little about asynchronous programming when studying Dart, and I investigated background tasks in Anvil, but couldn’t figure out how to have a background task on the server pass a large word list to the results_form object. Ideally, I’d have the big_word_list load in the background, while players are still playing the game (game_form methods), available as soon as the results_form is initialized. Is that possible? Are there other ways to avoid the delay when results_form is created?

2

You’re not the only one interested in this:

1 Like
3

A few options I can think of:

You can use a Timer on the form to load in the word list little by little (a few hundred at a time? You’d have to measure to see how many you can load in at once without slowing things down). Once the list is loaded, store it locally (if the browser memory can handle a list that big).

You can do the searching on the server, so the word list lives there permanently and you only pass back the bits that match.

You can store the word list in Assets, so it gets loaded when needed.

4

I have a similar use case and achieved fantastic results using a combination of duckdb, Anvil Data Files service and parquet files and just kept the large data file server side.

import duckdb
from anvil.files import data_files

def query_parquet():
    result = duckdb.query(f"""
SELECT * FROM read_parquet('{data_files['my_file.parquet']}')
WHERE <SPECIFY YOUR CONDITIONS HERE>
""").pl()

Duckdb can also read csv, excel and a host of other file types very efficiently, just visit their documentation ( DuckDB – An in-process SQL OLAP database management system). You’ll need to sharpen up your SQL skills but I’m achieving crazy results with this.

This way you can keep the big word list on the server, and you’ll notice the query time is VERY fast.

2 Likes
5

I didn’t have a great solution, in the end, but here’s what I’ve done:

  • I worked a little with the big_word_list file, converting it to a list and saving it in a data table on the server
  • I created a Globals module, with a function to get the list from the server and assign it to a variable
  • I call the function when the app starts
    There’s still a delay, of course, but I think users are more accustomed to the idea of waiting a bit for an app to load, rather than having to wait several seconds in the middle of use. I also kept the “loading spinner” visible (i.e., anvil.server.call() rather than .call_s() so folks will know something is happening.

Thank you all for your responses. This is a friendly, respectful community. It’s something I really appreciate these days.

6

That’s right, and your approach is quite common.

It’s also true that users typically open the app, glance at it for a moment, and only then begin interacting with it. I like to make use of that brief pause by starting to load the list with a short delay—with a 0.1 second time—after the form’s show event completes. This gives the impression of a fast startup, and the list is usually ready by the time users start clicking.

If a click happens before the list is populated, I simply disregard it.

The actual time to readiness doesn’t change, but the perceived responsiveness is much better.

2 Likes