Using JSON Files

Anvil Q&A
1

I am trying to build an application that provides statistics and data analytics to support cricket teams based on historical data. Currently, I have a database of hundreds of JSON files of previous match events, and I want to use the files in my server-side code. I have used the data files feature and have got a table of files as shown below:

Screenshot 2024-05-04 at 11.36.07 AM
Screenshot 2024-05-04 at 11.36.07 AM1588×1094 153 KB

However, when I try to reference the files as follows:

path_to_json = ‘T20Data’
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith(‘.json’)]
games =
for json_file in json_files:
try:
with open(json_file, “r”) as f:
games.append(json.load(f))
except:
pass
with open(“combined.json”, “w”) as f:
json.dump(games, f, indent=4)

I am getting a message that the T20Data Directory doesn’t exist.

Dos anyone have an idea of how to implement the JSON files into the Anvil code?

2

Since you stored the Jason files in a data table, you need to search the data table:

json_files = [row["file"] for row in app_tables.table_name.search() if row["file"] and row["file"].name.endswith("json")]

# rest of the code

The other code stays similar, but you won’t use open on the files since they’re anvil media objects. Here’s the documentation for that:

3

I’m not familiar with the file service, so a clone link would help understanding the problem.

If you don’t want to share the real app, you can create a small one to reproduce the problem and share the clone link for that.

I would like to add a little advice: I don’t know if it applies in this specific case, but in general, instead of opening text files, getting the text, converting the JSON text to Python objects, then going to the next file, rinse repeat, it’s better to work with a table with a simple object column.

Instead of uploading the files as files, you could create an uplink script that opens each file on your PC, gets the text, converts the text to Python objects, and adds rows to a table. The table can have a simple object (JSON) column or you could organize the data in columns at this time.

An uplink script has access to both the files in your computer and the app tables in the Anvil server.

Then the app would get Python objects from an easily searchable table. This is often more scalable, clean and easy to manage.

4

Hi,
Thank you very much for your advice. Here is a clone link for your reference: Anvil | Login

Also, the file service just creates a data table with each of the files.

I will try to use the uplink script to get the files as using the file service exceeded the runtime limit.

5

Hi,
Thank you very much! This seems to be working except that the process is exceeding runtime limits (probably because there are too many files. I am wondering if this could be done as a background task (I am pretty new to Anvil).

Here’s a clone link for your reference:

6

Probably could: