Load JSON data from table media column

Hi,

I’m fairly new to Python and I’m trying to convert a working class and script I already have to work in Anvil. I’ve run into two challenges that I think should be straightforward but haven’t found a good solution:

  1. I have one file with a couple classes where one opens a JSON file on my file system and loads the JSON data into a class dict variable and a second class that does some custom processing. A separate script file instantiates the classes and initializes them with the desired input file and calls the subsequent class functions to process the json into another set of dicts/lists for output.

I haven’t been able to figure out how to create a server module for the class and instantiate it in a separate server module, so I ended up combining into a single server module. Is there any documentation for best approach to doing this in Anvil?

  1. Now that I have the classes and scripts together in one server module, I’m trying to load the data into my class from a media type in a table as an equivalent of:

def LoadData(self):
** with open(self.file_obj, encoding=‘latin1’) as ds_file:**
** self.ds_raw_data = json.load(ds_file)**

I’ve tried various approaches unsuccessfully, including:

self.file_obj = app_tables.userfile.get(id=file_id)['datafile']
self.file_obj_data = self.file_obj.get_bytes()
self.my_json_data = json.loads(self.file_obj_data, encoding='latin1')

With the above, I get: TypeError: ‘NoneType’ object is not iterable

Other varied errors along the way:
TypeError: expected str, bytes or os.PathLike object, not LazyMedia
and with other variations at a later step: ‘LazyMedia’ object is not iterable

What is the best way to load from a media type into a dict? I’m guessing there is some additional decoding or deserialization missing.

Thanks!

Creating two server modules and importing one from the other should work without problems. What error message are you getting?

You could use simple object columns instead if media columns, so there is no need to convert to and from json.

Thanks for the suggestions.

I am already starting with a json file that I am uploading from my local system, which is a feature I need to include. My existing scripts are pretty simple and work correctly, so ideally, I would like to handle opening and loading the json as simply as possible.

If you write the json object to a simple object column instead of a media column everything should work smoothly.

Can you show the code you are using to manage your json data?

Thanks, I appreciate your help.

I’m starting with an existing JSON file that I need to load for processing. What I have in my original script that results in an iterable dict is essentially as follows (exerpted):

class DataSource:
def __init__(self, name, filetype, fullpath_filename):
    self.name = name
    self.filetype = filetype
    self.filename = fullpath_filename
    self.ds_raw_data = None

def LoadData(self):
       with open(self.filename, encoding='latin1') as ds_file:
          self.ds_raw_data = json.load(ds_file)

       for idx, event_item in enumerate(self.ds_raw_data):
            if event_item['host'] == self.ds_target_endpoint:
                self.ds_matched_endpoint_data[idx] = event_item

other processing follows, but I’m trying to get the initial self.ds_raw_data.

Thanks.

I don’t understand what you mean by loading a json file.

I made this example app with:

  • a file uploader that allows you to select a file from your computer, upload it, convert the text to json and save it to a Simple Object column
  • a button that allows you to get the json value from the database and show it in an alert.

I created a server module to write and read from the database rather than giving read and write access to the form because working with tables directly from the form is possible, but is unsafe and slower than accessing the database from the server and returning only the required data.

https://anvil.works/build#clone:JFNX5HTD6VZSSQAC=JPXJKMTFMP2XVGG7VA2X5OJR
The table has two columns key and value. The app uses the text "key" for both writing to and reading from the table.

2 Likes

Hi Stefano,

Sorry for the late reply and thank you for your response and example.

I have a file on my local system with contents already in json format. I wanted to be able to upload the entire file to the server and the file could potentially be very large so there is a possibility that it may work better to store the file in S3, so my thought was that I would initially try to store the file as media to see how that worked.

It may make sense for now to adapt my components to match your example and create the json without uploading the file. I expect it should work well.

Thank you!