Error storing a pickled object to database

Anvil Q&A
#1

I have a complex object with multiple nested dicts and lists which I’d like to save to the database.

I’ve tried pickling the object which seems to work but then get an error when setting/updating the database column:

anvil.server.SerializationError: Cannot serialize arguments to function. Cannot serialize <class ‘bytes’> object at msg[‘kwargs’][‘tl_factory_object_media’]

I’ve tried using both a simple object data type column as well as media. I suspect the object is either too big or not fully pickled, if that is possible, maybe due to the nested structure.

Any suggestions and/or clarification would be appreciated.

I’m also considering trying to write it out to a file and storing the file as media.

Thanks!

1 Like
#2

Hi @hasib, sounds complex!

Do you have a clone link you could share that is a minimal example of the problem you’re facing?

I presume this is all server-side stuff?

For simple object - I think better would be to do a json.dumps because a simple object is really a json object which is text-based whereas a pickle object probably would not be json enough for simple object table storage hence the serialization error…

Depending on how deep the rabbit hole goes you can store media objects in media columns…

#3

Hi,

Thanks for the response.

I don’t have a minimal example handy yet, but the object I’ve split out looks like this:

class TLFactoryDataObject:
    def __init__(self):
        self.default_event = {}
        self.tl_factory_out = {'events':[]}
        self.tl_csv_out = []
        self.config_filename = ''
        self.config_ds_group_key = []           
        self.config_ds_group_label = []
        self.factory_status = ''
        self.tl_grid_columns = []
        self.tl_grid_data = []

I’m thinking that to use.json.dumps it would all have to be a single dict, correct?

Thanks!

#4

And, yes, all server side.

#5

Correct - which I believe is the case for storing a simple object…

#6

Are you trying to store all of the above in the table?

What are the types of each item e.g.

self.config_ds_group_label = [] # List[str] or List[int]

And which object is breaking in storage?

#7

This makes me think that a string must be decoded.
Are you using python 2.7?

#8

I can’t tell which object is breaking.

self.config_ds_group_label is a list of dicts with each item having a couple [str] and an [int].

#9

I’m using Full Python 3.

I’m even trying just serializing a single dict and error occurs on the last line:

SessionTLDataList = [SessionTLData.tl_factory_out]
serial_SessionTLData = pickle.dumps( SessionTLDataList, protocol=pickle.HIGHEST_PROTOCOL )
data_files_row = app_tables.data_files.get(file_key=file_key)
data_files_row.update(tl_factory_object_media=serial_SessionTLData, file_process_indicator=“processed_TL2”)
#ALTERNATE: data_files_row[‘tl_factory_object’] = serial_SessionTLData

When I print serial_SessionTLData it does look pickled and there is no pickle error but errors on the update or the alternate method.

Thanks.

#10

Is this a Media column?
If this is the case, then you need to create a BlobMedia, not a json or a pickled string.
If you want to store a json structure you can create a Simple Object column, so you don’t even need to convert it to json.

1 Like
#11

Hi Stefano,

Yes, I now see that file should be a media type. This was a second column I was trying as I was thinking of trying to write to a file and storing to this column.
The original column is a simple object and I was getting the same error–I’ll try again.

Thanks!

1 Like
#12

Hi @stucork,

Here is a link to a minimal example with instructions on the page:

https://anvil.works/build#clone:WFMVHBDXBNAXWTIW=UR4BOUJSE4SU7WLUIQM6PDUD

I’ve tried a number of variations based on the suggestions and haven’t been able to spot what’s wrong.

Thanks in advance!

#13

ha - that doesn’t feel very minimal! I’ll have a look at some point soon and see how I get on!

#14

LOL–sorry! minimal compared to what I removed.

It is runSavePickleProcess() in the ServerModule_v0 (at bottom).

#15

This is the solution I needed!

Thanks!

1 Like
#16

phew :sweat_smile: That code was intimidating… glad you found a solution!