How to upload a pickle file from computer into server so that I can load it in my script?

Anvil Q&A
#1

What I’m trying to do:
I would like to use a pickle file in my script the way I use it on my local computer. The script code looks like this:
with open(r"cv.pickle", ‘rb’) as f:
cv = pickle.load(f)

What I’ve tried and what’s not working:
I’ve tried different code snippets from a variety of forum posts regarding loading files into anvil but I think I am fundamentally misunderstanding the process of pushing files into the server and how to then access them. For example, I tried adding a file loader to my app with uplink code like so:
@anvil.server.callable
def uplink_download(media):
with open(“cv.pickle”, “wb”) as f:
f.write(media.get_bytes())
anvil.server.wait_forever()

And uploading the pickle file that way, but I then get stuck on where the file is stored or how to access it within the server. I also tried to create a media data table and upload the file there but I again am unable to access it from my server.

#2

Can you provide a clone link?

It’s important to differentiate between “files” and Anvil “Media” objects. Files are bytes objects stored usually on a disk somewhere, and Media objects are Python objects that represent a file that can be passed between Anvil Server and Anvil Client. A ‘pickle’ file is just a special type of a regular file object. An Anvil Media object can be made from a pickle file.

If you are uploading your file in Anvil Client (in this case a pickle file) through an Anvil file uploader, then you are dealing with a Media object. It looks like you are then passing that media object to an Anvil server callable function, and want to reload that object.

Basically you need to convert your Anvil media object back into a basic file object so you can then load it using pickle. If you have the personal plan, you can write your media object as a temp file:

# Warning, untested code!
@anvil.server.callable
def uplink_download(media):
    # Convert to a regular file
    with anvil.media.TempFile(media_object) as file_name:
        # Treat file_name as a normal file:
        with open(file_name, 'rb') as f:
            cv = pickle.load(f)

If you are not on the personal plan (or if you want to just keep stuff in memory) you need to use the io library, something like:

# Warning, untested code!
import io
@anvil.server.callable
def uplink_download(media):
    # Convert to a regular file
    buffer = io.BytesIO(media.get_bytes())
    buffer.seek(0)
    cv = pickle.load(buffer)

I never get this stuff right the first try so these almost certainly don’t work. If you want to provide a clone link I could test.

#3

Hi,

I’m not really sure I can send a clone link as I basically have nothing set up right now, just failed attempts at different code snippets. What I want to do is add a permanent file into the back-end of my server that my code will be able to access. I want to ultimately remove the file uploader component from my form and have my app’s backend be able to access the pickle file to as an input (my code will ultimately use the pickled data as an nlp model). From my research it seems that I can’t just directly upload a file into the server, I have to create this front end solution that takes my file and converts it into a media object (or, it seems from your description, we want to convert the uploaded file into a basic file object actually). Is this understanding accurate?

Based on this thought process, I need a mechanism that stores the file within my server and will end up actually deleting the uplink_download function.

#4

One (just one of many) way to ‘upload’ a file for later use would be to create a new data table, you could name it for example my_files or something.

Then create two columns, a text column, and a media column, with file_name and…

actually Ill just create a clone link for you, give me a minute.

#5

Okay, you can then load the media object into a data table: Anvil Docs | Storing Data in Data Tables

You can just upload directly into a data table you create and get the media object using something like:

media = app_tables.my_table.get()['media_column'] # assumes a single row exists in table
# now media is a Media object so you can treat accordingly
1 Like
#6

https://anvil.works/build#clone:7R5PGLWH6S6U3VNJ=X3XXNZC7AASNPR2QTCPQ24IB

file_upload

In a server module:


def get_file_from_datatable_as_media_object(file_name):
  media_object = app_tables.my_files.get(file_name=file_name)
  
  if media_object is not None:
    return media_object['my_file']
  
  raise FileNotFoundError
1 Like
#7

Thank you for these instructions, they’re super helpful!
I followed your guide and am running this code:
cv = app_tables.table_1.get(text=“cv”)[‘media’]
## load model
with open(cv, ‘rb’) as f:
cv = pickle.load(f)
Getting the error below, I think the loading of the actual pickle file is causing an issue-- I think I am probably missing a line of two of code that should prepare the pickle to be loaded in correctly?
TypeError: expected str, bytes or os.PathLike object, not LazyMedia

#8

It returns a media object, not a file (yet).

I can give you the code that works, but if you have more questions about media files, you should search the forum or docs.

Try just:

cv = pickle.load(cv.get_bytes())

instead of the

part.

#9

The bytes object is not the correct format for this-- the types of media files I need are:
<class ‘sklearn.feature_extraction.text.CountVectorizer’>
<class ‘sklearn.feature_extraction.text.TfidfTransformer’>
<class ‘sklearn.linear_model.logistic.LogisticRegression’>

Sorry if this is elementary stuff, but I’m not able to find any documentation on how to load these types of formats into anvil. I’m also not seeing a way to transform a byte/string object into those.

FYI, this is what I was able to get working:

cv= app_tables.table_1.get(text="cv")['media']
with open("cv_temp.pickle", "wb") as file:
  pickle.dump(cv.get_bytes(), file)

with open("cv_temp.pickle", "rb") as file:
    cv= pickle.load(file)
#10

I suspect you’re going to need to provide error trace-backs and a clone link to move forward here.

You reference three python object types (all from sklearn). These are Python objects. Bytes are… bytes. I’m not sure what you mean by “bytes object”. The pickle library lets you read bytes and reload the pickled Python object.

Per ianbuywise’s answer, You can get the bytes stream from a media object using the get_bytes, and probably read that using pickle.loads.

1 Like
#11

Yes, it looks like I dropped the s when unpickling.

Example:
image