Uploading MP3 audio to AWS S3

[koontz2k4]

What I’m trying to do:
I’m trying to upload an mp3 file from my Anvil app into my data table and an S3 bucket.

The uploader is working correctly.

The mp3 is making it to my data table correctly.

The issue is when I try to send to S3. I’m getting the below error, and I’m not understanding why it’s telling me that there is no such file. It was uploaded successfully to the data table a few seconds before.

anvil.tables.TableError: Column ‘audio’ can only be searched with a media (not a string) in table ‘uploads’

What I’ve tried and what’s not working:
I’m uploading the file from my desktop, using the file uploader.

Code Sample:

@anvil.server.callable
#The background server function that allows me to get around the 30 second server timeout issue
def upload_file(file):
  with anvil.media.TempFile(file) as file_name:
    if file.content_type == 'audio/mpeg':
      app_tables.uploads.add_row(audio=file) #Line of code to send the MP3 file to the data table.
      upload_to_s3(file)
    else:
      print("error")
    pass
  
  
**#Upload the mp3 file to the S3 bucket
def upload_to_s3(file):
  s3_client = boto3.resource('s3')
  s3_client.meta.client.upload_file(app_tables.uploads.search(audio='wspy newscast.mp3'), 'wspyradio', file)
  

Clone link:
share a copy of your app

[shaun]

Hi @koontz2k4

This error

anvil.tables.TableError: Column ‘audio’ can only be searched with a media (not a string) in table ‘uploads’

Is happening because of this line:

s3_client.meta.client.upload_file(app_tables.uploads.search(audio='wspy newscast.mp3'), 'wspyradio', file)

In particular, it’s this bit

app_tables.uploads.search(audio='wspy newscast.mp3')

The audio column is a Media column, it stores a Media Object. When you’re doing search(audio='wspy newscast.mp3'), you’re trying to search for a particular row in that table. But you’re trying to match the contents of the audio column to a string ('wspy newscast.mp3'). This fails because the column contains Media Objects rather than strings.

One solution would be to store the name of the file in a separate column, called say filename.

      # in the upload_file function
      app_tables.uploads.add_row(filename=file.name, audio=file)

Then when you search, you can do this

app_tables.uploads.search(filename='wspy newscast.mp3')

I’m not 100% sure if your media object will have a .name - if not, you’ll have to invent one and keep track of it.

EDIT: Although Boto3’s upload_file actually wants the first argument to be the path to a file on disk, so I think you can use TempFile again to get that to work:

# Get the row from the Data Table
row = app_tables.uploads.search(filename='wspy newscast.mp3')[0]
# Create a temporary file on disk so Boto3 can see it
with anvil.media.TempFile(row["audio"]) as file_name:
  # Now we have to the path to the file on disk as 
  # the file_name variable, so...
  s3_client.meta.client.upload_file(file_name, 'wspyradio', file)

[koontz2k4]

Gotcha, that makes perfect sense. There will be a <.name>, that worked perfectly. However, now I’m getting another error. Any idea why?

@anvil.server.callable
#The background server function that allows me to get around the 30 second server timeout issue
def upload_file(file):
  with anvil.media.TempFile(file) as file_name:
    if file.content_type == 'audio/mpeg':
      app_tables.uploads.add_row(filename=file.name, audio=file)
      #app_tables.uploads.add_row(audio=file) #Line of code to send the MP3 file to the data table.
      upload_to_s3(file)
    else:
      pass 
    pass

#Upload the mp3 file to the S3 bucket
def upload_to_s3(file):
  s3_client = boto3.resource('s3')
  row = app_tables.uploads.search(filename='wspy newscast.mp3')
  with anvil.media.TempFile(row["audio"]) as file_name:
    s3_client.meta.client.upload_file(file_name, 'wspyradio', file)

[shaun]

I think it’s because app_tables.uploads.search(filename='wspy newscast.mp3') actually returns an iterator of rows, rather than just a single row.

So actually you want to put a [0] after the search call:

#Upload the mp3 file to the S3 bucket
def upload_to_s3(file):
  s3_client = boto3.resource('s3')
  row = app_tables.uploads.search(filename='wspy newscast.mp3')[0]
  with anvil.media.TempFile(row["audio"]) as file_name:
    s3_client.meta.client.upload_file(file_name, 'wspyradio', file)

My bad, I got it wrong in the original example! I’ll go back and fix it.

[koontz2k4]

No worries. I made that change, but now it’s throwing the below error. When I print the value of “file_name” I get “/tmp/n0e1ffe9t8vrgnufzpz8zd3rqbmb94yv”.

Could it be the “/” causing the issue?

    
#Upload the mp3 file to the S3 bucket
def upload_to_s3(file):
  s3_client = boto3.resource('s3')
  row = app_tables.uploads.search(filename='wspy newscast.mp3')[0]
  with anvil.media.TempFile(row["audio"]) as file_name:
    s3_client.meta.client.upload_file(file_name, 'wspyradio', file)

ParamValidationError: Parameter validation failed: Invalid type for parameter Key, value: <anvil._serialise.StreamingMedia object at 0x7fa018f82b50>, type: <class ‘anvil._serialise.StreamingMedia’>, valid types: <class ‘str’> at /usr/local/lib/python3.7/site-packages/botocore/validate.py, line 360 called from /usr/local/lib/python3.7/site-packages/botocore/client.py, line 729 called from /usr/local/lib/python3.7/site-packages/botocore/client.py, line 681 called from /usr/local/lib/python3.7/site-packages/botocore/client.py, line 388 called from /usr/local/lib/python3.7/site-packages/s3transfer/upload.py, line 694 called from /usr/local/lib/python3.7/site-packages/s3transfer/tasks.py, line 150 called from /usr/local/lib/python3.7/site-packages/s3transfer/tasks.py, line 126 called from /usr/local/lib/python3.7/site-packages/s3transfer/futures.py, line 265 called from /usr/local/lib/python3.7/site-packages/s3transfer/futures.py, line 106 called from /usr/local/lib/python3.7/site-packages/boto3/s3/transfer.py, line 279 called from /usr/local/lib/python3.7/site-packages/boto3/s3/inject.py, line 132 called from [ServerModule1, line 39](javascript:void(0)) called from [ServerModule1, line 28](javascript:void(0)) called from [WSPY, line 25](javascript:void(0))

[shaun]

That’s progress - the [0] did fix the error you had before.

The new error is because the final argument to upload_file should be a string, but you’re passing file which is a Media Object.

The final argument is the S3 Key which is S3’s equivalent of a filename. So if you pass something like "path/to/my/file/" + row["filename"] then it should work

s3_client.meta.client.upload_file(
  file_name,
  'wspyradio',
  "path/to/my/file" + row["filename"],
)

Although S3 might not like spaces in keys so perhaps "path/to/my/file" + row["filename"].replace(" ", "-")

[koontz2k4]

Yep, good progress! Just so I’m clear, when you say “path/to/my/file/” + row[“filename”]", you mean
lhe location in the datatable, correct?

I’m just thinking that if it’s the location on the desktop, it’s somewhat defeating the purpose of the file uploader.

[shaun]

By path/to/my/file/ I just mean whatever path you want the file to have inside S3… it doesn’t need to be any particular path. That third argument is telling Boto where to put the file within S3.

[koontz2k4]

Got it. I’m getting an “Access Denied” error, although I think that’s an S3 issue, and not an Anvil one. I made sure to make it public, so I’m not sure why my access is being denied. I’ll keep messing with it. I appreciate your help!

S3UploadFailedError: Failed to upload /tmp/zk6t4swzekrs9zvqu5jrug4bc4xn3t36 to wspyradio/wspy-newscast.mp3: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied at /usr/local/lib/python3.7/site-packages/boto3/s3/transfer.py, line 295 called from /usr/local/lib/python3.7/site-packages/boto3/s3/inject.py, line 132 called from [ServerModule1, line 46](javascript:void(0)) called from [ServerModule1, line 29](javascript:void(0)) called from [WSPY, line 25](javascript:void(0))

[shaun]

IIRC there are two kinds of S3 permission and you’ve just opened up one of them. There might be somewhere else to set the bucket as publicly writable or something.

Setting everything public is a good short-term solution to debug whether permissions are the problem, but you should definitely lock the permissions down if you’re going to make this app public.

You can log Boto in to AWS using your secret key, then it will be able to do anything your account can.

client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY_ID,
    aws_secret_access_key=SECRET_KEY,
)

See Boto docs

You can store your secret key in Anvil Secrets - they’re encrypted, so only you and your app can read them.

[koontz2k4]

Yep, I hear you. I won’t leave it publicly open for long. I’d just like to get it working at all first. S3 is a bit frustrating at the moment, I’ve never had this much difficulty with it before.

[shaun]

I think it’s a case of logging Boto in with your key… just below the public access bit you showed, there’s this bit that configures read/write access on a per-account basis.

I think the default permissions are read,write for your account, and all blocked for the public.

[koontz2k4]

I see the issue. The s3_client is getting created correctly.

But later on in the code, when I call the function that uploads the file to s3, their is another line of code that creates another client, but without the credentials, so that’s why the put object was getting blocked. I got rid of the the highlighted line, and it resolved that issue.

And then, I had to the change the s3_client defintion from <.client> to <.resource.>

And now, it’s working as expected. Thank you VERY MUCH for all of your help @shaun!! I really appreciate your time!

-Ryan