[FIXED] Intermittent Timeout in Background Task on Paid Account

Hi,

I am on a paid plan and am having an issue due to a background task that sometimes times out.

Twice a day, at scheduled times (via scheduled tasks), my application runs a background process that downloads data from AWS, processes it, and then sends out an email of the results.

About half of the time everything works fine. The other half of the time I get a timeout error.

For example, my scheduled task ran just fine this morning at 7:01, but it failed yesterday at 20:01. Below are the logs. Interestingly, the one that passed took 3 minutes and 23 to complete. The one that failed timed-out after just 47 seconds.

Please know that anytime I run the background task manually - via a button in the app - it runs fine and completes with no timeouts. The timeouts only occur when the background task is run via a scheduled task.

Could someone please help me with this? I have been searching the web for a solution but with no luck.

Thanks,
Matt


Thu 21 May 2020, 07:01
Launched Scheduled Task ( download_and_process_data )
getting data from s3
getting data from s3
Getting data from dynamodb
getting data from s3
saving data
Sending email
Done
Session ended Thu 21 May 2020, 07:04:23

Wed 20 May 2020, 20:01
*Launched Scheduled Task ( download_and_process_data )
getting data from s3
Session ended Wed 20 May 2020, 20:01:47
anvil.server.TimeoutError: Server code took too long|

This continues to happen. Yesterday the scheduled task ran fine at 20:00 but failed this morning at 07:00, after about the same amount of running time (1 minute and 40 seconds). I’m continuing to try and problem solve and the only thing I can think of is that the connection to AWS is somehow contributing to the timeout. I’m not sure why it would only cause timeouts when the background task is being run via a scheduled task and never cause it when the background task is being run via the client, and even then, only about 1/2 of the time.

Has anyone else had a similar experience? Right now I am using Boto3 for the AWS calls.

I attempted to fix the problem by replacing Boto3 with a web service I created. But I got the same strange result. I even tried multiple web service calls, each doing a fraction of the work so that it would return more quickly, but same result - unpredictable timeouts, and only when being run via scheduled task, never when being executed via the client.

I am fairly new to Anvil so appreciate any ideas or guidance with this.

Does your background task call a server function and it is the server function that times out?

1 Like

Hi stu, thanks for the reply.

Right now, I’m using Boto3 for pulling data from S3 and DynamoDB. It is certainly possible that it is timing out but I’m not seeing any ConnectTimeout errors, only anvil.server.TimeoutErrors. Any idea why it would occur when the background service is run via a scheduled task (about 1/2 the time) but never when the background service is run via the client?

Hi @stu, I think I misunderstood your question. To give a better answer, my background task does not call any functions that are decorated as @anvil.server.callable. .

I was curious why I’m seeing an anvil.server.TimeoutError being generated from a background task. As far as I can tell, the context appears as ‘server_module’ both in background functions and in server functions (at least as report by anvil.server.context.type).

My best working hypothesis right now is that if the background function is waiting too long for Boto3 to complete an operation - in this case, get a list of all filenames in a large bucket via s3.get_paginator('list_objects_v2') - it times out despite the fact that it is a background task on a paid plan. I don’t have high confidence in this, but continue to try and figure it out.

This was confirmed as an Anvil bug, and fixed last week!