How do I halt a background task temporarily?

Anvil Q&A
1

Hey people,
I have a project in which I am launching way too many background tasks all at once but as it turns out, because I’m only on a Personal plan, the server gives an ExecutionTerminatedError when a lot of those processes are running altogether.

An alternative I figured out is that I can see how many background tasks are there that are currently running, and if the number exceeds a certain threshold, I’ll halt the background task for a while until the backlog is cleared using time.sleep().
However, even that doesn’t work as using list_background_tasks() even returns the background tasks that have reached an end state (either completed, or failed), and so the program stops when there are 0 currently running tasks, but (say) 5 dead tasks. I tried running it through a parsing function that returns only the currently running tasks, but even that process overwhelms the server and it gives the ExecutionTerminatedError on the currently running tasks, and the task I’m trying to start.

So, I guess I’m asking, short of increasing the server quota, is there any other way of managing way too many background tasks all at once? Or is there a way to “queue” a background process such that it runs only when the backlog is finished, without making the server crash my apps?

2

Rather than pausing them, you should wait before starting them.

You could keep track of how many tasks are running and, if you already have 5, add a row to a table that lists the pending tasks and do nothing.

When a tasks ends, it sets the status column of its row to “completed”, then looks for the next row with status “pending” and if it finds one, it works on it.

This way you have a limited number of tasks running.

Using a table as the manager of the task queue and wrapping the queries for searching and setting the status in a transaction, will make your tasks “thread safe”. That is two tasks running in parallel will not find the same pending next task.

Using a table will also allow you not to use list_background_tasks(), which last time I checked was very very slow. I had thousands of tasks in the list and it was taking seconds every time I called it. This was long time ago. Things may have changed today, but since it’s difficult to test, I will stick with a table.

2 Likes
3

That seems like a very good way to do this process, but in a multi tenancy app with limited resources, I may end up exhausting the data tables limit on my plan.
Could I maybe store the tasks in a Google Sheet? I know it’d be a hell of a lot slower, but it may get the job done without consuming all the resources.

Or, is there another way to filter the running tasks?

4

If you delete the completed task rows every night, the number of task rows shouldn’t increase too much, unless you have thousands of concurrently pending tasks.

Interacting with a Google sheet containing thousands of concurrently pending task rows could be really slow, but it will work.

A better alternative would be one table with one simple object column, with one row where you store the whole list. This trick will take only one row away from your quota. (I don’t know if the quota includes limits in size.)

1 Like
5

I can assure you that the number of data table rows you can consume is far fewer than the number of tasks you’re going to be able to manage in memory at once! (And yes, there’s a 4MB size limit on Simple Objects, plus all the management of concurrent updates that would be required - so I’d advise against trying that.)

(And if you’re just using the data table as a queue of work - ie the results of your BG tasks are stored elsewhere in your database, so you can delete the row as soon as you start a background task to deal with it, or as soon as it’s finished - then you’re fine! If you have the ability to adopt that pattern, it’s a great one.)

2 Likes
6

I would try this, no doubt but my background tasks also store multiple media objects (PDFs and .txt files) and I feel that retrieval of these files will become difficult if I store them in a single simple object column.

I believe you, but at the same time, I may be able to store hundreds of pending BG tasks’ information in sheets or databases but the server quits the app when it has about 10-15 tasks concurrently running right now (in my debug environment, atleast). Is it that the server might quit after 10-15 tasks are launched in the same session?

As I’m typing this out, I have one more question. Can I store the bytes of media objects in simple object columns (given that the data does not exceed 4 MB)

7

Your question was about how to halt a background task, and you got the answer.

Now you are asking another question: how to store pdf files and text files, which has nothing to do with the previous one. This should be a new post.

Storing pdf files or text files has nothing to do with how they were generated. Whether it was a background task or not, it makes no difference.

Simple object columns store JSON data, and JSON data doesn’t allow to store files. You could encode the file in a text base format, so you can store it in a JSON object as a string, then decode it when you access it. The encoding will add a little overhead and will make the file larger.

So, yes, you can do it, but there are better ways, like a media column. If you don’t want to use Anvil database rows, you can use any external storage system.

1 Like
8

I tried this method out, and largely, this works very well.

Do you have any ideas how to prevent multiple background tasks from launching the same pending task multiple times?

This is a partial code I’ve written of a function that does this process.

@tables.in_transaction
@anvil.server.background_task
def foo(*args, pending=False, row=None, **kwargs):
    if pending:
        print("This is a previously pending task")
        print(row['task_id'])
    else:
        task_id = shortuuid.uuid()
    
        if len(app_tables.tasks.search(state='running')) >= 3:
            app_tables.tasks.add_row(task_id=task_id, task_name='foo'+task_id, state='pending', arguments={'args': args}, keyword_arguments=kwargs)
            print("Too many tasks running right. Task ID = " + task_id)
            return
        
        row = app_tables.tasks.add_row(task_name='foo'+task_id, state='running', arguments={'args': args}, keyword_arguments=kwargs)
        print("Added task to tasks table.")

    print("Executing function")
    time.sleep(10)
    print("Execution completed")

    
    row.delete()
    print("Row deleted")

    try:
        new_task = app_tables.tasks.search(state='pending')[0]
        if new_task:
            time.sleep(2)
            print("Pending task found. Starting task with ID:", new_task['task_id'])
            new_task['state'] = 'running'
            anvil.server.launch_background_task('foo', new_task['arguments'], pending=True, row=new_task, **{'task': 'previously pending'})
    except IndexError:
        return 'No pending tasks available'

Right now, this is linked to a click of the button, so when you click the button multiple times at once, it does store the extra functions in the data table as ‘pending’, but when the times comes of running the pending tasks, multiple functions kind of start the same function multiple times.

Here’s a demo of the app if you wanna see it for yourself: Anvil | Login

9

I didn’t clone or carefully analyze the question, but at a glance I see two problems:

  • A background task is decorated with @in_transaction. This will keep parts of the database locked as long as the task runs, maybe even more than just the rows used by the task. Do NOT do that! Keep the transactions as short as possible.
  • The background task is creating and deleting the rows, so there is no way for the other background tasks to know about them.

The rows should be created before launching the task and deleted after the task has ended, not inside the task. A start_task() function should create a row with the status column set to 'pending' and all the values required by the task in other columns, then launch the background task.

The background task should:

  1. Count the number of rows with status set to 'processing', and if it’s more than X, get out
  2. Search for the first row with status set to 'pending', and if it doesn’t find any, get out
  3. Get that row, set status to 'processing', get the values of the arguments from the row, and start working
  4. When it’s done, set the row status to 'completed' (and maybe another column to the returned value)
  5. Restart from point 1

This will keep the oldest X tasks running as long as there are X or more 'pending' rows, and all the new background tasks will exit immediately.

Keeping steps 1 through 3 inside their own @in_transaction decorated function will ensure that the same pending task is processed only once.

1 Like
10

The steps you mentioned are there, but perhaps what’s different is the in_transaction decorator. I’ll try that.

11

I see app_tables.tasks.add_row inside the background task. This will never work.

1 Like
12

It does work, and the function does recognize that there are pending tasks inside the table as well, but what’s happening is that if this is only one pending task and three running tasks, then after completion, all three running tasks launch the same pending task. If there are three pending tasks and three running ones, all three launch the first pending task first, and after the three iterations of the first pending task are completed, only then do they launch the second pending task.

13

You say it does work, then you explain that it doesn’t.

If you follow my directions it will work.

1 Like
14

No, I meant that using app_tables.tasks.add_row works, I wasn’t talking about the later part of the function.

15

As @stefano.menci points out, it is important to separate the management of the task queue from the execution of the tasks. These are two distinct responsibilities.

  1. The queue manager code can see the entire state of the queue, and so it can prevent queueing up redundant tasks. (It could even apply priorities, or re-schedule a task that has failed.)
  2. Doing the above outside the task should simplify the code that executes the task, as that code now has just one responsibility: get the task done.
1 Like
16

Ok so this is how I solved my problem.

Using @stefano.menci’s ideas, I changed that function to store the data of each task in a data table. If there are X number of “processing” tasks or more in the data table, then set the new task as “pending” and exit the function, storing all the arguments and whatnot in the data table itself.

When a “processing” task is completed, it’s status is set to “completed” and it is removed from the data table entirely.

Then I had a scheduled task running every 15 minutes to execute a specific number of “pending” tasks depending on how many maximum tasks I can run together, and the number of “processing” tasks.

That works very well for my use case, but if you guys foresee any problems with it, let me know.

Edit: I used a separate function to add the tasks to the data table, one that had the @tables.in_transaction decorator and removed that decorator from my original function.

1 Like
17

Sometimes it can be handy to reserve a column (or two) in the table to hold the task’s result. Then the result is still accessible even if other things don’t go as planned…

18

This is unnecessarily complex and can hide some bugs. It’s how my task manager was originally working, but was sometimes slow, because sometimes there were pending rows waiting for 14 minutes, and sometimes the logic was breaking because of race conditions or other special cases, and it was processing more than X tasks.

Since I changed the logic a described in my previous post, I got rid of scheduled tasks, simplified the logic and got rid of all random problems that would occasionally pop up.