Creating a Job Queue

[simon.antonelli]

Hi there,

I’ve been working on an automated video solution app which is coming along since thanks to Anvil

Lately, I’ve been struggling with the process of creating a job queue, so that multiple users entering information at the same time don’t clog up the process flow. It might be that I’m still somewhat of a novice when it comes to programming and I’m not 100 percent sure what workflow I should use.

My think has been to set this as a background task?

Code Sample:

# code snippet

Clone link:
share a copy of your app

[james2]

I’m a novice myself but I think the answer depends on what the users are entering and how soon they need an output from that response. One solution could be to use the Anvil scheduled tasks:https://anvil.works/docs/background-tasks/scheduled-tasks

You could one poll a table for jobs and process one at a time? Again I think it depends on the what.

[simon.antonelli]

@james2

e poll a table for jobs and process one at

Thanks for your reply, the idea was that the task could be queued as they get entered remotely, render on the server side and jump onto the next one.
That of using a poll could be good, I’ll give that a look

[ianb]

In this post to a different question I showed how one might lay out a requests table, using unix timestamps and user logins. (You could substitute session ID numbers if you did not want to have users create a login)

How to get media generated from background task to the client side

Great, as long as you don’t go over the row limit for the free plan with users or any other limits if you are on the paid plans then it should work forever. (Most use cases will not see 50000 or 150000+ different users but who knows) If someone else is reading this later and is interested in how to clean up the table (delete old rows) here is some code: from datetime import datetime, timedelta import anvil.tz when creating a new request for your media object and inserting it into a new row fo…

[simon.antonelli]

@ianb

one might lay out a requests table, using unix timestamps and user logins. (You could substitute session ID numbers if you did not wa

Apologies for the late reply. Thanks so much for sharing this! I think this might be the answer to a few things I’ve been struggling with, I’m going to give this a go

[p.colbert]

I tend to think of a “request” or “job” via a finite-state machine.

• It starts in state “pending”.
• When an appropriate processing engine takes it up, we set it explicitly goes to “accepted” (so that no other processing engine tries to pick it up).
• When the job is completed, we set it to “completed”, so that other code knows it’s ready.

This “state” variable would be a column in the “requests” table, or could be inferred from other fields. For example, if you have a timestamp for “accepted”, and another for “completed”, both starting out None, then you might not need a separate “status” column.

Edit: You may find a need for additional states, e.g., error conditions discovered along the way, and associated information. For these cases, you may not know, in advance, the size or shape of the diagnostic information. Then a SimpleObject column makes a nice catch-all.

[ianb]

Here is another mock-example with @p.colbert 's concepts in mind to give others some more ideas: