anvil.server.RuntimeUnavailableError: Downlink disconnected when confronted with 130 simultaneous requests

Anvil Q&A
1

I have a cms that is trying to send abt 130 simultaneous ish (not even really simultaneous just launched at the same time) http requests to the following set up)

import anvil.tables as tables
import anvil.tables.query as q
from anvil.tables import app_tables
import anvil.server


@anvil.server.http_endpoint("/trackeradd",cross_site_session=True)
def tracker_add(id, **params):
  try:
    req = anvil.server.request.form_params
    task = anvil.server.launch_background_task('backgroud_add',req=req)
    return "Successfully Sent To Background Proccess"
  except Exception as e:
    return (e) 

@anvil.server.background_task
def backgroud_add(req):
  try:
    row = app_tables.actives.add_row(levid=int(req['id']))
    return "Successfully Added To Tracker"
  except Exception as e:
    return (e)

I turned persistent server on and actually i modified the original script to send these as background tasks thinking that perhaps the insert may be slow? when i did that i was getting a server code taking too long error. I have done the exact same call on a flask app running on python anywhere and it handled it. Wondering what I am doing wrong here.

2

this worked for some reason ? is a try except really slowing this down that much ? or maybe did the server persistence just kick in ?

@anvil.server.http_endpoint("/trackeradd",cross_site_session=True)
def tracker_add(id, **params):
    req = anvil.server.request.form_params
    row = app_tables.actives.add_row(levid=int(req['id']))
3

Hi @scrockford,

“Persistent server not kicking in” would have been my first guess, but it should have kicked in immediately! A good diagnostic is to put a print() call in the main body of your module, so it prints when your module is loaded. If Persistent Server is working, you’ll only see that once at startup, then all the other calls will reuse the same loaded module. If Persistent Server is disabled, you’ll see it every time.

(Of course, the ideal is for you not to try to hit your app with 130 simultaneous requests, even on a Business plan, but I assume you’re aware of that and it’s a constraint from the external system.)

4

Actually, I just had a thought: is it perhaps failing only on the first time you hit the endpoint after changing your code? I wonder if hitting it with 130 simultaneous HTTP calls while the persistent server is trying to start is causing some performance issue? But if it’s already started up and handled one request, it can handle a burst without problems?

(To be clear, Anvil is designed to handle this sensibly, by queuing up the other 129 requests until the persistent server is loaded, but I could imagine a race condition where it tries to do some part of the initialisation 129 times and gets itself in a knot.)

5

Removing a try/except is not speeding up the code. The cause of the failure is somewhere else.

I once choked the server with 100ish calls to an uplink function. The function started a transaction, did a search and added a row in a table. 100 concurrent transactions started on an uplink were too many, and they started failing or timing out. I had not noticed it yet, when I received an email from the Anvil staff telling me that my database and all my apps were down.

They rebooted my dedicated server (I think).
I first made sure that bursts of calls wouldn’t happen, then I modified the app so the code inside the transaction would (1) run in the server rather than in a remote uplink and (2) it would be as short as possible.

Now I can manage bursts of 100 calls without problems.

Also other apps that had nothing to do with the 100 uplink calls stopped working. I’m mentioning this because you may need to investigate what else was happening. You have 130 functions trying to add rows to a table while… ? The persistent server kicking in could be a cause, but also other apps working with long transactions, or calling 100 other functions or …?

6

I made this cloneable app a while ago to show off background tasks and timers, you might modify its use to stress test your system / persistent server calls:

IIRC it had no problem launching 150 blocking background tasks, even without a persistent server.

1 Like