Another incident?

Anvil Q&A
1

Was there another ops incident around 11:10 UTC today (3 June)? API calls to do simple mathematical operation were timing out with “server code took too long”

2

Hi @simon.alger, does the error show after 30s or a few secs? Thanks

3

@Tony.Nguyen I’d say closer to 30 seconds and in the App Logs I see:

anvil.server.TimeoutError: Server code took too long

4

The API of one of my apps was down for 3 and a half hours this morning.

@centricengineer team - this being a developer tool, is there a status page to be informed of outages? You would probably get less of these posts on the forum if there was, and I’m sure the transparency would be appreciated by your paying customers (and their customers as well!)

6 Likes
5

Does it happen before? Might be your simple mathematical operation involve a lot of query from the database?

6

The code in question is:

@anvil.server.http_endpoint('/square_it', methods=['GET'])
def square_it(num):
  return float(num)**2

It always runs in <1s (as you would expect), but earlier it was timing out. It was back to normal by about 11:20 UTC

1 Like
7

My users also notified me of errors with the API and the webapp today.

Just now I was also unable to open the Anvil development environment.

1 Like
8

I’m running into this error right now:
TableError: Failed to connect to database: Too many transactions already in progress.

We can’t have more than a handful of users on right now. Other (of our app’s) users are complaining of errors too.

9

I’m getting the same error as well.
TableError: Failed to connect to database: Too many transactions already in progress.

10

@tcarter Thank you for confirming the issue.

11

FWIW, I’m getting this too many transactions error as well.

This is related to a function that has not yet been released to my users. So I am the only individual calling this function, so it seems odd to me that I’d run into a too many transactions in progress error.

Here is the relevant message from the app log on the server when I use the client code to access this server call.

anvil.tables.TableError: Failed to connect to database: Too many transactions already in progress.

<running on the server>
  called from /downlink/anvil/tables/__init__.py, line 55
  called from /downlink/anvil/tables/__init__.py, line 73

Out of curiosity, I called this server function which uses the @anvil.tables.in_transaction decorator from an uplink script on my local machine and same results (unsurprisingly)

Here is the stack trace from my machine (I removed my local directory tree)

      Traceback (most recent call last):
        File "<...>/python3.7/site-packages/anvil/server.py", line 286, in call
          return _do_call(args, kwargs, fn_name=fn_name)
        File "<...>/lib/python3.7/site-packages/anvil/server.py", line 278, in _do_call
          return _threaded_server.do_call(args, kwargs, fn_name=fn_name, live_object=live_object)
        File "<...>/lib/python3.7/site-packages/anvil/_threaded_server.py", line 393, in do_call
          raise _server._deserialise_exception(r["error"])
      anvil.tables.TableError: 'Failed to connect to database: Too many transactions already in progress.'

Just to explicitly dot the i’s and cross the t’s; Is there any known reason why a transaction decorated sever call wouldn’t work as expected over the uplink ?

I can envision a world where the transaction manager might not play nice with all the implications of how the uplink service works, but I can also envision a world where the uplink services are just another client participating in a distributed transaction across multiple services.

12

+1

I’m also getting the following error:

anvil.tables.TableError: Failed to connect to database: Too many transactions already in progress.

My DB operations work on some pages but I consistently get the above error on the same page.

Thanks

13

Would be nice if someone from Anvil could chime in here. Were there known issues yesterday? Are these issues ongoing? We greatly appreciate all the hard work you put in.

3 Likes
14

@meredydd @ryan
Are we able to get an official update as to what happened here and more importantly, how it’s being mitigated in the future?

Thank you,

Jeff

3 Likes
15

Not even so much as a “we’re looking into it”. At this point you have to assume the team’s priorities are focused elsewhere…

16

It was mentioned on another post that there was “outage which has now been resolved”.

I understand that it’s a small team and so they’ll have to triage the support they offer, with the forum probably taking last place. I love the company’s philosophy in general, and the product’s implementation in particular, but if what’s being sold here is a highly-available managed hosting solution – as opposed to the open source app server – it’s critical to have transparency around downtime.

I applaud all the fantastic, ground-breaking work you folks at Anvil are doing.

1 Like
17

I think you can be sure that the anvil team’s top priority will always be maintaining stability for its customers. If there’s a lack a response it’s not because they’re busy doing something else it’s because they’re busy doing just that…

2 Likes
18

@stucork That may be the case (and I very much hope it is), but we can only judge by what we see. ‘We hear you, we care, we’re looking into it’ takes less than 30 seconds to post. A lot quicker than writing out the full newsletter I just received!

19

Reference for anyone stumbling across this thread in the future: A note on last week's incident

I think this one can now be closed.