Announcement: Anvil outage [Resolved]

Announcements
1

Hi everyone,

We’re currently experiencing an outage affecting several Anvil services, including the development environment and apps. I’m sorry for the inconvenience, and I’ll post here again as soon as we have an update for you. We expect to have everything back up and running again in a few minutes.

2 Likes
2

All services should now be back online. Thanks for your patience, and apologies again for the interruption.

2 Likes
3

Thank you! We still seem to be experiencing the outage unfortunately

4

Having trouble with Uplink - sometimes working sometimes not - sometimes within the same minute! …

5

Yes - me too. Uplink doing some odd things. It says off line sometimes but then my logs show that the uplink processed my request.

Opened a new topic here :

6

I had all of my uplinks restart after failed connections at the 27th minute of the last hour, but they are all fine since:
image
(my scripts exit out after getting the uplink connection error, and a scheduler restarts them every one minute if they are no longer running)

7

Hmm, there’s definitely still something odd going on with Uplinks. We are investigating, and will update here as soon as possible.

8

If anyone is able to post an App ID (these aren’t secret or sensitive) along with the name of a sometimes-misbehaving Uplink function, that would be really helpful. Thanks!

9

Sure - app id IR6VEW7CLXOC4N2Q

function name : anvil_to_redis_command

10

Thank you @david.wylie. Would it be safe for me to try calling that function for the purposes of debugging? I’m assuming that you expect the behaviour to be that sometimes it works, and other times you get an UplinkDisconnected error?

11

Sure, but it always returns an error, but sometimes the function runs successfully and sometimes not.

12

Some more context -

The full set of errors is this :

`anvil.server.UplinkDisconnected: Uplink disconnected`

* `at NumberStuff, line 458`
* `called from NumberStuff, line 412`
* `called from A_Numbers.Numbers, line 86`

`AppOfflineError: Connection to server failed (error)`

* `at A_Main.Main, line 75`

That is the report from when it “ran successfully” but errored.

13

Thank you - this is really helpful!

How frequently are you seeing the UplinkDisconnected error? I just tried calling the function 1000 times, and it ran successfully every time. If this was the most common type of Uplink issue, I would expect the failure rate to be more like 50%, so this is going to take some more investigation. Are you able to get it to fail at all now?

14

ok, yes it’s working now every time (though I’ve only tried a few times).

I’ve got a few number allocations to do so I’ll report back in a few minutes.

15

Thanks @david.wylie. It’s possible that this issue resolved itself when the last of an old set of server nodes was rotated out of service at about 18:00 UTC. Anyone still seeing the problem, please let us know here, including an App ID and function name if possible. Thanks!

2 Likes
17

A post was split to a new topic: anvil.server.TimeoutError after outage

18

Do we have the same problem again? Can’t connect to uplinks since two days ago

Admin note: Please don’t resurrect old announcement threads - make a new post in the Q&A forum!

1 Like