We recently provisioned a small Linux server for running some Anvil uplink scripts - At first just in tmux shells but have recently moved them to running in Docker containers.
Things have generally been stable, but I have been having a sporadic issue where uplinks will crash out on ‘anvil.server.connect(key)’ with:
ConnectionResetError: [Errno 104] Connection reset by peer
When this happens, I can’t get any uplinks to connect for a while, and the solution is just to leave them off for a while then restart.
I have two scripts going and a test container, the two scripts can be running and working fine, but I’ll try and start up the test container for something and it will throw that error, and if I restart the two other scripts they’ll also then die with the same error.
I can run the same scripts locally on my machine and they start up no problem (Same network).
The error suggests the connection is getting through but is being closed off on Anvil’s end - Any idea why this might be? I wasn’t sure originally if it was to do with restarting too many times while testing changes but today they all failed on first restart.
I have experienced it in both just running the scripts from a Linux shell as well as in a container, so my only guess is that for some reason we’re being blocked remotely for a time?
Hmm…my first suspicion here is that something in your server’s networking is shooting down those connections. What’s the difference beween the server’s connection to Anvil and your local machine? Is it perhaps on a corporate network with a too-clever-for-itself “firewall” appliance that might be shooting down those connections? (That specific answer is a wild guess, although it’s a common one - if it’s working from machine and not from the other I’d be trying my best to narrow down what differs between the two of them!)
Thanks for the reply.
We have checked the firewall - No obvious issue or blocking, the machines all go through the same firewall. I also installed wscat on the Linux server and was able to manually open a websocket to wss://anvil.works/uplink as well with no issue.
Oddly enough, the one thing that fixed this instantly was cycling out the uplink key. The error was happening when launching the script on the Linux server, it was not happening launching it on a local machine with the same key, but getting a new key in Anvil made the Linux server script work fine again.
Is there any sort of blocking on Anvil’s end for suspicious connections etc? I did a full restart on the Linux server, as well as recreated the containers from scratch and that didn’t seem to fix it, so I wouldn’t think it was an environment issue…Especially where the websocket error seems to suggest Anvil is slamming the phone down on the connection after it’s successful.
We actually had 2-3 different uplink keys (for different Anvil projects) all fail with this error at seemingly the same time. Running in different containers/tmux shells as well.
There’s only so much I can talk about abuse countermeasures on a public forum, but the symptoms you’re describing don’t really match what you’d see if you were being deemed suspicious on our end. (They do match an oversuspicious “intrusion prevention system”, hence my first guess!) Resetting the Uplink key on an app, in particular, really ought not to do anything. My guess is that, as this is an intermittent issue, the associations you’re seeing are coincidence/noise.
If you want to debug further, I would suggest intercepting the websocket connection and looking at the transcript when something goes wrong. (You can use the url= parameter to direct the Uplink to a proxy, then either dump the connection with a proxy like wssip, or just use tcpdump to capture unencrypted leg and timing of the RST).
Will have to keep an eye on it and try to get more info if it happens again, thanks, at least its working for now. It is however difficult to believe its pure coincidence - We reset the key on one app, it worked right after. The 2nd app that wasn’t working, I tried running several times and it failed (as it has done all day), reset the key/updated it in the script and it started working instantly as well. Have been fiddling all day with it and it has done nothing but fail with the same error until the key refresh.
3 Python scripts and 2 different apps/keys, running in different environments (albeit on the same server) all failing continously all day, and all worked immediately after refreshing keys in Anvil with no other changes, very odd…