A few updates:
- I think I tried @ianbās workaround, but I am still getting weird behavior. I go into detail below for what I have been attempting.
-
@meredydd: I believe it is pretty reliably failing. I am going to try to construct a minimal example soon (hopefully later today, without the DB connection), but what I have now is fairly simplistic (it is unfortunately just part of a larger code)
What I have tried recently:
I created a Scheduler.py code on my desktop that just periodically calls a local program called connectDatabaseProduction.py in the same directory. All connectDatabaseProduction.py does is
parametersConnection = {
'sslmode': '',
'sslrootcert': '',
'sslcert': '',
'sslkey':'',
'hostaddr': '',
'port': '',
'user': '',
'password': ,
'dbname': ''
}
connection = psycopg2.connect(**parametersConnection)
cursor = connection.cursor()
def main():
anvil.server.connect('')
anvil.server.wait_forever()
main()
There are some other functions in here that get called by the Anvil server through the client. When I first load everything up, it all goes normally. After a few hours, things start to break down.
After @ianbās suggestion, I made this Scheduler.py code, which:
- runs
connectDatabaseProduction.py
- prints the time (in EST) every hour
- between 2 am and 3 am, it terminates
connectDatabaseProduction.py and restarts it
- repeats this for about 5 days
Here is Scheduler.py:
from datetime import datetime, date, timedelta
import time
import subprocess
# sleep every 1 hour and restart between 2-3 am every day
countDay = 0
now = datetime.now()
dateStart = date.today()
print('')
print('=== Scheduler === Start Date: ' + str(dateStart) + ' ' + now.strftime("%H:%M:%S"))
print('')
if datetime.now().hour > 3:
dateNext = dateStart + timedelta(days = 1)
else:
dateNext = dateStart
subprocessDatabase = subprocess.Popen(['python', 'connectDatabaseProduction.py'])
print("=== Scheduler === Waiting for 2 am of " + str(dateNext))
print('')
while countDay < 6:
now = datetime.now()
print("=== Scheduler === Checked at " + str(date.today()) + ' ' + now.strftime("%H:%M:%S"))
print('')
if dateNext == date.today() and now.hour == 2:
print('')
print("=== Scheduler === Restarted at " + now.strftime("%H:%M:%S"))
print('')
subprocessDatabase.terminate()
subprocessDatabase = subprocess.Popen(['python', 'connectDatabaseProduction.py'])
dateNext += timedelta(days = 1)
countDay += 1
print("=== Scheduler === Waiting for 2am of " + str(dateNext))
print('')
time.sleep(3600)
subprocessDatabase.terminate()
I ran this for about 2-3 days (between 6/28 and 6/30) and this was the output:
=== Scheduler === Start Date: 2021-06-28 16:09:49
=== Scheduler === Waiting for 2 am of 2021-06-29
=== Scheduler === Checked at 2021-06-28 16:09:49
Connecting to wss://anvil.works/uplink
Anvil websocket open
Connected to "Default environment (dev)" as SERVER
Anvil websocket closed (code 1006, reason=Going away)
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 311, in call
return _do_call(args, kwargs, fn_name=fn_name)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 303, in _do_call
return _threaded_server.do_call(args, kwargs, fn_name=fn_name, live_object=live_object)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\_threaded_server.py", line 404, in do_call
raise error_from_server
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 143, in heartbeat_until_reopened
call("anvil.private.echo", "keep-alive")
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 314, in call
raise _server._deserialise_exception(e.error_obj)
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
Reconnecting Anvil Uplink...
Connecting to wss://anvil.works/uplink
Anvil websocket open
Connected to "Default environment (dev)" as SERVER
OperationalError('server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n')
Restarted Connection
=== Scheduler === Checked at 2021-06-28 17:47:00
=== Scheduler === Checked at 2021-06-28 18:47:00
=== Scheduler === Checked at 2021-06-28 19:47:00
Anvil websocket closed (code 1006, reason=Going away)
Exception in thread Thread-5:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 311, in call
return _do_call(args, kwargs, fn_name=fn_name)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 303, in _do_call
return _threaded_server.do_call(args, kwargs, fn_name=fn_name, live_object=live_object)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\_threaded_server.py", line 404, in do_call
raise error_from_server
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 143, in heartbeat_until_reopened
call("anvil.private.echo", "keep-alive")
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 314, in call
raise _server._deserialise_exception(e.error_obj)
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
Reconnecting Anvil Uplink...
Connecting to wss://anvil.works/uplink
Anvil websocket open
Connected to "Default environment (dev)" as SERVER
Anvil websocket closed (code 1006, reason=Going away)
Exception in thread Thread-48:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 311, in call
return _do_call(args, kwargs, fn_name=fn_name)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 303, in _do_call
return _threaded_server.do_call(args, kwargs, fn_name=fn_name, live_object=live_object)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\_threaded_server.py", line 404, in do_call
raise error_from_server
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\gwebe\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 143, in heartbeat_until_reopened
call("anvil.private.echo", "keep-alive")
File "C:\Users\gwebe\anaconda3\lib\site-packages\anvil\server.py", line 314, in call
raise _server._deserialise_exception(e.error_obj)
anvil._server.AnvilWrappedError: 'Connection to Anvil Uplink server lost'
Reconnecting Anvil Uplink...
Connecting to wss://anvil.works/uplink
Anvil websocket open
Connected to "Default environment (dev)" as SERVER
=== Scheduler === Checked at 2021-06-30 09:39:29
=== Scheduler === Checked at 2021-06-30 10:39:29
I have not been able to quite figure out what is going on in this output.
In this case, Anvil reconnects in the first hour (sometimes it lasts longer), then the scheduler seems to pick itself back up about 1.5h later, then it prints every hour again for about 3 hours, then Anvil reconnects again, then the scheduler recovers a couple days later.
In the mean time, I was getting anvil.server.UplinkDisconnectedError: The uplink server for "<FunctionName>" has been disconnected and I believe a few other strange errors. Also during this time, I was using the same desktop (that was running the scheduler) and my laptop on the same internet with no internet connectivity issues on either device.
I intend to simplify this with a minimal app and get something repeatable and less chaotic, but I hope this can at least give some insight or maybe point to something I am doing wrong somewhere.
Thanks!
. However, even after fixing that, I am still seeing the same random errors - function disconnect exceptions or, more commonly, the client just hangs and then timeouts. I have found that printing a word on my Uplink program seems to help the hanging issue somewhat. But I have no idea if that is actually working or not.
