I’m getting an unusual error I can’t debug with a background task. I’m pulling some data from an external Postgres database using Pandas. Here is one of the failed tasks:
Here is the code where it is failing. Line 65 is where the data is pulling and it’s failing there. The dataframe it is pulling is about 50MB in size and it should take about 5-20 seconds to load from the database. But I notice the task fails within a few seconds of the print statement in line 64.
@anvil.server.background_task
def make_heatmap_data(end_date=None):
try:
print('Starting Map Data Creation')
conn_string = anvil.secrets.get_secret('air_pollution_db')
engine = db.create_engine(conn_string)
if end_date == None:
start_date = datetime.datetime.today() - datetime.timedelta(days=7)
end_date = datetime.datetime.today() - datetime.timedelta(days=1)
else:
start_date = end_date - datetime.timedelta(days=7)
start_date = start_date.strftime('%Y-%m-%d')
end_date = end_date.strftime('%Y-%m-%d')
print(start_date)
print(end_date)
print('Pulling pollution data from external database')
df = pd.read_sql(f'''select * from master where "date" >= '{start_date}' and "date" <= '{end_date}' and "source" = 'Stateair.mn' or "source" = 'Agaar.mn';''', engine)
df['aqi_mn'] = df['aqi_mn'].astype('float')
df['lat'] = df['lat'].astype('float')
df['lon'] = df['lon'].astype('float')
# Remove stations with less than the 84 required measurements for 1 week.
# We will first count the number of records per station per PM type, then filter out (using .loc) on the df.
df = df.dropna(subset=['aqi_mn'])
df_count = df.groupby(by=['type','station']).count().reset_index()
stations = df_count[df_count['aqi_mn'] > 84][['type','station']].reset_index(drop=True)
avg = df.groupby(by=['type','station']).mean().loc[stations.to_records(index=False).tolist()].reset_index()
# Create kernel
kernel = Matern(nu=2.5)
# Separate data into PM2.5 and PM10
pm10 = avg[avg['type'] == 'PM10']
pm25 = avg[avg['type'] == 'PM2.5']
print('Creating models')
## Bunch more code down here.
except:
## This sends an email with the error report if there's an error.
My guess is a memory issue or a Python environment issue. This background task was working a few months ago.