Long time to run a server function

Anvil Q&A
#1

Hi,

please see this video of the issue in action. A very simple server function is taking an age to even (I don’t know the right word here) “start”? “load”?

Its not just the first time this server function is called either. Here is a demo of the issue, and actually the long time to get user info, happening after being used a few times:

Any help MUCH appreciated. @stucork @meredydd @daviesian Thank you :slight_smile:

#2

in case its useful, here is a delay when just running a PRINT command from the server module:

#4

There’s a bunch of optimization posts on the forum.

My guess is that it’s related to the last section in this post:

(i.e. Move slow server imports inside the callable function)

2 Likes
#5

Hi @michaellavers,

Thanks for sharing this video - that really helps with diagnosis of the issue. You’re actually seeing two separate things going on here:

  1. Your app is large, with a lot of imports in your Server Modules, so you have (correctly) enabled the Persistent Server feature of your Business Plan. However, each time you change your app (to add or change some debug printing, for example), a new worker must spawn and load the updated code. This means that the first server call after tweaking your app will always be slower than subsequent calls, so you should always try a few times to get a clear picture of the steady-state performance you can expect. This effect is compounded by #2:

  2. You have a number of Scheduled Tasks running very regularly (every minute, or every two minutes). One or more of these tasks is completely saturating the CPU of your server runtime environment for up to 30 seconds every time, starving other calls of resources. This means that if you happen to call any server function (even from a cached persistent server environment worker) while these Scheduled Tasks are running, that call will have a long time to wait before it can receive enough CPU cycles to run. This is especially bad if you’ve just tweaked the app and the first server call has to load all your modules from scratch during one of the periods when the Scheduled Tasks are saturating the CPU. And these periods extend to more than half the available time!

As an experiment, I temporarily increased your available CPU quota slightly, and this did resolve the CPU starvation problem to some extent, so I’m confident that this fully explains the issues you’re seeing. The interaction between the two effects above explains why the delay feels inconsistent and unpredictable. There are two things you can do to tackle this problem:

  • Take a close look at your Scheduled Tasks, and see whether they can be optimised to be less CPU intensive. They must be doing quite a lot to saturate a whole CPU for 30 seconds every minute! Even running the CPU-intensive parts less often would improve things a lot.

  • Upgrade to the Dedicated Plan. This would likely resolve the issue completely, because you would be able to choose a dedicated server instance with sufficient CPUs to keep up with all your various Scheduled Tasks while maintaining performance of your live apps. If you’d like to go ahead with this, please drop us a line to support@anvil.works and we’ll talk you through the options.

I hope this helps!

5 Likes
#6

Another advantage of the dedicated plan is that you have access to SQL.

I have no clue what your long running background tasks do. Often performance problems in Anvil apps are caused by the lack of advanced filtering in table searches and you end up with slow loops scanning long lists of rows rather than doing a quick SQL select with the correct joins.

If your background tasks spend time slowly cycling through database search results, then very likely one well designed SQL query will cut the time down from minutes to hundredths of a second.

4 Likes
#7

I noticed you have a module labeled ‘digital ocean spaces’, if moving to the dedicated plan is not within your budget, maybe moving the specific process or database out of anvil (with more complication) could help.

If it is the lack of SQL access that is a problem as @stefano.menci mentioned, I know digital ocean does DBaaS with managed databases. Maybe even cashing your query results using a Redis server might help, you wouldn’t need as much complicated management of the data that way.

2 Likes
#8

Thank you so much everyone! I’m going to reduce the scheduled tasks and then if that doesn’t work I will move some tasks or DB to digital ocean and IF THAT doesn’t work then… I’ll have to sell a kidney and get the dedicated plan. But I have a spare!.

Thanks again my Anvil family, I cant tell you how much I appreciate the support. :heart:

3 Likes
#9

ok… who wants to hear a cautionary tale?

This is a story about an idiot. An idiot who brandished a mighty weapon (anvil) and then thought himself mighty. I hang him here as a warning to other idiots.

So, I basically would make a clone of my apps as a way of ‘saving’ them in case, I don’t know what, the ‘worst’ happened. Been doing this every 6 months or so since 2020.

(you can see where I am going with this)

Then I get slowdowns, sluggish response, the whole thing feels like treacle. I’m trying to re code everything. I even spent my weekend creating a new database and sending info there.

Then today a feeling of dread came over me as I wondered if any of those ‘backup’ apps had scheduled background tasks running.

Yes. I’m that guy.

25 mins later and all my worries are gone. I don’t need to upgrade. I don’t need to worry anymore. scheduled tasks are removed from all those other apps (HORRIFIC how much they were churning away!) and now response is instant.

So, all I can say is that I am sorry @stucork @meredydd and @daviesian. If there is an award for ‘forum dunce’ then I will wear it gladly.

Yours, in dramatic shame :slight_smile:

12 Likes
#10

You’ll need to get in the queue somewhere behind me for that one!

5 Likes
#11

And kudos to @michaellavers for fessing up so openly. Thst takes guts.

I’m fortunate that several of my clients pay for support contracts, so my best numpty impressions can be done to a more select audience.

Side note:
It amuses me greatly that Google seems uncertain whether numpy and numpty might be synonyms.

2 Likes
#12

Thanks for the cautionary tale, it is good for us users to learn about this type of potential problems.

But I would like to add a comment: when I see users acting like “an idiot” with software I wrote, I always have a feeling of failure and an urge to “fix” it, not of distributing “dunce” badges: if my software allows you to do something wrong, my software has a usability problem.

In this case your problem was caused by two problems in Anvil:

  1. The cloning process that does something without clearly telling you that it does it
  2. The lack of a dashboard that keeps you informed of what’s going on in your account

There are already feature requests about improving the cloning interface and creating a dashboard 1, dashboard 2, dashboard 3.

Your cautionary tale should be addressed to the powers that be, so they will address the problem :slight_smile:

5 Likes