Over the past few days, the app (which is still in the development stage) has been getting slower and slower. If I’m lucky, my app launches in a few seconds and the data appears fairly quickly (I have the Hobby Plan active during development and don’t expect extremely fast response times).
Since yesterday, however, it’s mostly been extremely slow; it takes 10 to 15 seconds to start up, and I often have to restart it just to get anything to happen. Data isn’t displayed until after 20 or more seconds—or not at all—and then I have to restart. This happens even when the relevant table contains just two records (no complex joins, etc.). This is no fun at all; I can forget about debugging like this.
As I said, it’s not always like this; early in the morning or late in the afternoon, it runs much faster. So I suspect the servers are simply overloaded.
Is it just me, or have others had this experience too?
Fun fact: As I’m writing this, the app is still loading in the background. And it barely has any data…
Please could you share the root cause for this? A business plan runs customer facing apps so there is always a need for detailed information when servers are slowing down.
What happened here was that we were presented with an unusual increase of load (from legitimate applications, albeit ones that were pushing the limits a bit!). Normally, what happens here is that Anvil prioritises work in its API servers based on usage: when work needs doing on behalf of any application or user, those who aren’t demanding so much go to the front of the queue, and those who are demanding a lot are serviced only when the others are done. This is applied hierarchically: between organisations, between apps within an organisation, and even between environments, so that (eg) a batch job running in a dev environment shouldn’t strangle a production user trying to get something done, and one app in your account shouldn’t strangle all the others. This allows us to remain responsive in the presence of aggressive or simultaneous spikes in load from our customers.
Unfortunately, this time, that didn’t work. The back-pressure didn’t get applied in the right places, and we ended up with ordinary use being squashed by hungry workloads instead of throttling the overactive apps. There were a few contributing factors (we were near a point of needing capacity expansion, and this machinery seems not to perform as well as we’d hoped in that regime!).
The response was a combination of a bit of debugging our work queuing system, a bit of “throw more cores at the problem”, and a bit of bottleneck reduction. We’ve also changed how we instrument this machinery in production, so that we should be able to better detect and respond to any collateral damage to well-behaved apps and respond faster.
Thanks very much for the detailed answer. Is there a url available where Anvil customers could view realtime network/server performance metrics? This will allow us to inform our customers pro-actively in case of these kind of issues.
Would love something like this. Because as of now I just come here and wait to see if other people are complaining. Some true “status” page would be really nice.