Huge delays after using anvil.pico.call

[Luke]

It was working fine before, now I don’t know what’s going on. I noticed it yesterday. When I change the data in the database, the change is visible almost immediately in the application. However, when I run Pico, which executes twice every second:

async def set_relay_state(relay_name, relay_mode):
     await anvil.pico.call("set_status", relay_name, relay_mode, 'abc123')

it’s all starting to slow down horribly. Both communication from Pico to the server is slow and communication from the application to the database is slow. After some time, the following errors appear in the console:

Task exception wasn’t retrieved
future: coro= <generator object ‘set_relay_state’ at 2001fd80>
Traceback (most recent call last):
File “uasyncio/core.py”, line 1, in run_until_complete
File “anvil_callable.py”, line 18, in set_relay_state
File “anvil/pico.py”, line 218, in call
Exception: Server code took too long

I changed the network, reset flash memory, reloaded the firmware, checked the limits in the free plan. I have no more ideas what I could do.

[Luke]

I tested a bit. In the application I have a form where I fetch a user and then fetch all the data from a table (3 rows) so there are 2 RPCs in the browser console. This is done every second.

Pico disabled:

one anvil.pico.call per second:

two anvil.pico.call per second:

three anvil.pico.call per second:

[Tony.Nguyen]

Executing twice every second seems to reach the the limits in the free plan.
So either upgrading to the paid one or reducing the time might help.

[Luke]

I was looking for information about the transfer limit and the limit of the number of connections, but I found nothing.

Besides, why earlier 2 queries from Pico didn’t cause such delays as now? Everything worked instantly. Now I can’t use it with two queries per second. See how it looks today for two anvil.pico.call queries, the time increases with each query:

If the limits are true, then they disqualify Anvil from being used in multi-device systems.

[ianb]

I don’t have any answers just an observation. When I usually see requests handled like that (the delays increasing with more steady requests) the first thing that comes to mind is soft throttling, the kind they might implement to stop abuse on the free plan. If there are no hard limits then I doubt they would spell out exactly where the ‘line’ is, but I am completely guessing.
I used anvil free for about a year before I had my company start paying for it and barely ran into issues.
By the time I needed to scale the business case for paying for it was already there.

Also, do you have two picos to test at the same time? (Or even another uplink running on another machine)

If it is throttling,

@Luke

being used in multi-device systems

each uplink on the server side gets its own unique connection to anvil, so I don’t know if the suspected throttling would be shared across all connections, or just handled individually per device. Or again, if it would just go away on any of the not free plans.

I did run into throttling here, when I tried to make a Free-Tier csv to table uploader for new users:

CSV to Anvil Data Tables For Beginners with Full Example App

https://anvil.works/build#clone:Z4O2BXJEL6TVJ4QK=2LL4HNW3TZQAG7FICS57W6JN [csvexample] I see many new anvil users struggling to load data sets into the data tables for the first time without having an example of how to avoid abusing the free resources while still loading your use case sample data under the 30 second limit provided by the server modules. So I created this full example app. Feel free to clone it and use it for testing, reference, ideas, or anything else. It is not fast. That’…

If you want to clone it and see what I’m talking about, you can definitely make the servers very angry quickly by loading a large file and setting the processing to auto adjust the sever module timeout limit.

[stefano.menci]

@ianb

the first thing that comes to mind is soft throttling

You could test if it’s this kind of throttling by increasing the time between requests.

This could give you an idea if what is the maximum rate for your pinging.

[daviesian]

Hi @Luke

We’d be happy to help work out what’s going on here. Please can you post a clone link for a minimal app demonstrating the problem, along with the full associated Pico code and instructions for triggering the problem? Once we have those, we can try to replicate the issue and investigate.

Thanks!

[Luke]

Hi, thanks for your interest in the topic.

I made a clone and removed most of the unnecessary things from my application. Removing element by element, I discovered that the problem is somehow related to the image that I put in the place where a logo should be.

Link to clone: Anvil | Login
Please use the following details to log in:
Email: 123
Password: 123
After logging in, you need to click on the link that says “test1”. This is a view refreshed every second and the console should show RPC times around 1000ms - a lot but it works.

Now run Pico with the files below:

boot.py:

import network
from time import sleep
import ntptime

WIFI_SSID = "XXXXXXXXXXXX"
WIFI_PASSWORD = "YYYYYYYYYYYYY"

sleep(1)
wlan = None
while not wlan or wlan.status() != 3:
    wlan = network.WLAN(network.STA_IF)
    wlan.active(True)
    wlan.connect(WIFI_SSID, WIFI_PASSWORD)
    while True:
        sleep(0.2)
        if wlan.status() in [-1, -2, 3]:
            break
ntptime.settime()

main.py

import anvil.pico
import uasyncio as a
from machine import Pin
import uasyncio

UPLINK_KEY = 'ZZZZZZZZZZZZZZZZZZZZZZZZ'

async def anvil_pico_call():
    await anvil.pico.call('set_status', 'AvAwrNVJtG73lPTYLH1NjLXdJ2KZ77CG')

async def my_function():
    while True:
        print('my_function()')
        uasyncio.create_task(anvil_pico_call())
        uasyncio.create_task(anvil_pico_call())
        # uasyncio.create_task(anvil_pico_call())
        # uasyncio.create_task(anvil_pico_call())
        # uasyncio.create_task(anvil_pico_call())
        await uasyncio.sleep(1)

async def main():
    uasyncio.create_task(anvil.pico.connect_async(UPLINK_KEY))
    await uasyncio.create_task(my_function())

uasyncio.run(main())

Remember to set your WiFi details and your Server Uplink Key. After starting Pico, as above, i.e. with two calls, the RPC times immediately start to increase to a value of several seconds, after which the application stops working.

Now the best part. After deleting the image (Main form), the RPC time drops to 400-700 ms, which is better than before starting Pico. How is this possible? What’s the picture got to do with it? Considering the very small amount of data transferred, shouldn’t this time be less than 100 ms?

Finally, I did a test, no picture and ten calls per second. Pico stops working and displays the error over and over again:

Task exception wasn’t retrieved
future: coro= <generator object ‘anvil_pico_call’ at 20020960>
Traceback (most recent call last):
File “uasyncio/core.py”, line 1, in run_until_complete
File “main.py”, line 9, in anvil_pico_call
File “anvil/pico.py”, line 200, in call
MemoryError: memory allocation failed, allocating 3112 bytes

What would it look like if 10 devices made one call per second? Would the same problem exist?

The image I used:
Dimensions 1664x1664
Size 3.66 MB

[daviesian]

Hi @Luke

Thanks for this! There are a couple of things going on here, and they are interacting badly to give the behaviour you’re seeing. First up, the giant app logo means that your app is going to take a long time to load on every server call. We can improve this somewhat from our end by optimising how different parts of your app are loaded and cached, but from your side there are two options:

1. Use a (much) smaller logo image
2. Keep your server environment running between server calls, so it gets reused (available on the Business Plan and above)

Okay, that’s the first part. The second part is the way you’re calling into Anvil from the Pico. You are correctly awaiting the anvil.pico.call, but then you’re throwing away this promise when you start two new tasks:

  uasyncio.create_task(anvil_pico_call())
  uasyncio.create_task(anvil_pico_call())

When you run this, you are asynchronously starting two new calls to Anvil (forgetting to await), then correctly waiting one second, then doing it again. If those calls take more than a second between them, you’re going to go around the loop and start the next pair of calls before the first pair are done. This pile-up will continue until the Pico runs out of memory. (Something similar is likely happening in the browser with your update timer, but you have a lot more headroom there before anything actually explodes.)

You can either await the call to uasyncio.create_task, or cut out the anvil_pico_call function entirely and just await anvil.pico.call directly inside your loop.

Once that’s done, and the logo is fixed, please try the application again and let me know how you get on. I’ll be happy to help with another round of debugging if any issues remain!

[Luke]

I know that images should not be this size, especially when presented as thumbnails. I just threw there the first picture I had on the disk to see how it (Anvil) works. Of course, it wasn’t the target logo However, I’m surprised that every request to the server for data reloads the pictures. I thought it would work like a standard HTML page where the browser caches the assets and doesn’t re-download them if it’s not needed. I thought it would be like a REST API request in Angular where only the data is fetched and the page is not reloaded. Fingers crossed that this will change in the future

As for “await”, I don’t know how to deal with it. I tried the following two methods but neither works.

async def my_function():
    while True:
        print(1)
        await uasyncio.create_task(anvil_pico_call())
        print(2)
        await uasyncio.create_task(anvil_pico_call())
        await uasyncio.sleep(1)

async def my_function():
    while True:
        print(1)
        await anvil.pico.call('set_status', 'AvAwrNVJtG73lPTYLH1NjLXdJ2KZ77CG')
        print(2)
        await anvil.pico.call('set_status', 'AvAwrNVJtG73lPTYLH1NjLXdJ2KZ77CG')
        await uasyncio.sleep(1)

Print(2) will never execute. Could you elaborate on that please?

[Luke]

@daviesian

let me know how you get on

Hi @daviesian, did you see my previous post? Let me know what I’m doing wrong because I can’t figure it out.