So, it actually had nothing to do with the imports or the time it takes to spin up an instance of the server code. Spinning up an instance took ~50-100 ms in my case, but launching the background tasks was still slow.
The issue was the size of the arguments (up to a few Mbs) being passed on to the background task, which I was able to address with some compression.
So I’m leaving this as a note to anyone who might stumble across the same issue in the future - if your backgroung tasks are slow to launch, the reason might be the size of your arguments.