I am trying to install the app server on a local VM following the instructions.
When I start it, it announces “Found Anvil App Server JAR in package directory”, then just hangs. It has not opened any port and logs no errors I’ve been able to find. This issue has been raised before but nothing in that thread hints at how to move me past this initial problem.
In case it is relevant, I am using openjdk-11 rather than openjdk-8. The machine is running Debian GNU/Linux 10 (buster).
Hi @roy,
That message is printed immediately before loading the JVM components of the App Server, so it looks like your JVM is taking a very long time to start up. The advice I would particularly point to in the previous thread is “try an instance with more RAM (or CPU)” - if you’re starting this on the smallest server available, or a rather old Raspberry Pi, the App Server can be really very slow to start. You can use the htop command (you might need to install it) in order to see how much memory and CPU your machine is using – my guess is you’ll find it pegged on CPU or RAM or both. You’ll either have to be very patient while the App Server boots, or (recommended) get a slightly more powerful machine 
Thanks Meredydd. I’ll admit the hardware hosting my VM is not in the first flush of youth but it’s fairly chunky: a four core, 4.2GHz AMD FX-4350 with 32Gb, of which 8Gb is allocated to the VM. htop shows CPU utilization hovering around 1.5% and I’m using 1.39Gb of memory.
I can think of about twenty random things still to try, including installing elsewhere. But a drunkard’s walk is not my favorite strategy. 
Any further advice is welcome.
I have managed to get it running on a different machine. There are a number of differences and I have no idea which if any is relevant. For the record, I’ve got it running on Windows on the same hardware, using java version “1.8.0_101” and Python 3.8.
That’s all great and encouraging, but I’d still like to get running where I originally intended. I shall report progress if any…
I have tried to eliminate some unknowns by harmonizing the Java version and the Python version I’m using on Linux and Windows (where the app server “just works”). The problem persists; it is still hanging on Linux.
I have obtained stack dumps from the working and hung servers. I am not at all proficient in reading them (and don’t aspire to become proficient) but at a glance I can see they are very different. There are ~45 threads running in the working instance but only 21 in the hanging instance. Would it be helpful to see them?
Hmm. Well, that’s the first hypothesis out of the way. Here are the next two most common issues we’ve seen:
1. Entropy depletion
The Linux kernel has historically had funny ideas about entropy, which cause it to pause when it doesn’t think it’s seen enough random events lately (such as IO times, network packet arrival times, or sometimes hardware randomness). This can be particularly exacerbated by running in a virtual machine, which offers less exposure to randomly-timed hardware events.
Unfortunately, by default, the JVM uses /dev/random, the randomness source most affected by funny ideas about entropy, and during startup we need a fair bit of it.
Consider configuring your JVM to use /dev/urandom instead (observe the warning about including /./ in the path – that one cost us a fair amount of pain in our production environment, a few years back), or configuring your hypervisor and/or VM to get hardware randomness.
2. Something strange with DNS
Sometimes, if the JVM (or, indeed, any UNIX program) hangs for ages when starting up, it’s because it tried to look something up in DNS, or do a reverse-lookup on some IP address, and is hanging waiting for a response. This can have any number of causes, but the best way to diagnose it is to use strace to dump the system calls the JVM is running, and see what it’s hanging on. You might also want to check your DNS resolution settings (/etc/resolv.conf and friends), and make sure you can resolve your machine’s hostname.
3. Filesystem shenanigans
This is a long shot, but one of the first things the App Server does in its default configuration is unpack its embedded Postgres server, and if you’re doing something funny with filesystems (eg NFS) this can sometimes lock up. Again, strace is usually your friend.
Appendix: Or, why we don’t offer much free support for the App Server
To take a peek behind the curtain for a moment, this post is a perfect demonstration of why we generally don’t promise support for the App Server (except on a support plan, and even then the App Server costs extra). More to the point, this is why we built Anvil as a cloud platform in the first place!
It’s not that we’re mean or miserly, or don’t believe in open source; it’s just that this sort of question rapidly descends into debugging strange idiosyncrasies of your particular computer. Because the causes are frequently so obscure, this tends to require a lot of time from quite senior engineers (writing out this post directly delayed releasing a new platform feature, and we’re a small team who can’t afford to do that very often - especially for a post that might not even solve your problem!). What’s worse, all this effort very rarely helps the next user, whose system will be idiosyncratic in a different way.
We happily offer support to everyone on this forum, including Free-plan users who will never pay us a penny, because in general answering their questions helps the next dozen people who have the same question. Unfortunately, “what strange thing is the App Server doing on my server?” rarely has those characteristics 
Thanks again Meredydd. I take your point about time taken to deal at length with potentially obscure problems. You have satisfied me that what I’m seeing is not a well-known problem with a stock answer, contrary to what I was hoping when I posted the question.
I’ll have a think about whether to work through your suggestions or whether I should just grit my teeth and use Windows. I am not totally stuck.
This is not an answer to your question, but it might help anyways.
Amoni, its used for local development and runs the app server in a docker container, this does not mean it will do everything you want, but it is very well documented, and the docs themselves may get you past at least a few hurdles, since they share many of the same concepts.
There is a daemon you can install called haveged (most distros have it in their repos), which provides artificial entropy when your server just doesn’t have enough randomness.
Works well for me for other applications when they take an age to start up.
For this record, a quick check using
systemctl | grep haveged
shows it is already running in my environment.
Thank you for the pointer. I will investigate.