We experienced some interruptions the last hour. Is something going on at Anvil?
1 Like
Yep, we’re on it! Load balancer threw its toys out of the pram, for reasons we’re still investigating, but I believe we’re now back.
Update: Yep, we’re satisfied we understand what went wrong here, and the situation is at least durably mitigated.
Behind-the-scenes details for nerds: DNS, it’s always DNS. In this case Docker and systemd-resolved are apparently not friends, and were resolving DNS gradually slower and slower until the load balancer gave up and decided all of Anvil’s serving cluster must have disappeared off the Internet. We’re confident we now have mitigation in place, and we have plans for a durable solution.
2 Likes