Background

After Elon Musk bought Twitter and started making a bizarre series of decisions about how to run it, people started logging into Mastodon to see what it’s all about. Lots of them. So, so many of them. In real numbers, Free Radical grew by 20% in the last week. Which is awesome, because it’s wonderful to see new faces excited and eager to join the fun. The downside is that new users, being new to it, tend to be understandably excited and exploratory, with lots of posting, following other new people, and doing the kinds of things that require server hardware to wake up and earn its living. I don’t have hard stats to back it up, but from eyeballing the logs, I estimate that the server load was about 4 times greater than it was 2 weeks ago.

In my 9 years in this city up until now, we’d lost power once. And now, in the last month, we’ve had three outages for completely unrelated reasons. Our lights went off about 3 hours ago and we don’t know when they’ll have the busted transformer to our part of the city fixed. Until then, Free Radical’s database server (and its Internet connection) will be offline. Sorry, gang – I don’t have the budget right now for a many-hour battery backup.

We had severeral hours of downtime yesterday because of a “fun” PostgreSQL problem (see Beware of your next glibc upgrade for details). Short version: because there are lots of ways to encode text, and they can change subtly when you upgrade your OS, some of your unique database keys may end up being not so unique. This resulted in quite a few Mastodon tables having duplicate rows, despite having unique indexes. It also meant that PostgreSQL was all like “indexes? lol never heard of ’em” and query performance was pretty awful.