In my 9 years in this city up until now, we’d lost power once. And now, in the last month, we’ve had three outages for completely unrelated reasons. Our lights went off about 3 hours ago and we don’t know when they’ll have the busted transformer to our part of the city fixed. Until then, Free Radical’s database server (and its Internet connection) will be offline. Sorry, gang – I don’t have the budget right now for a many-hour battery backup.
Posts with the tag operations:
We had severeral hours of downtime yesterday because of a “fun” PostgreSQL problem (see Beware of your next glibc upgrade for details). Short version: because there are lots of ways to encode text, and they can change subtly when you upgrade your OS, some of your unique database keys may end up being not so unique. This resulted in quite a few Mastodon tables having duplicate rows, despite having unique indexes.
Today I set out to migrate Free Radical’s database from a small, disk-limited server running on the same host as the FR application server to a vastly beefier system. This should make larger database operations a lot faster, and backup processes won’t bog down the system anymore. It also frees up a lot of RAM, CPU, and disk toward running the Mastodon services. I apologize for the longer than expected downtime.