Posts with the tag operations:

Less power to the people

In my 9 years in this city up until now, we’d lost power once. And now, in the last month, we’ve had three outages for completely unrelated reasons. Our lights went off about 3 hours ago and we don’t know when they’ll have the busted transformer to our part of the city fixed. Until then, Free Radical’s database server (and its Internet connection) will be offline. Sorry, gang – I don’t have the budget right now for a many-hour battery backup.

I fought PostgreSQL (and eventually won)

We had severeral hours of downtime yesterday because of a “fun” PostgreSQL problem (see Beware of your next glibc upgrade for details). Short version: because there are lots of ways to encode text, and they can change subtly when you upgrade your OS, some of your unique database keys may end up being not so unique. This resulted in quite a few Mastodon tables having duplicate rows, despite having unique indexes.

Database upgrades are (finally) done

Today I set out to migrate Free Radical’s database from a small, disk-limited server running on the same host as the FR application server to a vastly beefier system. This should make larger database operations a lot faster, and backup processes won’t bog down the system anymore. It also frees up a lot of RAM, CPU, and disk toward running the Mastodon services. I apologize for the longer than expected downtime.