Free Radical is now on Mastodon v4.2.1.

Mastodon 4.2.0 added per-user opt-in search indexing. I love this so much. People here have asked me to enable Elasticsearch on FRZ dozens of times. Of course users want to find toots easily without scrolling through hundreds of them! However, it presents significant privacy issues for people who don’t want their toots to be indexed. Who wants to make life easier for a stalker?

The new feature handles this thoughtfully. I can enable the indexing feature at FRZ, but it doesn’t process a user’s toots unless they deliberately turn it on for their own account. People who don’t want their toots to be indexed don’t have to do anything. I think this is the perfect balance between privacy and convenience: everyone gets to decide for themselves. Well done, Mastodon team. Well done.

Free Radical is now on Mastodon v4.2.0.

At 2:53AM Pacific, our normally rock solid Internet connection went down for 10 minutes. When it came back up at 3:03AM, that should’ve been the end of it. It wasn’t.

Short version: I had to reboot the firewall, then everything was fine.

Longer version:

• Wake up at 7-something.
• Yawn, stretch, pet the dog, look to see what happened online overnight.
• Dang it, FRZ’s down.
• Check email. See an email from a moderator from an hour earlier: hey, the site’s down!
• Run downstairs. Instantly remember and regret that the coffee pot died yesterday and I have no caffeine, nor will I any time soon. Curse quietly.
• SSH into FRZ. All looks OK from there, except that pgbouncer can’t connect to the database server.
• Check the database server; it’s up, running, and twiddling its thumbs in boredom. That’s weird.
• Use netcat from the FRZ server to verify that I can connect to the DB server. I can, but only with IPv6. IPv4 isn’t working. For ancient reasons. I had pgbouncer pinned to IPv4. Huh.
• Speculation here:
  • I think that when the outage was over, the firewall found itself bombarded with frantic inbound connections from the FRZ server, and either temporarily blocked them or overloaded some kernel table or such.
  • There’s no easily visible evidence that either of those happened.
    • If it autoblocked the FRZ server – and it shouldn’t have, but here we are talking about it – it didn’t log it or notify about it.
    • If it was because a NAT table filled up or such, I didn’t get an alert on that, either.
• Outbound IPv4 was just fine. I regret now that I didn’t check other inbound IPv4 ports on the firewall. I blame the lack of caffeine.
• Lacking anything else to go on, I rebooted the firewall, and ta-da!, we’re back.
• The Sidekiq queue has about 36,000-and-growing tasks to chew through. It’ll be a little while until we’re 100% back up to speed.

That was weird. I don’t know why that happened, and I don’t like that feeling. And as I write this, I remember that I’d pinned pgbouncer to IPv4 because one time IPv6 stopped working in a very similar way. Maybe the same thing happened then but in reverse?

Free Radical is now on Mastodon v4.1.6.

Free Radical is now on Mastodon v4.1.5.

Free Radical is now on Mastodon v4.1.4, because 2 upgrades in 2 days seemed fun.

Free Radical is now on Mastodon v4.1.3.

We’re updating the site to the new urgent security release v4.1.3. We may be offline or wonky for a few minutes at a time through the promise. Be back soon!

An urgent-ish software update has us down for maybe 20 minutes or so.