I configured Free Radical to use S3 to serve media assets, like images and videos, almost immediately after launching it. There are compelling reasons to use something like S3 instead of serving those files directly from the Mastodon server, such as freeing that server from getting clogged with a zillion media requests. In theory, it should be cheaper, too.

Well.

In October 2022, our AWS S3 storage bill was $12.54 for storing about 330GB of media files and serving about 113GB of files to users.

Background

After Elon Musk bought Twitter and started making a bizarre series of decisions about how to run it, people started logging into Mastodon to see what it’s all about. Lots of them. So, so many of them. In real numbers, Free Radical grew by 20% in the last week. Which is awesome, because it’s wonderful to see new faces excited and eager to join the fun. The downside is that new users, being new to it, tend to be understandably excited and exploratory, with lots of posting, following other new people, and doing the kinds of things that require server hardware to wake up and earn its living. I don’t have hard stats to back it up, but from eyeballing the logs, I estimate that the server load was about 4 times greater than it was 2 weeks ago.

I woke up to the terrible news that our good friends on another instance had lost their database during a software upgrade. Godspeed and good luck in bringing it back online. We’re pulling for you!

The Free Radical site backs itself up hourly to a private S3 bucket, and keeps a month’s worth of these snapshots. It’s configured to upload all media files to S3 and serve them from there. In the event of a complete server failure, I could – assuming all goes well – re-deploy the software on a new server and restore from backup without losing more than just users and posts created since the last hour’s backup.

Free Radical is running on a Digital Ocean VPS. Instead of deploying it manually, I turned the process into a couple of Ansible playbooks¹ that do the right things quickly and repeatably.

I describe what it does in the README, but that’s just a feature checklist. So why would I go through the effort? There are several reasons:

This morning, I moved all of the user-generated content on Free Radical from local storage to S3. It was completely painless and Just Worked – yay! There are a few reasons why this can be a great idea: