With the recent announcement of Elon Musk purchasing Twitter, myself (a long time Twitter user) and many of my professional peers and colleagues are finally tip-toeing away from the social media site.
Thus begun the hachyderm.io Mastodon instance.
So long Twitter 👋
I haven’t made the official jump yet (my account is still active) however I suspect it is only a matter of time before I stop contributing my content to a place that is actively harming my family, while legitimizing the christo-fascist state of the U.S and just abandon the platform all together.
In the mean time the small Mastodon server that myself and my Twitch crew set up a few months ago is now supporting upwards of 750 new users!
Mastodon Up and Running
So Mastodon is the epitome of the ruby monolith that we have all heard about. It’s a large stateful application runs several services on the backend. It communicates with remote SMTP APIs to send emails to the users, and requires a TLS encrypted ingress just to bring the service online.
Fortunately for us, we were currently looking for applications to prototype our new Kubernetes and Systemd alternative tool Aurae. Naturally a large stateful monolith was a great test-bed application for us. We configured the service running in Aurae cell isolation zones (cgroup and namespace boundaries) running on a Dell R630 server in my live streaming studio.
We host the service on a ZFS storage cluster that sits underneath the postgresql database, and underneath the mastodon storage volume we mount directly to /var/lib/mastodon.
We run the ruby code in an isolation zone directly on the host alongside an active Kubernetes cluster. We made the decision not to host the service on Kubernetes primarily because of the relationship with state which is that we have it… and it will likely grow. Additionally we need to prototype isolation zones with Aurae. In hindsight I feel we made the right decision as our scale needs aren’t high enough to merit a Kubernetes cluster at this time.
However if we did need to horizontally scale the node, I have some research that could be helpful.
We were able to completely migrate the service (while the service was online and serving requests) off the rack in the basement and onto a cloud node while my family moved from New York to Seattle. Here is how I did it.
The Data
Our Mastodon deployment had 2 main data locations, and a single production configuration file, and our LetsEncrypt TLS data.
/var/lib/mastodon
/var/lib/mastodon/.env.production -> /etc/mastodon.conf
/var/lib/postgresI was able to rsync about 50gb of data from the rack to the cloud in a few hours. Here are the rsync commands:
rsync -rv /var/lib/postgres root@alice.nivenly.com:/var/lib/postgres
rsync -rv /var/lib/mastodon root@alice.nivenly.com:/var/lib/mastodon
rsync -rv /etc/letsencrypt root@alice.nivenly.com:/etc/letsencryptAfter syncing the data I was able to start the services up on the new server. It was extremely easy, and extremely straightforward.
This leads me to believe that it would be viable to distribute the mastodon services behind a load balancer by simply keeping the local data storage in sync with the host. I suspect myself and the group of hackers that hang out in our crew will have an easy way to run mastodon replicas around the globe shortly. Stay tuned for our work!
The Traffic
We don’t have the best observability into the traffic patterns right now, however we plan on writing a Mastodon-specific observability tool that can be installed alongside an instance to provide insights into the data and traffic patterns. However we do have active graphs of the servers immediate peers and traffic relays over the gateway into the rack.
The main takeway is that the 50Gb of data we transferred into and out of linode dramatically outperforms any of the user generated traffic. The traffic patterns are relatively stable, and peaked around 1Mb per second.
Mastodon on the iPhone
I strongly suggest using the Toot! application for your iPhone. It has the best user experience and I catch myself doing the “dopamine scroll” on this application.
I am unsure if that is a good habit to be seeking out, however it definitely works. It is important to note that getting our little slice of the fediverse dialed in took some time before my iPhone felt anything like what Twitter used to do.
Android Users: We suggest Tusky.
Welcome To Hachyderm
One of the the things we have done is set out a solid set of official rules and establish about 15 volunteer Twitch/Mastodon moderators that organize in our Discord instance.
I think that the combination Discord/Mastodon is the right move for anyone considering moving their friends/community away from Twitter. The Discord channel is important to fill the gaps and create a venue to communicate during the awkward stages of getting started. Additionally it gives our mods a place to discuss things offline. It also serves as a place for our infrastructure volunteers to schedule upgrades, etc. Seriously — if you are considering managing a Mastodon instance with your friends, also establish a Discord or similar!
Privacy and Content
In full disclosure the privacy on Mastodon is surprisingly open. My immediate advice is to treat everything on Mastodon as if it is public data!
This includes your password. You should generate a password and use the built in 2fa mechanism with Mastodon. I will know if you don’t. 😉
Just to be crystal clear. Direct messages, toots, IP addresses, browser information, and even phone and geographic information we have access to. Seriously.
Yes! We can read your DMs! ⚠️
But like — we don’t — we promise. We are likely too busy dealing with the server itself.
As an administrator we have access to a LOT of your data! We also have a lot of controls and influence over the things you see.
The moderators are really the secret sauce behind a successful Mastodon instance. It took us to get to roughly 10k posts before the instance really started to “feel” like it had a good vibe and a good group of people.
However we have put a lot of work into it. We block domains, and manage users, emojis, hashtags, and more. We aren’t very active, however we could be — if we needed to be. I suggest taking a look at the official moderation docs if you haven’t already. This gives you an idea of the things we are capable of doing.
Ultimately the people on the instance do have a substantial influence on the content you see. (You should click that link and read it).
So myself and the nerds at Hachyderm are very proud of the group of hackers we have curated on Mastodon. We work hard to keep our content queer friendly, nerdy, and safe for everyone.
Welcome to Hachyderm
We wanted to welcome new folks to the instance. We will likely keep the doors open until we hit roughly 1,000 users. In which case it might move to invitation only until we can ensure the server can hold the load.
Please feel welcome to come and join the party. Just download the app for your phone and log in to hachyderm.io or visit the website directly in your browser.