Over the last few weeks, I’ve been working on migrating our Mastodon instance burningboard.net from its current Linux host to a modular FreeBSD jail-based setup powered by BastilleBSD.
This post walks through the architecture and design rationale of my new multi-jail Mastodon system, with aggressive separation of concerns, centralized firewalling, and a fully dual-stack network design.
Acknowledgements
This work is based on the excellent post by Stefano Marinelli:
Stefano’s article inspired me to try Mastodon on FreeBSD. My implementation takes that foundation and extends it for a more maintainable, production-ready architecture.
Design Goals
The motivation behind this move:
- Central PF firewall – all filtering, NAT, and routing are handled by the host only. Jails see a clean, local L2 view - no PF inside jails, no double NAT.
- Separation of concerns – every jail runs exactly one functional service:
nginx— reverse proxy + TLS terminationmastodonweb— Puma / Rails web backendmastodonsidekiq— background jobsdatabase— PostgreSQL and Valkey (Redis fork)
- Host‑managed source – Mastodon source tree shared via
nullfsbetween web and sidekiq jails. Common.env.production, shared dependencies, single codebase to maintain. - Clean dual‑stack (IPv4 + IPv6) – every component visible under both protocols; no NAT66 or translation hacks.
- Predictable networking – each functional group lives on its own bridge with private address space.
Jail and Network Overview
Example address plan (using RFC 5737 and 3849 documentation spaces):
| Jail | Purpose | IPv4 | IPv6 |
|---|---|---|---|
| nginx | Reverse proxy | 192.0.2.13 | 2001:db8:8000::13 |
| mastodonweb | Rails backend | 198.51.100.9 | 2001:db8:9000::9 |
| mastodonsidekiq | Workers | 198.51.100.8 | 2001:db8:b000::8 |
| database | PostgreSQL + Valkey | 198.51.100.6 | 2001:db8:a000::6 |
| Host | “burningboard.example.net” | 203.0.113.1 | 2001:db8::f3d1 |
Each functional bucket gets its own bridge(4) interface on the host (bastille0..bastille3) and its own /24 and /64 subnet.
Jails are created and attached to the corresponding bridge.
Schematic diagram
[ Internet ]
|
v
[ PF Host ]
├── bridge0 — nginx (192.0.2.13 / 2001:db8:8000::13)
├── bridge1 — mastodonweb (198.51.100.9 / 2001:db8:9000::9)
├── bridge2 — database (198.51.100.6 / 2001:db8:a000::6)
└── bridge3 — sidekiq (198.51.100.8 / 2001:db8:b000::8)
With the address plan established, the next step is creating the individual jails and assigning virtual network interfaces.
Jail Creation and Per‑Jail Configuration
Each jail was created directly through Bastille using VNET support, attaching it to its respective bridge.
For example, creating the nginx frontend jail on the bastille0 bridge:
bastille create -B nginx 14.3-RELEASE 192.0.2.13 bastille0
Bastille automatically provisions a VNET interface inside the jail (vnet0) and associates it with the corresponding bridge on the host.
Inside each jail, the /etc/rc.conf defines its own network interface, IPv4/IPv6 addresses, default routes, and any service daemons enabled for that jail.
Example configuration for the database jail (substituted with documentation addresses):
ifconfig_e0b_database_name="vnet0"
ifconfig_vnet0="inet 198.51.100.6 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:a000::6/64"
ifconfig_vnet0_descr="database jail interface on bastille2"
defaultrouter="198.51.100.1"
ipv6_defaultrouter="2001:db8:a000::1"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
valkey_enable="YES"
postgresql_enable="YES"
Each jail is therefore a fully self‑contained FreeBSD environment with native rc(8) configuration, its own routing table, and service definition. Bastille’s role ends at boot‑time network attachment - the rest is standard FreeBSD administration.
Host /etc/rc.conf
Below is a simplified version of the host configuration that ties everything together.
Each jail bridge subnet is assigned both IPv4 and IPv6 space; the host acts as gateway.
# Basic host config
hostname="burningboard.example.net"
keymap="us.kbd"
# Networking
ifconfig_vtnet0="inet 203.0.113.1 netmask 255.255.255.255"
ifconfig_vtnet0_ipv6="inet6 2001:db8::f3d1/64"
defaultrouter=="203.0.113.254"
ipv6_defaultrouter="2001:db8::1"
# Bridges for jails
cloned_interfaces="bridge0 bridge1 bridge2 bridge3"
ifconfig_bridge0_name="bastille0"
ifconfig_bridge1_name="bastille1"
ifconfig_bridge2_name="bastille2"
ifconfig_bridge3_name="bastille3"
# Bridge interfaces for individual networks
# Frontend (nginx)
ifconfig_bastille0="inet 192.0.2.1/24"
ifconfig_bastille0_ipv6="inet6 2001:db8:8000::1/64"
# Mastodon Web (Rails / Puma)
ifconfig_bastille1="inet 198.51.100.1/24"
ifconfig_bastille1_ipv6="inet6 2001:db8:9000::1/64"
# Database
ifconfig_bastille2="inet 198.51.100.2/24"
ifconfig_bastille2_ipv6="inet6 2001:db8:a000::1/64"
# Sidekiq (workers)
ifconfig_bastille3="inet 198.51.100.3/24"
ifconfig_bastille3_ipv6="inet6 2001:db8:b000::1/64"
gateway_enable="YES"
ipv6_gateway_enable="YES"
# Services
pf_enable="YES"
pflog_enable="YES"
bastille_enable="YES"
zfs_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
This provides proper L3 separation for each functional group.
In this layout, bastille0 → frontend, bastille1 → app, bastille2 → DB, bastille3 → worker pool.
/etc/pf.conf
The host firewall serves the dual purpose of NAT gateway and service ingress controller.
Below is an anonymized but structurally identical configuration.
# --- Macros ---
ext_if = "vtnet0"
jail_net = "198.51.100.0/20"
jail_net6 = "2001:db8:8000::/64"
host_ipv6 = "2001:db8::f3d1"
frontend_v4 = "192.0.2.13"
frontend_v6 = "2001:db8:8000::13"
# Trusted management networks (example)
trusted_v4 = "{ 203.0.113.42, 192.0.2.222 }"
trusted_v6 = "{ 2001:db8:beef::/64 }"
table <bruteforce> persist
set skip on lo0
set block-policy drop
set loginterface $ext_if
scrub in all fragment reassemble
scrub out all random-id max-mss 1500
# --- NAT ---
# Jails -> egress internet (IPv4)
nat on $ext_if inet from $jail_net to any -> ($ext_if)
# --- Port redirection ---
# Incoming HTTP/HTTPS -> nginx jail
rdr on $ext_if inet proto tcp to ($ext_if) port {80,443} -> $frontend_v4
rdr on $ext_if inet6 proto tcp to $host_ipv6 port {80,443} -> $frontend_v6
# --- Filtering policy ---
# Default deny (log for audit)
block in log all
block out log all
# Allow existing stateful flows out
pass out all keep state
# Allow management SSH (example port 30822) only from trusted subnets
pass in quick on $ext_if proto tcp from $trusted_v4 to ($ext_if) port 30822 \
flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)
pass in quick on $ext_if inet6 proto tcp from $trusted_v6 to $host_ipv6 port 30822 \
flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)
# Block all other SSH
block in quick on $ext_if proto tcp to any port 30822 label "ssh_blocked"
# ICMP/ICMPv6 essentials
pass in inet proto icmp icmp-type { echoreq, unreach }
pass in inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }
# Inter-jail traffic
# nginx -> mastodonweb
pass in quick on bastille0 proto tcp from 192.0.2.13 to 198.51.100.9 port {3000,4000} keep state
pass in quick on bastille0 proto tcp from 2001:db8:8000::13 to 2001:db8:9000::9 port {3000,4000} keep state
# mastodonweb -> database (Postgres + Valkey)
pass in quick on bastille1 proto tcp from 198.51.100.9 to 198.51.100.6 port {5432,6379} keep state
pass in quick on bastille1 proto tcp from 2001:db8:9000::9 to 2001:db8:a000::6 port {5432,6379} keep state
# sidekiq -> database
pass in quick on bastille3 proto tcp from 198.51.100.8 to 198.51.100.6 port {5432,6379} keep state
pass in quick on bastille3 proto tcp from 2001:db8:b000::8 to 2001:db8:a000::6 port {5432,6379} keep state
# Optional: temporary egress blocking during testing
block in quick on { bastille0, bastille1, bastille2, bastille3 } from $jail_net to any
block in quick on { bastille0, bastille1, bastille2, bastille3 } inet6 from $jail_net6 to any
# External access
pass in quick on $ext_if inet proto tcp to $frontend_v4 port {80,443} keep state
pass in quick on $ext_if inet6 proto tcp to $frontend_v6 port {80,443} keep state
This PF configuration centralizes control at the host. The jails have no firewall logic - just clean IP connectivity.
Shared Source Design
Both mastodonweb and mastodonsidekiq jails mount /usr/local/mastodon from the host:
/usr/local/mastodon -> /usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon
/usr/local/mastodon -> /usr/local/bastille/jails/mastodonsidekiq/root/usr/home/mastodon
Example fstab entry:
/usr/local/mastodon /usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon nullfs rw 0 0
That way, only one source tree needs updates after a git pull or bundle/yarn operation. The jails simply see the current state of that directory.
Logs and tmp directories are symlinked to /var/log/mastodon and /var/tmp/mastodon inside each jail for persistence and cleanup.
Service Boot Integration
Each Mastodon jail defines lightweight /usr/local/etc/rc.d scripts:
#!/bin/sh
# PROVIDE: mastodon_web
# KEYWORD: shutdown
. /etc/rc.subr
name="mastodon_web"
rcvar=mastodon_web_enable
pidfile="/var/run/mastodon/${name}.pid"
start_cmd="mastodon_web_start"
stop_cmd="mastodon_web_stop"
mastodon_web_start() {
mkdir -p /var/run/mastodon
chown mastodon:mastodon /var/run/mastodon
su mastodon -c "export PATH=/usr/local/bin:/usr/bin:/bin; \
export RAILS_ENV=production; export PORT=3000; \
cd /home/mastodon/live && \
/usr/sbin/daemon -T ${name} -P /var/run/mastodon/${name}_supervisor.pid \
-p /var/run/mastodon/${name}.pid -f -S -r \
/usr/local/bin/bundle exec puma -C config/puma.rb"
}
mastodon_web_stop() {
kill -9 `cat /var/run/mastodon/${name}_supervisor.pid` 2>/dev/null
kill -15 `cat /var/run/mastodon/${name}.pid` 2>/dev/null
}
load_rc_config $name
run_rc_command "$1"
Equivalent scripts exist for mastodon_streaming and the Sidekiq worker.
Everything integrates seamlessly with FreeBSD’s native service management:
service mastodon_web start
service mastodon_streaming restart
service mastodonsidekiq status
No Docker, no systemd, no exotic process supervisors.
Why It Matters
The resulting system is simple, observable, and robust:
- Firewall rules are centralized and auditable.
- Each jail is a clean service container (pure FreeBSD primitives, no overlay complexity).
- IPv4/IPv6 connectivity is symmetrical and clear.
- Source and configs are under full administrator control, not hidden in containers.
It’s also easy to snapshot with ZFS or promote new releases jail-by-jail using Bastille’s clone/deploy model.
Summary
In short:
- Host does PF, routing, NAT, bridges
- Each jail has exactly one purpose
- Source code lives once on the host
- Dual-stack networking, no translation
- Everything FreeBSD-native
This structure makes it easy to reason about - each moving part has one job.
That’s how I like my infrastructure: boringly reliable.