The old design when working with bridges and VLANs was like so:

• Create a bridge for a specific VLAN
• Create a VLAN interface off a physical interface
• Attach the VLAN interface to the bridge
• Now all your bridge members are able to communicate to that VLAN

As you can imagine this starts to get messy especially if you have a lot of VLANs to work with or try to deploy jails that are meant to communicate over multiple VLANs. An example from my old rc.conf looked something like this:

# Bhyve and VNET
ifconfig_ix1="up -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro"

vlans_ix1="vlan2 vlan3 vlan128"
create_args_vlan2="vlan 2"
create_args_vlan3="vlan 3"
create_args_vlan128="vlan 128"

cloned_interfaces="bridge0 bridge1 bridge2"

ifconfig_bridge0_name="vlan2bridge"
ifconfig_vlan2bridge="addm vlan2 up"

ifconfig_bridge1_name="vlan3bridge"
ifconfig_vlan3bridge="addm vlan3 up"

ifconfig_bridge2_name="vlan128bridge"
ifconfig_vlan128bridge="addm vlan128 addm"

It's a lot of work for something that feels so simple if you're familiar with real network gear.

The new design allows you to have a single bridge and specify tagged and/or untagged VLANs for each bridge member. Here's what all of that condenses down to.

# Bhyve and VNET
ifconfig_ix1="up -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro"

cloned_interfaces="bridge0"
ifconfig_bridge0="vlanfilter addm ix1 tagged 2,3,128"

That's it. Much simpler, isn't it? Please don't over look the vlanfilter flag on the bridge. If you are missing this you can add members to the bridge with untagged VLANs but you'll get this error when trying to add tagged VLANs:

ifconfig: BRDGSIFVLANSET ix1: Invalid argument (extended error VLAN filtering not enabled)

VNET jails posed a problem for me because I was relying on the "unofficial" /usr/share/examples/jails/jib script which can handle creating epair(4) devices for the jails. I'd copy this script into my PATH, make it executable, and then I could use it. A simplified example of this jail configuration looks like this:

webserver {
    exec.prestart += "jib addm $name vlan2";
    exec.poststop = "jib destroy $name";
    vnet;
    vnet.interface = "e0b_$name";
}

This would handle creating a stable name of epair(4) devices and attach it to the right bridge. e.g., "vlan2" becomes "vlan2bridge" in the jib script.

The jib script is old and has some features that I don't believe are necessary anymore. One feature tries to ensure your MAC addreses are stable, but this seems to be a native kernel feature of epair(4) now:

$ sysctl -d net.link.epair.ether_gen_addr
net.link.epair.ether_gen_addr: Generate MAC with FreeBSD OUI using ether_gen_addr(9)

And ether_gen_addr(9) says:

By default, ether_gen_addr attempts to generate a stable MAC address
using the hostid of the jail that the ifp is being added to. During
early boot, the hostid may not be set on machines that haven't yet pop-
ulated /etc/hostid, or on machines that do not use loader(8).

Great, we seem to be covered there. But I still need a way to make stable epair device names for the jails and attach them to the bridge the new way, so I wrote a script for that:

#!/bin/sh
# /scripts/vnetif

set -eu

ENAME=$1
BRIDGE=$2
VLAN=$3

NEW_EPAIR=$(ifconfig -D epair create -vlanhwfilter up)
EPAIR_NUM=$(echo ${NEW_EPAIR##epair} | tr -d '[a]')

# Do not even want ipv6 link local on here, layer3 not allowed
# anymore on bridge members
ifconfig ${NEW_EPAIR} inet6 ifdisabled -auto_linklocal -accept_rtadv no_radr

ifconfig epair${EPAIR_NUM}a name e0a_${ENAME}
ifconfig epair${EPAIR_NUM}b name e0b_${ENAME}
ifconfig ${BRIDGE} addm e0a_${ENAME} untagged ${VLAN}

This is not well designed but it gets the job done.

Note, on the epair device I had to set -vlanhwfilter -- another hidden requirement!

Now that this is sorted, my VNET jail config looks like this:

webserver {
    exec.prestart += "/scripts/vnetif $name bridge0 2";
    vnet;
    vnet.interface = "e0b_$name";
}

It works! The VNET interface gets added to the bridge with the correct VLAN membership. It even seems to start the jail faster now...

Notice I don't have an exec.poststop anymore. This seems to not be required as the epair interface is automatically destroyed when the jail is torn down now.

The Bhyve VM solution is not finalized for me right now. I'm using the vm-bhyve tooling to manage my VMs, but it has no support for this new bridging+VLAN functionality. I've resorted to defining a manually managed switch in the configuration and pre-creating tap interfaces for each VM. Technically I could use cloned_interfaces in /etc/rc.conf and add them all to the bridge when the bridge is created, but as the number of VMs grow the configuration for this is getting really long and messy. I've opted to sidestep it for the moment and create all the tap interfaces in /etc/rc.local, and then the VM config looks like:

loader="grub"
cpu="2"
memory="4G"
network0_type="virtio-net"
network0_switch="bridge0" # the predefined manual bridge config in /vm-bhyve/.config/system.conf
network0_device="tap133" # my precreated tap interface
disk0_type="nvme"
disk0_name="disk0.img"
uuid="2109c019-9fd0-11f0-9ab1-d05099db8057"
network0_mac="58:9c:fc:0f:74:60" # seems to be ineffective now
disk1_type="ahci-cd"
disk1_name="seed.iso"
disk1_dev="file"

So if you want a stable MAC address, you need to set it where you create the tap interface.

For the record, that /vm-bhyve/.config/system.conf looks like:

switch_list="bridge0"
type_bridge0="manual"
bridge_bridge0="bridge0"

and my /etc/rc.local has entries like:

# for VM such-and-such
ifconfig tap129 create up
ifconfig bridge0 addm tap129 untagged 2

It's not elegant, but it works well enough while I think of a better solution. When I have one I'll update this blog post. Hopefully this should be able to get you up and running with the new bridge+VLAN design in FreeBSD 15.0-RELEASE.