Note: these are my recommendations, but there are other completely valid ways to spend your money. For example, there's probably also some regime where rather than getting 4 rtx6kpros, you allocate most of your money to building out a linked 4x DGX Spark cluster for a total of 512GB VRAM and use that as the slow, big brain to drive Qwen3.7-27b to do the rote tasks quickly.

Here's the hardware I wound up purchasing for the 4x RTX 6000 pro machine.

A modest, last-gen EPYC system purchased in parts almost entirely from eBay.

I had to custom fabricate a wood enclosure for the PCI switch and GPUs, which took about a day.

I found the PCI switch's builtin fan very loud and seemingly useless, so I simply unplugged that from the board.

I save all model weights locally on a ZFS filesystem that's replicated across the two 8TB drives, which is mounted at ~/storage.

For any model I want to run, I first download the model using

hf download <model-name> --local-dir ~/storage/<model-name>

Once the model weights are cached locally, I have a specific directory for each model that contains a docker-compose.yml file that cordones off the running of each model to its own Docker container.

You can find these configurations in ./runners/.

Each container mounts in ~/storage/models in read-only mode to obtain the weights that I've cached locally.

I then use opencode hosted on a VM on another machine to access the models once they're serving on http://clank.j.co:5000.

I use a network-internal DNS server to point clank.j.co to the LLM machine, but you could simply do http://:5000 too.

There was a lot of fiddling with the BIOS in order to make sure the motherboard wasn't downregulating the PCI switch speeds.

# /etc/default/grub
GRUB_CMDLINE_LINUX="iommu=off amd_iommu=off nomodeset"
sudo update-grub

# nvidia_uvm P2P fix
echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf
sudo update-initramfs -u

Without iommu=off, NCCL hangs on multi-GPU P2P.

With ACS enabled (default), P2P traffic gets bounced through the CPU root port instead of staying inside the switch fabric, negating the switch entirely. pcie_acs_override requires a patched kernel, so we disable via setpci at runtime.

# /usr/local/bin/disable-acs.sh
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: must be run as root"
  exit 1
fi

for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
  setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
  if [ $? -ne 0 ]; then
    continue
  fi
  echo "Disabling ACS on $(lspci -s ${BDF})"
  setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done

Run on every boot via systemd oneshot:

# /etc/systemd/system/disable-acs.service
[Unit]
Description=Disable PCIe ACS for GPU P2P
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disable-acs.sh

[Install]
WantedBy=multi-user.target

Verify: lspci -vvv | grep ACSCtl should show all minus signs, and nvidia-smi topo -m should show PIX between all four GPUs (not PHB/NODE).

Use ./tools/measure-gpu-speed.sh to measure this easily.

In order to avoid installing a 220V circuit, I (probably unwisely) run this rig on a single 110V circuit, but I power regulate the cards.

Persistence mode + power cap applied at boot via systemd (install-gpu-power-limit.sh):

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 350    # 350W per GPU (default 600W)

350W/GPU = 1,400W GPU load, sized for the PSU budget. During the interim single-1700W-PSU phase (before the 240V circuit), cards ran at ~260W (4×260 = 1,040W GPUs + ~280W system ≈ 1,320W total).

Verify: nvidia-smi --query-gpu=index,power.limit,power.draw --format=csv

Upstream: Gen4 x16 (~30 GB/s to CPU). P2P through switch: 27.5 GB/s unidirectional / 50.4 GB/s bidirectional, 0.37–0.45 µs latency, i.e. Gen4 line rate. Note: lspci may still show downstream GPU links as "2.5GT/s (downgraded)" at idle if ASPM is active anywhere; this is cosmetic. Links retrain to Gen4 under load.