07Apr2026 https outage
Starting on February 25th and lasting a little over a month, acme.com
suffered intermittent network outages.
The symptoms were very high ping times and packet drops.
The outages would last a few hours and then go away for a while.
The problems started right after my internet provider, Sonic, did some scheduled maintenance and switched me to a new network. I worked with Sonic support to try and figure out if their network change, or my own config changes for the new network, could have caused the problems. I did find some things I did wrong, but fixing them didn't fix the outages.
A few days ago I was up at 1am, filled with anxiety at yet another outage, and decided to take a harder look at the traffic I was getting. I noticed some interesting things:
- Nearly all the incoming packets were web requests.
- Nearly all of them were for non-existent pages.
- Nearly all of them were on port 443 / https, not port 80 / http.
- Nearly all of them proudly announced in the user-agent that they were LLM scraper bots.
I run two web servers on ACME, one for http that is very fast, and another for https which is kind of slow. Maybe the slow https server was falling behind? I decided to test this: I closed port 443.
The problems went away immediately, and have not returned.
What I think was going on: before Sonic's February 25th maintenance, my https server was probably barely keeping up. Some change during the maintenance - maybe they increased the available bandwidth? - tipped things over into the web server falling behind, some of the time. Maybe when two different bots were pounding on it simultaneously. When the server fell far enough behind, the congestion spread to natd, the Network Address Translation daemon. When natd also got saturated, bam, packets started getting delayed and dropped.
Now closing https service is obviously just a temporary fix, because I do want to provide https, but it's actually not that bad. My legit web traffic is 90% http / 10% https, so I'm losing at most 10% of my traffic. But yeah, I'll be doing a better fix some time soon.
Also, this is a bigger problem than just me. The LLM companies are not picking on me in particular, they are pounding every site on the net. I know of two other hobbyist-level sites which are also having problems due to this. Someone really ought to do something about it.
Back to ACME Updates.
