My dream is to intercept the write-enable lines on the flash chips holding these firmwares so I can lock out updates. And schedule a daily reboot for any memory-resident-only crap.

That’s what we used to do on, ahem, satellite receivers, 20 years ago and maybe we all need to treat every device attached to the internet as having a similar susceptibility to “electronic counter-measures”.

Or at least monitor them for updates and light up a light when an update happens if it was my own equipment and I’d know if it should go off or not.

But what I don't get in this case is why it was not possible to reset the device to its original state. It seems like a misdesign if it's possible to destroy all of the firmware, including the backup.

I think that could work, to a degree. There's always the risk that your recovery mechanism itself it exploited, so you need to make it as small and hardened a target as possible and reduce its complexity to the bare minimum. That doesn't solve the problem, which might be inherently unsolvable, but it may reduce that likelihood of it to levels where it's not a problem until long past the lifecycle of the devices.

Almost all devices have something like that already in the form of a bootloader or SOC bootstrapping mode. But the idea breaks down if you want to do it OTA. The full storage/kernel/network/UI stack required to make that happen isn't ever going to run under "ROM" in the sense of truly immutable storage.

The best you get is a read-only backup partition (shipped in some form on pretty much all laptops today), but that's no less exploitable really.

Why not? I'm essentially describing a specialized OOB system, and it would just use a carved out small chunk of system RAM or ship with a minimal amount RAM of its own. If you mean actually impossible to change because it's physical ROM ("truly immutable"), that's less important to the design than there's no mechanism that allows that storage area to be written to from the system itself, whether that's just the very locked down and minimal recovery kernel it houses not allowing it, or a jumper.

You don't even need the rest of the device to contain any signing mechanism with keys that could be compromised, because using this method requires physical access, and any compromise that occurs from physical access can be detected or undone with same by checksumming or re-flashing the storage device again from a clean PC.

And you can also do signed firmware updates OTA without worrying that the device can be bricked by a vulnerability or signing key compromise, because it can always be restored via physical access.

Whether that's likely is entirely based on the cost of the device. Some things are simple and cheap and extra hardware cuts deeply into the profit. Others are not but this sort of thing is also important because they are remote and you don't want to have a person go out on site. When the device is expensive enough or sending someone to the site is expensive enough, "just ship a replacement" is not really a viable solution, unless you're installing it in a high-availability capacity where you can fail over to it without physical intervention.

Obviously it's not a solution for every circumstance. Nothing really is. I don't think it's useful for us to assume that a solution has to be, as that doesn't really help us in the many instances when it's good enough.

Granted, I use it once a year because lightning toasts many of my appliances and I have to wait for the replacement from the ISP.

At least my ISP modems can disable OTA updates. A happy oversight on their part.

Now, as it happens Apple (everyone really, but Apple is a leader for sure) has some great protections in place to prevent that. And that's great. But if you feel you can rely on those protections there's no need to demand the ROM recovery demanded upthread.

Still requires a truck roll but at least you don’t need a hot air workstation.

If the vendor's actually trying to lock down the platform they'll usually burn the JTAG fuses as well. It's hit or miss though, I've definitely come across heavily locked down devices that still have JTAG/SWD enabled.

Edit: To your question, JTAG is usually physical silicon, not part of the bootloader.

That seems like awful design? Can't you have an alternate immutable bootloader that can only be enable with a physical switch? Or via some alternate port or something? That way they can update the live one while still having a fallback/downgrade path in case it has issues.

However I assume that any malware doesn't want to be detected so I would have hard time knowing whether I should flip the switch or not, in a typical scenario.

1) The ISP exposed some form of external management they used to access them they shoudldn't have 2) The attacker overcame whatever security used on said management interface 3) Once in, the attacker could simply overwrite the first few sectors of the nand to make them unbootable without local hardware serial console. 4) There was no failsafe recovery mechanism it would seem

An actual "modem" would mostly likely prove volatile/immutable by nature, but anything with a "router" built into it is far more vulnerable that typically run for poorly secured tiny linux systems, and subject to Chinese enshittification.

This is getting attention because it wasn't incompetence this time.

But how does blank, unprovisioned equipment discover a path to its provisioning server? Especially in light of the new "trusted" push, this is an arms race in a market segment such as routers where there isn't any money for high end solutions - only the cheapest option is even considered.

tl;dr: a social and economic problem, likely can't be fixed with a purely technical solution

I think some routers still have a single flash partition and the update process here is a lot more hairy and will obviously not retain the previous version after an update.

Apart from attacks like this, there's absolutely no reason to have a protected read only copy of the factory firmware. 99.9999% all you would ever need to do to recover from a bad flash is to just fail back to the previous image.

A proper read only factory image would require an extra ROM chip to store it, as well as extra bootloader complexity required to load from ROM or copy to flash on failure. It's just barely expensive enough at scale to not be worth it for an extremely rare event.

Humor me; how would that work? If anything, I'd expect it to be easier to overwrite the inactive slot (assuming an A/B setup, ideally with read-only root). If you really wanted, you could have a separate chip that was read-only enforced by hardware, and I've seen that done for really low level firmware (ex. Chromebook boot firmware) but it's usually really limited precisely because the inability to update it means you get stuck with any bugs so it's usually only used to boot to the real (rw) storage.

But a switch on the route: Flip the switch the router reboots to a known safe OS, that downloads, verifies, and updates the firmware. Then it waits for you to flip the switch back before it will behave as a router again.

Unless attackers manage to steal key-signing codes, and also intercept and redirect traffic to their webserver to send a fake firmware, this seems secure to me. Only downside I'm seeing is that it would be impossible to put in a custom firmware. Maybe add a USB-key firmware option?

The nice thing about emulators is that you could intercept calls that you wanted and send your own response while still taking any and all updates. Hard to break when you have more control than they do.

Other providers that rolled out their own receivers had high control over the receiver firmware and once users figured out how to protect their cards, the receivers became an effective attack vector for the lazy.

But that’s where a lot of the public knowledge about JTAGs really started coming to light. Awfully nice of them to put in a cutout at the bottom of the receiver.

I'd expect that any attempt to lock write enable to the EEPROM would eventually result in your modem failing to provision.

Of course, I don’t think you’re supposed to make mods to your vendor provided equipment…

In the satellite world, this would happen too: old firmware would be cut off. That’s when you go legit for a while with your sub’d card, take the update, and watch your sub’d channels until the new update could be reverse engineered. And probably have some heroes learn the hard way of taking the update and having some negative impacts that are harder to reverse.

And it’s only that “rare unplanned outage” when a malicious update bricks your device. Much worse is a malicious update that doesn’t result in an outage. Probably still rare but that impact though.

Edit: would also add that there’s probably a big firmware chip that changes infrequently, and frequently changing config stored on a separate and smaller chip (like a 24c or 93 series eeprom that holds a few kilobytes). That way you don’t risk bricking your modem by unplugging it at the wrong time.

Imagine the damage that could be done by a malicious actor via the ISPs computers.

Or imagine someone being able to hack the system that does that update even without the ISP.

600K users would be a toy, they could do it to 6 Million.

Doesn't even have to be clever, just brick millions of cablemodems.

North Korea or some other government level entity could manage the resources to figure that out.

If you did want to go this route, a simple fix would be to have the write enable line gated by a hardware one-shot timer (think like a 555 timer) triggered by a physical button on the front (and the button would also reboot the router).

The firmware update sequence would go like: Router prompts user to update using button -> user presses update button -> router reboots to clear malicious software -> when the router comes back up, the write enable gate remains open for $n minutes (and maybe there's a GPIO that can hold the gate open if it is already open) -> router performs software update.

The problem is that there's pretty much no way to do this without adding a $1-2 to the BOM, and no manufacturer of (pro|con)sumer routers will do that.

Edit: Or do what stacktrust wrote and just have a toggle switch (update/secure).

Consumer grade routers often have automatic updates these days.

Some motherboards offer a physical jumper for firmware updates, including x86 PC Engines APU2 coreboot router.

A read-only partition can bootstrap recovery from cloud, if the main firmware or config is damaged.

Some updates are possible with live kernel patching of the memory-resident OS, which can also be used by malware.

Many router malware families don't even try to be persistent. Even Mirai - arguably the most famous router botnet - is not persistent [1]. In case the device is rebooted, it just gets infected again in a few minutes.

It's very important that all network connected devices have an update mechanism, working automatically in the background.

[1] At least the original version. After the code leak people were doing all kinds of updates, so there are some variants that try to be persistent in some cases.

edit they also come with a QR code you can scan on your phone to connect to the wifi without manually copy the password, so people just put up that qr code somewhere and connecting is easy enough.

"internet doesn't work" -> "send me a new router, it broke", problem solved.

Would they be sent back or would an engineer travel to wherever stuff is stored to do the job?

https://learn.microsoft.com/en-us/windows/iot/iot-enterprise...

> Unified Write Filter (UWF) is an optional Windows 10 feature that helps to protect your drives by intercepting and redirecting any writes to the drive (app installations, settings changes, saved data) to a virtual overlay. The virtual overlay is a temporary location that is cleared during a reboot or when a guest user logs off.. Increases security and reliability where new apps aren't frequently added. Can be used to reduce wear on solid-state drives and other write-sensitive media. Optimizing Application load timing on boot – it can be faster to resume from a HORM file on every boot rather than reloading the system on each boot. UWF replaces the Windows 7 Enhanced Write Filter (EWF) and the File Based Write Filter (FBWF).

Couldn't someone grab different firmware versions and compare them?

Looks like they are doing what everyone else is doing and using OpenWrt with a vendor SDK: https://forum.openwrt.org/t/openwrt-support-for-actiontec-t3...

What's interesting here is speculated the vendor send a malicious/broken update: https://www.reddit.com/r/Windstream/comments/17g9qdu/solid_r...

So why is there no official statement from the ISP? If it was an attack shouldn't there be an investigation?

I'm not familiar with how this is handled in the USA but this looks really strange.

Maybe these machines were bot infested and the vendor pushed an update that broke everything?

Maybe it's like in the article and it was a coordinated attack maybe involving ransom and everyone got told it's a faulty firmware update, keep calm?

which is also kind of bad, as the customer I'd like to know if there security incidents.

Has anyone links to firmware images for these devices? Or any more details?

We should assume a decision to make no statement was based on the outcome of an investigation.

I wonder how much of the replacement cost is insured. I am guessing none. Leaving the ISP at severe risk of, er, business discontinuity. Another good reason for no statement.

How does Black Lotus Labs global telemetry know which IP communicated with which other IP if they have control of neither endpoint? Who/what is keeping traffic logs?

If these guys can do it, remind me again how Tor is secure because nobody could possibly be able to follow packets from your machine, through the onion hops, to the exit node where the same packet is available unencrypted...

Tor does defeat this though. Rather than seeing the true destination of your traffic they see that of a Tor exit node.

Seeing a packet heading to a tor exit node and then a similarly sized packet heading onwards a fraction of a millisecond later is a pretty surefire way to spy on individual tor users.

If 1, 2, or possibly all 3 nodes are run by a malicious actor, deanonymization becomes easier. At one point 10% of nodes were run by a single malicious actor: https://therecord.media/a-mysterious-threat-actor-is-running...

IIRC There is an alternate method of connecting to an endpoint which uses a 3rd node as a rendezvous point which is meant to be better, but I forget the name of the process...

Of course, on the other end of the spectrum, the NSA has tapped into core internet links, is recording everything it possibly can, and is keeping it forever.

If we are generous and assume there a zettabyte of data a year that they want to store.

At consumer prices, you would have to pay $10B per year just buying hard drives yet alone the operational costs/redundancy.

The budget for all of the US intelligence services is ~$65B. I think if they wanted to actually do what you are describing it would be the single biggest intelligence expense they have and I don't see how you hide that.

i.e. they can prove I visited a bunch of cloudflare sites, but there are millions of those so who cares?

Backbone routers have no need to implement stateful TCP inspection or deal with the transport layer for TCP, dealing with IP is enough.

Hopefully this is made clear in Windstream's contract terms.

So to capture this, you at a minimum need to be logging every TCP connection's SRC IP and DST IP.

And they seem pretty confident in their worldwide map and fairly exact counts, so I would guess they must have probes covering most of the world doing this monitoring, and it likely isn't just 1-in-a-million sampling either...

You're supposed to be protected by the fact that you're going through multiple nodes before exiting TOR, and traffic should be mixed. Can you find some streams if you have most/all the nodes within your network and can analyze the traffic? Probably some, but the more traffic a node handles the harder it would be.

There is a simpler approach though, which is to just run exit nodes.[1]

1: https://en.wikipedia.org/wiki/Tor_(network)#Exit_node_eavesd...

I believe it does. I believe each destination IP you connect to uses a different exit node. And that'll even switch over time. And even if you connect to 2 destination IPs from the same exit node, I don't think there's a way for the exit node to know those 2 connections were from the same user.

It's likely they just do sampling (think netflow) to get some statistics over the data that's already transiting their network.

When I hear people say they use OpenWRT I assume they have their modem in bridge mode so that it doesn't even have an IP. OpenWRT would save you in that case.

Can't they just stage their updates? Surely, malware authors and users must be too cool for adopting standard prod practices.

Their economic pressures are just different, it's not their own hardware that they're bricking, nor are likely to be held liable for it.

This is just my interpretation, I also found it cryptic.

"Windstream" is mentioned in the first paragraph of the Ars article, while the Lumen post makes references to "a rural ISP" throughout the post.

I've purchased a couple of cheaper commercial WiFi access points, and I've placed them in my house with channels set up to minimize interference.

Prior to this I've gone through several iterations of network products from the likes of Apple, Google, and ASUS, and they all had issues with performance and reliability. For example infuriating random periods of 3-5 seconds of dropped packets in the middle of Zoom conferences and what not.

Since I've rolled my own I've had zero issues, and I have a higher degree of confidence that it's configured securely and is getting relevant security updates. In short, my home network doesn't have a problem unless some significant chunk of the world that's running the same well-known stable Linux distro also has a problem.

There are quite a few machines in this category, and what's in stock at any given time tends to rotate relatively quickly. I think the one I bought might still be available, but you will want to check to see if there is something with specs that will work better for your use case.

I'd be curious to know if it was actually meant to brick or someone f'ed the image and accidentally bricked them trying to be clever.

Also if it was a nation state why would you so publically burn your capability bricking residential routers on an ISP that seems to mostly serve rural areas, if they did it for testing that'd be real dumb.

“This included a drop of ~480k devices associated with Sagemcom, likely the Sagemcom F5380 as both this model and the ActionTec modems were both modems issued by the ISP.”

Not sure if that's enough evidence to say it was them... but I don't see another ISP's logo on these things.