It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain.

These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now!

Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack.

Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius.

[1] https://docs.github.com/en/actions/deployment/security-harde...

Yes, they aren’t perfect. They do some things that I disagree with.

But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this.

Thank you for the blog post!

- Insist that you have better integrity than your competitors

- share a few operational investigations after your latest security event

what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.

They mention a third-party audit was conducted, but thats not because they care about you. Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information, it needs to pass a yearly on-premise audit to ensure PCI compliance. if they didnt, major credit houses would stop processing their payments.

It's a very good writeup, as these things go. Cloudflare is huge. An ancillary system of theirs got popped as sequelae to the Okta breach. They reimaged every machine in their fleet and burned all their secrets. People are going to find ways to snipe at them, because that's fun to do, but none of those people are likely to handle an incident like this better.

I am not a Cloudflare customer (technically, I am a competitor). But my estimation of them went up (I won't say by how much) after reading this writeup.

IMO it’s more a risk reward trade off. I know some companies are paying relative peanuts in non-compliance fines rather than spend money on some semblance of security which they may still not be compliant with and have to pay the fines anyway…

I guess no need to specific names. I'm just using that as examples.

They clearly defined the scope of impact, and demonstrated that none of this impacts systems in scope for PCI. There was no breach to change management inside of BitBucket, and none of the edge servers processing cardholder data were impacted. They will have plenty of artifacts to demonstrate that by bringing in an external firm. So I am really not clear why you're bringing up PCI at all here. They made it clear no cardholder data was impacted so your perspective on the required "on-site" audits is moot.

Cloudflare operates two entirely different scopes for PCI; The first being as a Merchant where you the customer pays for the services. This is a very small scope of systems. The second is as a Service Provider that processes cards over the network. The network works such that it is not feasible to exfiltrate card data from the network. There are many reasons as to why this is, but they demonstrate year over year that this is not something that is reasonably possible. You can review their PCI AoC and get the details (albeit limited) to understand this better. Or you could get their SOC 2 Type 2 Report which will cover many aspects of the edge networks control environment with much better testing details. After reading that you can then come back to the blog and see that clearly no PCI scoped systems were impacted in a way that would require any on-prem audit to occur.

And they are not a card network. They are a PCI Service Provider because cards transit over their network. They are not at risk of being unable to process payments or transactions for their Merchant scope even if there are issues with their Service Provider scope. Because, again, these are two separate PCI audits that are done, testing two different sets of systems and controls.

And, as an aside, Cloudflare effectively always has on-prem PCI audits occur. Because the PCI QSA's need to physically visit Cloudflare datacenters to demonstrate not only the software side of compliance, but the datacenters deployed globally.

They did, and they admitted that it was their fault. I have to give them credit for that much.

> They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023...The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

Like I get cynicism, but they very clearly explained the lead-up to the accounts being compromised and the mistakes that caused that. They took full accountability of it. Which is frankly more than most companies dealing with security incidents. This entire write-up is more than most companies obligations or responses.

No competitors were mentioned in Cloudflares article, they explained what kind of information was breached, nothing to do with payment/card info... so I doubt you even read past the first few paragraphs/conclusion.

No payment card information was compromised.

For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company.

Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.

They had government engagements with Google's consent, and all those various engagements could be disclosed to each other?

If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?

Australians get the 'opportunity' to be part of that sort of that sort of espionage as a base level condition of citizenship [0].

As an upside, I guess it helps with encouraging good practices around zero trust processes and systems dev.

[0]: https://en.wikipedia.org/wiki/Mass_surveillance_in_Australia...

Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat.

The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.

How exactly ?

Nothing out of the ordinary/regular infiltration, investigation and attempt to move laterally is exposed.

[1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm)

I'm curious if they're rethinking being on Okta.

Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.

January 2022: https://blog.cloudflare.com/cloudflare-investigation-of-the-...

October 2023: https://blog.cloudflare.com/how-cloudflare-mitigated-yet-ano...

It's fair to "punch down" imo as that's how the credentials were originally compromised. I'd agree with you if CF were trying to minimize their own mistake but that doesn't seem to be what is happening here

I don't love CF, but IMO Okta deserves to be punched down on.

Is it really reasonable to come out and say your company utterly failed a pretty basic security practice when faced with a compromise but that it was really some other company's problem originally?

Of course it's not. It's still your company's failure. Own it.

If anything Okta is a bigger company (by revenue, by employee count) and they were founded a year earlier.

I am grandfathered in to an old MacBook that has absolutely no management software on it, from the “Early Days” when there was no IT and we just got brand new untouched laptops.

They offered me an upgrade to an M1/M2 pro, but I refused, saying that I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.

Since that would hugely disrupt my work, I can’t upgrade. Maybe I can use incidents like this to justify my beliefs to the IT department…

Okta doesn't make device management software, thats made by companies like Jamf. Okta can integrate with them but Okta isn't what manages your laptop at all.

> I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.

Do not do this, its not a personal device.

You think nobody's logged into their personal spotify on their work computer? All those guys wearing headphones in the office have brought in CDs to play in their laptop CD drives?

And that business traveller away from their partner and kids for a week+ isn't going to video call them? Or watch some netflix in their hotel room in the evening?

That's so unrealistic, you could write IT security policy for a Fortune 100 company :)

Why would I use a device to do personal things that they MITM everything I do on it? Privacy is too important to me to give it away like that. I'm sure all traffic on the corporate network is logged. Why open myself up for grounds for termination if my company hits hard times and wants to lay people off?

Been in the industry for a while now, no one cares if you pull out your phone. Generally, people treat others like adults not children.

It was a public case, but the essentially unanimous Supreme Court opinion in City of Ontario v. Quon [0, 2010] shows what expectations of privacy you should have on any work devices -- none.

[0] https://en.m.wikipedia.org/wiki/City_of_Ontario_v._Quon

Reasonable or limited private use of a work computer remains private.

https://edps.europa.eu/data-protection/data-protection/refer...

I've worked for large media companies where this is exactly the only way to have music available. The production network was blocked from accessing the www. To ensure content wasn't pirated, the original media had to be used. No CD-Rs were allowed. Personal devices were kept in lockers outside the restricted areas, so no streaming from them either.

Email was from a remote session. If you were emailed an attachment necessary for production work, there was an approved workflow to scan the data and then make it available to the production network.

So, while you were trying to be sarcastic, there are networks that are set up exactly like you thought didn't exist because it was too outlandish.

I refuse to carry more than one phone or one laptop, and I sure ain’t brining a personal device into a country I wouldnt go to on vacation.

Even companies with these policies preinstall Spotify on work computers.

Well... don't do that? Why would you ever have personal anything on a work computer?

I just don’t understand any rational otherwise.

So, would you care to more explicitly tell me what you think about my intelligence or ability to behave rationally compared to you? Or is there potentially some room for nuance here?

A more senior academic?

- Is the open-source software something that the company is sponsoring?

- If not, do you have permission to use company equipment for personal use?

> A more senior academic?

?

Do you do the above? If so, do you have a personal laptop? if yes, why utilize company property instead of personal, unless given permission to do so?

An example university policy [1]

> 11.5 reasonable personal use of College IT resources is permitted provided such use does not disrupt the conduct of College business or other users. Recreational use of the Halls of Residence network is also permitted, subject to these conditions;

We have a similar policy where I work. I have a personal laptop, but I don't take it to work. I am signed in to my personal GMail account on my work computer, along with many other accounts — like this HN account. If work needed to look at an employee's computer, we'd have someone from IT + someone from HR overseeing the process, and wouldn't look at anything clearly private, e.g. a personal email account. Doing otherwise would be a breach of the GDPR.

[1] https://www.imperial.ac.uk/admin-services/ict/self-service/c...

(& then just rotate the credentials on it when you part ways with the employer.)

Some of my co-workers even do a Github account per employment.

I'm sure there are plenty of people who have access to their company's private repos through their personal GitHub accounts.

No thanks. New account per job.

Involvement with private repositories is removed as soon as the organization removes the employee, or the employee removes themselves.

I think the horror stories could only happen if the individual's account has been used for generating many API keys or similar, but there are other reasons not to rely on that sort of thing.

Even if a tiny risk, it seems silly just to bolster the GH activity graph.

I think this is fairly common for people who work on open source projects.

* personal ChatGPT and copilot subscriptions, since company doesn’t pay for these

* Trello account for keeping track of my todo list (following up with people, running deploys)

* Obsidian for keeping notes, as a personal knowledge-base (things like technologies and reminders)

* Apple account for music, copy/paste, sharing photos from my travel with coworkers, synching docs related to my work visa and taxes

* Personal slack login for communicating with my partner in our private server

* personal GitHub account credentials for synching my private dotfiles repo with my neovim config. basically can’t work without my dotfiles, but I could theoretically email these to myself or something, to prevent this one.

And sure, I could be stubborn and not use any of this, but I’d be way less productive and kinda miserable.

* Stack Overflow

* Job Search sites

I don't remember if Jetbrains needs a password to get to personal licenses, but they definitely do to use their bug database. I suspect they're not the only one.

Letting other people blow off steam can be an act of self-preservation. Insisting that people only ever do 100% work things at work or on work hardware slightly raises your low-but-never-zero chances of being murdered by coworkers. Or less ironically, hilariously intense bridge-burning activities.

Also most of this conversation is happening during work hours so I think we can infer that grandparent is being a little hypocritical.

I’m now understanding how people get sued when going from company to company.

You are making your work take on an extraordinary risk in hiring you.

My notes are text files on the computer, so we’d have problems regardless if they got that. But maybe I should’ve left it out of the list above in that case… nothing else seems very damning.

But you do raise a valid concern, and it’s worth reevaluating!

> personal Obsidian for keeping meeting notes, and recording conversations as a personal knowledge-base

I'm not a lawyer, but I'm pretty sure these could subject a lot of your other personal data to potential subpoena should your employer get sued by a sufficiently determined attacker.

Don't cross the streams.

> Obsidian is free for personal and non-profit use. However, if you use Obsidian for work-related activities that generate revenue in a company with two or more people, you must purchase a commercial license for each user. Non-profit organizations are exempt from this requirement.

https://obsidian.md/license

> Q3. Can I buy a license for myself, or do I have to ask my company to buy it for me? > Yes, you can buy a license for yourself; just put your name in the company > field. You can use such a license to work for any company.

https://help.obsidian.md/Licenses+and+payment/Commercial+lic...

Does your workplace restrict you from bringing it in?

I’m fine with it because I know there’s no management software on this laptop, but yeah it’s a totally different story if I had to use a newer one with SSO and management software

At least that's how it works in the vast majority of companies.

It’s really easy to say ‘don’t use your personal stuff at work’, but when work is some locked-down behemoth whose view of productivity software is ‘just use Office’, and you’re really trying to be better at your job, using your own tools can be the only solution.

And in my situation, yeah, they didn’t want you bringing things in. I worked in a secure area.

Finding solutions to work around the systems, on your own time and dime, only hurts in the long run.

They think everything is fine. Nothing will ever get fixed. Voice these concerns.

The tools are the tools. There’s nothing me or my boss or theirs can do about it.

They just don’t care. But I care, because if nothing else it’s my reputation.

(HP used purely for size comparison. I’ve never worked there.)

You might feel a sense of social obligation or solidarity with the company. I usually do. But if I was placed in a dehumanizing situation like that – forced to work inefficiently due to overly rigid policies that assume everyone’s needs are the same – well, whether I worked around it or not, my empathy for the company would be at a nadir whenever I thought about it.

Also, people just do things for convenience. (Although I tend to pipe these passwords over an SSH connection, so that they're not resident on the work laptop. Though there is a good argument to be had about me permitting my work laptop SSH access to my personal laptop. From a technical standpoint, my employer could hack/compromise my personal laptop. From a legal and trust standpoint, I presume they won't.)

You trust all personnel with access to your employers network?

What's more surprising is that they trust you to setup adhoc ssh connections to arbitrary endpoints; unless you're the person in charge of network security?

Would anyone notice if you, or an intruder, dumped terabytes of data over that connection?

I don't work in IT but this just doesn't feel right to me.

Absolutely none of my personal stuff ever touches a corporate machine. Ever. I wouldn't even log in to the W2 downloading app as an employee from the work machine.

Granting work ssh keys access to your personal machine is crazy; if your work machine gets compromised, they steal your entire personal system's home directory too. Why would you unnecessarily expand the blast radius of a compromise like this?

You wouldn’t keylog “random devs”, you’d keylog all of the ones doing ops.

GP said nothing of the sort.

I wouldn't be surprised if they are working on first party IAM user support though.

For example, if Slack devops team were to exclusively communicate over Slack, then a Slack outage would be much harder to resolve because the team trying to fix it would be unable to communicate.

They are using zero trust and explained that it's why the scope of the security incident was extremely limited.

Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?

That is, I'd expect there was a flag in a database somewhere saying that those service accounts were "abandoned" or "cleaned up" or some other non-active status, but that this assertion was incorrect. Then they probably rotated all the passwords for active accounts, but skipped the inactive ones.

Speaking purely about PKI and certificate revocation, because that's the only similar context that I really know about, there is generally a difference between allowing certificates to expire, vs allowing them to be marked as "no longer used", vs fully revoking them: a certificate authority needs to do absolutely nothing in the first case, can choose to either do nothing or revoke in the second case, and must actively maintain and broadcast that revocation list for the third case. When someone says "hey I accidentally clobbered that private key can I please have a new cert for this new key," you generally don't add the old cert to the revocation list because why would you.

Great call out too

> Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.

> The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.

Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.

Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.

It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.

Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.

The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.

Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery.

And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.

Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.

(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)

https://twitter.com/taviso/status/1566077115992133634

> True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF?

For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare:

https://twitter.com/taviso/status/832744397800214528

> Could someone from cloudflare security urgently contact me.

This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone who worked as a fairly junior staffer at the FTC. On the employee’s personal time they mentioned being frustrated by how the disclosure took place to the person they were dating. I believe the employee’s frustration was because we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up.

There was never anything that Cloudflare or any executive raised with the FTC. And the FTC never took or even considered taking any action. The junior FTC staffer may have said something to Tavis or our employee may have said something about telling the staffer they were dating, but that was the extent of it.

I understand Tavis’s perspective, and agree it was inappropriate of the former Cloudflare employee, but this was two people not in any position of leadership at either Cloudflare or the FTC talking very much out of school.

This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline.

If you working there and having done research can get it this wrong while it's super easy to find the event log in the open, it doesn't give a very good vibe about the attitude inside cloudflare regarding what happened and fair disclosure.

Full even log on project zero is here : https://bugs.chromium.org/p/project-zero/issues/detail?id=11...

> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

Meanwhile link with Cloudflare went from this

> I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.

> I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.

To this

> Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.

> Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.

> They then told me Wednesday, but in a later reply started saying Thursday.

> I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks. https://security.googleblog.com/2013/05/disclosure-timeline-...

> If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.

> I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.

> They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.

> Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification.

So it was not project zero but cloudflare that moved the disclosure timeline around, and did so without keeping pzero in the loop, about an active in the wild exploit.

For context: you are answering to the co-founder & CEO of Cloudflare.

> However, Server-Side Excludes are rarely used and only activated for malicious IP addresses.

So… you’re celebrating that you only had buffer overruns for malicious IP addresses?

I can just imagine the attackers licking their lips when they first breached the data center.

Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)

I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext.

At the very least you need to tell the server what to trust.

That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user Cloudflare origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.

thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".

This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).

Maybe naively, I wish this assumption became universal.

Aha, the old replace-your-trusted-hardware trick.

If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this.

Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.

If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive.

† https://twitter.com/badthingsdaily?lang=en

You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.

obviously a customer data breach would be worse but this is really no bueno

The customer data next year is not the same as the customer data this year.

In the git universe there is a pretty short list of services, locally or hosted that you would probably use as an entity as large as cloud flare.

This odd to me - unused credentials should probably be deleted, not rotated.

1: "what are these accounts?"

2: "oh they're unused, they don't even appear in the logs"

1: "we should rotate them"

2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons"

?

I'd definitely consider a "silent" credential - a credential not registered centrally - to be a huge red flag. Either it could get stolen, or break and no one knows how to regenerate it. And it's pretty easy as devs to quickly generate an auth key that ends up being used permanently, without any documentation.

But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.

In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.

Am I missing something here?

There’s no machine cert used? AuthN tokens aren’t cryptographically bound?

This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”

If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.

This seems incredibly wasteful.

Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.

> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers.

It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad.

Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.

Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.

When does the US attack private enterprise?

Also remember the Google sniffing?

https://www.theregister.com/2013/11/07/google_engineers_slam...

As for covert ops, well, they're covert. I don't have any evidence (hence I said "I'm guessing") but that's how I understand secretive agencies do things. If you look at all of the agencies involved in Stuxnet, you'd get the idea that allied countries' secret services tend to work together (or for each other) to some degree when it suits them.

There's many such cases. They're well known for spying on Siemens as well. With allies like the United States, who needs enemies?

[1] https://www.spiegel.de/politik/ausland/wikileaks-enthuellung...

There was also inferences that they penetrated Huawei.

I might be willing to give you Huawei if you cite a source. They're a gray area (by design) due to China's strategy of military-civil fusion.

https://en.wikipedia.org/wiki/Military-civil_fusion