（评论）

（评论）
(comments)

原始链接: https://news.ycombinator.com/item?id=40990768

Kubernetes（通常缩写为“K8s”）设计用于托管共享公共基础设施的多方应用程序，称为多租户 Kubernetes。然而，由于其复杂性和快速变化，不正确的配置或设置过程中出现的错误可能会导致潜在的安全问题。信任经验不足的用户来管理如此复杂的系统可能会带来风险。与标准 Kubernetes 设置相比，在虚拟化之上部署集群的选项可以提高效率并提供更强的隔离。面向多个客户的服务部署最好利用 Kubernetes 进行更高级别的管理，而不是让单个客户创建自己的容器。 Kubernetes 开发的频繁更新和更改可能需要定期重组。小规模、管理良好的集群可以作为经济高效的选择，而不会影响安全性。为了严格的多租户目的，可能需要额外的工具和措施。参考链接： * [Kubernetes 中的多租户概念简介](https://kubernetes.io/docs/concepts/security/multi-tenancy/) * [集群 API 提供程序 NEST GitHub 存储库](https://github.com/kubernetes-sigs/cluster-api-provider-nest)

While I get that it's the AI product, the vulnerability here is the k8s configuration. It really has nothing to do with the AI product itself or AI training or anything related to machine learning or generative AI, it's more about poor cloud computing platform security.

Which is possibly worse lol, the fact SAP a company as big as they are with as much critical information as they have, fucking up basic cloud security, they didn't even fuck up something new they fucked up common shit from the sound of it.

Why do you say that? A bigger company has also more to loose with such a problem and therefore they care much more about such things.

Smaller companies can hide behind obscurity, not big company.

If the same thing happened to a company much smaller then SAP, it wouldn't have made hacker news.

And what consequences will this have for SAP? The same as with Microsoft who had major security fails over the last 20 years yet people still use their products and nearly every company uses Exchange.

A lot of companies are also Too Big To Fail/their products and security are secondary to service and customer relations. IBM can deliver failed product after failed product, and companies still buy from them.

The article doesn't say that it is an issue with the product itself though. It explains very well than infact it's the isolation of the AI training models.

> The root cause of these issues was the ability for attackers to run malicious AI models and training procedures, which are essentially code

It's being researched and investigated, to my understanding, due to the prevalence of AI products and the need to be mindful of the infrastructure.

The brand that sells is the brand at fault.

Securing it or knowing to secure it or testing it or never releasing it until it was secure is all things that are with the brand making the sale.

I hope SAP does a hard retrospective on why Wiz's research was not disrupted before they got full cluster admin. Like, I want to know from SAP's side whether they received any alerts for any of this activity and whether they investigated them properly. I wonder if there is any regulation SAP has to follow that requires them to have adequate alerting for suspicious network activity and whether this research can be used to show that they do not.

The problem is that people who do decisions don’t understand the technology. Most IT managers in Germany do not even know how programming works. There are exceptions, but the biggest players are people flying in blindsight.

That's not something I can confirm. I'm getting around in Europe and if anything, German management tends to be very technical, lots of engineers. Maybe not programmers, true, but bona fide engineers.

Usually security researchers are required to reach out to the target before escalating further into the systems, asking for permissions to proceed. This is also something bug bounty programs require as per their rules for their targets in scope. I’d expect this to be the case here as well, given the researcher is employed by a security company.

Researchers also usually mention which points they asked for additional permissions at in writeups, but now always.

You would be horrified if you would know how much pre-‘20s or even pre-‘10s software is still running in production out there. Here we are talking about a huge enterprise and a somewhat complex migration (from tiller) but you can easily find outdated software without these aggravating circumstances as well.

Software from 2019 is horrifyingly outdated? If updates with security patches exist but haven't been applied, sure, but that's not really a default scenario depending on the stack.

I’ve only used 2020 because of the example in question. Security patches might or might not have been applied in both my imaginary example and in real world.

In my experience, "deprecated" is often taken as "we can still use that, it is not removed yet", which I find somewhat disheartening sometimes.

This is really bad. They are running a single K8s cluster and expecting hard multi-tenancy guarantees?

All the major clouds use vm boundaries and separate K8s clusters between customers. Microsoft was similarly bitten a few years ago with one of their function products that expected K8s to be the primary security boundary.

Hard multi tenancy can't realistically be achieved in the same logical K8s cluster. And it is a moving target, which makes trying to secure it by admission controllers... not a great plan.

One needs to look into things like VirtualClusters to even begin to consider hard multi-tenancy with potential hostile tenants(https://github.com/kubernetes-sigs/cluster-api-provider-nest...). That is just about the control plane. It doesn't even touch the data plane.

How secure that is even with the extra layer, I do not know. Even in the VM land we have seen crazy VM escape exploits over the years.T

> They are running a single K8s cluster and expecting hard multi-tenancy guarantees?

Maybe I missed something in the article but where are they expecting any hard guarantees. If there is a model being trained for example (running arbitrary code) where does a multi K8 tenency play?

The main issue I see is all internal network communication was trusted once behind the proxy/firewall (Istio) but I probably just don't understand k8 clusters too well.

K8S done right is literally designed for multi tenancy. A separate cluster per customer would be insanely costly and terrible for the planet. Maybe in premium products where security is paramount, but a separate cluster per customer is basically setting your money on fire.

Multi-tenant Kubernetes[1] is a strcpy-level footgun IMO. It's perfectly fine as long as everyone involved does everything correctly and never makes a mistake.

Kubernetes itself is very complex. The "who needs a UI when you have configuration files and an API?" approach makes it even more opaque to the people who often end up responsible for it. The landscape changes very rapidly.[2] I'd trust Kelsey Hightower to set up a secure multi-tenant deployment, but probably not anyone else.

Is it not practical to deploy clusters on top of virtualization? That should make efficient use of hardware while still giving each tenant their own cluster, therefore providing stronger isolation than the typical Kubernetes configuration tends to.

[1] I am specifically referring to a Kubernetes deployment where different customers are running custom code on the same underlying hosts. Using Kubernetes to host a service that is multi-tenant at a higher level is not something I would recommend, but it's not as immediately dangerous as a model where customers run container-level custom code.

[2] This is not surprising for a relatively new technology, especially one that's as paradigm-shifting as Kubernetes was. But most people are not going to rearchitect and redeploy everything every six months just because the Kubernetes developers decided to replace a pod security or network security model with a non-backward-compatible alternative again.

> K8S done right is literally designed for multi tenancy

No it is not. Not in the way they are using it.

There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails

The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not.

Straight from the horse's mouth: https://kubernetes.io/docs/concepts/security/multi-tenancy/

There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small.

That said, if hard multi tenancy is what you need, then you need to use things like this: https://github.com/kubernetes-sigs/cluster-api-provider-nest...

(for the control plane - you still need to worry about the data plane)

Companies that penetrate networks uninvited looking for vulnerabilities to create blog content should be prosecuted IMHO. This piece in particular sounds like a hit piece thinly vailed as a vulnerability disclosure.

“We thanked them for their co-operation”. Sounds kinda like extortion.

> Companies that penetrate networks uninvited looking for vulnerabilities to create blog content should be prosecuted IMHO.

Your comment could be rephrased as, "Companies who carelessly collect and store sensitive user data insecurely should not be closely scrutinized, and should be left alone to continue exposing innocent user data to malicious cyber criminals."

Looks a lot different when you look at it from that angle, right?

It's a crime, anything who tries to hack a large company without being invited would get seriously prosecuted,

but as the law practice says "If you have billions of USD, laws don't apply to you anymore".

Using it and loving it. Security aspects aside, it's also the best tool I've tried for proper asset management in multi-cloud scenarios. With the graph feature you can write queries for basically anything, across all accounts if you wish.

This makes me glad I finally talked people at work into running our annual pentests of our products on production, and putting the entire production infrastructure in scope. Focus may be on a specific product or system, but everything is in scope.

And the first test is running, and no one is screaming yet, so fingers crossed.

when you say yearly I assume you're not conducting regular internal pentests?

any pentesting companies that you could recommend which do more than just drive-by shooting with metasploit?

The reported bugs have all been patched, and presumably the compromised secrets have been rotated. The blurring/pixelation is arguably unnecessary, regardless of its effectiveness. The censored data looks like local host names and some image hashes.

the sad part is that all this is going to accomplish is promote that sap has ai product their clients can purchase. it's not like anyone using sap know or care about security other than signing with a company that has all the ISO and whatnot, which is the reason they went with sap to begin with

I would assume for the prices SAP charges, it mah start as some kind of bulletin of how to properly secure the AI, and failing that a feature update to tighten some defaults.

But a security fist to a leaky side door? I’d bet that upsets some customers.

Many of these accounting systems are starting to sell AI to automate transactions, which may explain the read+write nature of the access described in the comments.

Excellent write up. This wasn’t a sophisticated attack. Seems like there is very little discipline at Salesforce when it comes to deploying production systems.

（评论） (comments)

（评论）
(comments)