CrowdStrike 前员工:“质量控制不是我们流程的一部分” CrowdStrike ex-employees: 'Quality control was not part of our process'

原始链接: https://www.semafor.com/article/09/12/2024/ex-crowdstrike-employees-detail-rising-technical-errors-before-july-outage

前员工指责网络安全公司 CrowdStrike 过于仓促的质量控制流程,导致客户数据意外泄露和监控系统间歇性故障等错误。 他们声称,由于开发时间缩短以及缺乏对处理基于云的系统的员工的专门培训,测试不足,特别是关于 2022 年推出的云威胁搜寻服务。 一些人还表示,在裁员和组织重组后,他们面临着工作量的大幅增加。 针对这些说法,CrowdStrike 否认了缺乏必要工具或仓促项目的指控,并表示他们为员工提供培训并不断增加员工队伍,同时增加研发 (R&D) 支出。 此外,他们将 7 月份的停电归因于传感器更新错误,而不是质量管理不善。 尽管受到批评,CrowdStrike 仍继续获得顶级雇主的认可,最近在一次黑客会议上被其总裁授予“最史诗般的失败”奖,强调问责制和持续改进。

Former employees accuse cybersecurity firm CrowdStrike of rushing quality control processes, leading to errors such as accidental disclosure of customer data and intermittent failures in their monitoring system. They claim that there was insufficient testing, especially regarding the 2022 launch of a cloud threat hunting service, due to shortened development timelines and lack of specialized training for staff handling cloud-based systems. Some also say they faced heavy workload increases after layoffs and organizational restructuring. In response to these claims, CrowdStrike denies allegations of lacking necessary tools or rushing projects, stating that they provide training for employees and consistently grow their workforce while increasing Research & Development (R&D) spending. Additionally, they attribute the July outage to a faulty sensor update rather than poor quality management. Despite criticism, CrowdStrike has continued to receive recognition as a top employer and was recently awarded for "Most Epic Fail" at a hacker conference by its president, emphasizing accountability and continuous improvement.


Some former employees said quality checks on software were rushed at times to get products launched quickly.

“It was hard to get people to do sufficient testing sometimes,” said Preston Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the tests completed by user experience developers that alerted engineers to bugs before proposed coding changes were released to customers. Sego said he was fired in February 2023 as an “insider threat” after he criticized the company’s return-to-work policy on an internal Slack channel. That’s the company’s designation for employees who present security risks. CrowdStrike declined to comment, saying it does not “discuss individual personnel matters.”

There were other issues. In one incident in the professional services department, one former employee described how a customer’s private information was accidentally uploaded to the wrong client’s folder three times, narrowly escaping sharing private client data with the wrong customer each time. CrowdStrike confirmed the incidents and said they occurred because of a “manual data entry error.” It said the data was “basic information like host names, IP addresses, and domain names,” and “checks are now run” to ensure private customer data isn’t sent to the wrong client.

Multiple people also cited issues with CrowdStrike’s Falcon LogScale service, which uncovers security and reliability issues in a customer’s systems. One recalled at least two instances where bad updates to LogScale briefly turned off its real-time alerts that notify customers of potentially malicious activity, which some of the engineers who built the updates blamed in internal meetings on tight deadlines. CrowdStrike denied the instances, saying it is not aware of any “‘bad update’ where alerts were lost and not received by customers.” The company also said the service isn’t designed to alert customers to potential data breaches in “real time.” It, instead, is designed to “rapidly shut down threats with real-time detection and blazing-fast search,” according to the company website.

A separate ex-employee said CrowdStrike rushed the 2022 launch of its cloud threat hunting service, called Falcon OverWatch Cloud Threat Hunting, where the company’s security professionals look for suspicious behavior that could indicate a breach on customers’ cloud setups, like Amazon Web Services. Engineers and threat hunters were given just two months for work that would normally take a year, according to a former senior manager who worked on the project. When the service launched, he said it lacked the internal tools that threat hunters used to fully monitor customers’ cloud systems for threats; employees ended up responding to alerts from existing security systems until at least last summer, about a full year after it was launched.

The former senior manager said CrowdStrike also used staff who had been trained to monitor customers’ computer systems — like laptops and desktops — and tasked them with looking for threats in cloud setups without mandating new training.

“AWS is a beast, and it takes a very special staff to be able to do that,” he said. CrowdStrike “took people who were like cops, looking for threats on the ground all day, and asked them to fly an airplane and look for threats in the sky.”

CrowdStrike confirmed that it used existing engineers instead of hiring a new team of “cloud threat hunters.” As a new service, it said, “there were no experienced ‘cloud threat hunters’ to be had, and it would not have been possible to hire individuals with specific training in a field that did not exist until CrowdStrike developed it.” The SANS Institute has been teaching courses and giving talks on cloud security since at least 2020, more than two years before the launch of CrowdStrike’s service.

“Any statement implying CrowdStrike employees were not trained to do their jobs is false,” CrowdStrike told Semafor. While the company confirmed that it didn’t mandate new training, it provided it for anyone who wanted it, the company said. “Employees routinely attend training appropriate to their position.”

“This service has worked as intended at all times,” CrowdStrike said. “Even before this novel service offering was launched the Falcon Overwatch team hunted on all public cloud environments and released research into this area.”

CrowdStrike also denied that its systems lacked the tools threat hunters needed and that it rushed the project. The company said the OverWatch product line has been around for more than a decade “and is routinely enhanced to meet the evolving threats and needs of our customers.”

Sego said temporary coding meant to keep projects moving — a common practice at tech companies — was often never improved. One former senior engineer said he asked unsuccessfully to be given time to fix old coding more than 20 times. CrowdStrike said “coding is an iterative process, and it is commonplace in the software industry to release and continuously improve upon code based on real-world experience with the product.”

Ex-employees cited increased workloads as one reason they didn’t improve upon old code. Several said they were given more work following staff reductions and reorganizations; CrowdStrike declined to comment on layoffs and said the company has “consistently grown its headcount year over year.” It added that R&D expenses increased from $371.3 million to $768.5 million from fiscal years 2022 to 2024, “the majority of which is attributable to increased headcount.”

CrowdStrike said it “receives, evaluates, and incorporates a range of feedback from its team,” and that it “focuses on always maintaining a high-performance culture.” The company also noted that it “has been recognized as one of the Fortune 100 Best Companies to Work For for the last four years.”

For the July outage, CrowdStrike has blamed a defect in an update to its Falcon Sensor. The episode has cost the company more than $21 billion in stock-market value and brought on a slew of lawsuits, including one potential suit by Delta Airlines, which pegged its losses at $550 million after thousands of flights were canceled.

At a hacker convention in August, CrowdStrike President Michael Sentonas accepted an award on stage for “Most Epic Fail.” He said it’s “super-important to own it when you do things horribly wrong.”

相关文章
联系我们 contact @ memedata.com