As part of standard incident-response practice, Ruby Central is publishing the following post-incident review to the public. This document summarizes the September 2025 AWS root-access event, what occurred, what we verified, and the actions we’ve taken to strengthen our security processes.

On September 30th, a blog post raised concerns that a former maintainer continued to have access to the RubyGems.org production environment after administrative access was removed from several accounts earlier that month. We want to share the outcome of our investigation including: what happened, the extent of what we verified, what we got wrong, and the actions we have taken to strengthen our security processes going forward.

When this situation came to light, our immediate concern was the integrity and safety of the RubyGems.org service and its data. We take seriously our responsibility to steward the open-source infrastructure that millions of developers rely on each day. While we have found no evidence that user data or production operations were harmed, we recognize that the existence of an unrevoked shared credential and unclear communication created understandable alarm and frustration. For that, we are sincerely sorry.

Incident Response Timeline

September 30 2025

• 17:23 UTC: A former maintainer, André Arko, emails the Director of Open Source at Ruby Central stating that he still has access to the RubyGems.org production environment and associated monitoring tools.
  Note: This is the first and only disclosure to Ruby Central about this access by Mr. Arko.
• 17:30 UTC: Joel Drapper (unaffiliated with Ruby Central) publishes a public blog post within minutes describing this access with screenshots taken earlier that day showing root account access.
• 17:51 UTC: Ruby Central engages its board members and OSS staff to verify the veracity of the report, assembles an incident team, and enumerates all services and credentials to assess exposure scope and ensure complete remediation. 
• 18:20 UTC: Ruby Central begins its emergency review and learns that the existing credentials for the AWS root account in our password vault are no longer valid.
• 18:24 UTC: Ruby Central initiates an AWS password-reset procedure, validates multi-factor authentication, and regains control of the AWS root account.
• 18:30 UTC: Ruby Central downloads a “Credentials Report” from the AWS console to understand why we could not access the root account, and learns that the root account password was changed by an unauthorized party on September 19th at 04:35 UTC.
• 20:45 UTC: After an examination of AWS CloudTrail logs, DataDog alerts, and IAM configurations, Ruby Central identifies and revokes all associated sub-accounts and legacy credentials, issues new MFA tokens to the remaining accounts, and migrates the new root access credentials into a secure vault under Ruby Central’s sole control.

Analysis of Events

By way of background, Ruby Central’s infrastructure runs on Amazon Web Services (AWS). The root account credentials, essentially the highest level of administrative control, are stored in a shared enterprise password manager in a shared vault to which only three individuals had access: two current Ruby Central staff members and one former maintainer, André Arko.

September 18 2025

• 18:40 UTC: Ruby Central notifies Mr. Arko, via email, of the board’s decision to remove his RubyGems.org production access, and the termination of his on-call services. During that transition, our teams remove the AWS security credentials belonging to Mr. Arko for accessing the production systems, but we fail to rotate the AWS root account password in tandem.

September 19, 2025

• 04:34 UTC: An unauthorized actor originating from a San Francisco, California IP address starts a root account session on the AWS Rubygems.org AWS account. 
• 04:35 UTC: The unauthorized actor changes the root account password.
  Note: After this point, and until the AWS root credentials were reset by Ruby Central on Sept 30th, all subsequent actions taken on the AWS root account originate from the unauthorized actor. 
• 04:37 UTC: The unauthorized actor removes authorized users from groups and detaches access policies which reduces the privileges of authorized Rubygems.org AWS account holders. 
• 04:39 UTC: The unauthorized actor rapidly enumerates the IAM posture of the entire AWS account. 

September 28th, 2025

• 05:49 UTC: An unauthorized actor originating from a Tokyo, Japan IP address starts a  root account session and uses IAM introspection API calls to check users’ group membership, last usage date, and last usage date of associated access tokens and policies.
  Note: This unauthorized access occurs adjacent to the Kaigi on Rails conference also in Tokyo, Japan from Sept 26th - 27th. As a result, we attribute this access to the same unauthorized actor.

September 30th, 2025

• 15:25 UTC: An unauthorized actor originating from a Los Angeles, California IP address starts a root account session.
• 15:35:24 UTC: The unauthorized actor issues a PutCredentials command to obtain user credentials, which match the screenshot shared in the blog post announcing the security vulnerability. The blog post asserts that this action was taken by Mr. Arko.

• 18:24 UTC: As we mentioned previously, Ruby Central performs the AWS password-reset operation to take back control of the root account.
  Note: After this point, all actions taken on the AWS root account can be attributed back to authorized actors.

Extent of the Incident

After a careful review, Ruby Central is relieved to report that we see no evidence that this security incident compromised end user data, accounts, gems, or infrastructure availability. In addition:

• RubyGems.org remained fully operational throughout.
• No personally identifiable information (PII) of RubyGems.org users nor Ruby Central financial data was accessed or transferred.
• The production database, S3 buckets, and CI/CD pipeline were unaffected.

Nonetheless, the existence of unrotated credentials and the public disclosure of continued access constitute a serious procedural failure, and we are treating it as such.

How Was The Incident Resolved?

After regaining control of the AWS account, Ruby Central:

1. Revoked all existing root and IAM credentials, created new MFA-protected access, and moved them to a restricted vault with per-user audit logs.
2. Rotated all related secrets and tokens, including DataDog, GitHub Actions, and other external system integrations.
3. Enabled AWS CloudTrail, GuardDuty, and DataDog alerting for any root login, password change, or IAM modification.
4. Reviewed all IAM roles and policies to ensure least-privilege access and removed legacy permissions.
5. Began a full end-to-end security audit with external advisors, covering infrastructure, credential storage, and incident-response procedures.
6. Updated the Ruby Central Security Runbook to include immediate password and key rotation upon personnel or role changes, quarterly credential reviews, and coordinated communication steps for any future incident.

Root Cause Analysis

After a post-mortem review, the root cause of the security incident was two-fold:

1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.  
2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.

Both of these events enabled the unauthorized actor to access RubyGems.org production infrastructure where they attempted unsuccessfully to lock out authorized personnel and frustrate recovery efforts.

What We Are Doing to Prevent Future Incidents?

RubyGems.org is a critical service that the entire Ruby community depends on, and we take that responsibility seriously. For RubyGems.org to succeed, it must not only maintain near-perfect operational uptime but also earn the community’s trust that it is operated professionally, that its operators can attest to the integrity of both the data and the code it serves to millions of Ruby applications worldwide, and that the privacy of the data we hold remains intact.

To that end, we commit to the following improvements:

1. Update our access revocation procedures and checklists to ensure access is also revoked to the Ruby Central enterprise password manager.
2. Update our access revocation procedures to ensure any non-federated credentials (particularly shared credentials) are rotated quickly after a personnel separation. 
3. Commission an independent security audit of Ruby Central’s systems and access.
4. Finalize formal Operator and Contributor Agreements to clearly define who may hold production access and under what conditions

Why Did Ruby Central Treat This Event as a Security Incident?

As part of our recent actions, we determined that many RubyGems.org systems were controlled by a single individual; an untenable situation for a service of this importance.

To provide additional context to the community about our decision to formalize production access through Operator and Contributor Agreements, and to explain why we treated this incident as a genuine security event, we are sharing context from conversations between Mr. Arko and Ruby Central personnel leading up to the September 18th access changes.

In early August 2025, Ruby Central began reviewing its open source contractor budget, which totaled approximately $762,000 in 2024. On-call coverage is critical for a service like RubyGems.org and allows us to ensure operational continuity and rapid response to production incidents. Every on-call shift has a primary who is directly responsible for responding to incidents, and a secondary who is there to serve as a back up and an escalation point, if and when needed.

For RubyGems.org, the secondary on-call rotation, which serves as a backup layer, was rarely activated. Ruby Central’s long-term goal was to transition this limited paid function into a distributed network of volunteer operators who could share those responsibilities without additional cost, ensuring both operational continuity and financial sustainability.

Following these budget adjustments, Mr. Arko’s consultancy, which had been receiving approximately $50,000 per year for providing the secondary on-call service, submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII). The offer would have given Mr. Arko’s consultancy access to that data, so that they could monetize it by analyzing access patterns and potentially sharing it with unrelated third-parties.

The board and leadership team determined that this proposal crossed important ethical and legal boundaries, introducing privacy, conflict-of-interest, and governance concerns inconsistent with Ruby Central’s responsibilities as stewards of the ecosystem. These concerns set in motion Ruby Central’s decision to adopt the new operating model and governance structure detailed in this blog post.  With this context in mind, when we discovered that Mr. Arko had retained access to production systems containing PII, it prompted us to consider it as a security incident and to respond immediately.

Based on our preliminary investigation, as of the publication of this post, we have no evidence to indicate that any RubyGems.org data was copied or retained by unauthorized parties, including Mr. Arko.

We recognize that these events have raised valid questions within the community and tested confidence in how Ruby Central fulfills its stewardship role. Our intent in sharing this level of detail is to be transparent about what occurred, what we have learned, and what we are doing to prevent it from happening again. We are hopeful that this openness marks a meaningful step toward rebuilding trust in our stewardship and demonstrating that accountability and collaboration remain central to how we serve the Ruby ecosystem.

We are deeply grateful to the community for holding us accountable and for the patience and professionalism shown during this process. Ruby Central remains committed to transparent, responsible stewardship of the RubyGems infrastructure and to maintaining the security and trust that the Ruby ecosystem depends on.

Sincerely,

Shan Cureton
Executive Director