During this time, code scanning default setup and code quality analyses were not triggered on newly opened pull requests. Additionally, newly created issues were not appearing on project boards.
The cause was a serialization error that prevented proper triggering of code scanning, code quality analyses, and project board updates.
We mitigated the issue by deploying a fix, restoring event publishing for code scanning and code quality. For project boards, an additional code change was deployed to update event consumers, followed by a reindex of affected project items.
We are working to prevent recurrence by strengthening our schema validations and improving monitoring for drops in publishing on critical hydro topics.
Apr 21, 05:04 UTC
Update -
The degradation has been mitigated. We are monitoring to ensure stability.
Apr 21, 04:18 UTC
Update -
The issue remains mitigated. Issues that were linked to projects during the incident may take approximately three more hours to render correctly while we complete a re-index.
Apr 21, 03:10 UTC
Update -
The degradation has been mitigated. We are monitoring to ensure stability.
Apr 20, 21:36 UTC
Update -
The degradation has been mitigated. We are monitoring to ensure stability.
Apr 20, 18:21 UTC
Monitoring -
The degradation has been mitigated. We are monitoring to ensure stability.
Apr 20, 18:20 UTC
Update -
The issue has been mitigated. Newly created issues linked to projects should now function as expected. Issues that were linked to projects during the incident may take approximately five hours to render correctly while we complete a re-index.
Apr 20, 18:20 UTC
Update -
A deployment to fix this issue of new issues not showing up in projects is underway.
Apr 20, 18:08 UTC
Update -
We continue to work on mitigation regarding new issues not showing on project boards.
Apr 20, 17:32 UTC
Update -
We continue to work on mitigation regarding new issues not showing on project boards.
Apr 20, 16:48 UTC
We are seeing problems with new issues not showing on project boards and are working on mitigation.
Apr 20, 16:16 UTC
Update -
We are continuing to work on a mitigation to unblock code scanning default setup and code quality features on pull requests.
Apr 20, 15:20 UTC
Update -
We are currently deploying mitigations that should unblock code scanning default setup and code quality features on pull requests.
Apr 20, 14:38 UTC
Update -
We are actively working to mitigate an issue affecting code scanning default setup and code quality features on pull requests. Users may experience pull request code scanning and code quality analyses not being triggered on new pull requests. Our engineering team has identified the root cause and working on mitigating the issue.
Apr 20, 13:57 UTC
Investigating -
We are investigating reports of impacted performance for some GitHub services.
Apr 20, 13:28 UTC