On April 5th, 2022, starting at 07:38 UTC, 775 Atlassian customers, representing 883 sites, lost access to their Atlassian products. The outage spanned up to 14 days for a subset of these customers, with the first sites being restored on April 8th and all customer sites progressively restored by April 18th.

The incident originated from an engineering task to delete instances of a standalone “Insight – Asset Management” app, whose functionality had become native to Jira Service Management. An existing script and process for app deletion were utilized. However, a critical communication gap led the team to provide IDs for entire cloud sites instead of the intended app IDs.

The API used for deletion lacked sufficient warning mechanisms, accepting both site and app identifiers and proceeding with deletion based on the input type without further validation. The peer-reviewed script, while focusing on the endpoint, did not cross-check the provided cloud site IDs to ensure they referred to the app rather than the entire customer site. This resulted in the immediate deletion of 883 sites between 07:38 UTC and 08:01 UTC on April 5th.

Customer impact included prolonged loss of access to their Atlassian products, although no customer lost more than five minutes of data. Initial communication with affected customers was also challenging due to the deletion of some customer contact information. Atlassian is implementing universal “soft deletes,” accelerating its Disaster Recovery program for multi-site/multi-product restoration, revising incident management processes for large-scale events, and creating a large-scale incident communications playbook to prevent similar occurrences.