A significant website outage occurred on 8/22/2024, affecting almost all customers, which resulted in overwhelming calls to customer support. The incident was triggered by a recent deployment that led to a spike in server errors. The response team, including JP Cheung, Prasanthi, and JJ, acted swiftly, pulling logs and analyzing data to confirm the rollback of the faulty deploy was necessary. Following the rollback, error rates decreased, and the website functionality was restored, allowing for a customer update to be sent out regarding the resolution.
Posted Aug 22, 2024 - 11:05 PDT
Update
A significant website outage occurred on 8/22/2024, affecting almost all customers, which resulted in overwhelming calls to customer support. The incident was triggered by a recent deployment that led to a spike in server errors. The response team, including JP Cheung, Prasanthi, and JJ, acted swiftly, pulling logs and analyzing data to confirm the rollback of the faulty deploy was necessary. Following the rollback, error rates decreased, and the website functionality was restored, allowing for a customer update to be sent out regarding the resolution.
Posted Aug 22, 2024 - 10:45 PDT
Investigating
A significant website outage occurred on 8/22/2024, affecting almost all customers, which resulted in overwhelming calls to customer support. The incident was triggered by a recent deployment that led to a spike in server errors. The response team, including JP Cheung, Prasanthi, and JJ, acted swiftly, pulling logs and analyzing data to confirm the rollback of the faulty deploy was necessary. Following the rollback, error rates decreased, and the website functionality was restored, allowing for a customer update to be sent out regarding the resolution.