The incident involved an outage of Elasticsearch (severity SEV2), which has affected customer access to the website and triggered numerous support calls. The issue was identified after an alert reported Elasticsearch being down, and it is suspected that the problem relates to a recent deploy that may have introduced breaking changes, evidenced by a spike in 5xx errors noticed in Datadog. The team, led by JP Cheung as Commander, is in the process of rolling back the last deploy and monitoring system metrics, with a customer update already drafted to inform users about the ongoing resolution efforts. The primary members involved in the response were Bach (Incident Commander), Stanley (SRE Lead), and JP (DevOps Engineer), who are working together to rectify the situation and ensure communication with customers and leadership.
The incident involved an outage of Elasticsearch (severity SEV2), which has affected customer access to the website and triggered numerous support calls. The issue was identified after an alert reported Elasticsearch being down, and it is suspected that the problem relates to a recent deploy that may have introduced breaking changes, evidenced by a spike in 5xx errors noticed in Datadog. The team, led by JP Cheung as Commander, is in the process of rolling back the last deploy and monitoring system metrics, with a customer update already drafted to inform users about the ongoing resolution efforts. The primary members involved in the response were Bach (Incident Commander), Stanley (SRE Lead), and JP (DevOps Engineer), who are working together to rectify the situation and ensure communication with customers and leadership.