Hi Katie (and everyone who posted here).
I wanted to close this one out by sharing some information about the incident, what we learned, and what plans we have for this to not happen again.
1. Outage
We had a failure of a single-point dependency in our US region's infrastructure. At the time of failure, the automatic notification to our on-call engineer also wasn't picked up immediately by our on-call engineer, exacerbating the respond/resolve time.
2. Notification
Ideally in a situation like the above, the internal incident notification system will contain a step for automatically notifying an engineering manager when an incident goes unacknowledged by the on-call engineer. This was not configured correctly for management notifications.
3. Communication
Ideally in a situation like the above, non-engineering staff are brought into the loop to manage public-facing communications like status.moodlecloud.com and the forums here. Due to the above breakdown in communications, neither of these steps were taken to let the community know about the issues we were having.
4. Outcomes
As Product Manager for MoodleCloud, I've taken on 17 discrete changes as a result of this. Some are already in place, others are longer term goals that will make more incremental improvements to how we provide this service to our users.
I would like to take this opportunity to apologise for the impact to your services, and I hope that this transparency goes some way to giving you confidence that we're working hard to improve every day here at the MoodleCloud team at Moodle HQ.
If you have questions or concerns, please reach out below. I'm always keen to hear your views.
Thanks heaps
Lee Goldsworthy
Product Manager - MoodleCloud
I wanted to close this one out by sharing some information about the incident, what we learned, and what plans we have for this to not happen again.
1. Outage
We had a failure of a single-point dependency in our US region's infrastructure. At the time of failure, the automatic notification to our on-call engineer also wasn't picked up immediately by our on-call engineer, exacerbating the respond/resolve time.
2. Notification
Ideally in a situation like the above, the internal incident notification system will contain a step for automatically notifying an engineering manager when an incident goes unacknowledged by the on-call engineer. This was not configured correctly for management notifications.
3. Communication
Ideally in a situation like the above, non-engineering staff are brought into the loop to manage public-facing communications like status.moodlecloud.com and the forums here. Due to the above breakdown in communications, neither of these steps were taken to let the community know about the issues we were having.
4. Outcomes
As Product Manager for MoodleCloud, I've taken on 17 discrete changes as a result of this. Some are already in place, others are longer term goals that will make more incremental improvements to how we provide this service to our users.
I would like to take this opportunity to apologise for the impact to your services, and I hope that this transparency goes some way to giving you confidence that we're working hard to improve every day here at the MoodleCloud team at Moodle HQ.
If you have questions or concerns, please reach out below. I'm always keen to hear your views.
Thanks heaps
Lee Goldsworthy
Product Manager - MoodleCloud