That reminds me of "when everything is important, nothing is important". Trying to keep focus with prioritization is hard, but necessary. We data engineers have a lot to learn and to get inspired from UX and Design team's disciplines.
Great stuff here - it's often super hard to place a management layer on top of alerting so that the noise is minimized and the benefit meaningful. Have you proposed limiting the total number of "serious" or "critical" alerts at any one time so that critical doesn't become the new p0?
That reminds me of "when everything is important, nothing is important". Trying to keep focus with prioritization is hard, but necessary. We data engineers have a lot to learn and to get inspired from UX and Design team's disciplines.
Thanks for writing this I think most Data Engineers suffer from the issues you lay out. Will be sharing this!
I wonder if at Monzo you looked at Data Observability solutions before/during making a low cost alert solutions in Slack?
Great stuff here - it's often super hard to place a management layer on top of alerting so that the noise is minimized and the benefit meaningful. Have you proposed limiting the total number of "serious" or "critical" alerts at any one time so that critical doesn't become the new p0?