Conf42: Site Reliability Engineering 2022

Smoke detectors in large scale production systems

Static alerting thresholds no longer cut it for modern distributed systems. With production systems scaling rapidly, using static alerting to observe critical systems is a recipe for disaster. Observability tools have recognized this need and provide a way to ""magically"" catch deviations from normal system behavior instead. - What is that magic? What goes in to deciding whether a spike or a...