On Thursday evening I got an alert that one of our server’s outbound network traffic exceeded normal level. By about 100-fold.
Why do we have these alerts? For security. I want to know if inbound or outbound traffic is too high. For inbound traffic, this could be the sign of a DoS attack. For outbound traffic this could indicate that one of our servers has been compromised and is now being used to send spam, distribute malware, or exfiltrate data.
I won’t get into the investigation here (the server wasn’t compromised). Instead I want to talk about how we’ve reduced the risk of a compromised server by harnessing the resiliency we designed into our operations. We can just blow everything away and start from clean images. This is a really big hammer; but, because of in-built resiliency, it’s non-disruptive.
Continue reading “Harnessing Resiliency for Security” →