In light of recent security incidents, we see again that restricting egress traffic through segmentation is an effective way to prevent attacks. In response to the Solarwinds breach, FireEye recommends organizations to “Block Internet egress from servers or other endpoints with SolarWinds software.” 1
The cybersecurity industry has not yet come up with an effective solution to prevent this kind of attack. We can assume similar sophistication in future attacks as it proves to be effective for the attackers.
But why is no one enforcing egress policies?
Egress filtering is simple to implement, yet few organizations do, why is that?
Based on customer feedback the common answers were- “it is too much of an overhead.”
When a firewall is protecting a large subnet, it is obviously very difficult to restrict outbound traffic because there are many servers on the internal side of the firewall trying to connect to many other endpoints on the other side. It would be extremely difficult for a security manager to assess all the connections that are coming through the firewall and determine which are needed and which should be blocked.
Managing a list of allowed destination IP addresses would be completely out of the question because the dynamic nature of CDNs, but even doing so by FQDN (DNS name) would be difficult because security managers would have to define the entire allow list of endpoints that are needed by their own applications.
Two other alternatives which are proposed by several security vendors but don’t prevent the sophisticated issues like the one in discussion are:
- Defining a deny list of IP addresses or DNS names: this requirement is a never-ending chase after hackers who will change IPs and DNS names to bypass the deny list. Even if the vendors update the lists from a central management plane (like a MISP), they can’t prevent a zero-day vulnerability until it's too late.
- Using next-gen malware detection: this works to some extent, but hackers are always improving their techniques of masquerading their illegal connections as legitimate ones so there is no guarantee that a firewall will be able to detect the bad connections and block them.
To make things worse, even if someone was able to define the allow list of egress endpoints, it would quickly become outdated because applications are always evolving with new features and endpoint dependencies.
So, the problem really comes down to policy management rather than enforcement: how can we define an allow list of legitimate egress DNS names?
Automating Security Policies – THE Effective Approach!
We believe that it is possible to control egress traffic effectively. To do so, we need a different approach to security policy management.
First, we need to reduce the scope. While it is impossible to define an egress policy for an entire VLAN, it should be quite easy for a single server or, even better, a single workload. Microservice architectures like Kubernetes and Serverless are ideal for this kind of security.
Secondly, no one should be writing security policies by hand. It is too difficult to maintain and it doesn’t scale throughout the frequent changes that applications go through.
Instead of writing policies manually we should be learning policies automatically by monitoring the application’s behavior.
But here is the catch 22 - if the workload whose traffic you are monitoring to automatically generate policies is already compromised, you’d be learning a bad connection.
To address this, we recommend to automatically learn the security policy early in the development process, for example, while running integration tests. These policies should consist a relatively small list of allowed egress endpoints. They can easily be reviewed by the developers and security teams. (Make sure to exclude penetration testing or similar activities from the learning patterns to avoid learning bad patterns)
3rd-party software should come with such a policy too which can be generated in the same fashion by the software vendor.
We call this “Immutable Policies”. They never get altered in production, only automatically generated every time the application is modified and then promoted to production in entirety. These policies are an integral part of the software and should be mandatory for every application.
Dictionary definition of immutable:
im·mu·ta·ble: unchanging over time
Our definition of immutable policies:
Immutable Policies: least privilege security policies that are generated automatically and are only updated when the application is deployed.
Dealing with Extreme Case of Supply Chain Security
In extreme cases, like the Solarwinds breach, the code itself was modified by the attackers so even the auto-generated policies would have included the malicious egress connections. Can this be prevented too?
We think it is possible. When the scope of the protected asset is small, it is easy to review each egress connection from a workload and approve it, just like we do code-reviews (think Policy-as-Code).
In the case of an infected binary, you will see a diff in the egress policy. A new connection would popup and the reviewer should easily be able to detect it as a suspicious connection.
- Define egress policies that only allow access to the essential and verified destinations.
These policies should be at much granular levels like Kubernetes workloads. Since each workload have a relatively smaller scope of external connections, defining egress restrictions is much easier and sustainable.
- Use automation to learn policies.
Learn the application connections to generate a policy of valid ingress and egress connections for every Kubernetes service.
- Learn policies as early as possible in the development cycle.
The application policies should be learned during application testing in order to generate a policy definition that can be committed to your code repository. This code can be then deployed to production as part of the CI/CD pipeline.