GuardDuty is a threat detection service from AWS service, and to jump straight to it, yes, you should enable GuardDuty! In this post I’ll first clear up some confusions about GuardDuty and convince you to enable it. Then, in another post, I’m going to show you how to receive GuardDuty alerts, along with Trusted Advisor and CloudTrail events, in near real-time, from multiple accounts and multiple regions, to a central location in a way that I believe is better than the architecture AWS tends to advocate.
How GuardDuty works
GuardDuty detects compromised IAM users and roles, and network resources, such as EC2s. It also recently added more of a policy violation check to detect if the root user logs in, and has had detection rules if the password policy is changed or CloudTrail is disabled. It’ll be interesting to see if they continue adding checks like that. GuardDuty has new detection rules added over time, with 47 detection rules currently.
It uses three log sources for this:
- VPC Flow Logs
- VPC DNS logs
- CloudTrail logs
You do NOT need to turn on VPC Flow Logs, or anything else, for GuardDuty to use these log sources. It has separate access to these. GuardDuty is also the only thing with access to the VPC DNS logs, you can’t see those.
Looking at the detection rules used by GuardDuty, what it calls Finding Types, roughly half of the detection rules use CloudTrail as their datasource.
A quarter of the rules use “machine learning”, which seems to just be the buzzword translation of the common SIEM rule “has this been seen in the past 30 days?”
GuardDuty is a regional service, which means if something happens in a region you don’t have GuardDuty turned on in, you could miss it. Global events, such as IAM related activity, are associated with us-east-1 on AWS, but GuardDuty will alert on those in all regions, so you don’t need to turn on GuardDuty in us-east-1 for those alerts, although you should turn on GuardDuty in all regions. When GuardDuty was first released, this meant that all regions you had GuardDuty on in resulted in charges for global events (so the cost was 16x more for global events if you had GuardDuty on in all regions), but this is no longer the case, so there is no reason not to turn GuardDuty on in all regions. If you don’t have activity in a region, GuardDuty shouldn’t cost you anything there.
GuardDuty is free for the first 30 days so you can understand how much it would have cost you. The pricing is based on how many events it analyzes.
Circumventing GuardDuty detections
Like any IDS or antivirus, GuardDuty can miss things or False Positive on things (ie. alert on things you don’t care about or aren’t real issues to you). For example, if an EC2 is compromised and calls out to a malicious domain, GuardDuty has a detection for this, but if the malicious domain is new, GuardDuty could miss it. Even more easily though, if the attacker uses a different DNS server, such as Google’s 8.8.8.8, when looking up the domain, GuardDuty will not detect this. Using that technique, GuardDuty’s entire use of VPC DNS logs can be circumvented. Another easy detection to circumvent are the ones related to the user-agent when making AWS API calls.
The most notorious detection to circumvent, which GuardDuty can and should fix, and has promised to fix for months now, is the “UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration” detection which detects if the IAM Role credentials used by an EC2 are used from somewhere else. This detection is great for detecting SSRF when an attacker manages to get access to the metadata service running on 169.254.169.254 and steal the session credentials (AWS also really needs to improve mitigations against that problem). The problem is, this detection will only alert if the attacker then uses those credentials outside of the AWS IP space. So if the attacker steals the session credentials, and then spins up an EC2 in their own account and uses them there, they can bypass this detection. Will Bengston from Netflix has blogged and presented at re:Invent and Black Hat about this problem.
Despite these limitations, GuardDuty is the best and cheapest solution available for detecting compromises on AWS. No one can compete due GuardDuty’s access to VPC Flow Logs being much cheaper than anyone else can get and because GuardDuty is the only thing with access to the VPC DNS logs.
Muting GuardDuty alerts
As mentioned, GuardDuty will have False Positives. Common rules to mute include:
- Recon:EC2/PortProbeUnprotectedPort: This detects port scans against your public hosts. Everything on the Internet is scanned all the time so most people just mute this rule entirely.
- UnauthorizedAccess:EC2/SSHBruteForce: This detects brute force login attempts against an SSH server. Not everyone can use SSM for their needs or lock down SSH logins to specific IPs, or has accepted the risk because they don’t allow password logins and possibly uses fail2ban. So this is another rule that is commonly muted due the accepted risk of having public SSH servers.
If you mute GuardDuty rules, I believe they should be muted from your SIEM, and not from GuardDuty (ie. don’t use the API UpdateFilter), so that you can have more control over what aspects you want to mute and to make this management easier.
Additional custom detections
Once you’ve enabled GuardDuty, you can take things a step further and add your own detections, but any detections you create should build on GuardDuty. It is not a good idea to think “GuardDuty isn’t perfect, so let’s build our own instead”. Instead think “We can add additional detection abilities to make up for the imperfections of GuardDuty.”
Some things you could detect are:
- Alerts for Access Denieds, or any type of failures, especially for services. For example, if you have an IAM role on an EC2 and it has read/write to a bucket and suddenly it makes a call to list all your buckets, that would odd, and not something GD might pick up on.
- Alerts around times. For example, if your employees don’t work on weekends or between 6pm and 6am. This might only work for some employees.
- Anything that violates your company’s own policies for best practices. For example, creating EC2’s without some tags, or where the user-agent doesn’t indicate terraform is being used, or where a console login happens from a browser you don’t use, etc.
Conclusion
GuardDuty has imperfections, but it is relatively inexpensive and will get you from no threat monitoring to a well-respected level of monitoring with ease, so enable it in all regions and keep an eye out for my next post on how to receive its alerts!