One way of detecting breaches on AWS is to sprinkle honey tokens throughout your environments and detect when those honey tokens are used. In the case of AWS, these would be AWS access keys and secret keys that have no privileges. These act like a tripwire, informing you of a breach. For example, if you had one of these sets of credentials sitting on your production server and received a notification that it was used, you could assume that the server had been breached as the only way these could have been found and used was if someone had obtained access to that server. This article will discuss more about how to use these honey tokens in practice.

How to create a honey token

To be a honey token, it needs to meet the following criteria:

• It needs to look enticing so an attacker would find and use it.
• It needs to be possible to detect when it has been used.
• It needs to not have any actual value other than detecting its use.

An AWS access key and secret key with no privileges is going to meet that criteria. Imagine finding the following at the top of some source code or bash script:

AWS_ACCESS_KEY = "AKIAJSIE27KKMHXI3BJQ"
AWS_SECRET_KEY = "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"

Or imagine finding a ~/.aws/credentials file with the contents:

[prod]
aws_access_key_id = AKIAJSIE27KKMHXI3BJQ
aws_secret_access_key = 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

An attacker would quickly punch those credentials into their own tools and try running aws s3 ls to list all of the S3 buckets in the account or start spinning up EC2 instances to mine bitcoin. A more advanced attacker might try looking around further and try to determine what access these keys have using something like dump_account_data.sh from Daniel Grzelak or one of the handful of other tools that try to call every AWS API call.

Method 1: Use canarytokens

The easiest way of creating fake AWS credentials as a honey token is to use canarytokens.org which is a free service from Thinkst for creating all sorts of different honey tokens, including AWS keys. When triggered these will email you or hit a webhook. Thinkst also has a paid service that makes some of the management of these easier, such as letting you change the webhook after you created them.

Method 2: Create your own

Create an empty AWS account and create an IAM user there. It is important that you do not give this user any privileges. By default IAM users will not have any privileges, but as an added precaution in case you accidentally grant all of your users privileges, you can additionally enforce the denial of privileges by adding this policy:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
  }
}

You do not have to create a separate empty AWS account, but it is helpful to avoid any mistakes, and additionally you can more easily setup detections.

For your detections, you’ll need to turn on CloudTrail and enable all logging (including the data level logs). You’ll then need to set up some way of detecting when this key has been used. The open-source tool StreamAlert from Airbnb can be used for this.

Method 3: Use a project

There are some projects to help you set up these honey tokens yourself, such as the open-source SPACECRAB from Atlassian. Thinkst also has their project open-sourced at github.com/thinkst/canarytokens.

Things to know about AWS credentials as honey tokens

An attacker has no way of knowing if AWS credentials are honey tokens until they try to use them, and even then an attacker may not be able to figure it out. I’ve tried looking for any way of correlating AWS credentials back to anything in my post AWS security credential formats and was not able to find anything.

AWS logs API calls to CloudTrail logs which take 15 minutes to record. This is a limitation of AWS. You can get faster detection of some API calls by using CloudWatch Events, but not all API calls are recorded this way.

An attacker can run aws sts get-caller-identity to find out the name of the user for these credentials. Even with all actions denied, this call always works as long as the keys are active. So if you call your user honeytoken1, this could tip off an attacker. You will have detected the attacker when they do this as that call is logged, but you may still wish to avoid letting the attacker know they found a honey token.

Not all AWS API calls are logged even when all possible settings are enabled in CloudTrail. I don’t want to give away the details of what those are, but I don’t think are of any value to an attacker. So in order to try to do anything interesting or find anything interesting, an attacker will need to make AWS API calls that are logged and thus can be detected.

Deployment phases

Deployment of honey tokens is best done in four phases:

1. Deploy honey tokens manually to critical systems.
2. Improve alerting framework.
3. Deploy widely via automation.
4. Improve host logging for better DFIR.

Stage 1: Deploy honey tokens manually to critical systems

In this phase, you are manually putting the honey tokens on maybe a dozen critical systems, such as some of your production servers, bastion hosts, VPN servers, and on the laptops of your critical employees that have access to these production systems.

You may deploy the same set of fake credentials to every one of your production servers. This won’t tell which server was compromised, but will tell you that a server was compromised. This is only the first step. We’ll improve this later.

You want to ensure that your alerting mechanism works and tells you which of your honey tokens was triggered. You may want to have a wiki page or some other way of documenting where these honey tokens were placed. Be mindful here that an attacker may find this wiki page in order to know what to avoid, but I think overly focusing on an omnipotent attacker here distracts from getting anything done.

Be mindful of delivery

Be aware of how these honey tokens get to where they’ll finally end up, because if the honey token is triggered, it could mean anything in the chain of custody was compromised.

For example, if you email this honey token to your devops person and tell them to add this to the production hosts, then this honey token exists in the following places (and more!):

1. Your laptop (briefly)
2. Your email sent items.
3. The devops person’s email inbox.
4. The mailserver, or if you use something like G Suite, then access could potentially be obtained by anyone with admin privileges to G Suite.
5. The devops person’s laptop.
6. The devops person’s phone that also has access to their email.
7. The git server where they checked this in for deployment via terraform (this is the only time it is ok to add “secrets” to git).
8. The laptop of every other devops person that has this repo checked out.
9. The third-party vendor that performs static analysis on your repos.
10. The production server where you wanted this deployed.

This means that if this honey token is triggered then any one of the above may have been compromised! So you may want to have them generate this honey token themself and then just let you know the AWS access key so that you can ensure your detection rules identify which honey token this is.

Stage 2: Improve alerting framework

In Stage 1, we were just getting the honey tokens out there and some of way of finding out if they were triggered. In this stage, you’ll want to ensure that your alerts provide as much context as possible about the honey token and what to do when triggered. So in Stage 1, you might have used canarytokens.org and just received an email. In this stage you want to tie this into your normal incident response paths. This probably means using a webhook instead of email and integrating this into Slack, Jira, and PagerDuty. I documented how to use StreamAlert to receive a webhook in this PR.

You want to ensure that your alerts tell you specifically where this honey token exists, which means you’ll likely want to add a lookup table in your detection rule, so key AKIAJSIE27KKMHXI3BJQ becomes “Key AKIAJSIE27KKMHXI3BJQ was placed on John’s macbook pro with serial number 012345 in ~/.aws/credentials”.

You also want to ensure your alerts describe your own DFIR responses. As this is now part of your monitoring, you want to ensure you have runbooks for handling these. Later in this article, I’ll discuss how to handle these incidents.

You may also want to build in orchestration at this point, or at least start thinking about it. This means that if a honey token is triggered, your alerting engine automatically takes action. This could mean automatically disconnecting the impacted EC2 instance from the production database, or disabling the real access of the devops person that was compromised. These are scary things to do and require a lot of consideration to avoid making a bad situation worse or accidentally having an outage due to a false positive. If it works as hoped for though, it can automatically help mitigate issues and speed up incident response.

Stage 3: Deploy widely via automation

At this point when an alert fires you have a good way of responding, so now it is time to deploy these honey tokens more broadly. Instead of a dozen systems, you now deploy these to every EC2 instance, every AWS lambda, every employee laptop, every email inbox, etc.

With more honey tokens floating around the likelihood of false positives increases. So you’ll want to take care to include a discussion of these honey tokens in your onboarding, provide a way for employees to ask “what is this?” when they come across one, and ensure that your runbooks account for checking to see how this token was triggered.

Stage 4: Improve host logging for better DFIR

In order to better understand incidents identified via these honey tokens and to better identify false positives, you’ll want to improve the logging you are doing on systems to understand how this honey token was found. This could be done with auditd or a proper EDR solution (ex. CrowdStrike, CarbonBlack, Veramine, etc.). With this you want to know what process and user on a system opened the file containing the honey token.

See my post Catching attackers with go-audit and a logging pipeline for some insight on how to use auditd to record when a file has been read.

Handling incidents from honey tokens

The first thing you want to do in an incident is to identify if this was a False Positive. This means that perhaps a curious devops person found this on the server. They might have been concerned that there were creds sitting there instead of an IAM role being used or other secret management solutions being used, wondered what this was, and triggered it while trying to investigate it. This might mean asking in Slack room for the devops team “Hey, we just had a honey token trigger on one of the web servers. Did anyone trigger it?” If you have better logging setup, you might quickly check the EDR to see what user might have accessed this.

Once you’ve decided this is a real event, you need to identify what was actually compromised, which as discussed earlier could be any number of systems on the chain of custody of this honey token on its way to its final destination. Again, an EDR could help.

Finally, as you’ve narrowed down the compromised system, your course of action is going to depend on your standard incident response plan. This could mean just wiping a laptop, or it could mean bringing in a third-party to perform forensics. It could mean disabling access to employee access, disconnecting servers, or shutting down the entire site.

Conclusion

Honey tokens are a great addition to your defensive arsenal. This document described the stages for deploying these in your own environment and how to respond.