Summit Route

Lightsail object storage concerns - Part 2

Thu, 17 Mar 2022 13:00:00 -0700

This is part two of a two part series about AWS’s new Lightsail object storage. In part 1, I looked at the new Lightsail access key capability. In this second part we’ll look more closely at the buckets created.

It was over 7 months ago when I posted part 1, so why the delay? I was waiting for AWS to fix something, and it was low enough risk and impact that I didn’t feel a need to “encourage” them with a public post, but I’m glad it’s finally fixed.

Making a public bucket

In part 1, I identified that Lightsail object storage creates an S3 bucket in a special Lightsail account. Let’s look at that bucket.

The first thing that jumped out at me in the announcement was an indication that these buckets could be made public. AWS has been trying for a decade to stop the almost weekly news stories of public S3 buckets found with sensitive data in them. Lightsail in one aggressive move decided to add yet another way of making these public, bypassing all the previously created defenses.

This S3 bucket is created in another AWS account outside of your own, so the S3 Public Block Access control that you may have set in your account did not apply to this. This was what I was waiting for them to fix. Now S3 Public Block Access does prevent this Lightsail bucket from being made public. I was asked to hold off on posting this until that was fixed, and I figure Lightsail is rare enough among folks that have stronger cloud security concerns and few people would expect “S3” public block access to impact Lightsail, so I agreed to hold off, then I forgot about it, then AWS fixed it and told me they fixed it, so now I’m posting it.

From the screenshot, there was an option for “individual objects can be made public”, which means S3 object ACLs can also be used, which is yet another violation of best practices, as you should avoid using those. This is however prevented with S3 Public Block Access.

Looking closer at the bucket

I was curious if I could get more info about the account this S3 bucket was creatd in, so I guessed that I would be able to read the object ACL from the public bucket, and I was able to.

$ aws s3api get-object-acl --bucket bucket-kgxqi8 --key test_object.txt
{
    "Owner": {
        "DisplayName": "lightsail-shadow+386-0944",
        "ID": "09ee0cfc526e2171e2fcc2848eb3218e6ff3627e7296cce899b09876c8a43eb4"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "lightsail-shadow+386-0944",
                "ID": "09ee0cfc526e2171e2fcc2848eb3218e6ff3627e7296cce899b09876c8a43eb4",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

So the name of the account is lightsail-shadow+386-0944. In the last post we figured out what the account ID was using the access key we had, but I actually had looked into the S3 bucket before I started using the access key, and identified the account ID using only the S3 bucket by translating the canonical ID of the account from the ACL. To do this, in an actual S3 bucket in my account, I changed the bucket policy to the following and saved it:

Once AWS saves an S3 bucket policy with a Canonical ID it will automatically translate it to an account ID.

So this S3 bucket is hosted in some AWS lightsail account with id 850260390076.

Why this bucket is a problem

Similar to the access key from part 1, having a resource in another account creates a number of problems. S3 Public Access Block now is able to stop this bucket from being made public, but enabling Cloudtrail data events in your account will not log access to this bucket. In November 2021, they did add the ability to enable S3 access logs, but those will not appear in your Cloudtrail logs like normal S3 object logs do.

Conclusion

You should apply an SCP from your organization to deny use of the Lightsail service. Lightsail is a useful service for personal users and maybe small businesses, but it’s not aligned with the interests of larger organizations. The addition of yet another ability to make data public, which originally circumvented the S3 public block access mitigation, and to make long-lived credentials that you have little visibility into, is a step in the opposite direction of the security best practices that AWS has tried to move people toward.

Lightsail object storage concerns - Part 1

Thu, 05 Aug 2021 13:00:00 -0700

This is part one of a two part series that will discuss AWS’s new Lightsail object storage. In this first part, we’ll look at the new access key capability and a security issue I discovered that has been fixed. In the second part we’ll look more closely at the buckets created.

What is Lightsail?

In 2016, AWS released Lightsail as a way of providing a simpler version of AWS in response to DigitalOcean. Instead of having to worry about configuring your networks and paying for bandwidth and all the other little details of setting up and running an EC2, Lightsail advertised an easier way for a flat rate of $5/mo. Over the years, Lightsail has expanded to incorporate more functionality than just a simpler EC2, and this service within AWS is now seen as being a mini-AWS within AWS that expands to achieve feature parity of where AWS was a number of years ago.

Here’s a table comparing the Lightsail features to AWS services and their respective launch dates.

Lightsail feature	AWS service
Compute and DNS, 2016	EC2, 2006; Route53, 2010
Load balancers, 2017	ELB, 2009
Block storage, 2017	EBS, 2008
Databases, 2018	RDS, 2009
Automatic snapshots, 2019	Backup, 2019
Containers, 2020	ECS, 2014
Resource monitoring and notifications, 2020	CloudWatch, 2009; SNS, 2011
CDN, 2020	Cloudfront, 2008
Object storage, 2021	S3, 2006

Trying out Lightsail object storage

On July 14, AWS announced Lightsail object storage, a capability that sounds like S3. The SDK commit mentions a new API CreateBucketAccessKey which seems like a wild circumvention of IAM credentials. This looks exciting.

I decided to start from the web console for Lightsail, which does not conform to any of the standards of the rest of the console.

I went over to create a bucket, and quite clearly you can see by the domain name, that this is an abstraction layer over S3.

I created my bucket and quickly jumped to the S3 service console, but did not see any trace of my Lightsail bucket! Sometimes one AWS service will provide an abstraction over another AWS service and resources will appear in your account for that other service, but this must be created in a different AWS account. We’ll look more at that bucket in part 2.

Getting an access key

From the Lightsail console I created an access key, which is said to be used only for Lightsail bucket access. The first thing I did was figure out what account that key belongs to:

$ aws sts get-access-key-info --access-key-id AKIA4L53X6C6KXW6KH6W
{
    "Account": "850260390076"
}

That account ID is not mine, so this access key was created in a special Lightsail account. Let’s go ahead and use it and find out more information.

$ aws sts get-caller-identity
{
    "UserId": "AIDA4L53X6C6AEGC7HIQA",
    "Account": "850260390076",
    "Arn": "arn:aws:iam::850260390076:user/bucket-kgxqi8.obj-mgmt"
}

So the username matches the bucket name, plus the .obj-mgmt suffix, which is interesting as that makes any IAM policies for this harder for AWS to write for restricting it to the same name as the IAM user.

This access key is used for S3 bucket access, so let’s try listing the buckets.

$ aws s3 ls
2021-07-23 11:13:23 bucket-kgxqi8
2018-12-26 00:57:12 cloudtrail-logs-850260390076-amazon-lightsail

Well now that is odd. I didn’t expect to be able to list the buckets in the account, and also didn’t expect there would already be another bucket there. I’m unable to list the objects in that cloudtrail S3 bucket, and that creation date is a few years old even though this is the first time I believe I’ve used lightsail in this account. That date is also not the same as the creation date of my account.

I wasn’t able to do much else with the key, beyond the expected use case of interacting with my bucket, and not in any ways I found interesting.

Back in my original account, I was curious to play with these new Lightsail APIs. You can see here that in order to find the Lightsail bucket access keys, you have to provide the Lightsail bucket name, so you have to iterate through all Lightsail buckets in the account to find possible credentials. For those that want to ensure they have no long-lived credentials in their accounts, this makes it slightly more annoying to check.

$ aws lightsail get-bucket-access-keys --bucket-name bucket-kgxqi8
{
    "accessKeys": [
        {
            "accessKeyId": "AKIA4L53X6C6KXW6KH6W",
            "status": "Active",
            "createdAt": "2021-07-23T12:48:56-06:00"
        }
    ]
}

Why this access key is a problem

A best practice on AWS is to avoid using long-lived IAM user access keys, so the creation of these seems like a step backward. Another best practice is applying a least privileges strategy, and specifically with S3 buckets, you should avoid granting the ability to list buckets. These keys allow listing the lightsail buckets and you cannot restrict this.

This key lives in another account and you do not have the same privileges over this key as you would if it was in your own account. ~~Specifically, you cannot get information about when this key was last used, so you you cannot identify unused access keys, which is another best practice to identify and revoke those.~~ Update 2021.08.08: AWS added the ability to see the last used time on these keys.

In the event of a credential compromise where you wanted to revoke that access key, you could revoke the key, but you could not apply an IAM policy in order to revoke any sessions, and because aws sts get-session-token is always possible with an access key, an attacker would be able to create their own sessions. This makes AWS’s recommendations for incident response impossible to perform.

The security issue: Cloudtrail logs

The above issues make me dislike this feature, but they don’t meet the bar of being a security issue that AWS needs to fix. So this brings us to an actual security issue with these that I reported and AWS fixed.

After playing with these for a bit, I looked at the Cloudtrail logs for the activities in my account, and everything aligned with my expectations, except when I looked at the call to CreateBucketAccessKey. This was in the responseElements

"accessKey": {
    "secretAccessKey": "gKk4ZzUFwWspe8Gx+x5iRfbDLfq7/tHH3Olss62T",
    "status": "Active",
    "accessKeyId": "HIDDEN_DUE_TO_SECURITY_REASONS",
    "createdAt": "Jul 23, 2021 6:48:56 PM"
}

They redacted the wrong value! They are supposed to redact the secretAccessKey (that’s why the name includes the word “secret”), not the access key id. This is a problem because Cloudtrail logs are not supposed to contain secrets.

I immediately reported this to AWS, which unfortunately was at 8pm Mountain time on a Friday on July 23. Two hours later they emailed me an acknowledgement of the issue, and on Saturday afternoon they had a call with me to let me know a fix was being deployed and by Monday it had been fixed globally. I really appreciate how quickly the AWS security team responds to issues and is able to get the relevant teams to produce fixes. This issue would have required an attacker to have access to the Cloudtrail logs and be able to find out the access key id, so personally I would have ranked this as low severity and not something the team needed to work late on or over the weekend, but I was glad to see how seriously AWS took it.

Unfortunately, the fix for this problem was to set the responseElements to null, which means you can no longer see information about which access keys were created. This means that if you do happen to use Lightsail access keys in your account, and you have a compromise where the attacker creates new Lightsail access keys, you will not be able to easily figure out which ones they created. You will need to identify these CreateBucketAccessKey calls and then look at the creation dates of the Lightsail access keys to identify which ones were created during the incident so you can revoke them.

Conclusion

Lightsail is a simplified version of AWS, and in order to simplify it, AWS has ignored some security best practices, such as not having long-lived access keys and issues that will be discussed in part 2. This service is directionally not aligned with the best practices advocated by the rest of AWS. Although Lightsail serves a certain type of customer, for most businesses, my recommendation is that you should apply an SCP over your organization to deny use of the Lightsail service, ideally by crreating an allow list of allowed services as described here.

S3 backups and other strategies for ensuring data durability through ransomware attacks

Tue, 03 Aug 2021 13:00:00 -0700

Ransomware has become a primary concern for infosec. This post will discuss options for ensuring the durability of data stored on S3, through protections in place and backup strategies. The AWS backup service on AWS unfortunately does not backup S3 buckets and a lot of discussions of backups and data durability on AWS do not describe the implementation in sufficient detail, which allows a number of potential dangers. This post will show you the two best options (s3 object locks and replication policies), explains how to use these, and what to watch out for.

Ransom attacks in AWS?

Wherever critical data lives, ransomware attackers will go. Although I’m not currently aware of ransomware groups that specifically targets AWS environments or S3, I have heard of ransom based attacks in AWS, along with scorched earth attacks where attackers just deleted all of the resources including data in the S3 buckets in accounts they compromised or in S3 buckets they were able to access. One of the most infamous breaches of an AWS account, the Code Spaces breach back in 2014, involved a decision not to pay a ransom, the deletion of everything in an AWS account, and the subsequent shutdown of the business within 12 hours of the breach as all data (and backups) had been destroyed.

On March 26, AWS added an inline policy to one of my IAM users (for flaws.cloud) that they believed was compromised by adding a deny on s3:*, and on April 21, AWS released a new IAM managed policy named AWSCompromisedKeyQuarantineV2 which included a deny on s3:DeleteObject and similar actions that would be used by an attacker to delete data. Both of those moves by AWS hint at some sort of attack they saw.

Diff of AWSCompromisedKeyQuarantine vs v2

Spencer Gietzen, who sadly passed away earlier this year, had published research in 2019 in his posts S3 Ransomware Part 1: Attack Vector and Part 2: Prevention and Defense, and presentation Ransom in the Cloud. Since that time, there have been some improved mitigations, that I’ll discuss.

The threat and constraints I’m considering

The threat I consider is having an attacker obtain admin access to an account that contains critical data in an S3 bucket. At one end of the spectrum, this could simply be a legit admin accidentally running an aggressive script that deletes the wrong S3 bucket, and at the other end this could be a sophisticated attacker that knows their way around an AWS environment and the potential mistakes that could be made in attempting to ensure data durability.

Even if you have many controls in place, ransomware groups are making enough money (supposedly over $100M/yr for REvil) that there is a possibility of them buying effective 0-days to target your AWS admins or IT staff (that likely has the ability to do things like password resets on your AWS admins or grant themselves privileged access to your AWS accounts). So we want to ensure our controls are as good as they can be.

S3 has 99.999999999% data durability of objects over a given year, and stores copies of your data redundantly across 3 AZs, meaning that even if two entire AZs in a region get destroyed at the same time, your data will persist. So we’re not concerned with a hard-drive failure at AWS resulting in our data loss. Of interest on this topic though, be aware that AWS has stated they store over 100 trillion objects in S3, so the way the math works out, this means they lose 1,000 S3 objects per year. At scale, rare events aren’t rare. Unfortunately I’ve never heard of what this looks like when it happens (does AWS notify you? Return an error when you try reading it? Return a corrupted object? Not show it listed anymore?).

For our solution, we want to maintain the existing usability of the S3 bucket as much as possible, meaning we still want the ability to modify and delete objects in this bucket, but we may wish to recover to an earlier time. For this reason, MFA delete is not an acceptable solution, because to delete an object you must use the AWS root account with an MFA device and provide the MFA code for every object deletion.

We want an RPO (Recovery Point Objective) on the order of 15 minutes (meaning we are willing to lose up to the most recent 15 minutes worth of data in an incident) and RTO (Recovery Time Objective) of a day (meaning we accept that it may take 24 hours to recover from an incident). Understand that if you want to improve those numbers, you may need to dramatically re-architect for a better solution.

We want to minimize costs. This mostly means we want to avoid having too many copies of our data and may want to use less expensive storage classes for any backups. Copying all our data to Snowballs everynight and shipping them to different physical locations is thus not acceptable as this minimally will cost $9K/mo to cover the $300/Snowball cost, but you’ll additionally have data transfer of $0.03/GB (which is more than normal S3 charges) and shipping fees. This also will not meet our RPO and RTO goals.

We assume that the need to recover is apparent when an incident happens. If an attacker has corrupted only a single file, and then waits a year to demand a ransom to tell you which file changed, it may be very difficult to detect and is out of scope of this article.

What about cross-region backups?

One concept you’ll often hear about for data durability is to backup your data to another region, but there are some issues with this:

Copying your data to another region incurs data transfer costs. It costs nearly as much for S3 to replicate your data to another region ($0.02/GB) as it does to store a GB in S3 for a month ($0.023), with some regional differences. Therefore if you backup your data to S3 standard in another region, you’ve nearly tripled your cost of that data. Attempting to use less expensive storage classes to save money will not work how you might expect as will be discussed later.
The possible situations that would result in the permanent loss of 3 AZs simultaneously may mean a physical disaster of significant enough impact that your business continuity may be the least of your worries.
If you’re concerned about risks that could impact an entire AWS region, your concerns are likely better addressed by backing up your data to another cloud provider in another region as well. Going multi-cloud for your backup strategy would ultimately address more risks more effectively than going multi-region within AWS.

You may need to backup to another region for compliance reasons though, and the ideas I propose will still be useful for that scenario.

Protecting the data in place

Least privilege

You should always strive to ensure least privileged access to your AWS accounts and avoid situations where an attacker would have access to your critical data. Putting your critical data in a dedicated AWS account is helpful for accomplishing this, especially if privileged users are regularly accessing your production account.

Restricting access to your S3 buckets can be further enhanced via VPC endpoints to only allow access from specific VPCs. This can help ensure that even if an admin user makes a terrible mistake or is compromised, they will not be able to impact the critical S3 buckets unless they do so from the allowed VPC.

When working with S3 buckets, you need to be very aware of how resource policies can grant access to principles even if that principle does not have associated IAM policies that have granted them access. The following graph warrants its own blog post, but the critical take-away is there are two green end states, meaning two ways you can grant someone access to a resource.

AWS policy evaluation graph

To explain this concept, a common issue I’ve seen is that an S3 bucket will have a statement similar to the following in its resource policy:

{
    "Sid": "DO_NOT_DO_THIS",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
        "arn:aws:s3:::bucket",
        "arn:aws:s3:::bucket/*"
    ],
    "Condition": {
        "StringEquals": {
            "aws:PrincipalOrgID": "o-abcd1234"
        }
    }
}

The person who set this policy wanted to make sure that it was possible for others in the organization to be able to access this bucket, but did not realize that they are actually granting access, especially to those within the same account as the bucket. When I would do assessments, I would be given SecurityAudit privileges, which should not allow me to read or write sensitive data, but because of this resource policy, myself and other vendors that had least privilege roles in the account would actually be given full access to that S3 bucket!

Random naming

As part of least privilege you should avoid giving the ability to list the S3 buckets in the account (s3:ListAllMyBuckets) or the objects within those buckets (s3:ListBucket). But let’s say you give only s3:PutObject for something that produces a critical set of logs each day that are named 2021-01-01.log, 2021-01-02.log, etc. If someone compromises that service and knows the naming pattern, they could overwrite these log files. There is not a way on AWS to give permissions to write an object, but not overwrite existing objects (at least not in the way most want, but object versioning discussed below helps with this). It would be better to name the files with something random, such as 2021-01-01-fdc1127f-7a1d-4721-a081-a66a536649ae.log. You’ll still be able to identify when this log was created, but an attacker would not be able to blindly overwrite it.

The foundations of our durability options

S3 object versioning

Many of the strategies to be discussed for data durability require S3 object versioning to be enabled for the bucket (this includes S3 object locks and replication policies). With object versioning, anytime an object is modified, it results in a new version, and when the object is deleted, it only results in the object being given a delete marker. This allows an object to be recovered if it has been overwritten or marked for deletion. However, it is still possible for someone with sufficient privileges to permanently delete all objects and their versions, so this alone is not sufficient.

When using object versioning, deleting old versions permanently is done with the call s3:DeleteObjectVersion, as opposed to the usual s3:DeleteObject, which means that you can apply least privilege restrictions to deny someone from deleting the old versions. This can help mitigate some issues, but you should still do more to ensure data durability.

Life cycle policies

Old versions of objects will stick around forever, and each version is an entire object, not a diff of the previous version. So if you have a 100MB file that you change frequently, you’ll have many copies of this entire file. AWS acknowledges in the documentation “you might have one or more objects in the bucket for which there are millions of versions”. In order to reduce the number of old versions, you use lifecycle policies.

Audit tip: It should be a considered a misconfiguration if you have object versioning enabled and no lifecycle policy on the bucket. Every versioned S3 bucket should have a `NoncurrentVersionExpiration` lifecycle policy to eventually remove objects that are no longer the latest version. For data durability, you may wish to set this to 30 days. If this data is being backed up, you may wish to set this to as little as one day on the primary data and 30 days on the backup.

If you are constantly updating the same objects multiple times per day, you may need a different solution to avoid unwanted costs.

Audit tip: In 2019, I audited the AWS IAM managed policies and found some issues, including what I called Resource policy privilege escalation. In a handful of cases AWS had attempted to create limited policies that did not allow `s3:Delete*`, but still allowed some form of `s3:Put*`. The danger here is the ability to call `s3:PutBucketPolicy` in order to grant an external account full access to an S3 bucket to delete the objects and versions within it, or `s3:PutLifecycleConfiguration` with an expiration of 1 day for all objects which will delete all objects and their versions in the bucket.

Storage classes

With lifecycle policies, you have the ability to transition objects to less expensive storage classes. Be aware that there are many constraints, specifically around the size of the object and how long you have to keep it before transitioning or deleting it. Objects in the S3 Standard storage class must be kept there for at least 30 days until they can be transitioned. Further, once an object is in the S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA, those objects must be kept there for 30 days before deletion. Objects in Glacier must be kept for 90 days before deleting, and objects in Glacier Deep Archive must be kept for 180 days. So if you had plans of immediately transitioning all non-current object versions to Glacier Deep Archive to save money, and then deleting them after 30 days, you will not be able to.

S3 object locks

S3 object locks are a powerful feature that allows you to ensure that an S3 object cannot be deleted until a set date, no matter what. This is great for our data durability, but this can also be potentially dangerous because an attacker could sync in a lot of large files from a public S3 bucket, and then lock these files for up to 100 years, and by design, not even AWS can undo this. The only way to stop paying for locked files that you don’t care about is to have AWS delete the account, which first means transitioning all your resources (except these locked files) to another account, which is often a big lift. To help mitigate this, the AWS documentation has a statement that you can apply to your bucket policy, or as an SCP, that restricts people from locking objects for too long.

{
    "Effect": "Deny",
    "Principal": "*",
    "Action":  "s3:PutObjectRetention",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": {
        "NumericGreaterThan": {
            "s3:object-lock-remaining-retention-days": "30"
        }
    }
}

Ian McKay put together a great list of privileges you should disable at the SCP level to limit some Denial of Wallet attacks and mistakes, such as someone using object locks (s3:PutObjectRetention) or setting a default object lock policy (s3:PutBucketObjectLockConfiguration). Denying the privilege s3:PutBucketObjectLockConfiguration will prevent someone from creating an S3 bucket that has S3 object lock enabled, but if one already exists, and the attacker has sufficient privileges, even with the above SCPs in place, the attacker could change the bucket policy to allow access from their attacker account, and because SCPs do not apply to principals in external accounts, the attacker could then fill the bucket up with objects that are locked for the next 100 years.

Audit tip: Ensure you apply SCPs to your accounts to prevent Denial of Wallet attacks using object locking and prevent locking objects for more than a duration you are comfortable with.

Despite the above concern, S3 object lock is the most powerful feature on AWS for ensuring your data cannot be deleted. You can also apply a default object lock retention period to an S3 bucket which means how you interact with the S3 bucket will not change. Objects can still be marked for deletion as happens normally with S3 object versioning, but the object cannot be deleted until the lock expiration date happens.

The problem I have with S3 object lock from a data durability perspective, is there isn’t a concept of a rolling lock. I want to say “Allow objects to be marked for deletion, but don’t allow them to be permanently deleted for 30 days after that”. To explain this, imagine you have an object lock configuration that locks your objects for 30 days. You then put some critical objects in it. If your account is breached on day 1, the attacker will not be able to permanently delete these objects. If the attacker breaches the account on day 31 with sufficient privileges, they can wipe out all your critical data that you had put there on day 1.

One possible solution is to run a nightly process to iterate through all your objects and put an object lock on the latest object versions. This can be done with the S3 Batch service as documented here.

Replication policies

The classic approach to data durability is to create backups and ensure the backups are separate from the primary data. In AWS, in order to have a strong security boundary, this must be done in a separate AWS account. Ideally, this would be completely separate from your other accounts (separate AWS organization, not accessed through your normal SSO process, accessed only with special unmanaged laptops, etc.), but there are good arguments for not going to this extreme, especially because it often means attempting to duplicate many of your security controls. Another important concern is that you will likely have multiple sets of critical data that have potentially been strongly isolated, all being backed up to the same AWS account. In doing so, the backup account becomes an attractive target full of sensitive data.

Audit tip: Ensure sufficient isolation of the backup account. You should be fully aware of, and acknowledge the risks or take steps to mitigate, situations where:

The same people have access to both the primary data and backup data.
The IT or help team has the ability to reset credentials or deploy software updates to the people with access to the backup data, or to add themselves to the groups that have access.
Any cross-account AWS roles that grant write or delete access into the backup account from other accounts.
Monitoring or security controls on the backup account are not aligned with how other accounts are secured.

The easiest way to setup S3 backups is through replication policies, which can copy objects to an S3 bucket in a different account in the same region or a different region. To do this, you need to:

Create an IAM role that will replicate this data.
Set the bucket resource policy in the target bucket in the backup account to allow the source account to send data to it, as documented here.
Set replication policy on the source bucket in primary account.
- Ensure delete markers are replicated.
- Include changing the replica owner to the policy.
- Set a less expensive storage class for the replicated data.
- Optional: Ensure Replication Time Control is enabled to ensure the objects are replicated within 15 minutes for 99.99% of objects.
Set lifecycle polices on the source and destination buckets to clean out the non-latest version of objects.
Optional: Monitor notification events for replication failures.

Audit tip: You must ensure the bucket policy on the destination (backup) bucket is not too open. If this grants the source account too much access, then the attacker, from the source account, will be able to delete the logs in the backup bucket. The bucket policy should only allow the IAM role in the source account the following actions, limited to the backup bucket: `s3:ReplicateDelete`, `s3:ReplicateObject`, `s3:List*`, and `s3:GetBucketVersioning`. The documentation lists `s3:PutBucketVersioning` but this seems unnecessary.

Audit tip: If the backup objects are encrypted, you should consider whether the attacker can delete the key that would be used to decrypt the backups, or else your backups could become useless in an incident.

Recovery

In the event that an attacker has compromised and destroyed your primary data, your backup bucket will have deletion markers on it that you’ll need to remove. You’ll then need to copy your data back from your backup bucket to the primary bucket. Be mindful in how you do this that you do not enable an attacker that may still be around to have write access to your backups. AWS is not clear on what the fastest method is for this, but I think AWS DataSync may be the best solution. You should work with your TAM or AWS support on this recovery process. A problem with many ransomware cases I’ve heard is that recovering from backups takes longer than just paying the ransom and recovering that way, but I don’t think this would be the case for data stored in S3.

Further considerations

With object locking and replication policies, when these are applied to an existing S3 bucket, the objects that already exist in the S3 bucket will not be impacted. So if you have an existing S3 with critical data in it, you must take actions to replicate or lock the existing files. AWS support can assist with replicating files and enabling object locking on existing S3 buckets.

Detections

You should setup monitoring to detect and alert on the following:

Detect s3:PutBucketLifecycleConfiguration as this can be used to delete objects. Mark it has high severity if it is less than some amount, say 30 days, or when it is not on the non-current object version.
Detect s3:PutBucketPolicy calls on any critical S3 buckets.
For data events:
- Detect calls to s3:DeleteObjectVersion.
- Detect access denied calls to s3:DeleteObject
Detect the creation of any S3 bucket with object locking enabled and auto-remediate.

Cloudtrail data events can get expensive (both to record and to monitor), but you can use advanced event selectors to look for specific object level APIs and filter on specific S3 buckets.

Conclusion

Ensuring S3 data cannot be deleted by an attacker is not entirely trivial, but hopefully this guide has explained the better options (s3 object locking and replication policies) and pointed out some common problems to watch out for.

References

Protecting Amazon S3 Data from Ransomware by Ben Potter

AWS security project ideas

Tue, 16 Feb 2021 12:00:00 -0800

I’m excited to announce that I’ve taken a new job and am shutting down my consulting business. This post will discuss some project ideas I never got to, but first I want to briefly discuss this move. It’s weird to move on from something I built over the past 3.5 years and that was by all definitions a success. I’ve had dozens of clients across 5 continents, was quoted in the WSJ, keynoted a conference in Switzerland, travelled to South Africa to train people, obtained over 10,000 followers on Twitter, worked with Duo Security to create some of the most popular open-source cloud security tools, and generally have become one of the goto people in the world for AWS security.

It all started with the release of flaws.cloud almost 4 years ago to this day, which motivated me to quit my job and start that new adventure. I’ve had a ton of personal and professional growth along the way. I highly recommend considering that life path and have written about how to do something similar here and here. I’m excited for this new opportunity where I can focus on challenges that require deeper integrations, architectural changes, and longer time horizons than short-term contract work.

This post will describe project ideas I didn’t get around to as I suspect I’ll be a bit too busy for a while to get them. If you’d like to kick start a consulting business, draw attention to the engineering talent of your organization, or possibly create a SaaS business to get some extra revenue, these are some ideas that I think the world of AWS security would benefit from.

Security Group optimizer using VPC Flow Logs

In much the same way as tools such as repokid and cloudtracker have recommended or auto-remediated changes to IAM privileges based on the privileges actually used (as evidenced by Access Advisor or CloudTrail logs) vs the privileges granted, the concept here would be to take the existing Security Groups (ie. network access granted) and available VPC Flow Logs (ie. network access used) and perform a diff. The resulting output would allow you to recommend changes, so you could say “This EC2 has never received traffic on port 80, therefore that Security Group can be changed”. I suspect that much like the problems I encountered in graphing network diagrams with CloudMapper, this may be more difficult in larger environments. You would also likely need to take CloudTrail logs into consideration in order to understand what EC2s (or other resource) existed at the time of the flow logs. AWS has a blog post here that mentioned some analysis of VPC Flow Logs for Security Group improvements, but is minimal in terms of what all could be done.

Investigate memory capture using EC2 hibernation

When AWS updated their Incident Response whitepaper in June 2020, the biggest change was the mention of using EC2 hibernation for memory capture. Historically it has been very easy to take a disk snapshot of an EC2, but doing a memory capture required third party tools. Having AWS native functionality for memory capture is very interesting, but there are a ton of limitations and requirements for this to be possible, and it’d be interesting in seeing this explored. See some discussion here.

IAM privilege aggregator

I have often found myself trying to understand what all the privileges a role has when taking into consideration the IAM policies, Permission Boundaries, and SCPs being applied to it. At a minimum it would be nice to just concat all these together into one place, especially for SCPs which can be applied to the account or multiple levels of OUs. Ideally though you’d like something that can understand how these operate together and give you the actual privileges in some minimized way. For example, if an IAM role has an admin policy, but a permission boundary that only allows access to S3 and an SCP that denies changing GuardDuty, you’d like to see that the role can only access S3. This functionality is needed for some later projects.

Access Denied explainer

Similar to the “IAM privilege aggregator”, a situation that is going to become an increasingly worse problem on AWS, is debugging why something was denied. When you get an Access Denied error, there is no further context, either within the API response or CloudTrail logs. Imagine trying to access an S3 bucket and getting an Access Denied. Was it because you didn’t have privileges, or because of a permission boundary, a bucket policy, or SCP denying access? This problem is made worse because from an account you do not have an ability to see what SCPs have been applied to you. Is the access restricted based on some set of conditions? Maybe the tags on the bucket don’t match the tags required for your access?

As SCPs become more common and as AWS advocates greater use of ABAC, this situation is going to happen more frequently and become harder to debug. Minimally, you want to at least tell someone the relevant statements and conditions for the privilege involved. The ultimate goal wold be to tie this to the CloudTrail events in EventBridge to look for Access Denied messages in real-time and then automatically send a Slack message to someone telling them “Hey, looks like you tried to create an EC2, but you’re only allowed to use t3.micro instances in that account”. There are some common conditions like that which you could have pre-planned conversational text for. If you build that, there are people I know right now that would write you checks. Another option would be to use CSM to identify these Access Denieds locally or in special cases, such as when unit tests are run in a CI/CD or staging environment.

SCP baseline creator

Once in an engagement I was given access to a newly created account that had been initialized with the company baseline that included IAM roles for a vendor, enabled GuardDuty, EventBridge rules, etc. I was asked to create an SCP that could protect those configurations, so for example if a developer runs aws-nuke on a sandbox account, it doesn’t modify this baseline. It should be fairly easy to generate an SCP based on some common features like that which has built-in functionality for exception conditions so that those features can still be changed by a certain Organization accessible admin role.

Along with this, it would be nice to have a differ and some sort of linter, such that you could identify that an existing SCP, for example, does not properly protect GuardDuty.

Tagging policy SCP generator

Many companies would like to ensure all their resources are tagged with certain keys, such as an Owner tag. You might think that the Organization feature Tag Policies could do this, but you would be sorely dissapointed, just like everyone else that ever tried using that feature to do anything meaningful. One way of accomplishing this is to auto-remediate or otherwise detect improperly tagged resources after the fact, but I believe a better way would be to enforce this via an SCP. Before doing this, you’ll want the “Access Denied explainer” project to exist, or else you will bring sadness and frustration to your developers.

AWS List/Describe proxy

Many companies have multiple tools scanning their accounts looking for security, tagging, cost, operations, and other issues. As these tools make List and Describe calls, in large environments, eventually they are rate limited and companies try to make use of AWS Config, which requires you to change how you make queries and also lacks coverage across a number of services and features. It would be better to have a single tool collect this information, possibly leveraging AWS Config and also possibly leveraging the CloudTrail data to keep it updated without querying (or at least without querying as often). Netflix began down this path with their Historical project, but no longer maintains that project. Ideally, this would work fairly transparently, such that you could trick an application into visiting your server instead of amazonaws.com (such as via /etc/hosts and the SDK environment variable CA_BUNDLE so you get through HTTPS), then you’d either respond with the data or relay the request to AWS. In environments that use AWS Config, you might convert the request into an Athena query against the data in S3 as explained here.

Improving existing tools

The ideas above were for some new projects. There are lots of features and improvements to CloudMapper that I haven’t gotten to as well, in addition to just general maintenance and bug fixes of CloudTracker. One valuable focus area for CloudMapper is on the identification and visualization of trust relationships between accounts. CloudMapper, via the weboftrust command, can identify trust relationships between accounts of IAM roles, S3 buckets, and VPC peerings. There are lots more relationships that can exist, and one could start by going through the list of aws_exposable_resources. The visualization of these could also be improved by simply creating a table as one option. In theory Access Analyzer could find many of these relationships, but that tool lacks the coverage you would expect.

Conclusion

I’m not going away, but I might not ever get to some of these things, and I don’t want these ideas to die, so go forth and do awesome things!

AWS Security Maturity Roadmap 2021

Tue, 12 Jan 2021 04:00:00 -0800

This is the third annual release of my “AWS Security Maturity Roadmap” to give companies a series or actionable steps to improve the security of their AWS environments. Each year I update this with the latest features of AWS, along with improvements based on my discussions with clients, other consultants, and more who have gone through some of these activities or encountered challenges, or have mentioned new ideas. This includes companies of all sizes, risk tolerances, and integration (from lift and shift to completely serverless).

Changes include:

Added that your root email addresses should be under your company domain and on email distribution lists as I’ve found many accounts during assessments that were associated with old aol.com emails and other personal accounts. This is how AWS communicates with you and is an important part of account recovery.
Added opting out of the AI services from sending your data outside of the regions you put it in.
Added budget alarms to Stage 1. Although less likely to provide as much value for larger enterprises, these are very important for individuals and small businesses. Also mentioned Cost Anomaly Detection and Budget Actions.
Mentioned that you should consider using GuardDuty’s S3 monitoring, which for monitoring, is as an alternative to Macie.
Mentioned the use of CloudFormation StackSets with Organizations which is an Organization feature that deploys a CloudFormation StackSet whenever a new account is created to ensure it is initialized with your baseline.
Discussed using a ticketing system for alerts, as I’ve found the ways in which companies receive alerts to vary in maturity which impacts their ability to properly monitor and respond to incidents.
Discussed turning on more log sources, such as VPC DNS queries, VPC Flow Logs, S3 access logs, etc. and linked to a post by Matt Fuller for a list of log sources on AWS: How to Enable Logging on Every AWS Service in Existence (Circa 2021).
Mentioned that IAM user access keys, when required, should ideally be put in a single account and assume roles into other accounts as it makes detection of compromise easier.
Mentioned that Access Advisor now shows the use of s3:ListAllMyBuckets, so you should use that to remove that specific privilege when not needed.
Discussed the need to plan how accounts will be connected, as the security boundaries of separate accounts can become blurred due to trust relationships of resource policies, network connections, and other mechanisms.
Expanded on the discussion of using SCPs to mention how exceptions can be made for specific roles, and SCPs can vary between accounts. Therefore, you can initialize an account by configuring it’s network configuration in advance and restricting changes via SCPs, such that a dev account may be restricted from making any resources public. Also listed possible SCPs to deploy, which are described in my post AWS SCP Best Practices.
Mentioned former2 for helping transition to IaC (Infrastructure as Code).
Mentioned that IaC scanning tools can be used, referencing Christophe’s post Shifting Cloud Security Left — Scanning Infrastructure as Code for Security Issues.
Removed mention of Tag Policies as this service is limited in various ways that make it not worth bothering with.
Merged “Stage 8: Enhance detection” and “Stage 9: Auto-remediation and privilege refinement” into one stage. Mostly because I wanted this to be 10 stages. :)
Mentioned Client Side Monitoring as part of the least privilege improvements, and linked to the cloudonaut article Record AWS API calls to improve IAM Policies.
Mentioned the AWS Network Firewall as a solution for monitoring and protecting your networks.
Mentioned the use of microservices to access resources instead of direct access so that better monitoring, restrictions, and rate limiting can be performed.

Download the AWS Security Maturity Roadmap for 2021
...Read more ...

Opting out of AWS AI data usage

Wed, 06 Jan 2021 04:00:00 -0800

Update 2022.02.22: AWS has no team managing this feature and now some services work slightly differently depending on if this feature is enabled or not. Some caution must therefore be taken when enabling this. AWS is working on better documenting what these differences are.

This post will discuss why you should opt out of the AI data usage on AWS, how to do that, and how to confirm you did it correctly.

A general rule on AWS is that your data will not leave the region you put it in. AWS customers rely on this for compliance, data sovereignty, and other reasons. Another general rule is that AWS will not access your data. AWS customers rely on this because they want to use AWS services, but do not want Amazon to use their data to create competing products or improve their services in such a way that it would directly benefit existing competitors.

In 2017 AWS made exceptions for these rules for their Machine Learning and Artificial Intelligence services in the AWS Service Terms. Specifically, under section 50.3, by using AWS services such as Lex, Transcribe, and more, you automatically opt in to allowing AWS “to develop and improve AWS and affiliate machine-learning and artificial-intelligence technologies” and that they may store your data outside of the region you put it in. Section 50.4, as I understand it, then informs you that it is your responsibility to disclose this use by AWS to your own users in order to abide by various laws, such as the Children’s Online Privacy Protection Act (COPPA).

The AWS terms are not often read or understood, other than the regular chuckle about Amazon Lumberyard having a clause related to zombies (section 42.10). So it wasn’t until AWS added functionality to the SDK for Organizations and subsequent announcement in July 2020 that I noticed the section about the AI services. They added functionality to the SDK to opt-out of this data usage, which is described in the docs here. Opting out of this still allows you to use these services. Opting out just means that now AWS won’t copy your data or use it for their own benefit. There is no negative to opting out other than that maybe these services may not be improved as much as they would otherwise.

The functionality has all sorts of complexity to allow opting out of specific services, default allowing some services, having different policies for different accounts, and whether accounts can make exceptions. This is ridiculous because anyone that learns about this and takes action is only going to ensure that all of their accounts are opted out.

The remainder of this article will explain how to turn this on and how to confirm it is on.

Opting out

You should opt out of this data usage. You can do this by following the docs to enable the opt-out ability, creating a policy, and associating that policy with your organization root. The policy you create should be:

{
    "services": {
        "@@operators_allowed_for_child_policies": ["@@none"],
        "default": {
            "@@operators_allowed_for_child_policies": ["@@none"],
            "opt_out_policy": {
                "@@operators_allowed_for_child_policies": ["@@none"],
                "@@assign": "optOut"
            }
        }
    }
}

This policy is documented here and will opt you out of all services from using your data for all accounts, and will not allow member accounts to create exceptions. As mentioned, these policies are needlessly complex.

To opt-out via terraform you can use this gist from Christophe Tafani-Dereeper.

This process can be done from the command-line with the following with a session to the Organization management account:

# Get root id
export ROOT_ID=`aws organizations list-roots | jq -cr '.Roots[0].Id'`

# Enable the feature
aws organizations enable-policy-type --root-id $ROOT_ID --policy-type AISERVICES_OPT_OUT_POLICY

# Create the policy
aws organizations create-policy --type AISERVICES_OPT_OUT_POLICY --name optout --description "Opt out of AI services using our data" --content "{\"services\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"default\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"opt_out_policy\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"@@assign\":\"optOut\"}}}}"

# Get the policy id
export POLICY_ID=`aws organizations list-policies --filter AISERVICES_OPT_OUT_POLICY|jq -cr '.Policies[0].Id'`

# Attach it to the root
aws organizations attach-policy --policy-id $POLICY_ID --target-id $ROOT_ID

Confirming you opted out

To ensure this has been applied correctly, I’ll list the steps to confirm this. This can be used by auditors to ensure this step has been performed. For these commands, ensure you have the latest version of the AWS CLI. When I run aws --version the response includes aws-cli/2.1.15 so ensure you are using at least that version.

Confirm feature is enabled

First, ensure the opt out functionality has been enabled for the organization. In the organization management account, run:

 aws organizations list-roots

This should return something like:

 {
    "Roots": [
        {
            "Id": "r-p0xn",
            "Arn": "arn:aws:organizations::123456789012:root/o-abcd123456/r-abcd",
            "Name": "Root",
            "PolicyTypes": [
                {
                    "Type": "AISERVICES_OPT_OUT_POLICY",
                    "Status": "ENABLED"
                },
                {
                    "Type": "TAG_POLICY",
                    "Status": "ENABLED"
                },
                {
                    "Type": "SERVICE_CONTROL_POLICY",
                    "Status": "ENABLED"
                }
            ]
        }
    ]
}

You want to confirm AISERVICES_OPT_OUT_POLICY is set to ENABLED.

Confirm the policy exists

Confirm an opt-out policy exists by running:

$ aws organizations list-policies --filter AISERVICES_OPT_OUT_POLICY
{
    "Policies": [
        {
            "Id": "p-0000000000",
            "Arn": "arn:aws:organizations::123456789012:policy/o-abcd123456/aiservices_opt_out_policy/p-0000000000",
            "Name": "optout",
            "Type": "AISERVICES_OPT_OUT_POLICY",
            "AwsManaged": false
        }
    ]
}

Then get the contents of this policy (replacing your policy id):

$ aws organizations describe-policy --policy-id p-0000000000
{
    "Policy": {
        "PolicySummary": {
            "Id": "p-0000000000",
            "Arn": "arn:aws:organizations::123456789012:policy/o-abcd123456/aiservices_opt_out_policy/p-0000000000",
            "Name": "optout",
            "Type": "AISERVICES_OPT_OUT_POLICY",
            "AwsManaged": false
        },
        "Content": "{\"services\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"default\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"opt_out_policy\":{\"@@operators_allowed_for_child_policies\":[\"@@none\"],\"@@assign\":\"optOut\"}}}}"
    }
}

You can pipe that through jq '.Policy.Content|fromjson' for a more readable policy. Confirm the policy matches the policy text shown earlier.

Confirm the policy is attached to the root

Finally, run the following (replacing your policy id):

$ aws organizations list-targets-for-policy --policy-id p-0000000000
{
    "Targets": [
        {
            "TargetId": "r-p0xn",
            "Arn": "arn:aws:organizations::123456789012:root/o-abcd123456/r-abcd",
            "Name": "Root",
            "Type": "ROOT"
        }
    ]
}

Confirm that the type in the response is “ROOT”.

This confirms that you have opted out all the accounts in your organization from having their data moved to other regions or used for reasons outside of your control.

Setting up personal G Suite backups on AWS

Tue, 24 Nov 2020 04:00:00 -0800

In giving training to companies on their AWS security, I advise they have backups. There was an infamous breach of a company named CodeSpaces that ran on AWS who had all of their data deleted by an attacker and the company had to shutdown within hours of the breach. I didn’t have my own house in order as well as I would have liked, so this week I set up automated backups, and did so in a generalized way that others can use via my cdk app backup_runner.

My business relies on Google Workspaces (previously known as G Suite) for email and Google Drive. My account is part of the Advanced Protection Program, but in a worst case scenario an attacker that obtained code execution on my laptop could wipe/ransom this data. I don’t know how Google would handle such an incident, but other scenarios also exist, however unlikely, that make me want to have backups of this data. My business is only myself, so using a featureful vendor solution isn’t needed, and it is nice to avoid some third-party risk of a vendor having access to all my email and company data, so I setup backups myself in AWS.

Tools used

I used got-your-back and rclone for the email and drive backups respectively. Got-your-back is made by Jay Lee, a manager at Google for Enterprise support, and the creator of GAM which is the de facto standard for interacting with the GSuite APIs. I could have used GAM for the drive backups, but it doesn’t maintain directory structure, which is important to me, so I turned to rclone.

Architecture

Historically, a number of folks have setup an EC2 and S3 bucket for this purpose, but I decided to run a nightly ECS container with an attached EFS, which itself is backed up via AWS Backup. The primary benefit of this architecture is AWS Backup takes care of incremental backups for me, meaning it only stores the changes each night, and makes it easy to recover the state of my data from any day in the past 35 days (this could be set longer).

Backup architecture

The output of the script run by the ECS is recorded to CloudWatch Logs, where a CloudWatch Alarm monitors for the word “ERROR”. If found, the alarm is sent to an SNS that I’m subscribed to. The AWS Backup service will also generate events that are sent to the SNS under error conditions.

Security considerations

Encryption

AWS Backups are always encrypted. I configured my EFS to be encrypted at rest, and the access from the ECS is configured to use encryption in transit.

Business continuity

I am storing the backup Vault in the same account as the EFS as my primary motivation is to get the data copied to AWS. AWS did just release cross-account backups for AWS Backups, but my data already exists in two places (the live data in GSuite and the copy in the AWS account).

I did set the access policy on the Vault to restrict some ways of destroying the backups, but there seem to be limitations in the CDK for deploying an ideal policy here, so a privileged attacker could get around this by editing/deleting the policy. One thing that AWS Backup does not offer is the concept used by S3 Object Lock and Glacier Vault locking which enforces strong WORM (Write Once Read Many) controls.

Secrets management

As the EFS contains all my email and data, it seems reasonable to just put the secrets used to download this data directly on that EFS, as compromising those creds does not extend the access beyond what is already on the EFS.

Costs

By running a nightly ECS, I avoid some of the compute costs of running an EC2, but EFS costs $0.30/GB vs $0.023/GB for S3 (EFS is 13x more!). Files in the EFS should move to the less expensive Infrequent Access Storage tier after 7 days which is only $0.025/GB which is roughly the same cost as S3, but S3 itself has an infrequent access tier at only $0.0125, and can be made even less expensive by using Glacier Deep Archive at only $0.00099/GB. However, given that I have 30GB in GSuite, my costs should end up around $9/yr for using EFS (once most of the data moves to infrequent access), so I’m not going to waste time trying to shave pennies off that.

AWS Backup is $0.05/GB, and does have a cold-storage tier at $0.01/GB, but that requires storing your data for over 90 days, and I only store mine for 35 days. The backups are incremental, meaning it automatically figures out what the new files are and only charges me for those.

Having run this setup for a month, my total costs are looking to be about $50/yr.

Using S3 and its storage tiers would result in cost savings, but the time saved in having AWS Backup handle some of these backup concepts instead of me figuring them out is worth the expense.

Deploying

In order to setup this architecture I created a CDK app. The repo to clone is called backup_runner.

The ECS just calls a script named nightly.sh that it expects to find on the EFS. As a result, this app is very generalized for any backup solution, so it could easily backup Office365 or other data.

Once the CDK app is deployed, it won’t actually copy over any data to it, because again, it just runs a script. So in order to set this up you need to spin up an EC2 in the subnet where the EFS is , access a command-line in the EC2 (via SSH, EC2 Instance Connect, or Session Manager), attach the EFS to the EC2, and then setup got-your-back and rclone, or whatever else you had in mind. Setting those up is outside of the scope of this post, but it involves setting up some oauth creds. Be sure to copy the ~/bin that got-your-back creates and ~/.config that rclone creates to the EFS. You’ll want a larger instance, like a t3.large for got-your-back, and after setting this up and testing it, you can terminate that instance.

The state of ABAC on AWS

Mon, 02 Nov 2020 04:00:00 -0800

Two years ago, in November 2018, AWS announced new conditions keys aws:PrincipalTag and aws:RequestTag, and started to push the concept of Attribute Based Access Control (ABAC). This post will describe what this is, the difficulties with implementing this strategy, and what AWS needs to do for customers to be successful with this concept.

What is ABAC?

A long standing problem with AWS security has been that if you had two projects in a single AWS account, it was often impossible to ensure that some principals (meaning the users and roles there) could only interact with the resources of one project and not the other. In order to implement a least privilege strategy, you want to isolate the actions each principal can take to only certain resources to ensure they cannot impact or exfil data from the other project.

The solution many customers have been forced to adopt is to isolate their projects into separate AWS accounts, but that’s not always ideal. For example, it can be difficult to take an existing account and move resources into another account as an account grows. So AWS started focusing on tagging resources and restricting access via tags. Over time, many privileges started to be able to work with the condition key aws:ResourceTag so that you could restrict who could interact with an existing resource. But what if you wanted the principal to create new resources, but restrict what tags they could use, so they couldn’t create a resource with the tag of another project? For this AWS released aws:RequestTag.

What if you had many principals and projects and you didn’t want to create separate IAM policies for each one? You want a single policy that you can apply to all principals that says “Only interact with resources that match the same tag as you have” or the common request of “You can only interact with resources you created.” To implement this concept, AWS released aws:PrincipalTag, so you could now use a conditions such as:

StringEquals: { "aws:RequestTag/project": "${aws:PrincipalTag/project}" }

Attribute-based access control (ABAC) is an authorization strategy that defines permissions based on attributes, which on AWS means tags. Two of the best resources on this concept are Brigid Johnson’s re:Inforce talk Scale Permissions Management in AWS w/ Attribute-Based Access Control and Michael Chan’s blog post Working backward: From IAM policies and principal tags to standardized names and tags for your AWS resources.

Public dataset of Cloudtrail logs from flaws.cloud

Fri, 09 Oct 2020 05:00:00 -0700

In order to advance research into AWS security, I’m releasing anonymized CloudTrail logs from flaws.cloud. I don’t know of any other public datasets of CloudTrail logs and the logs from flaws.cloud are a unique collection, as they are largely attacks within a simple AWS environment. They cover over 3.5 years, and have involved many attackers and types of attacks. This post will describe what these logs represent and how these logs were anonymized. It will describe how to load these into Athena. With there being 240MB of log data, roughly every 10 Athena queries will cost you a penny. The logs are available here.

Denial of Wallet Attacks on AWS

Mon, 08 Jun 2020 05:00:00 -0700

The AWS incidents that make the news are normally data loss incidents (ex. a public S3 bucket), but one of the common ways people find out about a compromise is through their AWS bill, because a common incident that isn’t made public is a compromised AWS key that is used to spin up EC2s to mine bitcoin. That attack is used for the personal gain of the attacker, but it is possible that an attacker just wants to bring hurt to you. Historically, this would have taken the form of a DDoS attack, but in the age of the cloud, that attack can be modified to be a Denial of Wallet attack, where the goal is to cause a high bill such that you run out of money.

When you have servers in a datacenter, and an attacker just wants to bring you hurt, they can DDoS you and your site goes down. When you run in the cloud, an attacker can do things such that your site might stay up, but you’ll be bankrupt. This post will describe this concept, how it can be abused, and how it can be avoided.

This post comes about from this tweet:

As I’m going to be discussing improvements on the attack, mitigations, and other technical issues related to this, I want to start by briefly touching on some of the non-technical issues involved here:

Black lives matter.
I don’t condone illegal or unjust activities, whether that is done by protestors or police officers.
Increasing police department expenses may only be shifting the burden to tax payers.
This tweet could have instead been about an app that allows protestors to report on police somehow that was overwhelmed.

Now onto the technical issues.

Billing alerts

One of the first things you should do on an AWS account is create a billing alert. It is better to find out about a dozen expensive EC2s running on the day they are spun up, so you can take action, rather than 20 days later after they’ve incurred 20 days worth of charges against your account.

If you’re just trying AWS out for the first time and you hear about this “free tier” concept, you might be surprised to discover that nothing stops you doing things that actually cost money, and you won’t know about this until the end of the billing cycle. There is no way to tell AWS “Don’t charge me more than X for the month, and terminate my application if it exceeds that amount.” You however can set alerts for when your estimated charges exceed a threshold. I normally set this to something like $10 for accounts I expect to be free (often you end up with bills for using AWS for something like $0.49, so setting the threshold to $0 causes alerts you aren’t too worried about).

For enterprises with monthly spend in the millions, this can be more difficult, but can still be helpful. If you’re in that category, you should talk to folks like Corey Quinn of the Duckbill Group and Thomas Dullien (aka halvarflake) of optimyze who each focus on different areas of cloud spend improvements. This is not a paid ad. I have a lot of respect for both of those people.

One strategy commonly used is to set multiple alarms that you expect to trigger throughout the month. For example, if you expect your bill to be $1000, then set the following:

Alarms for $250, $500, $750, and $1000 and name them based on the week you expect them to trigger. If your $500 alarm triggers in week 1, you need to investigate.
Alarms for $1500 and $2000 so you know when your expenses have increased dramatically.

For my free tier accounts I have alerts for $10 and $100. When the $10 alert triggers, and I’m not around my computer, I’ll casually make a note to look into it when I get back to a computer. Usually it’s a domain name being renewed for $12. When the $100 triggers, I run to my computer because something bad must be happening.

You can have these alerts not only email you, but also text you. You can set up to 10 alarms for free, after which it is $0.10/mo.

Service Quotas

AWS has a number of hard and soft limits that among other possible reasons, help mitigate run-away code. Many people have stories about infinite loops happening in AWS that caused resources to be created over and over again, or a Lambda to trigger that caused itself to trigger again. This is a common enough problem that the Cloudwatch Event Rule documentation even has a warning about it.