This is part one of a two part series that will discuss AWS’s new Lightsail object storage. In this first part, we’ll look at the new access key capability and a security issue I discovered that has been fixed. In the second part we’ll look more closely at the buckets created.

What is Lightsail?

In 2016, AWS released Lightsail as a way of providing a simpler version of AWS in response to DigitalOcean. Instead of having to worry about configuring your networks and paying for bandwidth and all the other little details of setting up and running an EC2, Lightsail advertised an easier way for a flat rate of $5/mo. Over the years, Lightsail has expanded to incorporate more functionality than just a simpler EC2, and this service within AWS is now seen as being a mini-AWS within AWS that expands to achieve feature parity of where AWS was a number of years ago.

Here’s a table comparing the Lightsail features to AWS services and their respective launch dates.

Trying out Lightsail object storage

On July 14, AWS announced Lightsail object storage, a capability that sounds like S3. The SDK commit mentions a new API CreateBucketAccessKey which seems like a wild circumvention of IAM credentials. This looks exciting.

I decided to start from the web console for Lightsail, which does not conform to any of the standards of the rest of the console.

I went over to create a bucket, and quite clearly you can see by the domain name, that this is an abstraction layer over S3.

I created my bucket and quickly jumped to the S3 service console, but did not see any trace of my Lightsail bucket! Sometimes one AWS service will provide an abstraction over another AWS service and resources will appear in your account for that other service, but this must be created in a different AWS account. We’ll look more at that bucket in part 2.

Getting an access key

From the Lightsail console I created an access key, which is said to be used only for Lightsail bucket access. The first thing I did was figure out what account that key belongs to:

$ aws sts get-access-key-info --access-key-id AKIA4L53X6C6KXW6KH6W
{
    "Account": "850260390076"
}

That account ID is not mine, so this access key was created in a special Lightsail account. Let’s go ahead and use it and find out more information.

$ aws sts get-caller-identity
{
    "UserId": "AIDA4L53X6C6AEGC7HIQA",
    "Account": "850260390076",
    "Arn": "arn:aws:iam::850260390076:user/bucket-kgxqi8.obj-mgmt"
}

So the username matches the bucket name, plus the .obj-mgmt suffix, which is interesting as that makes any IAM policies for this harder for AWS to write for restricting it to the same name as the IAM user.

This access key is used for S3 bucket access, so let’s try listing the buckets.

$ aws s3 ls
2021-07-23 11:13:23 bucket-kgxqi8
2018-12-26 00:57:12 cloudtrail-logs-850260390076-amazon-lightsail

Well now that is odd. I didn’t expect to be able to list the buckets in the account, and also didn’t expect there would already be another bucket there. I’m unable to list the objects in that cloudtrail S3 bucket, and that creation date is a few years old even though this is the first time I believe I’ve used lightsail in this account. That date is also not the same as the creation date of my account.

I wasn’t able to do much else with the key, beyond the expected use case of interacting with my bucket, and not in any ways I found interesting.

Back in my original account, I was curious to play with these new Lightsail APIs. You can see here that in order to find the Lightsail bucket access keys, you have to provide the Lightsail bucket name, so you have to iterate through all Lightsail buckets in the account to find possible credentials. For those that want to ensure they have no long-lived credentials in their accounts, this makes it slightly more annoying to check.

$ aws lightsail get-bucket-access-keys --bucket-name bucket-kgxqi8
{
    "accessKeys": [
        {
            "accessKeyId": "AKIA4L53X6C6KXW6KH6W",
            "status": "Active",
            "createdAt": "2021-07-23T12:48:56-06:00"
        }
    ]
}

Why this access key is a problem

A best practice on AWS is to avoid using long-lived IAM user access keys, so the creation of these seems like a step backward. Another best practice is applying a least privileges strategy, and specifically with S3 buckets, you should avoid granting the ability to list buckets. These keys allow listing the lightsail buckets and you cannot restrict this.

This key lives in another account and you do not have the same privileges over this key as you would if it was in your own account. Specifically, you cannot get information about when this key was last used, so you you cannot identify unused access keys, which is another best practice to identify and revoke those. Update 2021.08.08: AWS added the ability to see the last used time on these keys.

In the event of a credential compromise where you wanted to revoke that access key, you could revoke the key, but you could not apply an IAM policy in order to revoke any sessions, and because aws sts get-session-token is always possible with an access key, an attacker would be able to create their own sessions. This makes AWS’s recommendations for incident response impossible to perform.

The security issue: Cloudtrail logs

The above issues make me dislike this feature, but they don’t meet the bar of being a security issue that AWS needs to fix. So this brings us to an actual security issue with these that I reported and AWS fixed.

After playing with these for a bit, I looked at the Cloudtrail logs for the activities in my account, and everything aligned with my expectations, except when I looked at the call to CreateBucketAccessKey. This was in the responseElements

"accessKey": {
    "secretAccessKey": "gKk4ZzUFwWspe8Gx+x5iRfbDLfq7/tHH3Olss62T",
    "status": "Active",
    "accessKeyId": "HIDDEN_DUE_TO_SECURITY_REASONS",
    "createdAt": "Jul 23, 2021 6:48:56 PM"
}

They redacted the wrong value! They are supposed to redact the secretAccessKey (that’s why the name includes the word “secret”), not the access key id. This is a problem because Cloudtrail logs are not supposed to contain secrets.

I immediately reported this to AWS, which unfortunately was at 8pm Mountain time on a Friday on July 23. Two hours later they emailed me an acknowledgement of the issue, and on Saturday afternoon they had a call with me to let me know a fix was being deployed and by Monday it had been fixed globally. I really appreciate how quickly the AWS security team responds to issues and is able to get the relevant teams to produce fixes. This issue would have required an attacker to have access to the Cloudtrail logs and be able to find out the access key id, so personally I would have ranked this as low severity and not something the team needed to work late on or over the weekend, but I was glad to see how seriously AWS took it.

Unfortunately, the fix for this problem was to set the responseElements to null, which means you can no longer see information about which access keys were created. This means that if you do happen to use Lightsail access keys in your account, and you have a compromise where the attacker creates new Lightsail access keys, you will not be able to easily figure out which ones they created. You will need to identify these CreateBucketAccessKey calls and then look at the creation dates of the Lightsail access keys to identify which ones were created during the incident so you can revoke them.

Conclusion

Lightsail is a simplified version of AWS, and in order to simplify it, AWS has ignored some security best practices, such as not having long-lived access keys and issues that will be discussed in part 2. This service is directionally not aligned with the best practices advocated by the rest of AWS. Although Lightsail serves a certain type of customer, for most businesses, my recommendation is that you should apply an SCP over your organization to deny use of the Lightsail service, ideally by crreating an allow list of allowed services as described here.