This article discusses the different options for logging AWS API calls. In November, 2013, AWS CloudTrail was announced to record API calls to an S3 bucket. It’s important to realize that not all APIs are recorded. As discussed in AWS IAM vs API vs CloudTrail, some API calls aren’t recorded anywhere. Specifically “data” level APIs, such as S3 object access, unless you turn on additional flags, but even then, there are still APIs that aren’t recorded. Some of these are due to being “data” level calls, but also AWS just doesn’t have a baseline set of requirements for new services, so sometimes a new service won’t log to CloudTrail for months after its release. A recent example is Amazon Transcribe, which was made generally available on April 4, 2018, not getting CloudTrail support until June 28, 2018.

CloudWatch Events

One frustration many have with CloudTrail is that the logs are delayed by 15 minutes, so you can’t respond immediately to actions. AWS resolved this by announcing CloudWatch Events in January, 2016, which are real-time logs of actions. One example use case for this is you can immediately change the policy on an S3 bucket if it has been made public, thereby setting it back to private again. A major limitation for CloudWatch Events though is that only actions that result in changes are recorded, so List and Describe calls are not recorded. Using these logs is somewhat trickier, but integrates with real-time log services better. To get access to these logs you have to set up a CloudWatch Rule to send these events somewhere, such as a Kinesis Stream.

CloudTrail Event History

Two problems with both CloudTrail and CloudWatch Events are that you have to turn these features on and an attacker could turn them off. So AWS announced CloudTrail Event History in August, 2017. This originally recorded only 7 days of logs, but in December, 2017, this was increased to 90 days. It also only recorded actions that caused changes, like CloudWatch Events, but as of June, 2018 this service additionally records read calls. The main drawback of this service is the API access is clunkier or non-existant. You have to query AWS for events, but you can only query once per second, and you can only retrieve 50 events per request. However, you can download all events from the web UI as a bulk file as will be explained later.

Comparison

This table summarizes the above information

CloudTrail Event History access

The only time you’ll likely use CloudTrail Event History is after an incident where CloudTrail logs had either not been turned on, or had been disabled.

In the web console, you’re presented with a way of querying the data.

It has filters for different attributes and for the time ranges.

You can click on events and see the json logs that are the same as what you’d see in a normal CloudTrail log.

The most important capability in the web view is the ability to bulk download the logs.

There is no mention of the limits this has, and given that the downloaded file is uncompressed and that the some accounts might have millons of events per month, this could be a very large download. The download will take your filters into account though, so that can help reduce the download size. Unfortunately, there is no API call to download these, so you have to perform the download through the web UI.

You can make API calls for these events via LookupEvents, but there is no bulk download, and you are limited to one request per second with 50 results max returned.

Here is an example call:

aws cloudtrail lookup-events --start-time 1533600000 --end-time 1533700000

Returns:

{
  "Events": [
      {
            "EventId": "4a147606-f960-4def-a99a-680205f1c6b9",
            "Username": "summitroute",
            "EventTime": 1533660617.0,
            "CloudTrailEvent": "{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAJI5HFRR3SFKANJZ5C\",\"arn\":\"arn:aws:iam::954574370272:user/summitroute\",\"accountId\":\"954574370272\",\"userName\":\"summitroute\"},\"eventTime\":\"2018-08-07T16:50:17Z\",\"eventSource\":\"signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"65.1.1.1\",\"userAgent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 Safari/537.36\",\"requestParameters\":null,\"responseElements\":{\"ConsoleLogin\":\"Success\"},\"additionalEventData\":{\"LoginTo\":\"https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true\",\"MobileVersion\":\"No\",\"MFAUsed\":\"Yes\"},\"eventID\":\"4a147606-f960-4def-a99a-680205f1c6b9\",\"eventType\":\"AwsConsoleSignIn\",\"recipientAccountId\":\"954574370272\"}",
            "EventName": "ConsoleLogin",
            "EventSource": "signin.amazonaws.com",
            "Resources": []
        },
...

Extracting the CloudTrailEvent field gives the usual CloudTrail log data.

Conclusion

So which service should you use to record API calls? The answer is all three.

You should turn on CloudTrail to log to an S3 bucket in a separate Security account, and an additional trail possibly to log events to an S3 bucket within the account. The logs sent to the Security bucket can have permissions set on the bucket so they cannot be modified or deleted. These logs will primarily be used for debugging or incident response, and additionally alerting, but with a 15 minute delay.

You should also turn on CloudWatch Events and have those sent to a Security account where they will be used for alerting. The reason you’ll alert off of both CloudTrail and CloudWatch Events is because CloudWatch Events do not capture all of the events that CloudTrail does, but the ones that do come through are real-time.

Finally, if you end up with an incident where CloudTrail logs were not enabled or were disabled, you can use CloudTrail Event History. It can also be used for debugging simple issues.