In December, 2018, AWS added code to the SDK for a feature called Client Side Monitoring (CSM). According to the docs, this feature is “available only to customers with an Enterprise Support subscription.” This article will discuss what this is and how we can use aspects of it without a fancy support contract.

Trying out CSM

If an application that uses the AWS SDK has the environment variable AWS_CSM_ENABLED set, or the ~/.aws/config for the profile includes csm_enabled = true, then the application will send info locally to the CloudWatch Agent (or something else) listening on localhost on UDP port 31000.

We can try this out locally by running a netcat listener in one window:

nc -kluvw 0 localhost 31000

In another window run:

export AWS_CSM_ENABLED=true
aws s3 ls s3://flaws.cloud

In your netcat window, you should then see the following (text formatted for clarity):

{
  "Version": 1,
  "ClientId": "",
  "Type": "ApiCallAttempt",
  "Service": "S3",
  "Api": "ListObjectsV2",
  "Timestamp": 1590430065174,
  "AttemptLatency": 243,
  "Fqdn": "s3.amazonaws.com",
  "UserAgent": "aws-cli/1.18.51 Python/3.7.6 Darwin/19.4.0 botocore/1.16.1",
  "AccessKey": "ASIAXXXXXXXXXXXXXXXX",  
  "Region": "us-east-1",
  "SessionToken": "XXX",
  "HttpStatusCode": 301,
  "XAmzRequestId": "0000000000000000",
  "XAmzId2": "XXX",
  "AwsException": "PermanentRedirect",
  "AwsExceptionMessage": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."
}
{
  "Version": 1,
  "ClientId": "",
  "Type": "ApiCallAttempt",
  "Service": "S3",
  "Api": "ListObjectsV2",
  "Timestamp": 1590430065418,
  "AttemptLatency": 249,
  "Fqdn": "s3.us-west-2.amazonaws.com",
  "UserAgent": "aws-cli/1.18.51 Python/3.7.6 Darwin/19.4.0 botocore/1.16.1",
  "AccessKey": "ASIAXXXXXXXXXXXXXXXX",
  "Region": "us-west-2",
  "SessionToken": "XXX",
  "HttpStatusCode": 200,
  "XAmzRequestId": "0000000000000001",
  "XAmzId2": "XXX"
}

We can see a call to ListObjectsV2 was made in one region, and then redirected to the region containing the bucket, resulting in two calls.

Additional configuration

In July, 2019 this was extended to allow an environment variable called AWS_CSM_HOST to send this data to any host. You can also set AWS_CSM_PORT to change the port from the default 31000, and AWS_CSM_CLIENT_ID to set an identifier to differentiate multiple processes you might have configured to send to the same listener.

The golang SDK provides more documentation on how to use this feature from your code, for example to start and stop monitoring programmatically.

Use case

You could get this same information by wrapping the SDK somehow, increasing debug logging, hooking API calls, or doing network monitoring. Much of it also shows up in CloudTrail logs. However, CSM is potentially an interesting feature due to the ease that it can be enabled (no recompiling or source code access needed), and the focus on AWS API calls (so you’re not digging through a network dump of other traffic). In comparison to CloudTrail logs, this records every AWS API call, such as the object level ListObjectsV2 which is not recorded by CloudTrail by default, and records them in real-time. It is however more limited than CloudTrail logs in that it does not record any of the parameters (for example in the above logs you don’t know what bucket the list call was made for).

Using this, you could potentially have as part of your unit testing, a record of every AWS call made, which could let you ensure that the IAM role of the application was restricted to only those privileges that you had unit tests for. You might also use this for monitoring your application to detect errors or latency problems without having to make code changes to do that.