This post explains how to send GuardDuty events, along with Trusted Advisor and CloudTrail events, in real-time from all regions, from all your AWS accounts, to a single region in one account. This uses CloudWatch Events, which is in contrast to the strategy AWS tends to advocate for GuardDuty, where they make use of a Master/Member structure of accounts with invite/accept steps. I’ll explain the architecure I use, show you how to do it, and why I like it.

Description of the architecture

The architecture I use views GuardDuty as a source of alerts, and not as a single pane of glass that some SOC analyst is going to check throughout the day. I assume security teams are working mostly in chat apps like Slack and ticketing systems like JIRA in order to see alerts, with high severity, high signal alerts paging them via text messages or push notifications to their cell phones via something like PagerDuty. I described this in more detail in my post How to write security alerts.

My goal is therefore to get GuardDuty alerts into the log collection pipeline and treated the same as any other log events, and then let the SIEM, in this case the open-source StreamAlert, make the decision for which events to alert on and where to send those.

StreamAlert, in the bottom right, is based around alerts coming into a Kinesis Stream and a Lambda function analyzing them. The diagram shown here for StreamAlert isn’t entirely accurate, but is sufficient for our discussion. StreamAlert also use a Kinesis Firehose in order to record all of the events it sees to an S3 bucket so that these can be searched later using Athena. You can use Splunk or another solution in place of StreamAlert. The main point is to have a single place where events are sent in order to simplify updating your detection rules or making configuration changes for outputs.

GuardDuty alerts, along with Trusted Advisor, CloudTrail, and some other event types, are received by default to the CloudWatch Event bus. You don’t need to do any configuration to make this happen. In the case of GuardDuty, you just need to enable GuardDuty and the alerts it generates end up in CloudWatch Events by default. In the case of Trusted Advisor, you need to have at least a Business support contract (this is an annoying requirement I think AWS should change). For CloudTrail, it just happens, whether or not you enabled CloudTrail logging. It’s up to us though to get those events from the event bus, to somewhere useful, which is done by a CloudWatch Event Rule.

A frustrating limitation of CloudWatch Event Rules is they cannot send events cross-region, although they can send them cross-account. So although the Security account will be somewhat complicated, the other accounts will have an identical setup in every region to send events to the corresponding region in the Security account.

Setting up the non-Security accounts

If we start on the left, for every account, for every region, we’ll need to do the following:

1. Enable GuardDuty.
2. Create a an IAM Role that the CloudWatch role can use that grants it events:PutEvents so the events can be sent to the corresponding region’s Event bus in the Security account. (This only needs to be one done once for the account, not each region).
3. Create an events Rule for the EventPattern {"source":["aws.events", "aws.guardduty", "aws.trustedadvisor"]}
4. Create a target for the Rule to use the newly created IAM role to send the events to the default Event bus in the Security account (arn:aws:events:SECURITY_ACCOUNT_ID:REGION:event-bus/default)

Setting up the Security account

In the Security account, we’ll have one special region, which will be us-east-1 for our needs, where all of the events will be sent to and where StreamAlert will analyze them. We’ll call this the final destination region. We’ll assume StreamAlert has been set up there so its Kinesis Stream exists that all events will be sent to.

In the other regions, we’ll do the following:

1. Enable GuardDuty like we did for the other accounts.
2. Add permissions to the default Event bus to allow the other accounts to send to it.
3. Create a Kinesis Stream, except for in the final destination region.
4. Create an Event Rule to send the events to the from the event bus to the Kinesis Stream.
5. Except for the final destination region, associate a Lambda function with the Kinesis Stream to send the events to the Kinesis Stream in the final destination region.

The policy needed for the Event Bus is:

{
  'Action': 'events:PutEvents',
  'Principal': '*',
  'Condition': {"Type" : "StringEquals", "Key": "aws:PrincipalOrgID", "Value": ORGANIZATION_ID}
}

For the Lambda function, you can use the AWS project aws-lambda-fanout, but beware that you will need to reduce the IAM privileges of the role created by that, as it is badly over-privileged. That code also works as a generic solution, which for our needs results in over-complicating things by making use of DynamoDB and a deployment process that involves more steps than a more specific solution would require.

Why I like this architecture

The architecture for something similar that AWS tends to advocate for is described in this post How to Manage Amazon GuardDuty Security Findings Across Multiple Accounts. In their architecture, they create a Master/Member structure that is part of GuardDuty’s functionality, so that the GuardDuty services in the Member accounts send their alerts directly to the Master (ie. Security) account, and requires an invite/accept process to setup. They then advocate that once you get the alerts to the Master account, that you use CloudWatch Events in order to send the events to a Kinesis Firehose to centralize them for your SIEM. So whereas I have the “member” accounts use CloudWatch Events immediately, this setup waits until they get to the “master” account.

The reason I prefer my approach is because you get the Trusted Advisor and Cloudtrail events for free in this setup. The AWS architecture is still going to have to do what I did separately, for no benefit for what they did with GuardDuty. You want the Trusted Advisor events because those will tell you if an access key was put on Github, if an S3 bucket was made public, and a few other useful alerts. You want the CloudTrail events because these come in at near real-time whereas the logs to S3 take 15 minutes. You should have both the CloudTrail to S3 configuration and the CloudWatch Events collection though, because the CloudTrail events sent to CloudWatch are a subset of those sent to S3 (CloudWatch does not receive List, Get, or Describe* actions from CloudTrail).

With the Master/Member approach, you also need to take actions in two accounts every time a new account is needed, as the Master account needs to send the invite. This can be automated (see aws-samples/amazon-guardduty-multiaccount-scripts), but I just find it unnecessary.

AWS also sheepishly avoids mentioning much about aggregating your alerts to a single region, other than commenting you can use Kinesis Firehose to get the alerts to an S3 bucket for your SIEM to then collect from, which is a slower setup.

As far as I’m aware, the architecture I’ve described will get you alerts with the least amount of delays while also having one of the simplest configurations. There are a number of similar approaches, involving SNS, Kinesis Firehose, and more, but this architecture seems to be the best trade-off.

Some arguments against my architecture include:

• If you really like the GuardDuty UI and want to see events from all accounts there, and you enjoy clicking through multiple regions every time. :)
• Kinesis Streams do require some minimal periodic maintenance to increase their shard count as needed if the number of events they process increases dramatically. They also cost some money to be running in all regions.
• The AWS code for the fanout Lambda is overly complex for our needs and requires tuning the IAM privileges, as was mentioned.
• Edit 2019.03.06 thanks to @mattarnao: This architecture does not allow you to centrally manage threat lists and trusted IPs used by GuardDuty, but I hadn’t heard of anyone using those features.

Using the architecture I described, you’ll have events from important services coming in from all regions and all accounts in near real-time to a central place for them to be analyzed. For more information on why you should be using GuardDuty, see my recent article Should you use GuardDuty? and if you’re interested in having me set this up for you, reach out!