You can ingest Events, Alarms, and CloudTrail data by configuring Amazon Simple Queue Service (SQS) and Simple Notification Service (SNS). Amazon Simple Notification Service (SNS) works with Amazon SQS to provide a powerful messaging solution for building cloud applications that are easy to scale.

Configure Amazon SNS and Amazon SQS

This guide assists in discovering all your AWS resources. Upon completion, make sure to conclude by proceeding to the Collect Metrics section to initiate the collection of metrics from the identified resources.

Step 1: Create an Amazon SNS topic

1. Log into your AWS management console.
2. Create an SNS topic. See Getting started with Amazon SNS for more details.
   The topic Name, ARN, Display name (optional), and Topic owner AWS account ID are displayed in the Details section.

AWS events

If the Amazon CloudWatch rules or Amazon SNS topics are created through automation, add the following code snippet to your policy because this is not included in the default policy. When a CloudWatch rule is created manually, AWS automatically adds this snippet to the SNS policy.

{
   "Sid": "AWSEvents_RULE-NAME_IdRANDOM-NUMBER-STRING",
   "Effect": "Allow",
   "Principal": {
     "Service": "events.amazonaws.com"
   },
   "Action": "sns:Publish",
   "Resource": "SNS-TOPIC-ARN"
 }

Amazon CloudTrail

If the Amazon CloudTrail or SNS topics are created through automation, add the following snippet to your policy because this is not included by default. AWS automatically adds this snippet to the SNS policy when an SNS is added manually to a trail in the AWS console.

{
      "Sid": "AWSCloudTrailSNSPolicyRANDOM-NUMBER-STRING",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "SNS-TOPIC-ARN"
    }

Step 2: Create a queue in Amazon SQS

1. Navigate to Amazon SQS in the AWS console.
2. Create a queue. See creating a queue for more details.
3. Configure an Access policy as defined in the given table according to the service.
4. Amazon SQS creates the queue and displays the queue details page. The queue details such as ARN, URL, and type are displayed in the Details section.
5. Subscribe to the Amazon SNS topic created in step 1.
6. Copy the Amazon SQS URL to the clipboard or a text editor such as Notepad. The Amazon SNS topic and Amazon SQS URL must be unique for Amazon CloudWatch Alarm, CloudWatch Event, and CloudTrail.

Access Policy under the Permissions tab.

Amazon CloudTrail

AWS CloudTrail generates events when there is a change such as launching or termination of an instance. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. You can create resources in the platform by integrating with these CloudTrail event streams. The system supports below methods to ingest and process CloudTrail data:

• SQS-Based: Stream events from an Amazon SQS queue that is subscribed to a CloudTrail log delivery stream.
• CloudTrail Lake: Use AWS CloudTrail Lake to collect and query events in a centralized data store.
• Master Account: Consolidate events from multiple AWS accounts using a master (management) account.

Configure Amazon CloudTrail - SQS-Based

AWS CloudTrail generates events when there is a change such as launching or termination of an instance. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. These events are captured through the CloudTrail SQS URL to create events.

Prerequisites

1. Create an Amazon S3 bucket where all log files can be stored.
2. Create an Amazon SNS topic.
3. Create an Amazon SQS subscription and link it to the Amazon SNS topic.

Steps

1. Log in to your AWS management console.
2. Navigate to Amazon CloudTrail.
3. On the Dashboard, click Create Trail. See AWS documentation on Creating a trail
4. While creating the trail, under Storage location, click Advanced.
5. Set Send SNS notification for every log file delivery to Yes.

1. Select the Amazon SNS topic and click Create Trail.

   The SNS topic should be unique to the trail created. Use the Amazon SQS URL associated with the Amazon SNS topic to configure Amazon CloudTrail while creating or updating the AWS integration in OpsRamp.

Amazon CloudTrail is now configured to send events to OpsRamp.

Configure Amazon CloudTrail - CloudTrail Lake

To capture and query AWS activity logs using CloudTrail Lake, follow these steps to set up an Event Data Store and ensure proper permissions.

Step 1: Create an Event Data Store

1. Sign in to the AWS Management Console and open the CloudTrail console.
2. From the navigation pane, under Lake, choose Event data stores.
3. Choose Create event data store.
4. On the Configure event data store page, in General details, enter a name for the event data store.
5. Specify a retention period for the event data store.

6. Choose Next to configure the event data store.

7. On the Choose events page, choose AWS events, and then choose CloudTrail events.

8. For CloudTrail events, choose at least one event type. By default, Management events is selected.

9. To have your event data store collect events from all accounts in an AWS Organizations organization, select Enable for all accounts in my organization.

   Note

   You must be signed in to the management account or delegated administrator account for the organization to create an event data store that collects events for an organization.

10. To capture events from all regions, opt for the All Regions setting.

11. Choose Next to review your choices.

12. On the Review and create page, review your choices.

13. When you are ready to create the event data store, choose Create event data store.

14. After creating the event data store, open its details page and copy the ARN. You will need this ARN to configure CloudTrail Lake in the AWS integration within OpsRamp.

Step 2: Enable CloudTrail Lake Across AWS Organizations (Optional)

• If you are using AWS Organizations, enable CloudTrail Lake at the organization level to automatically include events from all linked member accounts.
• For standalone AWS accounts, repeat the setup steps above on each account individually to enable CloudTrail Lake.

Step 3: IAM Permissions for Query Access

Ensure that the AWS Identity and Access Management (IAM) role or user used by your application has the necessary permissions to query CloudTrail Lake. Required permissions may include:

{
  "Effect": "Allow",
  "Action": [
    "cloudtrail:StartQuery",
    "cloudtrail:GetQueryResults",
    "cloudtrail:LookupEvents",
    "lakeformation:GetDataAccess"
  ],
  "Resource": "*"
}

Configure Amazon CloudTrail - Master Account

In environments where direct access to configure CloudTrail Lake across all individual AWS accounts is not available, you can designate one account as the Master Account to centralize event collection.

In this setup, CloudTrail Lake is configured only in a single AWS account (referred to as the Master Account), and all required CloudTrail events are streamed into this account. This eliminates the need to individually configure CloudTrail Lake in each account.

In the AWS Integration page, select the Master Account option and choose the configured master account from the dropdown list.

Benefits of the New CloudTrail Lake Integration Approach

The new CloudTrail Lake-based integration offers several advantages over the existing SQS-based method, particularly in multi-region and multi-account environments.

• Multi-Region Event Collection:
  • Unlike the existing SQS-based approach, which is limited to a single region, the CloudTrail Lake method enables comprehensive multi-region event aggregation—improving visibility across your entire AWS footprint.
• Simplified Setup with CloudFormation:
  • To streamline onboarding, a ready-to-use CloudFormation template will be provided:
    • Automatically sets up CloudTrail Lake.
    • Generates the required event datastore ARN.
    • Reduces manual configuration steps.
• Flexible Account Support:
  • Single Account Setup: Simply provide the generated datastore ARN to integrate CloudTrail Lake.
  • Multi-Account (Master-Child) Setup:
    • Set up CloudTrail Lake in the Master Account.
    • Install the Master Account in the OpsRamp platform.
    • Enable CloudTrail in Child Accounts without additional CloudTrail Lake configuration.

Configure Amazon CloudWatch alarms

Amazon CloudWatch monitors your AWS services and the applications you run on AWS in real-time. You can use CloudWatch Alarms to collect and track metrics. You can define alarms to send notifications or automatically make changes to the resources you are monitoring when a threshold is breached.

Prerequisites

• Create a unique SNS topic to CloudWatch Alarm.
• Create a unique SQS subscription and link to the SNS topic.

Steps

1. Log in to your AWS management console.
2. Navigate to Amazon CloudWatch.
3. Click Create Alarm. To set up CloudWatch alarm, see How to create a CloudWatch Alarm Based on a Static Threshold.
4. Specify the metric conditions and click Next.
5. From the Configure Actions page, select the In Alarm (The metric or expression is outside of the defined threshold) option.
6. Select an Amazon SNS topic created earlier to send notification for CloudWatch alarms.
7. Click Add notification.
8. Select the OK (The metric or expression is within the defined threshold) option.
9. Select an SNS topic and click Next.

Choose the same SNS topic for both the options of notifications.

1. Enter a unique name for the alarm and a description (optional).
2. Click Next. A preview page opens displaying the configuration details of the alarm.
3. Click Create alarm.

Amazon CloudWatch Alarm is now configured to send notifications to OpsRamp.