Alert Management

Monitor-generated events indicate the presence of device or resource anomalies. Alerting helps you interpret and act on events by providing a single aggregation and response system for all events. Events can also be produced by diagnostic and third-party tools.

Much of the work of interpreting and responding to an alert is automated. Alerting correlates related-cause alerts, automatically suppresses redundant alerts, notifies operators, and creates incident tickets for alerts that need attention.

The following figure shows the alert handling workflow:

Note: The information alerts do not activate actions within our alert policies. The information alerts are considered to provide information and are distinct from regular alerts.

Alert terminology

The following terms are used in alert management:

Alert types

There are two alert types:

• Forecast alerts
• Change detection alerts

Forecast alerts

When a monitor triggers a forecast alert, the alert is displayed on the Alerts page. The following example shows the forecast alert display:

The details and updates on a forecast alert are displayed in the comments section:

Change detection alerts

Change detection helps monitor sudden changes in metric behavior, especially on metrics with an indefinite threshold. This feature utilizes machine learning with a sliding window where change is calculated as soon as a data point becomes available.

Assign the change detection parameters to the required monitors/metrics to detect the changes and trigger alerts when signification change is detected.

Key considerations include:

• If a significant change is detected continuously, an alert is appended to an existing alert.
• If no change is detected after two hours, an alert is automatically resolved.
• A minimum of four hours of real-time data is required to process change detection. Hence, change detection graphs/widgets will be populated only after 4 hours after assigning change detection monitor to a resource.
• When a monitor triggers a change detection alert, the alert is displayed on the Alerts page.

A sliding window of two hours is used to calculate the change score percentile and determine if an alert should be triggered.

The following example shows the forecast alert display on the Alerts page:

Change detection is summarized on the Alert Details page:

Alert lifecycle

The alert lifecycle describes alert status transitions, from Open status to Closed status, as a result of actions applied to the alert.

Alert action

The following actions can be applied to an alert:

For correlated alerts, an action can be performed on the entire inference, but not on a single alert.

Alert status

Alert status describes a logical condition of an alert within the alert lifecycle. Alert status should not be confused with alert state, which can be critical, warning, or OK.

Both automatic and manual alert actions can cause an alert status change, as shown in the following figure:

You can monitor alert status in the Alert Details page comments section:

After waiting until the problem no longer is displayed as an alert the alert is placed in the OK state. Alerts in the OK state are not displayed in the alert browser. In the OK state, if the same alert reoccurs, a new alert is created. Otherwise, the repeat count is incremented for the alert.

Alert filters

The following filters can be applied to alerts:

Native attributes filter criteria

Below are the description of the native attributes that can be applied to alerts in alert browser criteria and Alert Management Policies filter criteria.

Alert noise

Alert noise or alert floods might make it difficult to recognize and deal with more important alerts. The platform can reduce alert noise in the following ways:

• Automatically ignore repeat alerts
• Manually stop processing alerts temporarily

Automatically ignore repeat alerts

When the platform receives more than one alert with the same resource, metric, component, and state combination within one minute, the platform ignores the alerts after processing the first alert.
Ignoring the repeat alerts conservers resources. Outgoing email can be blocked if the platform continues to process the noise. When there are no repeat alerts with the same resource, metric, component, and state combination within a minute, the platform resumes normal alert processing.

Manually stop processing alerts temporarily

When the platform receives an alert flood from a resource or metric, the operations team stops the processing of the resource or metric until they ensure the flooding has stopped. For example, flooding occurs when you get 100s of alerts within a few minutes from a single metric on a resource or a resource sends 100s of alerts within a few minutes. This flood of alerts affects the processing of other alerts for the client, which might be important alerts. Also, this flood of alerts from one client might affect alert processing for other clients as well because of the load generated by these noisy alerts. When this flooding occurs, the operations team changes the server configuration to stop processing these alerts and informs the corresponding client of the issue. After the issue is resolved, the operations team changes the configuration back to normal to process all alerts.

Alert repeat and occurrence history

Older alert occurrences are not always available because they are purged after 180 days.

• On the Alerts page, the link in the Repeat column does not show the number of alerts indicated when the occurrences are older than 180 days.
• On the Alert Details page, the Total Occurrences link does not show the number of alerts indicated when the occurrences are older than 180 days.

Machine learning concepts

Machine learning (ML) status shows the stages of machine learning implemented in a policy used with analyzing a sequence to suppressing alerts:

ML Status	Description
	Insufficient data. The policy is temporarily disabled. Due to insufficient data, the machine learning model cannot detect any alert sequences and the policy is temporarily disabled. The policy becomes active when the machine learning model has sufficient data.
	Training ML model is queued. When a policy is created or a CSV file is uploaded to a policy, the training can be queued. If a policy is in training, the new policy is queued. After the training on the existing policy is completed, the status of the new policy moves to training initiated.
	Training ML model is initiated. After completion, training on the machine learning model is initiated. The status then moves to training started.
	Training ML model is started. After completion, training on the machine learning model is started. Training progress is shown on the progress bar.
	Training ML model is in progress. After completion, training on the ML model continues with the percentage of progress shown in the progress bar.
	ML model training is complete. Predictions will commence and the ML model detects alert sequences and suppression.
	ML training encountered an error. Contact support for assistance.

ML seasonality patterns

Alert seasonality patterns help detect seasonal patterns of alerts. These patterns are based On unmodified alert information retrieved from existing data. OpsQ suppresses an alert that matches with the seasonality pattern detected by ML. For example, if an alert is displayed at 10 PM on Mondays, OpsQ suppresses the alert triggered at 10 PM on Mondays.

ML retrieves three months of alerts and groups alerts based on three attributes (resource, metric, and component). Seasonal patterns of alerts are studied for each group. Alert groups that have seasonal patterns are listed and displayed when a group is selected.

Ninety percent of the alerts must be seasonal to qualify as seasonal alerts.

Seasonality pattern graphs

Seasonality pattern graphs have the following attributes:

Seasonality groups in the graph refer to alert groups that have seasonal patterns.

The following graph provides the details of the seasonal patterns for alert data and describes when OpsQ suppresses alerts due to matching seasonal patterns:

This is a sample recommendation by the OpsQ bot for a suppressed seasonal alert:

The following graph shows the seasonal pattern of a suppressed alert: