Threshold Alert
A change alert evaluates the difference between a value N minutes ago and now. On each alert evaluation Datadog will calculate the raw difference (not absolute value) between the series now and N minutes ago then compute the average/minimum/maximum/sum over the selected period. An alert is triggered when this computed series crosses the threshold
Step 1. Choose Threshold detection method
Choose the detection type Threshold Alert
Step 2. Define the search query
Add any required log filters in the “Log Filters” search box
Log count based charting
Use the rate or count_over_time aggregate to chart the count of log lines based on the log filter
Log Facet based charting
Select a log facet to extract and chart. For instance, select a duration facet “took”.
Apply a normalization function like 'duration' to interpret the duration string as seconds
Choose a aggregation function to aggregate log events in time and generate a time-series
Add any grouping facets to “Group by” section. This can reduce the number of time-series
For more details on Log derived metrics, see Log Derived Metrics section in Logs overview
Step 3: Populate condition and Evaluation
Populate the Condition section by defining the:
aggregate to be used on the query result from the drop-down.
query or expression from the drop-down
thresholds that should be breached for the alert to be firing
Populate the Evaluation section by defining the:
evaluation frequency that determines the frequency at which alert expression/query must be evaluated (must be a multiple of 10 seconds. For example, 1m, 30s, etc.) and
specify the duration for which the condition must be true before an alert fires
(Note: Once a condition is breached, the alert goes into the “Pending” state. If the condition remains breached for the duration specified in “For”, the alert transitions to the “Firing” state, otherwise it reverts to the “Normal” state)
Step 4: Populate Name and Title details
Choose the folder to which the alert definition should be saved. (If you need to create a separate folder, then create one using the “new folder” option in the drop-down menu).
Rule Name: set a descriptive name for the rule.
Group Name: Specify a group name. Rules within a group are run sequentially at regular intervals, with the same evaluation time.
Populate title and summary with variables to include additional information in the alert.
Step 5: Configure a contact point
Choose how notifications are sent to your teams (email, Slack, PagerDuty, etc). Choose an existing contact point from the drop-down menu for notifications when this alert fires, or create a new one. To configure a new contact point, please see details for each type of contact point in this section. Once done, click “Create Rule”
