Kloudfuse Managed Alerts
- 1 Creating an alert
- 2 Contact points for notifying your teams
- 3 Manage alerts
- 4 Functions in alert queries
- 5 Advance Auto-Alerting
- 6 Contact points settings
- 6.1 Setting email notifications
- 6.2 Setting Notifications to PagerDuty
- 6.3 Getting the slack webhook URL
- 6.4 Setting webhook contact point integration
- 6.5 Setting Flock contact point integration
- 6.6 Setting up Microsoft Teams contact point
- 6.7 Setting up OpsGenie contact point
- 6.8 Setting up Google Chat contact point
- 7 Kloudfuse Provided Out of the box control plane alerts
Kloudfuse-managed alerting aims to provide an opinionated view of how alerting framework should be. It allows for, what we believe are, more intuitive workflows that are easy to follow and less cluttered.
Using the Kloudfuse platform you can configure alerts, notify your teams, and manage alerts at a glance.
Note: Currently the Kloudfuse platform supports threshold-based category alerts on Metrics and Logs type of telemetry data. Please contact us to know about upcoming changes to other alert categories (automatic tracking changes in behavior or using explicit anomaly algorithms) and alert types (APM & Events)
Creating an alert
To create a monitor in the Kloudfuse platform follow these steps:
Navigate from the main page to Alerts > Rules and then click “Create New Alert”
Select an alert type corresponding to the kind of telemetry you want to alert on. Read further for each alert type:
Contact points for notifying your teams
Contact points are used to define how your contacts are notified when an alert fires. The Kloudfuse platform supports the following types of contact points. The Kloudfuse platform supports Grafana message templates to customize notification messages for the contact point types.
Create contact point
Click “Contact points” to open the page listing existing contact points.
Click “Add New contact point”.
Add a descriptive name for the contact point In the “Name” field.
Use the “Choose contact point type” drop-down to select a type of contact point and fill out mandatory fields.
Some contact point types, like email or webhook, have optional settings. So fill them as necessary.
See contact point specific settings here.
Optionally test that the contact point is functioning appropriately using the “Test” button.
“Save contact point” to save your changes.
Edit contact point
To edit a contact point click “Contact points” to open the page listing existing contact points. Find the contact point to edit, then click the pen icon to edit the contact point. Make necessary changes, optionally test, and click “Save contact point”.
Delete contact point
To delete a contact point click “Contact points” to open the page listing existing contact points. Find the contact point to delete, then click the trash icon to delete the contact point. Confirm the choice when a window pops up.
Dealing with multiple contact points
In cases when multiple contact points are used within a single alert, the default one to one mapping provided by Kloudfuse UI may not work. Some additional configuration may be required based on your specific requirements. Please make sure to review these steps. here to understand how to structure the notification flags.
Example 1. Notification to be triggered to more than one contact points for a single alert.
Make sure that the contact points within a group allow matching other sibling policies for notification to be attempted to other policies and related contact points. In this example, for the alert a grafana email and opsgenie contact points are enabled. The notification policy for grafana contact point ensures that rule is further matched with sibling policies. See following.
Contact point types
The following list of contact points is currently supported. This list will be enhanced as support for additional contact points is added.
Email
PagerDuty
Slack
Webhook
Microsoft Teams
OpsGenie
Google Hangouts
Manage alerts
The Alerts > Rules page lists the installed alerts in the system. A list of alerts can be searched using their state, labels, title, etc., intuitively on this page. Each alert can be further investigated by clicking on that alert from the list where the alert’s properties, current evaluation graph, and history are displayed.
Functions in alert queries
PromQL functions
Kloudfuse platform underlying query is always a PromQL query. These queries can be further extended using all https://prometheus.io/docs/prometheus/latest/querying/functions/ .
Kloudfuse Advance Functions/Algorithms
In addition to supporting full promql for alerting, Kloudfuse platform has some additional advance functions that can be used for detecting anomaly, outliers, etc. These functions are made available directly through detection methods in each stream. For example metric streams provides these detection methods.
Advance Auto-Alerting
Kloudfuse platform provides the ability to do alerting on all service endpoints of your services out of the box with a simple configuration update to enable. Enabling this feature results in the Kloudfuse platform starting to monitor the communication between various endpoints without requiring any instrumentation to your applications. On the collected data, it evaluates anomalies in streaming fashion using state-of-the-art statistical learning algorithms. Please see the advanced analytics section for more details on this.
Contact points settings
Setting email notifications
Grafana lets user send email notification on alerts if configured. Grafana can be configured in more than one ways, but here we’ll give example of configuring grafana with gmail based notifications. (Other methods of authentication are weaker and therefore not listed here. This is the recommended way)
Firstly, grab an email address that you will be using to send the notifications. we’ll use grafana_alerts@domain.com.
create an app password in gmail. You will need to be using 2FA for the account (grafana_alerts@domain.com) to be able to create app password. https://support.google.com/mail/answer/185833?hl=en . Note down the app password as you will need it in step 3.
make sure you are connected to the cluster where kloudfuse stack is installed and you are in kfuse namespace
# connect to your cluster kubectx <cluster-name> kubens kfuse
create a kubernetes secret with the username and password you created in step 1.
kubectl create secret generic grafana-smtp-user-password --from-literal=user=grafana_alerts@domain.com --from-literal=password=<generated-app-password>
edit the values.yaml to uncomment settings related to smtp in grafana section (to look like the snippet below). Update following settings:
update host to your smtp mail server
update from_address to the smtp user you want to use
update from_name if needed
grafana: grafana: # grafana.ini - Grafana server configuration settings grafana.ini: ... # start -- Uncomment the following to enable smtp smtp: enabled: true host: your_smtp_hostname_colon_port skip_verify: true from_address: your_smtp_user@domain.com from_name: AlertsAdmin envValueFrom: GF_SMTP_USER: secretKeyRef: name: grafana-smtp-user-password key: user GF_SMTP_PASSWORD: secretKeyRef: name: grafana-smtp-user-password key: password # Uncomment the following to enable smtp -- end
issue the same kfuse helm install command which you used to install kfuse cluster again.
Please make sure to update the default email address in grafana-default-email otherwise
Setting Notifications to PagerDuty
Setting Kloudfuse (and grafana) alert notifications to PagerDuty is done in 2 simple steps:
Obtaining service integration key from PagerDuty
If you already have an existing service (using grafana-incoming-incidents service in the following example), then use following steps to get the integration key.
Otherwise, create a new service (test-incoming-notifications) and use that service’s integration key
Use the integration key obtained above in Kloudfuse platform by choosing “PagerDuty” as the contact point type.
Getting the slack webhook URL
Use a Slack webhook to create a contact point in slack.
Follow the link below to create a slack webhook and get its url- https://api.slack.com/messaging/webhooks
One the webhook is created, enter its URL in the webhook field while creating the contact point.
Use the Optional Slack Settings to mention specific user/group or send alerts to entire channel.
Setting webhook contact point integration
Add the webhook URL in the URL (mandatory) field.
Add any additional settings as necessary.
Setting Flock contact point integration
Setting flock integration with Kloudfuse alerting is same as setting up webhook based integration. However, the webhook url needs to be obtained from your flock app and some configuration at flock end is required.
Get a webhook url by creating an “incoming webhooks” from flock and pay attention to the transformations. Please add following transformation configuration in the incoming webhook you create:
Use the URL from above step (it would look like: https://api.flock.com/hooks/sendMessage/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx) to create a webhook contact point as described here.
Setting up Microsoft Teams contact point
Setting Microsoft Teams integration with Kloudfuse alerting is similar to setting up a webhook based integration.
First head to Microsoft Teams and create a channel for alert notifications. Using
test-notifications
channel name in this example, and create a connection of type “incoming webhook” as shown below. Copy the created “URL” (example:https://kloudfuse.webhook.office.com/webhookb2/257d29a4-xxx
)Go to the Kloudfuse->Alerts->Contact Points->Add New Contact Point. Choose Microsoft Teams from the drop down menu. Use url from above in the “URL” field and save. (Test will send a test notification).
Setting up OpsGenie contact point
Navigate to Grafana tab in the Kloudfuse UI.
Create a OpsGenie-Grafana integration with steps in the https://support.atlassian.com/opsgenie/docs/integrate-opsgenie-with-grafana/
After completing the steps, navigate to Alerts → Contact Points on Kloudfuse UI.
Choose Create Contact Point and fill the required details
Now use the contact point from Kloudfuse UI to any of the alert.
Setting up Google Chat contact point
Create a new Google Workspace space for alerting or you can use an existing space.
Create an incoming webhook for space
Navigate to Alerts tab in the Kloudfuse UI. Select Contact Point and Click on New Contact Point.
Choose Create Contact Point and fill the required details
Now use the contact point from Kloudfuse UI to any of the alert.
Kloudfuse Provided Out of the box control plane alerts
Kloudfuse provides a number of out of the box alerts for getting the stats for data plane these alerts thresholds or other parameters can be updated as per each deployment. These alerts are part of kfuse-cp
folder in alerts. Following are the default thresholds for these alerts.
Type | Check | Alert Condition |
---|---|---|
Kubernetes Pods | In Failed state | For 5 mins |
Restarting multiple times | For 5 mins | |
CrashLoopBackOff | For 5 mins | |
Deployments | Lesser replicas than desired | For 15 mins |
Statefulsets | Lesser replicas than desired | For 15 mins |
Nodes | Unschedulable | For 10 mins |
Not Ready | For 5 mins | |
High CPU Usage | > 90% for 5 mins | |
Disk Usage | > 90% for 5 mins | |
Data Lake (pinot) | Segments in error condition | > 0 for 5 mins |
Segment creation threshold breached | > 10 mins | |
Persistent Volumes | Current Usage | > 90% |
Forecast Usage | Notify when it will run out of space | |
Agent/Collector | Not sending data | For 5 mins |
These alerts do not have any default contact point associated with them. The contact point for these alerts need to be updated as per each deployment requirement.