Monitoring all of your infrastructures in one place would be incomplete without the ability to know when critical changes are occurring. The Kloudfuse platform gives you the ability to create alerts that can actively check for configured invariants in data coming from metrics, logs & traces.
The Kloudfuse platform has taken a flexible, yet simple and powerful approach to alerting. It keeps net new users who are new to any alerting platform, users who are used to vendor platforms, and users who are used to Grafana alerting in mind at the same time. To that effect following options are available.
Migrating alerts from vendor platforms to open telemetry-compatible (PromQL) platforms (such as Kloudfuse)
Grafana alerting is embedded within Kloudfuse UI for users who want to use Grafana alerting with a Prometheus-compatible data source for metrics & logs.
Kloudfuse-managed alerting for everyone.
In addition, Kloudfuse platform comes bundled with infrastructure alerts. To learn more click here.
To import alerts from another source, please follow the instructions at Dashboards and Alerts Onboarding
We understand if you are used to Grafana and would like to continue using it for your alerting. Therefore Kloudfuse embeds Grafana UI in its UI where you can use Grafana just the way you’d like.
From here on all functionalities that you use Grafana with should be available as is. For more on how to use Grafana Alerting, you can follow Grafana Alerting documentation.
Kloudfuse stack comes installed with a few very useful infrastructure (K8s and physical resource) alerts which are designed to be functional in any k8s based environment out the box. It also has in-built alerts which are useful for “self-monitoring” (to get notified if the Kloudfuse stack itself is having functional issues). However, the user is required to configure the notifications so that they can be notified when the alert is firing. The preinstalled alerts can be reviewed and edited from the main alerting page.
Kloudfuse-managed alerting aims to provide an opinionated view of how alerting framework should be. It allows for, what we believe are, more intuitive workflows that are easy to follow and less cluttered.
Using the Kloudfuse platform you can configure alerts, notify your teams, and manage alerts at a glance.
Note: Currently the Kloudfuse platform supports threshold-based category alerts on Metrics and Logs type of telemetry data. Please contact us to know about upcoming changes to other alert categories (automatic tracking changes in behavior or using explicit anomaly algorithms) and alert types (APM & Events)
To create a monitor in the Kloudfuse platform follow these steps:
Navigate from the main page to Alerts > Rules and then click “Create New Alert”
Select an alert type corresponding to the kind of telemetry you want to alert on. Read further for each alert type:
Contact points are used to define how your contacts are notified when an alert fires. The Kloudfuse platform supports the following types of contact points. The Kloudfuse platform supports Grafana message templates to customize notification messages for the contact point types.
Click “Contact points” to open the page listing existing contact points.
Click “Add New contact point”.
Add a descriptive name for the contact point In the “Name” field.
Use the “Choose contact point type” drop-down to select a type of contact point and fill out mandatory fields.
Some contact point types, like email or webhook, have optional settings. So fill them as necessary.
See contact point specific settings here.
Optionally test that the contact point is functioning appropriately using the “Test” button.
“Save contact point” to save your changes.
To edit a contact point click “Contact points” to open the page listing existing contact points. Find the contact point to edit, then click the pen icon to edit the contact point. Make necessary changes, optionally test, and click “Save contact point”.
To delete a contact point click “Contact points” to open the page listing existing contact points. Find the contact point to delete, then click the trash icon to delete the contact point. Confirm the choice when a window pops up.
The following list of contact points is currently supported. This list will be enhanced as support for additional contact points is added.
PagerDuty
Slack
Webhook
The Alerts > Rules page lists the installed alerts in the system. A list of alerts can be searched using their state, labels, title, etc., intuitively on this page. Each alert can be further investigated by clicking on that alert from the list where the alert’s properties, current evaluation graph, and history are displayed.
Kloudfuse platform underlying query is always a PromQL query. These queries can be further extended using all https://prometheus.io/docs/prometheus/latest/querying/functions/ .
In addition to supporting full promql for alerting, Kloudfuse platform has some additional advance functions that can be used for detecting anomaly, outliers, etc. These functions are made available directly through detection methods in each stream. For example metric streams provides these detection methods.
Kloudfuse platform provides the ability to do alerting on all service endpoints of your services out of the box with a simple configuration update to enable. Enabling this feature results in the Kloudfuse platform starting to monitor the communication between various endpoints without requiring any instrumentation to your applications. On the collected data, it evaluates anomalies in streaming fashion using state-of-the-art statistical learning algorithms. Please see the advanced analytics section for more details on this.
Grafana lets user send email notification on alerts if configured. Grafana can be configured in more than one ways, but here we’ll give example of configuring grafana with gmail based notifications. (Other methods of authentication are weaker and therefore not listed here. This is the recommended way)
Firstly, grab an email address that you will be using to send the notifications. we’ll use grafana_alerts@domain.com.
create an app password in gmail. You will need to be using 2FA for the account (grafana_alerts@domain.com) to be able to create app password. https://support.google.com/mail/answer/185833?hl=en . Note down the app password as you will need it in step 3.
make sure you are connected to the cluster where kloudfuse stack is installed and you are in kfuse namespace
# connect to your cluster kubectx <cluster-name> kubens kfuse |
create a kubernetes secret with the username and password you created in step 1.
kubectl create secret generic grafana-smtp-user-password --from-literal=user=grafana_alerts@domain.com --from-literal=password=<generated-app-password> |
edit the values.yaml to uncomment settings related to smtp in grafana section (to look like the snippet below). Update following settings:
update host to your smtp mail server
update from_address to the smtp user you want to use
update from_name if needed
grafana:
grafana:
# grafana.ini - Grafana server configuration settings
grafana.ini:
...
# start -- Uncomment the following to enable smtp
smtp:
enabled: true
host: your_smtp_hostname_colon_port
skip_verify: true
from_address: your_smtp_user@domain.com
from_name: AlertsAdmin
envValueFrom:
GF_SMTP_USER:
secretKeyRef:
name: grafana-smtp-user-password
key: user
GF_SMTP_PASSWORD:
secretKeyRef:
name: grafana-smtp-user-password
key: password
# Uncomment the following to enable smtp -- end
|
issue the same kfuse helm install command which you used to install kfuse cluster again.
helm upgrade --create-namespace --install kfuse . -f [gcp|aws].yaml -f custom_values.yaml --set global.orgId=<your-company-name> |
Getting integration key from an existing service in PagerDuty
Create a new service and use that service’s integration key
Use a Slack webhook to create a contact point in slack.
Follow the link below to create a slack webhook and get its url- https://api.slack.com/messaging/webhooks
One the webhook is created, enter its URL in the webhook field while creating the contact point.
Add the webhook URL in the URL (mandatory) field.
Add any additional settings as necessary.
Setting flock integration with Kloudfuse alerting is same as setting up webhook based integration. However, the webhook url needs to be obtained from your flock app and some configuration at flock end is required.
Get a webhook url by creating an “incoming webhooks” from flock and pay attention to the transformations. Please add following transformation configuration in the incoming webhook you create:
{"text": "{
status: \"$(json.status)\"
alerts: $(json.alerts),
groupLabels: $(json.groupLabels),
commonLabels: $(json.commonLabels),
commonAnnotations: $(json.commonAnnotations),
externalURL: \"$(json.externalURL)\",
version: \"$(json.version)\",
groupKey: \"$(json.groupKey)\",
truncatedAlerts: $(json.truncatedAlerts),
orgId: $(json.orgId),
title: \"$(json.title)\",
state: \"$(json.state)\",
message: \"$(json.message)\"
}"} |
Use the URL from above step (it would look like: https://api.flock.com/hooks/sendMessage/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx) to create a webhook contact point as described here.