Release Notes

Owned by Pankaj Thakkar
Last updated: Oct 09, 2024 by Roza Leyderman
6 min read

 

2.7.4 (Latest)

September 24, 2024

This release includes many innovations to the Kloudfuse platform:

Service Catalog

Until this release, Kloudfuse's APM(Traces) interface only displayed the services that sent data within the selected time block. We now enable you to see all services, including the ones that may be stopped or paused.

To see all services, simply toggle the Show Inactive Services selector. You can see that two new services in this example, frontend-web and loadgenerator, appear in the service list. Because the services were inactive during the relevant time interval, they have empty columns.

service-catalog.png

Metrics Roll Up

Kloudfuse supports roll up and aggregation of metrics data, computed directly from the raw data stream during ingestion. For longer time intervals, our approach significantly improves query performance, and reduces chart loading times and I/O costs. Depending on the time span or step size of the query, Kloudfuse calculates results either from raw data, or from rolled up data. In the shorter time spans, we continue to use raw metrics because the calculation approach could potentially smooth out the data and miss important signals, such as outliers.

Metrics roll up is off by default, and the default aggregation is for 5 minutes. Contact us to turn on metrics roll up, and help you configure your environment.

For more information about this feature, see Metrics roll up.

Facet exploration and analytics

Logs default sidebar

facets-in-left-bar.png

Kloudfuse automatically extracts facets attributes from logs during ingestion. Before this release, we surfaced facets in the Logs side bar, organized under Sources.

Because the number of sources and facets can quickly become very large, the troubleshooting tasks become more difficult and the feature is not as useful as we want it to be.

In this release, we re-organized the side bar to include expandable sections as these components:

  • Filters

  • Labels

  • Facets

As you can see in the image on the right, we now organize Facets in their own section. Users can create custom groups that include the facets they want to monitor, across any number of sources.

You can group facets by source, and use the search bar to find relevant groups.

When you expand a group (here, the logs-parser), you can see some of the facets tracked in the group. We also added a search bar in each group to help you locate a facet by name.

 

 

 

 

Facet Explorer

Click the icon on the Facets title line, and Kloudfuse opens the new Facet Explorer. When you choose a source (logs-parser to continue this demonstration), you can typically see many more named facets than you see in the Logs sidebar.

 

 

 

 

 

Facet Favorites

This release also introduces Facet favorites that optimize facet design and usability; you can hover over one of the facets, and click on the hollow icon to “favorite” that facet. In the Edit Favorite interface, simply decide if you want to add the facet as a favorite to the existing group, or create a new group. You can even change the display name of the favorite facet.

Because of these changes, and other usability improvements, you can now control the organization and visibility of facets much more effectively:

  1. From the log event detail pane, you can select a facet and add it do the side bar.

  2. You can organize facets by user-defined groups, so the side bar has a single-level folder structure.

  3. The Logs page now has a Facet Explorer option that enables you to browse across all facets, add facets, and remove them.

Changes to existing Kloudfuse UI:

  • In the Logs interface left sidebar, Facets replace Sources.

  • When creating/changing Favorites, you can “remember” the source by selecting that option.

  • Upgrade note: The log facets that appeared in the sidebar before this release are now hidden. After upgrading to Kloudfuse Release 2.7.4, Use the Facet Explorer to identify the facets you want to see in the sidebar as “favorites”.

Log Archive and Hydration

You may have to save transactional information for compliance, legal, or other regulatory requirements. In addition to processing logs for observability and analytics, this release of Kloudfuse introduces a supplementary mechanism for archiving pre-processed logs (with identified filters, facets, and so on) into longer-term storage, and a separate mechanism to hydrate these logs to examine them for the relevant data.

The benefits of this approach extend beyond basic regulatory compliance:

  • You store important historical data in a cost-effective compressed format in a location that you own and control.

  • When uncompressed, the logs are human-readable and highly searchable because of the high level of indexing through labels and other data attributes.

  • You can configure the archival instructions in a manner that categorizes data consumption by internal cost center.

We currently support log archive and hydration for AWS S3.

Contact us at support@kloudfuse.com to enable this feature in your Kloudfuse cluster.

Archiving

Based on your own set of archival rules and configurations, you can add an archive section to the deployments.yaml file to specify the logs you need to write to your own archive storage.

archive: enabled: true prefix: "<Example_Cluster/Example_Folder>" # Optional, can specify as "" useSecret: true # Security, 4 methods, see note createSecret: true secretName: "<Example_Secret>" type: s3 # AWS storage s3: region: <Example_Region> # Such as us-west-2 bucket: <Example_Bucket_Name> # You MUST create the bucket in you archival storage location accessKey: <Example_Access_Key> secretKey: <Example_Secret_Key> rules: |- - archive: # Define first archive args: archiveName: a1 conditions: - matcher: "#source" value: "s1" op: "==" - matcher: "@label" value: "l1" op: "==" - archive: # Define next archive args: archiveName: a2 conditions: - matcher: "#source" value: "s1" op: "=="

The archive rules apply in order, and must match all conditions. In the preceding example, a log line from source “s1” gets mapped to archive a1 if it also contains label l1. If it does not contain label l1, it gets mapped to archive a2.

You must grant access for Kloudfuse to write archives into the specified storage. There are four security approaches

  1. createSecret = true, useSecret=true
    The helm creates the k8s secret based on the provider’s accessKey and secretKey.
    The deployments.yaml file gets the value from secret.

  2. createSecret=false, useSecret=true
    The customer creates the k8s secret.
    The deployments.yaml file works automatically because it picks up the env var from secret.

  3. createSecret=false, useSecret=false
    This approach assumes that the customer already configured the node IAM role to have permission to access the S3 bucket, so there is no requirement to set env var.

  4. Customer creates a service account with access permission to S3.
    That service account maps to serviceAccountName. There is no requirement to set env var, and the pod will inherits the permissions of the service account.

Kloudfuse then writes the logs to the specified archive location after ingestion, applying facets, labels, and so on.

Note that individual files are named by original log stream name, with .log.zst extension.

Hydration

Whenever you need to examine a record, you can access it directly in your archival storage because of our simple storage and compression rubric: by year, by month, by date. When you decompress and open a log file, you can see all the facets, labels, and other tags attributed to the log by Kloudfuse.

Additionally, you can hydrate the archived logs into Kloudfuse. We run them through the metadata analysis, labeling, and so on. For older logs, this gives us the opportunity to apply the current (newer) set of grammar and rules, making them compatible (and comparable) with current logs.

To begin log hydration, select the Logs tab, and then the Hydration option in the dropdown menu.

The Logs Hydration interface opens on the Archives tab, and lists all logs registered In Config (the Kloudfuse configuration) and In S3. Every archive includes:

  • First archived record (date and time)
    This is the hour-granularity-location of the oldest archive object. Kloudfuse uses this timestamp when starting the hydration job.

  • First object created (date and time)
    This points to the timestamp of creation of the first object for that archive rule.
    It helps you to roughly identify when your organization installed the rule.

  • Name of the archive

  • Approximate size of the archive

  • In Config stipulates that the archive rule is active

  • In S3 denotes that the customer at some point in the past created an archive rule that is now deleted
    The archives for that archive rule persist in archival storage

  • Location directory in the archive location

  • Filters that were used to select the log lines for the archive.

For example, if the user installed a new archive rule on 2024/09/24 15:00, and started archiving old logs from 2023/01/02 9:00 onwards, then:

  • First Archive Record is 2023-01-02 9:00:00

  • First Object Created is 2024-09-24 15:00:00

To hydrate specific logs from an archive, hover the cursor over the last column in the table; when the Hydrate button appears, click it.

In the Hydrating interface, follow these steps:

  1. In Set Time Range (required), select the start and end time. Note that you can specify time measurement either in UTC or in local browser time.
    If you don’t explicitly specify the time range, Kloudfuse uses the default of 1 week after the start of the archive.
    The time selection options are:

    • Absolute time range using the time picker.

    • Quick ranges, such as “Next 5 minutes”, “Next 15 minutes”, and so on, through to “Next 3 days”.
      This assumes the start at the beginning of the first record of the object, and ending calculated by adding the quick range selection.

  2. Set filter to hydrate log entries that match the specified key/value pairs.
    You can specify multiple filters; Kloudfuse combines them through AND logic. So, a record must match all filters to be hydrated.
    If you don’t specify filters, Kloudfuse potentially hydrates all log records that match the other criteria.
    Filter settings are optional.

  3. Set tags to add the same attribute to all hydrated records.
    Specify key/value pairs for each tag. You can specify multiple tags.
    If the log entry already uses a specified key, Kloudfuse overwrites the value attribute.
    Tag settings are optional.

  4. Set retention (required) to specify how long you want to keep the hydrated records on the Kloudfuse platform.
    Default retention is 7 days.

  5. Click Hydrate to start log hydration.

  6. The system notifies you that it started the Hydration job.

Switch to the Jobs tab to monitor the progress of the hydration job. This interface provides a historical overview of all archival jobs by Name, Start Time and End Time, the Status of he job, Progress for the currently running job, Filters, Tags, Created at (date and time), and Retention period of the hydrated logs.

Each job also has a unique job id that you can use to locate the hydrated logs in the logs search page. All hydrated logs have the label __kf_hydration_job : <unique_job_id>.

The Status of a hydration job is one of the following:

INIT The job is waiting to be processed

HYDRATING The job is running

FAILED An error occurred before the job completed

CANCELED The user cancelled the job before it completed

INDEXING Hydration service finished processing the job (all logs);
Kloudfuse is indexing the logs as soon as they are hydrated, essentially in parallel.
This ensure that the user can see their data in the platform.

DONE The job completed successfully

After successful hydration and indexing, the log is inside the Kloudfuse system, and can be reviewed and analyzed like other logs. The user can see the logs from the hydration job by clicking the Logs button and opening a new tab filter set set to the job id.

You can cancel a job in progress. Any log lines that have been hydrated before the cancel command remain visible, through the retention time specified in the hydration job.

Log label cardinality analysis

Log labels have two main source: many are attached by the agent that delivers the logs to the Kloudfuse platform, and potentially even more are defined by users. Some labels are meaningful at the time they are defined, yet loose their relevance over time. Some are created accidentally, or as a result of not fully understanding the common use cases or the purpose of log tracking. Automatically-generated labels, from agents, are often cryptic and unnecessary.

Seeing cardinality analysis of a log helps users to remove unnecessary labels, improving data accessibility and making the collected information more actionable.

To determine log label cardinality, select the Logs tab, and then the Cardinality Analytics option in the dropdown menu.

The Logs Cardinality report shows the overall cardinality; in this case, the cardinality is 335. It further breaks out the data by Label, showing the Value Count (1h) for the preceding hour, the Value Count for the specified most recent time range (the last 5 minutes is the maximum time range and the default setting), and the Value Chart, which is a simple bar chart representation of the count of unique values over the selected time range.

To find the prevalence of a specific label value, use the filter at the top of the page to select a label, the operator, and the comparison value.

Graviton support

Starting with this release, Kloudfuse ingests data from AWS EC2 sources that run on AWS Graviton processors.

For more information, see AWS Graviton Processors documentation.

 

 

 

 

RBAC improvements

User Management

This page now displays all groups to which the user belongs, and their role: Admin or Member. You can change:

  • The name of the user

  • The user’s role ( Admin ↔ Editor ↔ Viewer],

  • Re-assign group membership level ( Admin ↔ Member )

  • Remove the user from any assigned group

User Status

At the top right, where we show logged in user’s information, Kloudfuse UI now displays the user’s role.

 

 

Error display improvements

Starting with this release, we are replacing error message toasters with more specific error messages that display directly on the components that throw the error.

 

 

 

 

 

 

 

2.7.3.P2

September 17, 2024

We improved the performance on the Trace Errors interface.

2.7.3.P1

September 6, 2024

OpenTelemetry on Docker deployments

In this release, we added support for data collection through OpenTelemetry on Docker containers. For details, see https://kloudfuse.atlassian.net/wiki/x/u4CpOg.

Trace (APM)

Prettify JSON

For the Trace > Log details interface, we improved the display of the JSON format of the log line.
Select the Pretfify JSON option to see the more familiar and easy to read rendering of the log line.

Trace Span Details

When examining Trace Latency Breakdown detail, under the Logs option, we added a new filter option. Depending on the nature of your technical stack, you can now select to filter on:

  • Kubernetes: pod_name

  • Docker: container_id

  • Host: hostname

To select the relevant filter:

  1. On the top navigation bar, select APM > Traces.

  2. {Optional}
    Use Search to find the relevant service.

  3. In the list of services, click the relevant Trace Latency Breakdown diagram.

  4. In the detail, under the Flame Chart, select the Logs tab.

  5. The header of the list starts with the Filter, set to traceId by default. Click, and select the relevant filter from the dropdown.

 

2.7.3

August 28, 2024

Logs

We made significant improvements to the performance of log search with efficient indexes; comparable searches are now an order of magnitude faster.

APM

Bug fixes.

RBAC

We added default access policies for users without explicit policy assignment. See Default Policy.

2.7.2

August 19, 2024

Kloudfuse 2.7.2 release builds on 2.7.1 and introduces new & improved RBAC(Role Based Access Control) with UI support for easier user management. Additionally, we are introducing support for Multi-key authentication and targeted performance improvement for APM.

This release also includes additional features, bug fixes as listed below.

APM

 

RBAC

Starting with this release, KloudFuse does not support RBAC and policy configuration through editing of raw configuration files.

  • Follow these Upgrade Instructions to fully migrate your existing KloudFuse RBAC configuration to release 2.7.2.

Multi-key auth ingestion

2.7.1

August 12, 2024

Kloudfuse 2.7.1 release builds on 2.7.0 version and improves analytics experience for Traces and logs. Additionally, we are introducing new Forecast and Anomaly models for metrics.

This release also includes additional features, bug fixes as listed below.

APM

  • Support for pushdown queries for better performance & user experience.

  • Improved Analytics formula for correctness with optimized joins for top k and bottom k results.

  • Support for easy query param selection for multiple queries.

Logs

  • Support for pushdown queries for better performance & user experience.

  • Improved Analytics formula for correctness with optimized joins for top k and bottom k results.

  • Support for easy query param selection for multiple queries.

Advanced Alerts

Others

2.7.0

July 29, 2024

Kloudfuse 2.7 is a major release many enhancements and critical bug fixes.

This release has a kafka upgrade that requires specific steps - please refer to upgrade instructions.

APM

UI

Logs

Others

2.6.7 Patch 1

May 30, 2024

Kloudfuse 2.6.7 is patch release with some critical bug fixes.

Bug Fixes

  • Fix for RBAC exception when navigating to live trace and APM Derived Metrics - only applicable if RBAC is enabled.

  • Fix for Alignment of values between “Total Requests” and side bar span counts in the Traces screen

  • Fix for No Data/Exec State alert condition issue

2.6.7

May 15, 2024

Kloudfuse 2.6.7 release builds on 2.6.5 version and continues to improve APM and logs user experience. We have added support for defining Service Level Objectives (SLOs) at service level among other improvements and enhanced logs search with further search operators.

This release also includes many performance improvements, bug fixes and other minor enhancements as detailed below.

APM

Logs

  • Performance improvements to reorder filters based on their efficiency while executing queries in the database. This can have a large effect on user experience.

  • Improved and reduce disk reads in termsExist filter execution.

Earlier releases

Review the release notes for older releases.