2.7.3.P2 (Latest)

September 17, 2024

We improved the performance on the Trace Errors interface.

2.7.3.P1

September 6, 2024

...

OpenTelemetry on Docker deployments

In this release, we added support for data collection through OpenTelemetry on Docker containers. For details, see https://kloudfuse.atlassian.net/wiki/x/u4CpOg.

Trace (APM)

Prettify JSON

For the Trace > Log details interface, we improved the display of the JSON format of the log line.
Select the Pretfify JSON option to see the more familiar and easy to read rendering of the log line.

...

Trace Span Details

When examining Trace Latency Breakdown detail, under the Logs option, we added a new filter option. Depending on the nature of your technical stack, you can now select to filter on:

• Kubernetes: pod_name

• Docker: container_id

• Host: hostname

To select the relevant filter:

1. On the top navigation bar, select APM > Traces.

2. {Optional}
   Use Search to find the relevant service.

3. In the list of services, click the relevant Trace Latency Breakdown diagram.

4. In the detail, under the Flame Chart, select the Logs tab.

5. The header of the list starts with the Filter, set to traceId by default. Click, and select the relevant filter from the dropdown.

2.7.3

August 28, 2024

Logs

We made significant improvements to the performance of log search with efficient indexes; comparable searches are now an order of magnitude faster.

APM

Bug fixes.

RBAC

We added default access policies for users without explicit policy assignment. See Default Policy.

2.7.2

August 19, 2024

Kloudfuse 2.7.2 release builds on 2.7.1 and introduces new & improved RBAC(Role Based Access Control) with UI support for easier user management. Additionally, we are introducing support for Multi-key authentication and targeted performance improvement for APM.

This release also includes additional features, bug fixes as listed below.

APM

...

RBAC

Multi-key auth ingestion

2.7.1

August 12, 2024

Kloudfuse 2.7.1 release builds on 2.7.0 version and improves analytics experience for Traces and logs. Additionally, we are introducing new Forecast and Anomaly models for metrics.

This release also includes additional features, bug fixes as listed below.

APM

Logs

Advanced Alerts

Others

2.7.0

July 29, 2024

Kloudfuse 2.7 is a major release many enhancements and critical bug fixes.

This release has a kafka upgrade that requires specific steps - please refer to upgrade instructions.

APM

UI

Logs

Others

2.6.7 Patch 1

May 30, 2024

Kloudfuse 2.6.7 is patch release with some critical bug fixes.

Bug Fixes

• Fix for RBAC exception when navigating to live trace and APM Derived Metrics - only applicable if RBAC is enabled.

• Fix for Alignment of values between “Total Requests” and side bar span counts in the Traces screen

• Fix for No Data/Exec State alert condition issue

2.6.7

May 15, 2024

Kloudfuse 2.6.7 release builds on 2.6.5 version and continues to improve APM and logs user experience. We have added support for defining Service Level Objectives (SLOs) at service level among other improvements and enhanced logs search with further search operators.

This release also includes many performance improvements, bug fixes and other minor enhancements as detailed below.

APM

Logs

• Performance improvements to reorder filters based on their efficiency while executing queries in the database. This can have a large effect on user experience.

• Improved and reduce disk reads in termsExist filter execution.

2.6.5

April 10, 2024

Kloudfuse 2.6.5 is a major release that significantly enhances the APM user experience by introducing the concept of service identifier. Service identifier allows APM services to be uniquely identified by user configurable set of attributes with a reasonable default that includes key cloud and kubernetes attributes.

This release also includes many performance improvements, bug fixes and other minor enhancements as detailed below. It also includes various security fixes to address CVEs.

APM

Infrastructure

2.6.0

February 16, 2024

Kloudfuse 2.6.0 is a major release that includes significant UI improvements, many new APM and Dashboard features.

UI Enhancements

Dashboards

Metrics

APM

2.5.4

January 6, 2024

kloudfuse 2.5.4 is a minor release with a few enhancements, performance improvements and bug fixes.

2.5.3

December 26, 2023

kloudfuse 2.5.3 is a minor release with logs facet autocomplete with typeahead, APM trace search enhancements, performance improvements and bug fixes.

Logs

APM

2.5.2

December 13, 2023

Kloudfuse 2.5.2 is a minor release with performance improvements and bug fixes.

2.5.1

December 10, 2023

Kloudfuse 2.5.1 is a minor release with improved alerting capabilities and performance and bug fixes.

Alerts

2.5.0

November 21, 2023

Kloudfuse 2.5.0 is a major release with lots of improvements and features and bug fixes.

Logs

APM

Metrics

Alerts

RBAC

...

...

3.0(Latest)

November 6, 2024

This release includes two significant innovations in Kloudfuse platform, Real User Monitoring and FuseQL query language. It also adds significant new discoverability and analysis tools to the APM system by introducing Trace Heatmap and K-Lens.

Real User Monitoring (RUM)

RUM allows you to capture and analyze data from real users as they interact with your web application, providing insights into their experience and identifying performance issues from the user's perspective.

Read about our implementation in the following pages:

• Real User Monitoring

• Event Exploration

• Views

• Actions

• Resources

• Errors

• Performance Monitoring

FuseQL

Kloudfuse developed its proprietary query language, FuseQL, for range of applications. It has flexible parameters for answering highly complex questions.

To understand the defining characteristics of FuseQL and how to use it, see the following pages:

• FuseQL

• Cheatsheet for FuseQL

• Aggregation operators

• Advanced functions and algorithms

• Mathematical operators

APM

This release introduces two powerful newAPM observability tools - Trace Heatmap and K-Lens to assist in service-level issue detection and advanced debugging using APM Trace data

• Trace Heatmap is an interactive trace data visualization chart that helps you to visually detect deviations and outliers in the latencies reported by APM data.

• K-Lens is a proprietary analysis tool that helps you to determine what is wrong in the system; it compares thousands of attributes across span events to their baseline performance, and displays ranked results based on significance of deviation.

2.7.4

September 24, 2024

This release includes many innovations to the Kloudfuse platform:

• Service Catalog

• Metrics Roll Up

• Facet exploration and analytics

• Log Archive and Hydration

• Log label cardinality analysis

• Graviton support

• RBAC improvements

• Error display improvements

Service Catalog

Until this release, Kloudfuse's APM(Traces) interface only displayed the services that sent data within the selected time block. We now enable you to see all services, including the ones that may be stopped or paused.

To see all services, simply toggle the Show Inactive Services selector. You can see that two new services in this example, frontend-web and loadgenerator, appear in the service list. Because the services were inactive during the relevant time interval, they have empty columns.

...

Metrics Roll Up

Kloudfuse supports roll up and aggregation of metrics data, computed directly from the raw data stream during ingestion. For longer time intervals, our approach significantly improves query performance, and reduces chart loading times and I/O costs. Depending on the time span or step size of the query, Kloudfuse calculates results either from raw data, or from rolled up data. In the shorter time spans, we continue to use raw metrics because the calculation approach could potentially smooth out the data and miss important signals, such as outliers.

Metrics roll up is off by default, and the default aggregation is for 5 minutes. Contact us to turn on metrics roll up, and help you configure your environment.

For more information about this feature, see Metrics roll up.

Facet exploration and analytics

Logs default sidebar

...

Kloudfuse automatically extracts facets attributes from logs during ingestion. Before this release, we surfaced facets in the Logs side bar, organized under Sources.

Because the number of sources and facets can quickly become very large, the troubleshooting tasks become more difficult and the feature is not as useful as we want it to be.

In this release, we re-organized the side bar to include expandable sections as these components:

• Filters

• Labels

• Facets

As you can see in the image on the right, we now organize Facets in their own section. Users can create custom groups that include the facets they want to monitor, across any number of sources.

You can group facets by source, and use the search bar to find relevant groups.

When you expand a group (here, the logs-parser), you can see some of the facets tracked in the group. We also added a search bar in each group to help you locate a facet by name.

Facet Explorer

...

Click the icon on the Facets title line, and Kloudfuse opens the new Facet Explorer. When you choose a source (logs-parser to continue this demonstration), you can typically see many more named facets than you see in the Logs sidebar.

Facet Favorites

...

This release also introduces Facet favorites that optimize facet design and usability; you can hover over one of the facets, and click on the hollow icon to “favorite” that facet. In the Edit Favorite interface, simply decide if you want to add the facet as a favorite to the existing group, or create a new group. You can even change the display name of the favorite facet.

Because of these changes, and other usability improvements, you can now control the organization and visibility of facets much more effectively:

1. From the log event detail pane, you can select a facet and add it do the side bar.

2. You can organize facets by user-defined groups, so the side bar has a single-level folder structure.

3. The Logs page now has a Facet Explorer option that enables you to browse across all facets, add facets, and remove them.

Log Archive and Hydration

You may have to save transactional information for compliance, legal, or other regulatory requirements. In addition to processing logs for observability and analytics, this release of Kloudfuse introduces a supplementary mechanism for archiving pre-processed logs (with identified filters, facets, and so on) into longer-term storage, and a separate mechanism to hydrate these logs to examine them for the relevant data.

The benefits of this approach extend beyond basic regulatory compliance:

• You store important historical data in a cost-effective compressed format in a location that you own and control.

• When uncompressed, the logs are human-readable and highly searchable because of the high level of indexing through labels and other data attributes.

• You can configure the archival instructions in a manner that categorizes data consumption by internal cost center.

We currently support log archive and hydration for AWS S3.

Contact us at support@kloudfuse.com to enable this feature in your Kloudfuse cluster.

Archiving

Based on your own set of archival rules and configurations, you can add an archive section to the deployments.yaml file to specify the logs you need to write to your own archive storage.

  archive:
    enabled: true
    prefix: "<Example_Cluster/Example_Folder>"    # Optional, can specify as ""
    useSecret: true                               # Security, 4 methods, see note
    createSecret: true
    secretName: "<Example_Secret>"
    type: s3                                      # AWS storage
    s3:
      region: <Example_Region>                    # Such as us-west-2
      bucket: <Example_Bucket_Name>               # You MUST create the bucket in you archival storage location
      accessKey: <Example_Access_Key>
      secretKey: <Example_Secret_Key>
    rules: |-
      - archive:                                  # Define first archive
        args:
          archiveName: a1
        conditions:
          - matcher: "#source"
            value: "s1"
            op: "=="
          - matcher: "@label"
            value: "l1"
            op: "=="
      - archive:                                  # Define next archive
        args:
          archiveName: a2
        conditions:
          - matcher: "#source"
            value: "s1"
            op: "=="

The archive rules apply in order, and must match all conditions. In the preceding example, a log line from source “s1” gets mapped to archive a1 if it also contains label l1. If it does not contain label l1, it gets mapped to archive a2.

Kloudfuse then writes the logs to the specified archive location after ingestion, applying facets, labels, and so on.

...

Note that individual files are named by original log stream name, with .log.zst extension.

Hydration

...

Whenever you need to examine a record, you can access it directly in your archival storage because of our simple storage and compression rubric: by year, by month, by date. When you decompress and open a log file, you can see all the facets, labels, and other tags attributed to the log by Kloudfuse.

Additionally, you can hydrate the archived logs into Kloudfuse. We run them through the metadata analysis, labeling, and so on. For older logs, this gives us the opportunity to apply the current (newer) set of grammar and rules, making them compatible (and comparable) with current logs.

To begin log hydration, select the Logs tab, and then the Hydration option in the dropdown menu.

The Logs Hydration interface opens on the Archives tab, and lists all logs registered In Config (the Kloudfuse configuration) and In S3. Every archive includes:

• First archived record(date and time)
  This is the hour-granularity-location of the oldest archive object. Kloudfuse uses this timestamp when starting the hydration job.

• First object created (date and time)
  This points to the timestamp of creation of the first object for that archive rule.
  It helps you to roughly identify when your organization installed the rule.

• Name of the archive

• Approximate size of the archive

• In Config stipulates that the archive rule is active

• In S3 denotes that the customer at some point in the past created an archive rule that is now deleted
  The archives for that archive rule persist in archival storage

• Location directory in the archive location

• Filters that were used to select the log lines for the archive.

For example, if the user installed a new archive rule on 2024/09/24 15:00, and started archiving old logs from 2023/01/02 9:00 onwards, then:

• First Archive Record is 2023-01-02 9:00:00

• First Object Created is 2024-09-24 15:00:00

...

To hydrate specific logs from an archive, hover the cursor over the last column in the table; when the Hydrate button appears, click it.

In the Hydrating interface, follow these steps:

1. In Set Time Range (required), select the start and end time. Note that you can specify time measurement either in UTC or in local browser time.
   If you don’t explicitly specify the time range, Kloudfuse uses the default of 1 week after the start of the archive.
   The time selection options are:

   • Absolute time rangeusing the time picker.

   • Quick ranges, such as “Next 5 minutes”, “Next 15 minutes”, and so on, through to “Next 3 days”.
     This assumes the start at the beginning of the first record of the object, and ending calculated by adding the quick range selection.

2. Set filter to hydrate log entries that match the specified key/value pairs.
   You can specify multiple filters; Kloudfuse combines them through AND logic. So, a record must match all filters to be hydrated.
   If you don’t specify filters, Kloudfuse potentially hydrates all log records that match the other criteria.
   Filter settings are optional.

3. Set tags to add the same attribute to all hydrated records.
   Specify key/value pairs for each tag. You can specify multiple tags.
   If the log entry already uses a specified key, Kloudfuse overwrites the value attribute.
   Tag settings are optional.

4. Set retention (required) to specify how long you want to keep the hydrated records on the Kloudfuse platform.
   Default retention is 7 days.

5. Click Hydrate to start log hydration.

6. The system notifies you that it started the Hydration job.

Switch to the Jobs tab to monitor the progress of the hydration job. This interface provides a historical overview of all archival jobs by Name, Start Time and End Time, the Status of he job, Progress for the currently running job, Filters, Tags, Created at (date and time), and Retention period of the hydrated logs.

Each job also has a unique job id that you can use to locate the hydrated logs in the logs search page. All hydrated logs have the label __kf_hydration_job : <unique_job_id>.

...

After successful hydration and indexing, the log is inside the Kloudfuse system, and can be reviewed and analyzed like other logs. The user can see the logs from the hydration job by clicking the Logs button and opening a new tab filter set set to the job id.

You can cancel a job in progress. Any log lines that have been hydrated before the cancel command remain visible, through the retention time specified in the hydration job.

Log label cardinality analysis

...

Log labels have two main source: many are attached by the agent that delivers the logs to the Kloudfuse platform, and potentially even more are defined by users. Some labels are meaningful at the time they are defined, yet loose their relevance over time. Some are created accidentally, or as a result of not fully understanding the common use cases or the purpose of log tracking. Automatically-generated labels, from agents, are often cryptic and unnecessary.

Seeing cardinality analysis of a log helps users to remove unnecessary labels, improving data accessibility and making the collected information more actionable.

To determine log label cardinality, select the Logs tab, and then the Cardinality Analytics option in the dropdown menu.

The Logs Cardinality report shows the overall cardinality; in this case, the cardinality is 335. It further breaks out the data by Label, showing the Value Count (1h) for the preceding hour, the Value Count for the specified most recent time range (the last 5 minutes is the maximum time range and the default setting), and the Value Chart, which is a simple bar chart representation of the count of unique values over the selected time range.

...

To find the prevalence of a specific label value, use the filter at the top of the page to select a label, the operator, and the comparison value.

Graviton support

...

Starting with this release, Kloudfuse ingests data from AWS EC2 sources that run on AWS Graviton processors.

For more information, see AWS Graviton Processors documentation.

RBAC improvements

User Management

This page now displays all groups to which the user belongs, and their role: Admin or Member. You can change:

• The name of the user

• The user’s role ( Admin ↔ Editor ↔ Viewer],

• Re-assign group membership level ( Admin ↔ Member )

• Remove the user from any assigned group

...

User Status

...

At the top right, where we show logged in user’s information, Kloudfuse UI now displays the user’s role.

Error display improvements

...

Starting with this release, we are replacing error message toasters with more specific error messages that display directly on the components that throw the error.

2.7.3.P2

September 17, 2024

We improved the performance on the Trace Errors interface.

2.7.3.P1

September 6, 2024

...

OpenTelemetry on Docker deployments

In this release, we added support for data collection through OpenTelemetry on Docker containers. For details, see https://kloudfuse.atlassian.net/wiki/x/u4CpOg.

Trace (APM)

Prettify JSON

For the Trace > Log details interface, we improved the display of the JSON format of the log line.
Select the Pretfify JSON option to see the more familiar and easy to read rendering of the log line.

...

Trace Span Details

When examining Trace Latency Breakdown detail, under the Logs option, we added a new filter option. Depending on the nature of your technical stack, you can now select to filter on:

• Kubernetes: pod_name

• Docker: container_id

• Host: hostname

To select the relevant filter:

1. On the top navigation bar, select APM > Traces.

2. {Optional}
   Use Search to find the relevant service.

3. In the list of services, click the relevant Trace Latency Breakdown diagram.

4. In the detail, under the Flame Chart, select the Logs tab.

5. The header of the list starts with the Filter, set to traceId by default. Click, and select the relevant filter from the dropdown.

2.7.3

August 28, 2024

Logs

We made significant improvements to the performance of log search with efficient indexes; comparable searches are now an order of magnitude faster.

APM

Bug fixes.

RBAC

We added default access policies for users without explicit policy assignment. See Default Policy.

2.7.2

August 19, 2024

Kloudfuse 2.7.2 release builds on 2.7.1 and introduces new & improved RBAC(Role Based Access Control) with UI support for easier user management. Additionally, we are introducing support for Multi-key authentication and targeted performance improvement for APM.

This release also includes additional features, bug fixes as listed below.

APM

RBAC

Multi-key auth ingestion

2.7.1

August 12, 2024

Kloudfuse 2.7.1 release builds on 2.7.0 version and improves analytics experience for Traces and logs. Additionally, we are introducing new Forecast and Anomaly models for metrics.

This release also includes additional features, bug fixes as listed below.

APM

Logs

Advanced Alerts

Others

2.7.0

July 29, 2024

Kloudfuse 2.7 is a major release many enhancements and critical bug fixes.

This release has a kafka upgrade that requires specific steps - please refer to upgrade instructions.

APM

UI

Logs

Others

2.6.7 Patch 1

May 30, 2024

Kloudfuse 2.6.7 is patch release with some critical bug fixes.

Bug Fixes

• Fix for RBAC exception when navigating to live trace and APM Derived Metrics - only applicable if RBAC is enabled.

• Fix for Alignment of values between “Total Requests” and side bar span counts in the Traces screen

• Fix for No Data/Exec State alert condition issue

2.6.7

May 15, 2024

Kloudfuse 2.6.7 release builds on 2.6.5 version and continues to improve APM and logs user experience. We have added support for defining Service Level Objectives (SLOs) at service level among other improvements and enhanced logs search with further search operators.

This release also includes many performance improvements, bug fixes and other minor enhancements as detailed below.

APM

Logs

• Performance improvements to reorder filters based on their efficiency while executing queries in the database. This can have a large effect on user experience.

• Improved and reduce disk reads in termsExist filter execution.

Earlier releases

Review the release notes for older releases.