Overview

This document describes the various search functionalities offered by our log search engine. We support three main search types:

• Facet Search: Enables searching based on auto-detected key-value pairs (facets) associated with your logs. This is the fastest search method. Facets also have a notion of a data type.

• Term Search: Focuses on individual terms or keywords within your log messages. This method offers a good balance between speed and flexibility.

• Grep Search: Provides exact pattern matching for precise searches. This is the slowest but most accurate search type.

In addition to these search types, we support a variety of operators to refine your searches.

Operators:

• Equals (=): Matches exact values for facets.

• Not Equals To (!=): Excludes entries where the facet has a specific value.

• Greater Than/Less Than (>, <): Used with numerical facets for value comparisons.

• Greater Than or Equal To (>=, <=): Similar to above, but includes the specified value.

• NOT (!): Negates a search term.

• FacetTermsExist (==): Searches for terms within a facet

• FacetTermsExist (!==): Negates FacetTermsExist

• Contains (**): Finds logs where a facet value contains the specified text.

• Regex (=~): Regular expression

• Not Regex (!~): Regular expression

• StartsWith (*~): Search for log line which starts with 

• EndsWith (~*): Search for log line which ends with

Facet Data Types

• String

• Number

• Duration

• Size

• IP

• UUID

Unions/Intersections

• AND

• OR

Search Types

1. Facet Search

Facet searches allow you to filter logs based on specific key-value pairs (facets) present in your logs.

• Supported Data Types:

  • Each Facet is associated with a data type.

  • All data types (String, Number, etc.) are supported in this search.

• Operator Support:

  • All operators (Equals to, Not Equals to, Greater/Lesser etc.) are supported.

  • Certain operators are only supported with a subset of data types

• How to do Facet Search?

  • Type the symbol @ - and then start typing the characters. UI would try to populate the existing facets. OR,

  • Pick detailed view of a log line and select facet name/value. OR,

  • From the side bar, pick any of the existing labels

Examples:

Facet Combinations/ Unions:

Following example lists all possible union combinations with Facet Search

Not supported:

OR based search of two (or more) facet searches are not supported as of today.

• Facet1-Value OR Facet2-Value

Facet Key Existence:

One can search for the existence of specific facets (keys) within the logs.

Example: 

Not supported:

• key exists="Facet1 OR Facet2"

Facets V/s Labels

Our facet search functionality leverages two distinct data sources to provide comprehensive filtering capabilities:

1. Automatically Extracted Facets: These facets are dynamically generated by parsing the log message content using predefined heuristics. This approach identifies key-value pairs inherent to the log data itself. For example, in a log line like "User=Vikas logged into an EC2 machine with e2_instance_id=0xABCD at time=10am", facets like "User=Vikas", "e2_instance_id=0xABCD", and "time=10am" would be automatically extracted.

2. Labels: Similar to automatically extracted facets, labels are also searchable using the same search heuristics. However, these labels are pre-configured key-value pairs associated with each log line that provide contextual details about the environment in which the log originated. This information typically includes data like pod name, Kubernetes namespace, cloud region, or any other relevant metadata attached to the log during ingestion. Common examples of environment labels include:

   • pod_name: Identifies the specific pod that generated the log message (e.g., pod_name=request_handler).

   • cloud_region: Indicates the cloud region where the log originated (e.g., cloud_region=us-west-2c).

   • kube_namespace: Specifies the Kubernetes namespace associated with the log (e.g., kube_namespace=kloudfuse).

This distinction between automatically extracted facets and predefined environment labels empowers you to filter logs based on both the inherent content of the message and the broader environment in which it was generated. This two-pronged approach offers a powerful and flexible way to pinpoint the specific logs you need for troubleshooting and analysis.

2. Term Search

Term search focuses on identifying logs containing specific words or phrases (terms).

• Supported Data Types: All data types (text is extracted from the log message) as string

• To do a term search, type the needed input and press enter without any quotes.

• Operator support: There is no operator support except Not Equals.

  • Does not support operators like Contains, StartsWith, or EndsWith.

• Term Search is case-insensitive

Term Search Combinations:

• AND: Combine terms using spaces (or use a different UI chip) to find logs containing all specified terms.

• OR: Use the OR keyword within the chip function to find logs containing any of the specified terms.

• NOT: Prefix a term with ! to exclude logs containing that term.

Examples:

The term chip is used to denote the UI element which encapsulates the searched term(s).

3. Grep Search

Grep search provides a way to perform exact pattern matching on your log lines. To use grep, enclose the pattern within double quotes (“...”).

Grep Search is case-insensitive

• Supported Data Types: All data types (treated as a single string)

• Operators: There is no notion of operators (including regex) associated with Grep.

  • Use TermExists search for these use cases

Examples:

• Find logs containing the exact pattern "GET /api/users": "GET /api/users"

• Find logs starting with "INFO ": "INFO " (note the trailing space)

• Find logs ending with "connection closed": "connection closed"

Grep Search Combinations:

• AND: Combine multiple grep patterns using spaces to find logs matching all patterns.

• OR: Use the OR operator within quotes to find logs matching any of the specified patterns.

• NOT: Use ! (exclamation) before the quotes to negate the search pattern

Important Notes:

• Grep search is case-insensitive.

• Wildcards are not directly supported within Grep searches. However, you can achieve similar functionality for basic pattern matching using term search.

Special Characters

Our search engine supports searching for logs containing a wide range of characters. However, to ensure accurate results when using special characters like double quotes (") and backslashes (\), proper escaping is necessary. Here's a breakdown of how to handle these special characters:

1. Double Quotes (")

Double quotes are used to perform exact phrase matching within your search queries. To search for a literal double quote character within your logs, you need to escape it using a backslash (\) before the quote.

• Example: Let's say you want to find logs containing the error message User="Vikas" logged in at 10am. Here, the double quotes are part of the error message itself, and you need them to be included in the search. The correct query would be:

2. Backslash (\)

Backslash has a special meaning in search queries because it is used for escaping other characters. If you want to search for a literal backslash character, you need to escape it with another backslash (\).

• Example: Imagine you're looking for logs containing a path like C:\Users\Vikas\AppData. In this case, the backslashes are part of the path, and you need to escape them to perform an exact match. The correct query would be:

Other Special Characters:

For most other special characters, you don't need any special handling. They can be included directly in your search queries without the need for escaping. This includes characters like commas (,), periods (.), exclamation points (!), and so on.

By following these guidelines for handling special characters, you can construct effective search queries that precisely target the logs you need within your system.