Introduction

Log enrichment allows you to enhance your log data by adding or updating attributes based on specific rules. This is done using Lookup Tables, Source Tags, AuthZ Tags, Transform & Lookup Tag, and Enrichment Configuration under Logs Configuration. The enriched data provides more context and value, making it easier to analyze and act upon logs.

To create an alert definition:

  1. Navigate to Infrastructure > Logs.
  2. On the left side of this page, click the Menu icon.
  3. From the MY LOGS VIEWS page, under QUICK LINKS, select Logs Configuration.
    The configuration page is displayed.
  4. From the configurations page, select the Enrichment tab.
    The enrichment details page is displayed with the two sub-tabs:
    • Lookup Tables
    • Enrichment Configuration

Enrichment Configuration

After setting up a Lookup Table, you can create Enrichment Configs that specify how these attributes are applied to your logs.

To create a configuration:

  1. Click the ENRICHMENT CONFIGS tab.
  2. Click +ADD.
    The ENRICHMENT CONFIGURATION page is displayed.


  1. On the ENRICHMENT CONFIGURATION page, provide below information:
    • GENERAL DETAILS:
      • Name: Enter a name for the configuration.
      • Priority Order: Set the priority for this configuration. This determines the order in which configurations are applied if multiple rules match.
      • Lookup Table: Click the Lookup Table button, and in the Lookup Table dropdown, select the lookup table for which you want to create a configuration.
      • Source Tags: Click the Source Tags button, page displays Enrichment Rules section with enables to enter maching conditions.
      • AuthZ Tags: Click the AuthZ tags button to enter the matching conditions and when all conditions are met, the selected AuthZTags are added to the logs.
      • Transform & Lookup: Click the Transform & Lookup button to refine and extract yhe log information by defining conditions and applying transformations.
      • LOG FILTER CRITERIA: Click the +ADD FILTER option. Select the available attributes to define the filter criteria. This will determine which logs are enriched by this configuration.
      • ENRICHMENT RULES: Define conditions and matching fields under the Enrichment Rules section. These rules specify how and when the enrichment occurs.
      • LOG FIELDS: Choose the log fields that should be appended or updated when all conditions are met.
  2. After configuring all the necessary settings, click ADD CONFIGURATION to save your enrichment setup.


Lookup Tables

Lookup Tables are essential for enriching logs. They define the attributes that will be added or updated in your logs based on matching rules.

To create a lookup table:

  1. Click the LOOKUP TABLES tab within the Enrichment section.

  2. Click the +ADD.
    The ADD LOOKUP TABLE screen is displayed.

  3. Enter a name for your lookup table.

  4. Upload a CSV file containing the attributes you want to use for enrichment. The CSV file should be structured according to your enrichment requirements.

  5. Once the name and CSV file are provided, click ADD LOOKUP TABLE to save the changes.

Source Tags

Source Tags enable identification of the originating resources for logs, especially in centralized logging environments where logs are forwarded through intermediate devices like log hosts. This feature ensures that logs maintain resource-specific traceability and are enriched with contextual data such as Department, Region, or Hostname, even when a Resource UUID is missing from the source.

To enable source Tags:

  1. Go to ENRICHMENT. To access ENRICHMENT tab, see
  2. CLick ENRICHMENT tab.
  3. Click the ENRICHMENT CONFIGURATION tab.
  4. Click +ADD.
  5. Click Source Tags button.


Custom Attributes for Logs with Resource UUID:

Source Tags enable identification of the originating resources for logs, especially in centralized logging environments where logs are forwarded through intermediate devices like log hosts. This feature ensures that logs maintain resource-specific traceability and are enriched with contextual data such as Department, Region, or Hostname, even when a Resource UUID is missing from the source.


You can now have the option to enable the Log Value checkbox in the Custom Attributes section. This determines which attributes are included in enriched logs.


The following is the functional behavior in different Scenarios.

ScenarioFunctional Behavior
Single Lookup Table in a Single Enrichment ConfigurationLogs are enriched using one lookup table and one setup. Filters and rules extract data from the table and add relevant attributes to the logs.
Single Lookup Table in Multiple Enrichment ConfigurationsOne lookup table is used in multiple setups. Each setup applies its own rules and filters, enriching logs sequentially based on a defined priority order.
Multiple Lookup Tables in Multiple Enrichment ConfigurationsEach setup uses its own lookup table. Logs are enriched with attributes from multiple tables, processed one at a time, in the specified priority order.
Single Enrichment Configuration with Source TagsLogs are enriched by applying filters to identify relevant logs, adding resource details (like IDs and custom attributes) to the matching logs.
Enrichment Configuration with Both Lookup Table and Source TagsLogs are enriched using a mix of lookup tables and tags. Multiple setups are processed sequentially, each adding attributes to logs based on its logic.

AuthZ tags

AuthZ tags in logs are key-value pairs used in Role-Based Access Control (RBAC) to enforce fine-grained access control, ensuring users can only perform actions on logs based on their assigned tags.



Transform & Lookup

The Transform & Lookup tag is an enhancement in log enrichment that enables advanced data transformation and extraction. It introduces functions like splitString, splitRegex, parseJson, and regexExtract, to allowi users to refine log data efficiently.

  • splitString – Splits a string based on a specified delimiter and extracts the desired part.
  • splitRegex – Uses regular expressions to dynamically split a string, offering precise pattern-based segmentation.
  • regexExtract – Applies regex matching to extract specific parts of a string, ensuring efficient data retrieval.

splitString Function

The splitString function is used to divide a string field into multiple parts based on a specified delimiter (e.g., :, -, _). It then extracts a specific part using a 0-based index. This is particularly useful when dealing with composite fields like container image names (gcr.io/my-image:latest) or log strings that follow a predictable formatt.

Syntax

splitString(field=<field_name>, by="<delimiter>", index=<number>) [AS alias_name]

Parameters

ParameterDescriptionRequired/Optional
fieldThe field that contains the string you want to split.Required
byThe delimiter used to split the string.Required
indexPosition to extract.Required
AS alias_nameOptional alias for the result of the function.Optional

Default Behaviors

  • If index is not specified, it defaults to 0.
  • If alias is not specified, the default alias_name is <field_name>[<index>].

Functionality behaviour

  • If the condition matches, the extracted value will be appended to the alias_name key. If not, the value remains empty.
  • If the field do not exist, alias_name key will be created and assigned an empty value.
  • If the alias_name already exists, the value will be replaced with the new transformed data.
  • A Lookup Table is optional—if selected, matching conditions will ensure relevant values are added from the table.

Examples

  • Extract the first part of the image (default index=0)
      splitString(field=kubernetes_container_image, by=':')
      Alias: kubernetes_container_image[0]
  • Extract the second part with a custom alias
      splitString(field=kubernetes_container_image, by=':', index=1) AS image_tag

SplitRegex Function

The splitRegex function works like splitString, but uses a regular expression pattern for splitting. This is ideal when the delimiter is not fixed or is more complex.

Syntax

splitRegex(field=<field_name>, by='<regular expression>', index=<number>) [AS alias_name]

Parameters

ParameterDescriptionRequired/Optional
fieldThe field to split.Required
regexRegular expression pattern to split on.Required
indexPosition to extractRequired
AS alias_nameOptional alias for the result of the function.Optional

Default Behaviors

  • If index is not specified, it defaults to 0.
  • If alias is not provided, the default alias is <field_name>[<index>].

Functional Behaviors

  • If the condition matches, the extracted value will be appended to the alias_name key. If not, the value remains empty.
  • If the field do not exist, alias_name key will be created and assigned an empty value
  • If the alias_name already exists, the value will be replaced with the new transformed data.
  • A Lookup Table is optional—if selected, matching conditions will ensure relevant values are added from the table.

Examples

  • Default index and alias (splits on whitespace)
	splitRegex(field=log, by='\\s+')
    Creates alias: log[0]

  • Custom index with default alias
	splitRegex(field=log, by='\\s+', index=2)
  Creates alias: log[2]

  • Custom index with custom alias
		splitRegex(field=message, by='[-_:]', index=3) AS segment

regexExtract Function

The regexExtract function identifies and extracts specific parts of a string using a regular expression that contains named groups.

Syntax

regexExtract(field=<field_name>, regex="<regex_with_named_groups>")

Parameters

ParameterDescriptionType
fieldSpecifies the name of the field to which the regex will be applied.String
regexA regular expression that includes only one named capturing groups.String

Default Behaviors

  • If the regular expression does not match, the function will still append named group key, but the value will be empty string.
  • Only capturing group defined with explicit name is extracted.

Functional Behaviors

  • If the condition matches, the extracted value is appended with corresponding named group key.
  • If the field do not exist, named groups key is created but with empty value.
  • A Lookup Table is optional. If selected, it uses matching conditions to add relevant values from the table.

Examples

regexExtract(field=“date”,regex="\d{4}-(?P<month>\d{2})-\d{2}")
Extracts month from date