Introduction
Log enrichment allows you to enhance your log data by adding or updating attributes based on specific rules. This is done using Lookup Tables, Source Tags, AuthZ Tags, Transform & Lookup Tag, and Enrichment Configuration under Logs Configuration. The enriched data provides more context and value, making it easier to analyze and act upon logs.
To create an alert definition:
- Navigate to Infrastructure > Logs.
- On the left side of this page, click the Menu icon.
- From the MY LOGS VIEWS page, under QUICK LINKS, select Logs Configuration.The configuration page is displayed.
- From the configurations page, select the Enrichment tab.The enrichment details page is displayed with the two sub-tabs:
- Lookup Tables
- Enrichment Configuration
Enrichment Configuration
After setting up a Lookup Table, you can create Enrichment Configs that specify how these attributes are applied to your logs.
To create a configuration:
- Click the ENRICHMENT CONFIGS tab.
- Click +ADD.The ENRICHMENT CONFIGURATION page is displayed.
- On the ENRICHMENT CONFIGURATION page, provide below information:
- GENERAL DETAILS:
- Name: Enter a name for the configuration.
- Priority Order: Set the priority for this configuration. This determines the order in which configurations are applied if multiple rules match.
- Lookup Table: Click the Lookup Table button, and in the Lookup Table dropdown, select the lookup table for which you want to create a configuration.
Note
You can only add one lookup table for a configuration. - Source Tags: Click the Source Tags button, page displays Enrichment Rules section with enables to enter maching conditions.
- AuthZ Tags: Click the AuthZ tags button to enter the matching conditions and when all conditions are met, the selected AuthZTags are added to the logs.
- Transform & Lookup: Click the Transform & Lookup button to refine and extract yhe log information by defining conditions and applying transformations.
- LOG FILTER CRITERIA: Click the +ADD FILTER option. Select the available attributes to define the filter criteria. This will determine which logs are enriched by this configuration.
- ENRICHMENT RULES: Define conditions and matching fields under the Enrichment Rules section. These rules specify how and when the enrichment occurs.
- LOG FIELDS: Choose the log fields that should be appended or updated when all conditions are met.
Note
You can include multiple rules within a single enrichment configuration.
- GENERAL DETAILS:
- After configuring all the necessary settings, click ADD CONFIGURATION to save your enrichment setup.
Lookup Tables
Lookup Tables are essential for enriching logs. They define the attributes that will be added or updated in your logs based on matching rules.
To create a lookup table:
Click the LOOKUP TABLES tab within the Enrichment section.
Click the +ADD.The ADD LOOKUP TABLE screen is displayed.
Enter a name for your lookup table.
Upload a CSV file containing the attributes you want to use for enrichment. The CSV file should be structured according to your enrichment requirements.
Once the name and CSV file are provided, click ADD LOOKUP TABLE to save the changes.
Source Tags
Source Tags enable identification of the originating resources for logs, especially in centralized logging environments where logs are forwarded through intermediate devices like log hosts. This feature ensures that logs maintain resource-specific traceability and are enriched with contextual data such as Department, Region, or Hostname, even when a Resource UUID is missing from the source.
To enable source Tags:
- Go to ENRICHMENT. To access ENRICHMENT tab, see
- CLick ENRICHMENT tab.
- Click the ENRICHMENT CONFIGURATION tab.
- Click +ADD.
- Click Source Tags button.
Custom Attributes for Logs with Resource UUID:
Source Tags enable identification of the originating resources for logs, especially in centralized logging environments where logs are forwarded through intermediate devices like log hosts. This feature ensures that logs maintain resource-specific traceability and are enriched with contextual data such as Department, Region, or Hostname, even when a Resource UUID is missing from the source.
You can now have the option to enable the Log Value checkbox in the Custom Attributes section. This determines which attributes are included in enriched logs.
Note
This feature is only available only Log resources and new UI.The following is the functional behavior in different Scenarios.
Scenario | Functional Behavior |
---|---|
Single Lookup Table in a Single Enrichment Configuration | Logs are enriched using one lookup table and one setup. Filters and rules extract data from the table and add relevant attributes to the logs. |
Single Lookup Table in Multiple Enrichment Configurations | One lookup table is used in multiple setups. Each setup applies its own rules and filters, enriching logs sequentially based on a defined priority order. |
Multiple Lookup Tables in Multiple Enrichment Configurations | Each setup uses its own lookup table. Logs are enriched with attributes from multiple tables, processed one at a time, in the specified priority order. |
Single Enrichment Configuration with Source Tags | Logs are enriched by applying filters to identify relevant logs, adding resource details (like IDs and custom attributes) to the matching logs. |
Enrichment Configuration with Both Lookup Table and Source Tags | Logs are enriched using a mix of lookup tables and tags. Multiple setups are processed sequentially, each adding attributes to logs based on its logic. |
AuthZ tags
AuthZ tags in logs are key-value pairs used in Role-Based Access Control (RBAC) to enforce fine-grained access control, ensuring users can only perform actions on logs based on their assigned tags.
Note
Users with the “All” AuthZ tag can perform add, update, and delete actions with full administrative control.Transform & Lookup
The Transform & Lookup tag is an enhancement in log enrichment that enables advanced data transformation and extraction. It introduces functions like splitString, splitRegex, parseJson, and regexExtract, to allowi users to refine log data efficiently.
- splitString – Splits a string based on a specified delimiter and extracts the desired part.
- splitRegex – Uses regular expressions to dynamically split a string, offering precise pattern-based segmentation.
- regexExtract – Applies regex matching to extract specific parts of a string, ensuring efficient data retrieval.
splitString Function
The splitString function is used to divide a string field into multiple parts based on a specified delimiter (e.g., :, -, _). It then extracts a specific part using a 0-based index. This is particularly useful when dealing with composite fields like container image names (gcr.io/my-image:latest) or log strings that follow a predictable formatt.
Syntax
splitString(field=<field_name>, by="<delimiter>", index=<number>) [AS alias_name]
Parameters
Parameter | Description | Required/Optional |
---|---|---|
field | The field that contains the string you want to split. | Required |
by | The delimiter used to split the string. | Required |
index | Position to extract. | Required |
AS alias_name | Optional alias for the result of the function. | Optional |
Default Behaviors
- If index is not specified, it defaults to 0.
- If alias is not specified, the default alias_name is <field_name>[<index>].
Functionality behaviour
- If the condition matches, the extracted value will be appended to the alias_name key. If not, the value remains empty.
- If the field do not exist, alias_name key will be created and assigned an empty value.
- If the alias_name already exists, the value will be replaced with the new transformed data.
- A Lookup Table is optional—if selected, matching conditions will ensure relevant values are added from the table.
Examples
- Extract the first part of the image (default index=0)
splitString(field=kubernetes_container_image, by=':')
Alias: kubernetes_container_image[0]
- Extract the second part with a custom alias
splitString(field=kubernetes_container_image, by=':', index=1) AS image_tag
SplitRegex Function
The splitRegex function works like splitString, but uses a regular expression pattern for splitting. This is ideal when the delimiter is not fixed or is more complex.
Syntax
splitRegex(field=<field_name>, by='<regular expression>', index=<number>) [AS alias_name]
Parameters
Parameter | Description | Required/Optional |
---|---|---|
field | The field to split. | Required |
regex | Regular expression pattern to split on. | Required |
index | Position to extract | Required |
AS alias_name | Optional alias for the result of the function. | Optional |
Default Behaviors
- If index is not specified, it defaults to 0.
- If alias is not provided, the default alias is <field_name>[<index>].
Functional Behaviors
- If the condition matches, the extracted value will be appended to the alias_name key. If not, the value remains empty.
- If the field do not exist, alias_name key will be created and assigned an empty value
- If the alias_name already exists, the value will be replaced with the new transformed data.
- A Lookup Table is optional—if selected, matching conditions will ensure relevant values are added from the table.
Examples
- Default index and alias (splits on whitespace)
splitRegex(field=log, by='\\s+')
Creates alias: log[0]
- Custom index with default alias
splitRegex(field=log, by='\\s+', index=2)
Creates alias: log[2]
- Custom index with custom alias
splitRegex(field=message, by='[-_:]', index=3) AS segment
regexExtract Function
The regexExtract function identifies and extracts specific parts of a string using a regular expression that contains named groups.
Syntax
regexExtract(field=<field_name>, regex="<regex_with_named_groups>")
Parameters
Parameter | Description | Type |
---|---|---|
field | Specifies the name of the field to which the regex will be applied. | String |
regex | A regular expression that includes only one named capturing groups. | String |
Default Behaviors
- If the regular expression does not match, the function will still append named group key, but the value will be empty string.
- Only capturing group defined with explicit name is extracted.
Functional Behaviors
- If the condition matches, the extracted value is appended with corresponding named group key.
- If the field do not exist, named groups key is created but with empty value.
- A Lookup Table is optional. If selected, it uses matching conditions to add relevant values from the table.
Examples
regexExtract(field=“date”,regex="\d{4}-(?P<month>\d{2})-\d{2}")
Extracts month from date