Note
All regular expressions specified in this document follow syntax mentioned at Syntax.Introduction
Supported Functionality
- Inclusion and exclusion of log files
- Parsing logs based on Regex and JSON
- Specifying custom OTLP operators using custom formatting
- Filtering based on tag
- Scrubbing/Redaction of text
Components of a log format
In OpsRamp, all collected logs are structured into three sections, making them easy to query and organize.
- Log Resource Attributes: These attributes accompany each log. They contain details about the source from which the logs are collected, such as host, file name, file path, and other sources.
- Log Record Attributes: These attributes are specific to the individual log records. They are user-defined and can vary based on the specific needs and requirements of the organization. By default, this section remains blank unless the user explicitly defines it.
- Body: This field contains the raw log line as collected unless other transformations using Regex or JSON parsers are applied. You can transform the log line to extract fields using regex or a JSON parser. The body is then structured to contain key-value pairs. Once the body contains these key-value pairs, you can specify whether these extracted fields should be treated as record or resource attributes. Any attributes and resource attributes set in this manner will be indexed within OpsRamp.
File Configurations
In Linux and Windows systems, configuration files play a crucial role in defining the behaviour of applications and services.
For OpsRamp Agent, log configuration files are in the following paths:
Linux File Configurations
Default Location:
Path: /opt/opsramp/agent/conf/log/log-config.yaml
User-Defined Config Location:
Path: /opt/opsramp/agent/conf/log.d/log-config.yaml
A user is supposed to modify the User Defined Config as per the configuration located in the /opt/opsramp/agent/conf/log.d/log-config.yaml
path. If configuration is not specified in the user-defined config file, the system collects the logs for Syslog and Journal ID using the default configuration.
Windows File Configuration
Default Location:
Path: C:\Program Files (x86)\OpsRamp\Agent\conf\log\log-config.yaml
Default Location:
Path: C:\Program Files (x86)\OpsRamp\Agent\conf\log.d\log-config.yaml
A user is supposed to modify the User Defined Config which is in the “” path. If no configuration is specified in the user-defined config file, the default configuration is used by which it collects application event error logs.
FAQ
If the User Defined Config is the only configuration required by the user, what is the purpose of the Default Config file?
- The default configuration is used concurrently with the auto-detection of applications. So, if the agent identifies any application, the log collection becomes active for those applications as well.
How can we verify the applications detected through auto-detection?
- Detected applications are recorded in the /opt/opsramp/agent/conf/log.d/log-config.yaml.sample in Linux and C:\Program Files (x86)\OpsRamp\Agent\conf\log.d\log-config.yaml.sample in Windows. This file is updated each time auto-detection occurs, allowing users to review the list of detected applications.
Note:
- All configuration modifications must be made in the User-Defined Config.
- If a User-Defined Config exists, the Default Config is entirely disregarded.
- Auto-discovery of applications is deactivated for the User-Defined Config.
Simple File Log Collection
Simple log file contains usually contains a basic text file with log messages. You can configure the log collection within OpsRamp by directly sending raw logs to the portal using only Regex or JSON parsing using Simple File log collection approach. Consider the following limitations before implementing this approach:
- The timestamps in these logs are based on when they are collected, rather than when they are generated.
- Log levels are determined using a basic string-matching technique within the log message for common log-level strings. However, this method may be unreliable if the log line contains multiple log-level substrings.
- Timestamp parsing and severity mapping fields are disregarded in this type of log collection.
inputs:
source_name:
type: file
source: ''
include:
- /var/log/application_name/log_name.log
exclude:
- ''
fingerprint_size: 1000
File Log Collection Using Regex
File log collection using Regex involves configuring the system to extract structured information from log files based on predefined patterns. For example, This section explains the process of reading and parsing an Nginx log.
inputs:
nginx_access:
type: file
source: nginx_access
include:
- /var/log/nginx/access.log
exclude:
- ''
parser_type: regex
regex: ^(?P<remote_addr>[^\s]*)\s-\s(?P<remote_user>[^\s]*)\s\[(?P<timestamp>[^]]*)]\s*"(?P<request>[^"]*)"\s*(?P<status_code>\d*)\s*(?P<body_bytes_sent>\d*)\s*"(?P<http_referer>[^"]*)"\s*"(?P<http_user_agent>[^"]*)"$
timestamp:
layout_type: strptime
layout: '%d/%b/%Y:%H:%M:%S %z'
location: UTC
Here, Regex parser is used to parse Nginx logs. This involves specifying the parser_type as Regex and providing a valid regular expression that matches the log line format of the application. Note: You can validate the regex using tools like REGEX 101.The following example explains how Regex extracts information from the sample Nginx log:Sample Nginx logs:
172.26.32.120 - - [05/Oct/2021:12:57:04 +0000] "GET /nginx_status HTTP/1.1" 200 110 "-" "Go-http-client/1.1"
172.26.32.120 - - [05/Oct/2021:12:58:04 +0000] "GET /nginx_health HTTP/1.1" 200 110 "-" "Go-http-client/1.1"
172.26.32.120 - - [05/Oct/2021:12:59:04 +0000] "GET /nginx_info HTTP/1.1" 200 110 "-" "Go-http-client/1.1"
172.26.32.120 - - [05/Oct/2021:13:00:04 +0000] "GET /nginx_status HTTP/1.1" 200 110 "-" "Go-http-client/1.1"
Regex used:
^(?P<remote_addr>[^\s]*)\s-\s(?P<remote_user>[^\s]*)\s\[(?P<timestamp>[^]]*)]\s*"(?P<request>[^"]*)"\s*(?P<status_code>\d*)\s*(?P<body_bytes_sent>\d*)\s*"(?P<http_referer>[^"]*)"\s*"(?P<http_user_agent>[^"]*)"$
As observed below, anything enclosed in (?P<group_name>regex_expression
) captures and extracts specific fields. These extracted fields can be customized using advanced configurations such as custom formatting, filtering, scrubbing, and more. These will be explained further in the Advanced Configurations section.
The subsequent step in the configuration is the timestamp field. This feature enables you to synchronize the timestamp of the log in OpsRamp with the time it was originally generated. To accomplish this, parse the time field using formats like strptime, gotime, or epoch using Regex. See Supported Timestamp Formats for more details.
File Log Collection Using JSON
You can configure the system to extract structured information from log files formatted in JSON.log-config.yaml:
eventstore:
type: "file"
source: "eventstore"
include:
- /var/log/eventstore/**/*.json
parser_type: "json"
custom_formatting: |-
[
{
"type":"move",
"from": "body.[\"@t\"]",
"to":"body.timestamp"
}
]
timestamp:
layout_type: strptime
layout: "%Y-%m-%dT%H:%M:%S.%s%z"
In the above code snippet, enables you to parse the EventStore logs, which are structured in JSON format instead of plain text logs. To complete this task, extract relevant fields from the JSON-formatted logs using JSON parser.The following code snippet is an example of an event store log parsing process:sample eventstore log:
{"@t":"2020-08-18T10:57:05.6398910Z","@mt":"SLOW BUS MSG [{bus}]: {message} - {elapsed}ms. Handler: {handler}.","@l":"Debug","bus":"MainBus","message":"BecomeShuttingDown","elapsed":102,"handler":"WideningHandler`2","SourceContext":"EventStore.Core.Bus.InMemoryBus","ProcessId":2950,"ThreadId":12}
{"@t":"2020-08-18T10:57:05.6560627Z","@mt":"SLOW QUEUE MSG [{queue}]: {message} - {elapsed}ms. Q: {prevQueueCount}/{curQueueCount}.","@l":"Debug","queue":"MainQueue","message":"RequestShutdown","elapsed":124,"prevQueueCount":0,"curQueueCount":8,"SourceContext":"EventStore.Core.Bus.QueuedHandlerMRES","ProcessId":2950,"ThreadId":12}
{"@t":"2020-08-18T10:57:05.6623165Z","@mt":"========== [{httpEndPoint}] Service '{service}' has shut down.","httpEndPoint":"127.0.0.1:1113","service":"StorageWriter","SourceContext":"EventStore.Core.Services.VNode.ClusterVNodeController","ProcessId":2950,"ThreadId":12}
The above log is transformed into the following and the @t
key is now renamed to “timestamp” which allows you to parse the timestamp.
When the JSON format is not used to parse the logs, not all fields will be parsed, resulting in incomplete information. Consequently, in the UI, the logs may not display correctly in the expanded view.
Advanced Configurations
Multiline Configuration
The multiline configuration refers to the ability to handle log entries that span multiple lines. If set, the multiline configuration block instructs the file source type to split log entries on a pattern other than newlines.The multiline configuration block must contain exactly one line_start_pattern or line_end_pattern. These are regex patterns designed to match either the beginning of a new log entry or the end of a log entry.
For example:Consider the following logs as input.sample logs:
2022/09/07 20:20:54 error 373 Exception in thread "main"
java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
2022/09/07 20:20:54 info 374 recovery sucess
2022/09/07 20:20:54 info 375 random log line
2022/09/07 20:20:54 warn 376 another random log line
Following is the configuration to parse the multiline:
log-config.yaml
inputs:
application:
type: file
include:
- "/var/log/application/*.log"
parser_type: "regex"
multiline:
line_start_pattern: ^(?P<timestamp>\d{4}\/\d{2}\/\d{2}\s*\d{2}:\d{2}:\d{2})\s*(?P<level>\w*)
When the collector attempts to read a line, it interprets everything from the matched pattern until the subsequent match as a single line.
In the above example, the first line, which spans across multiple lines, will be treated as a single line by the collector.
Severity From
The severity_from refers to the method or source from which the severity level of log entries is derived or determined. In log management, the severity level indicates the importance or severity of a log entry.log-config.yaml:
redis:
type: "file"
source: "redis"
include:
- /home/lokesh/experiments/otel_collector/test_otel_collector_integration/redis-server-tampered.log
multiline:
line_start_pattern: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)
parser_type: "regex"
regex: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)\s*(?P<level>\W)(?P<message>.*)$
timestamp:
layout_type: strptime
layout: "%e %b %H:%M:%S.%f"
severity_from: "body.lvl"
Severity Mapping
Severity mapping is the process of correlating or mapping severity levels from log entries to standardized severity levels for better organization, analysis, and prioritization of logs. It enables you to define rules or criteria for assigning severity levels to logs that lack explicit severity information.
Following is the example of Redis logs that lack proper log levels:
18842:M 27 Jun 10:55:21.636 * DB saved on disk
18842:M 27 Jun 10:55:21.636 * Removing the pid file.
18842:M 27 Jun 10:55:21.637 # Redis is now ready to exit, bye bye...
24187:C 27 Jun 10:55:21.786 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
You can configure the severity mapping as follows:
redis:
type: "file"
source: "redis"
include:
- /home/lokesh/experiments/otel_collector/test_otel_collector_integration/redis-server-tampered.log
multiline:
line_start_pattern: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)
parser_type: "regex"
regex: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)\s*(?P<level>\W)(?P<message>.*)$
timestamp:
layout_type: strptime
layout: "%e %b %H:%M:%S.%f"
severity_priority_order: [ "error", "fatal", "warn", "info", "debug", "trace" ]
severity_mapping:
warn: [ "#" ]
info: [ "*" ]
debug: [ "." ]
trace: [ "-" ]
error: []
fatal: []
In the above scenario, the mapping of characters to severity levels is as follows. This mapping dictates how log entries containing these characters will be represented in terms of severity levels.
#
is mapped to WARN*
is mapped to INFO.
is mapped to DEBUG-
is mapped to TRACE
If the severity levels map is blank, it will be ignored. We demonstrate the basis of the mapping by displaying it as a log attribute named Custom Severity.
Encoding
Encoding is a method or scheme used to represent characters or data within log entries. It is useful when the log is not following standard UTF-8 encoding.
application_name:
type: "file"
source: "application_name"
encoding: "utf-8"
include:
- /var/log/applicaton/*.log
The valid encoding types include:
Key | Description |
---|---|
nop | No encoding validation. Treats the file as a stream of raw bytes |
utf-8 | UTF-8 encoding |
utf-16le | UTF-16 encoding with little-endian byte order |
utf-16be | UTF-16 encoding with big-endian byte order |
ascii | ASCII encoding |
big5 | The Big5 Chinese character encoding |
User Defined Constant Labels
The User Defined Constant Labels are the custom labels or tags that users can define and assign to log entries or events. You can configure up to five resource labels. If more than five labels are specified, only the first five labels are considered, and any additional labels are ignored.
application:
type: "file"
include:
- /var/log/application/*.log
labels:
label_key_1: "lable_value_1"
label_key_2: "lable_value_2"
label_key_3: "lable_value_3"
Setting ParseTo
The Setting ParseTo specifies the target location where the parsed log fields should be stored or mapped within a log management system. Using ParseTo can introduce undefined behaviour and potentially disrupt functionalities such as level and timestamp parsing.It is recommended to exercise caution when employing ParseTo and only utilize it if there is a clear benefit, especially when combined with custom formatting.When using a Regex or JSON parser, you can specify the location of parsed result within the log management system. Following are three destinations:
- Body - Parsed results are stored within the body of the log entry itself.
- Attributes - Parsed results are stored as attributes associated with the log entry.
- Resource - Parsed results are stored as resource attributes, which are specific to the resource or entity represented by the log entry.
application:
type: "file"
include:
- /var/log/application/*.log
parser_type: "regex"
regex: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)\s*(?P<level>\W)(?P<message>.*)$
parse_to: "body"
timestamp:
layout_type: strptime
layout: "%e %b %H:%M:%S.%f"
Set Values
Setting values in record attributes and resource attributes from parsed fields involves extracting information from log entries using parsers like regex or JSON and then assigning this extracted information to attributes associated with the log entry or the resource being monitored.This is useful if the you want to set certain parsed fields as either record or resource attributes.
application:
type: "file"
include:
- /var/log/application/*.log
parser_type: "regex"
regex: ^(?P<pid>\w*):(?P<role>\w*)\s*(?P<timestamp>\d*\s*\w*\s*\d*\s*\d*:\d*:\d*.\d*)\s*(?P<level>\W)(?P<message>.*)$
timestamp:
layout_type: strptime
layout: "%e %b %H:%M:%S.%f"
attributes: [ "role", "pid" ]
resource_attributes: [ "role" ]
In the above example, the preference for role
and pid
to serve as record attributes is indicated2, while role
is designated as a resource attribute as well.
It is important to note that anything defined as an attribute or resource attribute will be indexed within OpsRamp.
Custom Formatting
If you require full control over the parsing process, you must employ custom formatting to tailor log entries according to their needs. The Custom Formatting field accepts a list of operators, from which information can be extracted.
See OpenTelemetry collector operators for more details.
Filter Logs
Filtering logs involves selecting specific log entries from a larger set based on defined criteria. To filter logs based on specific criteria, you must define the rules to filter in the filtering section for their respective input.filtering syntax:
filters:
- key: ""
include: ""
- key: ""
exclude: ""
- key: ""
include: ""
- attribute_type: "resource"
key: ""
exclude: ""
.
.
.
Tag Name | Description |
---|---|
exclude | Removes the records that match the specified regular expression pattern. |
include | Keeps the records that match the specified regular expression pattern. |
key | Tag for which the respective filtering rule must be applied. |
attribute_type | Accepted values are body, attributes, and resource (defaults to body if the field is ignored). |
The filter rule must have either include or exclude but not both at the same time. It is recommended to use either include or exclude within a single filter rule.However, if there is a need to incorporate both include and exclude conditions, it is advised to create separate filter rules and chain them together. This chaining allows you to be more precise and ensures flexible filtering criteria.
- key: ""
include: ""
- key: ""
exclude: ""
Filtering occurs in the sequence in which the rules are specified. If the attribute_type is designated as body and the key is left empty, the filtering process assumes that no regex or JSON parsers are applied and attempts to filter directly on the body of the log entry.
Note: When using this approach, as it may result in errors if the body is not a raw string.
Masking/Scrubbing Logs
Masking or scrubbing logs involves obscuring sensitive or personally identifiable information (PII) found within log entries. This practice aims to safeguard privacy and maintain compliance with data protection regulations. To mask any sensitive data present in the logs, you must define masking rules within the masking section corresponding to their respective log inputs.
Masking Syntax:
masking:
- text: ""
placeholder: ""
- text: ""
placeholder: ""
Tag Name | Description |
---|---|
text | Regular Expression for the text that needs to be masked. |
placeholder | String which will be used for replacing the text specified. |
Multiple Regexes for Single Source
Note
This functionality can only be accessed with agent versions equal to or greater than 17.1.0.The Multiple Regexes for Single Source refers to the use of multiple regular expressions (regexes) to parse and extract information from log files. In cases where a log file contains entries with multiple formats, you can utilize the following configuration:
inputs:
source_name:
type: file
include:
- /var/log/file1.log
- /var/log/file2.log
multiline:
line_start_pattern: ""
parser_type: "regex"
multi_regex: # specify as many regex expression as required (only works when parser_type is regex)
- regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}) (?P<level>[A-Z]*) (?P<message>.*)$'
timestamp:
layout_type: strptime
layout: "%b %e %H:%M:%S"
severity_from: "body.lvl"
severity_priority_order:
["error", "fatal", "warn", "info", "debug", "trace"]
severity_mapping:
warn: ["#"]
info: ["*"]
debug: []
trace: []
error: []
fatal: []
filters:
- attribute_type: "body"
key: ""
include: ""
- key: ""
exclude: ""
masking:
- text: ""
placeholder: ""
attributes: []
resource_attributes: []
custom_formatting: |-
[
{
"type": "copy",
"from": "body.message",
"to": "body.duplicate"
}
]
- regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}) (?P<message>.*)$'
Point to remember:
- Ensure that the multiple line start, or end pattern is configured to match all the regex patterns defined for the source, given that log line splitting occurs before regex matching.
- All the specified regex patterns are compared in the order in which they are defined, and the first one that matches is considered, with all subsequent patterns being ignored.
Additional File Type Settings
Fingerprint Size
The parameter fingerprint_size determines the number of bytes used from the beginning of a file to uniquely identify it across polls. Since this is the only criterion for identifying file uniqueness, it is essential to adjust this value if the initial 1000 bytes of multiple files are identical.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Fingerprint Size
#########################################################################################################
fingerprint_size: 1000
#########################################################################################################
Poll and Flush intervals
Poll and flush intervals dictate how frequently data is collected and transmitted in systems.
- Poll Interval: This refers to the frequency at which a system retrieves or collects data from a source.
- Flush Interval: This determines how often collected data is transmitted or flushed to its destination.
For example,
- poll_interval: It is the duration to check log files for new changes.
- force_flush_period: Determines the period elapsed since the last data read from a file. Once this interval is completed, the currently buffered log will be forwarded to the pipeline for processing. A setting of 0 disables forced flushing, a configuration usually not necessary in the majority of cases.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Poll and Flush Settings
#########################################################################################################
poll_interval: 200ms
force_flush_period: 500ms
#########################################################################################################
Retry on Failure
The Retry on Failure refers to the capability of log collection systems to automatically retry sending log data to a central repository or processing pipeline when a failure occurs during the initial attempt.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Retry On failure Settings
#########################################################################################################
retry_on_failure:
enabled: false
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 5m
#########################################################################################################
Max Concurrent Files
When processing logs, the agent restricts the maximum number of files it can open at any given point in time. This setting determines that value on a per-source basis, with a default of 5 files. It is advisable to proceed with caution when considering increasing this number.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Max Concurrent Files Settings
#########################################################################################################
max_concurrent_files: 5
#########################################################################################################
Refresh Interval
This configuration parameter specifies the interval at which the list of files is updated for the specified paths to monitor in the include configuration.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Refresh Interval
#########################################################################################################
refresh_interval: 60s
#########################################################################################################
Ordering Criteria
The agent selectively collects a limited number of files based on their modified time, adhering to a predetermined upper limit.
inputs:
source_name:
type: "file"
include:
- "/var/log/application_name/log_name.log"
#########################################################################################################
# Ordering Criteria
##########################################################################################################
ordering_criteria:
top_n: 5
max_time: 2h
#########################################################################################################
Note
Starting from agent version 17.1.0-1 and above, the agent automatically sets ordering criteria for all files by default.type:
file, it defaults to: ordering_criteria:
top_n: 5
max_time: 2h
In the case of Kubernetes and Docker, these limits are set to:
ordering_criteria:
top_n: 500
max_time: 2h
To understand the function of these settings, let us examine the following example:
Suppose we have an application that generates logs in the directory /var/log/application/*.log
, we utilize the following configuration to collect the logs from that path:
inputs:
application:
type: "file"
include:
- "/var/log/application/*.log
-rw-r--r-- 1 root wheel 0 Apr 14 16:35 file_1.log
-rw-r--r-- 1 root wheel 0 Apr 14 16:45 file_2.log
-rw-r--r-- 1 root wheel 0 Apr 14 16:55 file_3.log
-rw-r--r-- 1 root wheel 0 Apr 14 17:05 file_4.log
-rw-r--r-- 1 root wheel 0 Apr 14 17:15 file_5.log
-rw-r--r-- 1 root wheel 0 Apr 14 17:25 file_6.log
-rw-r--r-- 1 root wheel 0 Apr 14 17:35 file_7.log
-rw-r--r-- 1 root wheel 0 Apr 14 18:00 file_8.log
-rw-r--r-- 1 root wheel 0 Apr 14 18:30 file_9.log
-rw-r--r-- 1 root wheel 0 Apr 14 19:36 file_10.log
Case 1: If the present time is 20:00The agent will only watch the following files:
- file_10.log
- file_9.log
- file_8.logThis is because the max_time is set to 2hr by default and any file that has a modified time that is older than 2hr is ignored.
Case 2: If the present time is 19:40The agent will only watch the following files:
- file_10.log
- file_9.log
- file_8.log
- file_7.log
- file_6.logHere, the other logs which are still under 2 hrs ignored since we hit the top_n limit of 5 by default.These defaults ensure that the agent does not try to watch historic files which never get updated and are only used for bookkeeping. Setting these values to a high number is usually not advised since it comes with a performance penalty.