Introduction
Custom Formatting allows users to modify the log content by chaining different operations. Below are a few example scenarios where this custom formatting proves advantageous.
Parse JSON embedded within a parsed field
This section refers to the process of extracting and interpreting JSON data that is nested within a larger text field that has already been parsed using regular expressions (regex).
Scenario 1: JSON is a part of the message
In cases where the JSON data is included as part of a parsed message or another group, as explained in the example below, we can employ the following approach:logfile sample:
2024-02-25 [1024] Info something important happened with details {"key1": "value1", "key2": "value2"} proceeding
log-config.yml
inputs:
test:
type: file
source: normal_tst
include:
- /var/log/normal.log
parser_type: "regex"
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2})\s\[(?P<pid>\d+)\]\s(?P<level>[^\s]*)\s(?P<message>.*)$'
custom_formatting: |-
[
{
"type": "regex_parser",
"parse_from": "body.message",
"parse_to": "body",
"regex": "(?P<parsed_json>{[^}]*})"
},
{
"type": "json_parser",
"parse_from": "body.parsed_json",
"parse_to": "body"
},
{
"type": "remove",
"field": "body.parsed_json"
}
]
Having matched the log line, we now have a JSON object: {"key1": "value1", "key2": "value2"}
. To extract and process this data, you need to employ few custom formatting operators.To get the JSON part from the message, you can use (?P<parsed_json>{[^}]*}) regular expression:
You can specify this regex match using the regex_parser
operator as follows:
After extracting the JSON using the regex_parser
operator, you can further parse its keys and values using the JSON parser by specifying the following:
Finally, to eliminate redundant entries in logs, we remove the parsed_json
regex capture group, which was initially used to match the JSON. This can be achieved by specifying the following:
Scenario 2: The entire message is a JSON
If the matched group is a valid JSON structure, you can proceed as follows:
logfile sample
2024-02-25 [1026] Info {"user_name":"vivek","message":"There was an error processing the request","request_id":"1234567890","user_id":"abcdefghij"}
log-config.yml
inputs:
test:
type: file
source: normal_tst
include:
- /var/log/normal.log
parser_type: "regex"
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2})\s\[(?P<pid>\d+)\]\s(?P<level>[^\s]*)\s(?P<message>.*)$'
custom_formatting: |-
[
{
"type": "json_parser",
"parse_from": "body.message",
"parse_to": "body"
}
]
Since the entire message is already a well-formed JSON, you can directly use the json_parser
to parse the JSON without the need for prior regex matching.
The resulting output will be displayed in the portal as follows: