{"id":105543,"date":"2025-03-12T12:51:37","date_gmt":"2025-03-12T04:51:37","guid":{"rendered":"https:\/\/version-2.com\/?p=105543"},"modified":"2025-03-12T15:01:40","modified_gmt":"2025-03-12T07:01:40","slug":"graylog-parsing-rules-and-ai-oh-my","status":"publish","type":"post","link":"https:\/\/version-2.com\/en\/2025\/03\/graylog-parsing-rules-and-ai-oh-my\/","title":{"rendered":"Graylog Parsing Rules and AI Oh My!"},"content":{"rendered":"
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

<\/p>

In the log aggregation game, the biggest difficulty you face can be setting up parsing rules for your logs.\u00a0To qualify this statement: simply getting log files into Graylog is easy. Graylog also has out-of-the-box parsing of a wide variety of common log sources, so if your logs fall into one of the many categories of log for which there is either a dedicated Input; a dedicated Illuminate component; or that uses a defined Syslog format; then yes, parsing logs is also easy.<\/p>

\u00a0<\/p>

The challenge arises when you have a log source that does not neatly fall into one of these parsed out-of-the-box categories. A Graylog Raw\/Plaintext<\/strong> input will accept just about any log format you can find, so getting the message into Graylog without parsing isn\u2019t hard.<\/p>

The difficulty is usually then turning your message from being a block of raw text that looks like this:<\/p>

\"\" <\/picture><\/p>

\u00a0<\/p>

Into a useful array of fields that can be searched and aggregated, like this:<\/p>

\"\" <\/picture><\/p>

It is difficult to provide a step by step process on how to parse a log message. Log messages do not obligingly follow a widely agreed-upon format. Developers often make up their own log formats on the fly, and don\u2019t necessarily do so with a lot of thought to how easy it will be to parse later. It follows that the process of breaking log messages down into fields is usually bespoke. It is a common joke in the field that even as technology gets better, parsing data that can be given in a wide array of different formats \u2013 in particular, timestamps \u2013\u00a0 remains very challenging.<\/p>

<\/picture><\/p>

\u00a0<\/p>

Since there is no one-size-fits-all approach, and we understand that you are too good-looking and busy for an exhaustive manual on every single approach to parsing, this guide will instead just try to provide useful quick examples and links to the primary methods of parsing logs. We will assume in all the subsequent examples, that the text that needs parsing is in the $message.message<\/strong> field \u2013 when lifting Pipeline rules from this guide, remember to replace this field in the code block with the field from which you are trying to parse text.<\/p>

1. Look for Delimiters<\/h3>

Fields that are consistently<\/em> separated by a delimiter \u2013 a comma, a pipe, a space \u2013 are very easy to parse.For example, the message:<\/p>

Graylog 100 awesome<\/code><\/pre>
Let\u2019s say this message lists a software; its review score; and one word review summary. The following pipeline rule will parse named fields out of the contents of $message.message<\/strong> (eg. the message field), delimited by a \u201c \u201c (a space). Changing the character within those speech marks allows you to delimit by other characters. The fields are extracted (and so named) in the order they appear.<\/p>
Rule \"Parse fields from message\"\nwhen   \ntrue\nthen\n    let pf = split(\n           pattern: \" \",\n           value: to_string($message.message)\n           );\nset_field(\"fieldname_1\",pf[0]);\nset_field(\"fieldname_2\",pf[1]);\nset_field(\"fieldname_3\",pf[2]);\n\nend<\/code><\/pre>
For example, if the message field is currently \u201cGraylog 100 awesome\u201d, this rule would create three new fields with the current values:<\/p>
fieldname_1:<\/strong> \u201cGraylog\u201d<\/p>
fieldname_2:<\/strong> \u201c100\u201d<\/p>
fieldname_3:<\/strong> \u201cawesome\u201d<\/p>
Very easy!<\/p>
We can also change the delimiter to be \u201c,\u201d or \u201c, \u201c or \u201c|\u201d as needed by changing the value in the pattern <\/strong>field. Now, sometimes a message is very nearly consistently<\/em> separated by a delimiter, but there are some annoying junk characters messing the parsing up. For those cases, here is an example of the same pipeline rule, but which first removes any annoying square bracket characters from the message, before then parsing it into space delimited fields.<\/p>
rule \"Parse fields from message\"\nwhen   \ntrue\nthen\n\n    let cleaned = to_string($message.message);\n    let cleaned = regex_replace(\n\n           pattern: \"^\\\\[|\\\\]$\",\n           value: cleaned,\n           replacement: \"\"\n   );\n    let pf = split(\n           pattern: \" \",\n           value: to_string(cleaned)\n           );\nset_field(\"fieldname_1\",pf[0]);\nset_field(\"fieldname_2\",pf[1]);\nset_field(\"fieldname_3\",pf[2]);\n\nend<\/code><\/pre>
This technique of \u201ccleaning\u201d values from messages before parsing can of course be copy-pasted to act before any other parsing method.<\/p>
2. Look for Key Value Pairs<\/h3>
Messages that consist of a list of key value pairs are also very easy to parse.
For example, the message:<\/p>
fieldname_1=graylog fieldname_2=100 fieldname_3=awesome<\/code><\/pre>
Key Value Pairs is also the extraction method you would employ if the contents of $message.message (eg. the message field) looked like this:<\/p>
\u201cfieldname_1\u201d=\u201dgraylog\u201d \u201cfieldname_2\u201d=\u201d100\u201d \u201cfieldname_3\u201d=\u201dawesome\u201c<\/code><\/pre>
Or like this:<\/p>
fieldname_1=\u2019graylog\u2019,fieldname_2=\u2019100\u2019,fieldname_3=\u2019awesome\u2019 Or like this:\u201cfieldname_1\u201d,\u201dgraylog\u201d \u201cfieldname_2\u201d,\u201d100\u201d \u201cfieldname_3\u201d,\u201dawesome\u201c<\/code><\/pre>
Any consistent format that lists a field name followed by a value is a good target for this parsing approach.<\/p>
There is a nice Graylog Blog post that talks about Key Value Pair extraction in great detail here<\/a> and documentation on using the function here<\/a>. For the reader who is too executive to have time to read a whole blog post right now, here is a pipeline rule that would parse that last example (observe that we are trimming the \u201c characters from both the key and values, and that \u201c has to be escape-character-ed to be \\\u201d):<\/p>
rule \u201ckey_value_parser\u201d\n\nwhen\ntrue\nthen\nset_fields(\n   fields:key_value(\n   value: to_string($message.message),\n   trim_value_chars: \"\\\"\",\n   trim_key_chars:\"\\\"\",\n   delimiters:\" \",\n   kv_delimiters:\",\"\n)\n);\nend<\/code><\/pre>
This rule would again create three new fields with the current values:<\/p>
fieldname_1:<\/strong> \u201cGraylog\u201d
fieldname_2:<\/strong> \u201c100\u201d
fieldname_3:<\/strong> \u201cawesome\u201d<\/p>
3. Look for JSON Format<\/h2>
JSON formatted messages are easily recognized from their structured organization of brackets and commas. JSON logs work nicely with Graylog, since the format provides not only the values but also the field names. Graylog can parse JSON format logs very simply using JSON flattening, which is detailed in the Graylog documentation here<\/a>.<\/p>
If we take the below JSON message as an example:<\/p>
{\n   \"type\": \"dsdbChange\",\n   \"dsdbChange\": {\n       \"version\": {\n           \"major\": 1,\n           \"minor\": 0\n       },\n       \"statusCode\": 0,\n       \"status\": \"Success\",\n       \"operation\": \"Modify\",\n       \"remoteAddress\": null,\n       \"performedAsSystem\": false,\n       \"userSid\": \"S-1-5-18\",\n       \"dn\": \"DC=DomainDnsZones,DC=XXXXX,DC=XXXX,DC=com\",\n       \"transactionId\": \"XXXX-XXXX-XXXX-XXXX\",\n       \"sessionId\": \"XXXX-XXXX-XXXX-XXXX\",\n       \"attributes\": {\n           \"repsFrom\": {\n               \"actions\": [{\n                   \"action\": \"replace\",\n                   \"values\": [{\n                       \"base64\": true,\n                       \"value\": \"SOMELONGBASE64ENCODEDVALUE\"\n                   }]\n               }]\n           }\n       }\n   }\n}<\/code><\/pre>
We can parse this effortlessly with a generic JSON parsing Pipeline Rule, below:<\/p>
rule \"JSON FLATTEN\"\nwhen\n   true\nthen\n   let MyJson = flatten_json(value: to_string($message.message), array_handler: \"flatten\", stringify: false);\n   set_fields(to_map(MyJson));\nend<\/code><\/pre>
This will parse all the fields out of the JSON structure, fire and forget.<\/p>
<\/picture><\/p>
4. Look for a consistent format for Grok<\/h2>
OK, so your logs don\u2019t follow a format that Graylog can parse out-of-the-box, are not consistently delimited, are not set up in key value pairs, are not in a JSON format. But the format is at least consistent, even if the way the fields are broken up maybe isn\u2019t. There is a structure here that we can parse using Grok. For example, the message:<\/p>
2023-02-22T09:29:22.512-04:00 \u00a0 XXX.XXX.XXX.XXX\u00a0 <179>50696: Feb 22 13:29:22.512: %LINK-3-UPDOWN: Interface GigabitEthernet1\/0\/11, changed state to down<\/p>
This log format is all over the place with delimitation of fields, but there is still a consistent pattern of fields we can see: timestamp, ip_address, priority, process_id, event_timestamp, interface_name, interface_state. <\/strong>In this situation, the easiest way to extract these fields is to use Grok<\/a>. You can read more about using Grok within a Pipeline Rule in the Graylog documentation here<\/a>.<\/p>
Grok might look a bit intimidating, but it\u2019s actually pretty easy once you get started. Online Grok de-buggers, such as this one<\/a>, are your best friend when writing a Grok rule. The key to writing Grok is to focus on capturing one word at a time before trying to capture the next, and to remember that whitespace \u2013 including trailing whitespace, which often catches people out \u2013 is included in the pattern.<\/p>
Here is the Grok to parse this message:<\/strong><\/p>
%{TIMESTAMP_ISO8601:timestamp}\\s+%{IPORHOST:ip_address}\\s+<%{NUMBER:priority}>%{NUMBER:process_id}: %{MONTH:month}\\s+%{MONTHDAY:day}\\s+%{TIME:time}: %{GREEDYDATA:interface_name}: %{GREEDYDATA:interface_state}<\/p>
Seen here in the Grok debugger https:\/\/grokdebugger.com\/<\/a> in which it was written:<\/p>
<\/p>
\u00a0<\/p>
Once you have a Grok pattern that works \u2013 and check it against multiple examples of the log message, not just on one, to make sure it works consistently<\/em> \u2013 the next step is to convert your Grok pattern into a Graylog Pipeline Rule. Note that all escape characters within your Grok string need to be prefaced with a \\, including \u201c\\\u201d.<\/p>
Here is the pipeline rule for parsing the message field using this Grok rule:<\/strong><\/p>
rule \"Parse Grok\"\nwhen\n   true\nthen\nlet MyGrok = grok(\n   Pattern: \"%{TIMESTAMP_ISO8601:timestamp}\\\\s+%{IPORHOST:ip_address}\\\\s+<%{NUMBER:priority}>%{NUMBER:process_id}: %{MONTH:month}\\\\s+%{MONTHDAY:day}\\\\s+%{TIME:time}: %{GREEDYDATA:interface_name}: %{GREEDYDATA:interface_state}\",\n   value: to_string($message.message),\n   only_named_captures: true\n);\nset_fields(\n   fields: MyGrok\n);\nend<\/code><\/pre>
5. Nothing is consistent? Time for Regex<\/h3>
If the field you need to extract from your data is really inconsistently placed, and none of these techniques are useful, then it\u2019s probably time to write some Regex.<\/p>
Regex can be used in Pipeline Rules much the same as Grok, though it is better suited to scalpelling out a single tricky field than trying to parse a whole message into fields. There is a Graylog Documentation page on using Regex in Pipeline Rules here<\/a>. Regex is especially useful when capturing errors or stacktraces, which can blow out to many lines of text and otherwise confuse your parsers.<\/p>
For example, the message:<\/strong><\/p>
26\/03\/2023 08:03:32.207 ERROR:  Error in EndVerifySealInBatch()Rep.dingo.Library.Serialisation.dingoHelperException: The exception has occured in one of the dingo Helper classes: ISL_LINK                \nServer stack trace:\n   at Rep.dingo.Library.Serialisation.DataFrame.VerifySeal(dingoSecurity2 itsSecure, Boolean dyeISRN, Byte[]& native, shipmentType shipmentType)\n   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.AsyncVerifySeal(Boolean decryptIsrn, DataFrame df, Byte[]& dfNative)\n   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)\n   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)\nException rethrown at [0]:\n   at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)\n   at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)\n   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.AsyncVerifySealDelegate.EndInvoke(Byte[]& dfNative, IAsyncResult result)\n   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.EndVerifySealInBatch()<\/pre>
If you want to capture the first 3 words after the first occurrence of \u201cERROR\u201d in your log message, you could use a Regex rule.<\/p>
We would highly recommend the free online Regex tool available at https:\/\/regex101.com\/<\/a> for the purposes of composing your Regex.<\/p>
In this example, the Regex rule would be: [E][R][R][O][R].\\s+(\\S+\\s\\S+\\s\\S+)<\/p>
This would capture the value \u201cError in EndVerifySealInBatch()Rep.dingo.Library.Serialisation.dingoHelperException:\u201d<\/p>
\u00a0<\/p>
<\/picture><\/p>
Once your Regex rule is working in https:\/\/regex101.com\/<\/a> then it is time to put it into a Graylog Pipeline Rule. Note that all escape characters within your Regex string need to be prefaced with a \\, including \u201c\\\u201d.Here is the Pipeline Rule for capturing the first 3 words after the first occurrence of \u201cerror\u201d in the message field using this Regex rule:<\/p>
rule \"Regex field extract\"\nwhen\ntrue\nthen\n let MyRegex = regex(\"[E][R][R][O][R].\\\\s+(\\\\S+\\\\s\\\\S+\\\\s\\\\S+)\", to_string($message.message));\n set_field(\"MyFieldname_1\", x[\"0\"]);\n\nend<\/code><\/pre>
This rule would create a new field with the current value:<\/p>
MyFieldname_1:<\/strong> \u201cError in EndVerifySealInBatch()Rep.dingo.Library.Serialisation.dingoHelperException:\u201d<\/p>
Very cool!<\/p>
6. Stuck? Look for Extractors in the Graylog Marketplace<\/h2>
Extractors are a legacy feature of Graylog, providing an interface for extracting fields from messages hitting an input using Regex. We recommend against creating your parsing rules using the Extractors interface, as it is rather fiddly and outdated. You can read more about Extractors and how they work in the legacy Graylog Documentation here<\/a>.<\/p>
Extractors have been around for many years, so there is A merit to continuing to use this functionality: the Graylog Open community has created a lot of useful Extractor Parsing rules over the years, and these are all available to download from the Graylog Marketplace<\/a>.<\/p>
If you require a parser for the complex logs of a common<\/strong> hardware device or software suite, it can be worth checking if the Graylog Open Community has already produced them. Work smarter not harder: downloading someone else\u2019s ready-made parser is often quicker than writing your own \ud83d\ude0e<\/p>
Be mindful however<\/u> that this option is presented late in this guide because it is something of a last resort. Extractors are a vestigial mechanism, and being community written and maintained, carry no guarantee on being correct, up to date, or even working. There will often be a bit of TLC required to get such content working and up to date.<\/p>
7. Stuck? ChatGPT can write both Graylog Pipeline Rules and GROK\/Regex Parsing\u2026 sometimes.<\/h2>
Technology is a beautiful thing! ChatGPT<\/a>, the AI that needs no introduction, can write Graylog Pipeline rules. It can also write GROK or Regex parsers \u2013 just paste in your log sample and ask nicely. This is really useful in theory and can often point you in the right direction, but be warned that in practice, the AI will make various mistakes. Rather than entering your requests into ChatGPT directly, we recommend checking out this useful Community tool that leverages OpenAI\u2019s GPT API and an extensive prompt designed to improve results.<\/p>
https:\/\/pipe-dreams.vercel.app\/<\/a><\/p>
\u00a0<\/p>
<\/picture><\/p>
AI is far from perfect at these tasks at this stage, but still very useful \u2013 particularly at showing syntax and structure. Please note the tabs on the top left that switch between Pipeline and GROK parsing modes.<\/p>
<\/picture><\/p>
\u00a0<\/p>
8. I am still stuck \u2013 Parsing logs is hard!<\/h2>
Yes, parsing logs can be hard. If you really get stuck, and you still can\u2019t parse your logs, there are several avenues for assistance you might pursue.<\/p>