POC: Writing a Log Source Extension

Summary

A Universal DSM, or UDSM, can be used to integrate log sources that do not have an official

DSM. A Log Source Extension, or LSX, is then applied to the UDSM to provide the logic for

parsing the logs. The LSX is based on Java regular expressions and can be used against any log

protocol, e.g. Syslog, JDBC, LFPS, etc. Values can be extracted from the logs and mapped to all

common fields within QRadar. The unsupported log source can then be fully utilized within

QRadar.

Obtaining A Log Sample

Creating The Log Source

The first step in building a Universal DSM is to create the log source within QRadar. This

prevents the logs from being automatically classified and allows you to export the logs for

review.

1. From the Admin tab, create a new source using the Log Sources icon.

2. Specify the Log Source Name.

3. Specify Log Source Type as Universal DSM.

4. Specify the protocol that will used for Protocol Configuration. This is the method that

QRadar will obtain the logs from the unsupported log source.

5. Enter the IP address or hostname of the unsupported log source as the Log Source

Identifier.

6. Click Save to save the new log source. Close the window.

7. Click Deploy Changes from the Admin tab.

Alternatively, log samples can be used if provided by a customer. This is typically done before

the start of a POC. The log samples must be in the same format as the logs being sent to

QRadar. Typically this is only possible if the log samples are from Syslog or FTP/SFTP. Other

methods, e.g. screenshots of a mainframe screen or a description of an SQL schema, do not

provide logs in a format that can be utilized by QRadar.

Exporting The Logs

The next step is to export the logs that were obtained from the previous step. Typically you will

want a significant amount of logs for review. Depending on the EPS rate of the unsupported, it

may take several hours to obtain a comprehensive log sample.

1. Create a search for exporting the logs.

1. From the Events tab, select Search -> Edit Search.

2. Specify the Time Range as an adequate amount of time from when the log source

was created, e.g. 6 hours.

3. Under Search Parameters, select Log Source Equals and specify the log source

that was created in the previous step.

4. Click Search to view the results.

2. Export the results of the search.

1. After the search has completed, select Actions -> Export to CSV.

2. You will now be prompted to download a compressed file. Open the compressed

file and then open the resulting CSV file using Microsoft Excel.

3. In Microsoft Excel, copy column AM. This column should contain the raw

payloads from the unsupported log source.

4. Paste the results into a raw text file using an application such as Notepad++.

5. Save the file. The file name should be based on the unsupported log source name

and not include spaces, e.g. Fakeware_Logs.txt.

Reviewing The Logs

A critical step in creating a Universal DSM is reviewing the logs for usability. At a bare

minimum, the logs should have a value that can be mapped to an event name. The event name

should be a unique value that can distinguish the various log types.

• An example of usable logs: o May 20 17:16:14 dropbear[22331]: bad password attempt for 'root'

from 192.168.50.80:3364 o May 20 17:16:26 dropbear[22331]: password auth succeeded for

'root' from 192.168.50.80:3364 o May 20 16:42:19 kernel: DROP IN=vlan2 OUT=

MAC=00:01:5c:31:39:c2:08:00 SRC=172.29.255.121

DST=255.255.255.255 PROTO=UDP SPT=67 DPT=68 • An example of slightly usable logs:

o Oct 26 08:12:08 loopback 1256559128 autotrace[215824]: W: trace:

no map for prod 49420003, idf 010029a2, lal 00af0008 o Oct 26 16:35:00 sxpgbd0081 last message repeated 7 times o Nov 24 01:30:00 sxpgbd0081 /usr/local/monitor-rrd/sxpgbd0081/.rrd

(rc=-1, opening '/usr/local/monitor-rrd/sxpgbd0081/.rrd': No such

file or directory)

Creating The Log Source Extension

Determining Which Fields Can Be Used

The LSX template provides a large number of fields that values can be mapped to. These values

are obtained from the unsupported logs. This includes fields such as the source IP, destination

port, username, etc. The first step in creating the log source extension is reviewing the

unsupported logs and determining what fields will be used. At a bare minimum, the EventName

field must be used.

1. Download the LSX template and save it to your desktop. Be sure to right-click and use

the "Save As" option so it's not opened by Internet Explorer.

2. Rename the file to reflect the unsupported log source. For example, for Fakweare,

rename the file to Fakeware_LSX.xml.

3. Open the file using a text editor such as Notepad++. A list of fields will be displayed.

4. Determine which values in the unsupported log source can be mapped to the fields in the

LSX template. It is not necessary to use all of the fields in the default LSX template.

o Example log entry:

May 20 17:24:59 kernel: DROP MAC=5c:31:39:c2:08:00 SRC=172.29.255.121

DST=192.168.100.25 LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9582 PROTO=UDP

SPT=67 DPT=68 LEN=331

o The following fields are usable for this example:

EventName, i.e. DROP

SourceMAC, i.e. MAC=5c:31:39:c2:08:00

SourceIp, i.e. SRC=172.29.255.121

DestinationIp, i.e. DST=192.168.100.25

Protocol, i.e. PROTO=UDP

SourcePort, i.e. SPT=67

DestinationPort, i.e. DPT=68

5. Remove any unused fields and their corresponding Pattern IDs from the LSX.

6. Rename the Pattern IDs to a unique name. This saves confusion if multiple patterns are

used and helps distinguish between field and Pattern ID names. In most instances, the log

source type can be appended to the end of the Pattern ID as the unique name. For

example, EventName can be renamed to EventName-Fakeware as the Pattern ID name.

7. Change the Pattern IDs defined in the fields to match the renamed Pattern IDs.

8. Any other values in the logs that do not have a corresponding field can be utilized later

using Custom Event Properties.

Determining Regular Expression Patterns

The next step in creating a UDSM is building the regular expression patterns. Regular

expressions, or regex, are used to match strings of text from the unsupported log source.

Finding The Strings To Match

Visually analyze the unsupported log source to identify unique patterns. These patterns will later

be translated into regular expressions. When possible, you should include characters before and

after the actual value to provide a basic level of error checking. This will prevent similar values

from being unintentionally matched. The actual value can be isolated from the extra characters in

the later steps.

• Example log entry:

May 20 17:24:59 kernel: DROP MAC=5c:31:39:c2:08:00 SRC=172.29.255.121

DST=10.43.2.10 LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9582 PROTO=UDP

SPT=67 DPT=68 LEN=331 May 20 17:24:59 kernel: PASS MAC=5c:14:ab:c4:12:59 SRC=192.168.50.10

DST=192.168.10.25 LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9583 PROTO=TCP

SPT=1057 DPT=80 LEN=331 May 20 17:24:59 kernel: REJECT MAC=5c:ad:3c:54:11:07 SRC=10.10.10.5

DST=192.168.100.25 LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9584 PROTO=TCP

SPT=25212 DPT=6881 LEN=331

• Pseudo-code for matching patterns. The quotes can be ignored. The event names are

DROP, PASS, and REJECT. Please note that the space character is included to denote the

beginning and end of a pattern whenever possible.

o EventName: " kernel: VALUE "

o SourceMAC: " MAC=VALUE "

o SourceIp: " SRC=VALUE "

o DestinationIp: " DST=VALUE "

o Protocol: " PROTO=VALUE "

o SourcePort: " SPT=VALUE "

o DestinationPort: " DPT=VALUE "

In the next steps we will translate the pseudo-code to regular expressions.

Common Regular Expressions

There is essentially an infinite numbers of regular expressions that can be used to match a

desired string of text. The following are several common regular expressions:

Type Expression

IP Address \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Port Number \d{1,5}

MAC Address (?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}

Protocol (tcp|udp|icmp|gre)

Device Time \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}

White Space \s

Tab \t

Match Anything .*?

Additionally, the escape character, or "\", is used to denote a literal character. For example, in

regex the "." character means "any single character" and would match A, B, 1, X, etc. To specify

a literal match, you would use "\." instead. This would only match the "." character itself.

Escaping any non-digit or non-alpha character is usually the best way to ensure you do not

accidentally match another character.

Translating Pseudo-Code to Regular Expressions

The translations are based on the common expressions provided in the previous step. The quotes

can be ignored. Always substitute a space with the \s regular expression. Please note that any

non-digit or non-alpha character has been escaped, e.g. = becomes \= and : becomes \:.

Field Pseudo-Code Regular Expression

EventName " kernel: VALUE

" \skernel\:\s.*?\s

SourceMAC " MAC=VALUE " \sMAC\=(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}\s

SourceIP " SRC=VALUE " \sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

DestinationIp " DST=VALUE " \sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

Protocol " PROTO=VALUE " \sPROTO\=(tcp|udp|icmp|gre)\s

SourcePort " SPT=VALUE " \sSPT\=\d{1,5}\s

DestinationPort " DPT=VALUE " \sDPT\=\d{1,5}\s

Specifying Capture Groups

A capture group isolates a certain value in the regular expression. For example, in the

SourcePort pattern in the previous example, it is not possible to pass the entire value since it

includes spaces and SRC=<code>. Instead, you would specify just the port number using a capture group. The value in the capture group is what is passed to

the relevant field in QRadar. Simply place parenthesis around the values you

would like to capture:

Field Regular Expression Capture Group

EventNam

e <code>\skernel\:\s.*?\s \skernel\:\s(.*?)\s

SourceMA

C

\sMAC\=(?:[0-9a-fA-

F]{2}\:){5}[0-9a-fA-F]{2}\s \sMAC\=((?:[0-9a-fA-

F]{2}\:){5}[0-9a-fA-F]{2})\s

SourceIP \sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}

\.\d{1,3}\s \sSRC\=(\d{1,3}\.\d{1,3}\.\d{1,3}

\.\d{1,3})\s

Destination

Ip

\sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}

\.\d{1,3}\s \sDST\=(\d{1,3}\.\d{1,3}\.\d{1,3}

\.\d{1,3})\s

Protocol \sPROTO\=(tcp|udp|icmp|gre)\s \sPROTO\=((tcp|udp|icmp|gre))\s

SourcePort \sSPT\=\d{1,5}\s \sSPT\=(\d{1,5})\s

Destination

Port \sDPT\=\d{1,5}\s \sDPT\=(\d{1,5})\s

Migrating Regular Expression Pattern Into The LSX

The next step is to migrate the patterns with capture groups into the LSX. These can be copied

and pasted from the previous step into the inner-brackets of the relevant Pattern ID.

For example, [CDATA[]] becomes [CDATA[\sDPT\=(\d{1,5})\s]].

Please see the screenshot below for a complete example.

At this point, the LSX is complete and can be used to parse the unsupported log source within

QRadar.

Uploading And Applying The LSX

The completed LSX must be uploaded to QRadar and applied to the Universal DSM. The logic

from the LSX is then used to parse the logs from the unsupported log source.

Uploading The LSX

1. From the Admin tab, click the icon labeled Log Source Extensions. A new window will

open.

2. Click Add in the top-left corner to add a new LSX.

3. Assign the LSX a name, e.g. Fakeware.

4. Select the Use Condition as Parsing Override.

5. Do not set the LSX as the default for a Log Source Type.

6. Browse to the LSX you have saved and click Upload. Assuming there are no errors in the

LSX, it will be uploaded to QRadar.

7. Click Save and close the window.

Applying The LSX

1. From the Admin tab, click the icon labeled Log Sources. A new window will open.

2. Double-click the Universal DSM that was created in the previous step so it can be edited.

3. Select the LSX that was uploaded as the Log Source Extension.

4. Select Parsing Override as the Extension Use Condition.

5. Click Save and close the window.

The LSX has now been successfully uploaded and applied to the Universal DSM.

Mapping Events

Mapping Unknown Events To A QID

Initially, all of the events from the Universal DSM will appear as unknown in the Event tab. This

is normal. Although the event names might appear as understandable values in the logs, e.g.

DROP, DENY, and ACCEPT, QRadar has no understanding of what these values actual represent.

From QRadar's perspective, these value are strings of text that are not mapped to any known

values. It is necessary to manually map all unknown events to their equivalents in the QID map.

The value will then appear as expected and treated as normalized events.

1. From the Events tab, select Search -> Edit Search.

2. Specify the Time Range as an adequate amount of time from when the LSX was applied

to the Universal DSM, e.g 15 minutes.

3. Under Search Parameters, select Log Source Equals and specify the log source that was

created in the previous step.

4. Click Search to view the results. All of the events will appear as unknown.

5. Double-click on an unknown entry to view the event details.

6. Click Map Event in the event toolbar. A new window will open.

7. The value Log Source Event ID should display an EventName value from the LSX, e.g.

DROP, DENY, or ACCEPT. The value should not be blank. A blank value indicates an error

with the LSX.

8. The value displayed as the Log Source Event ID must now be mapped to an appropriate

QID.

1. Using Browse By Category and/or QID Search, find a value that best matches the

Log Source Event ID value. For example, the value DROP can be mapped to the

QID Firewall Deny - Event CRE.

Please Note: whenever possible, use the QID with Event CRE in the name. Most events

are tied to a particular log source type. For example, mapping to a random Firewall Deny

QID is essentially mapping the Universal DSM to events from another log source type.

The QID entries containing the name Event CRE are "generic" and not tied to a particular

log source type.

2. Select the value that the Log Source Event ID should be mapped to and click OK.

The window will close.

9. Continue repeating these steps until all unknown events are mapped successfully. A

method for quickly mapping all of the events can be found below.

At this point, any further events from the Universal DSM containing that particular Log Source

Event ID will appear as the specified QID. Events prior to the QID mapping will still remain

unknown. There is no supported method for mapping previous events to a current QID. This

process must be repeated until all of the unknown event types are successfully mapped to a

QID. A screenshot of this can be found below:

Helpful Hints

Using The logrun.pl Script To Quickly Map Events

The log sample from the previous step can be replayed within QRadar to quickly generate a large

number of events for mapping.

1. Copy the log sample to the /root directory on QRadar using an SCP client.

2. Log into QRadar as root.

3. Run the following command to replay the logs on a loop at 5 EPS. The logfile name

should be substituted with the correct value. The IP address should be changed to the IP

address or hostname of the unsupported log source.

/opt/qradar/bin/logrun.pl -f Fakeware_Logs.txt -l -u 192.168.50.200 5

4. From the Events tab in QRadar, select Search -> Edit Search.

5. Specify the Time Range as Real Time (streaming).

6. Under Search Parameters, select Log Source Equals and specify the log source that was

created in the previous step.

7. Click Search to view the results. The replayed logs will be streaming in real-time.

8. Click the Pause icon in the upper-right corner to stop the real-time view.

9. Double-click on any unknown event and map the event as described in the previous step.

10. Return to the search results and click the Play icon to resume streaming. Mapped events

will appear correctly.

11. Continue repeating step 8 - 10 until all of the unknown events are mapped.

Reducing Event Mappings

In many instances, such as an IDP/IDS, there are literally thousands of events that would need to

be mapped. This would be nearly impossible with a Universal DSM. In situations such as this, it

is more reasonable to map a category as the event name as opposed to the actual event itself. For

example:

date: "Feb 25 2010 00:43:26"; name: "SQL Slammer v312"; category: "Worm

Activity"; source ip: "192.168.200.200"; date: "Feb 25 2010 00:43:26"; name: "Code Red v412"; category: "Worm

Activity"; source ip: "192.168.200.200"; date: "Feb 25 2010 00:43:26"; name: "Annoying Toolbar"; category: "Malware";

source ip: "192.168.200.200";

Instead of using the name field for the Event Name, use the category field instead. This will

reduce the number of mappings. The actual event name, e.g. Code Red v412 can be displayed

using a custom property.

Parsing a CSV Log Format

It is not always necessary to create multiple Pattern IDs when parsing certain log types. For

example, a CSV formatted log can use a single parser with multiple capture groups. Take the

following log sample:

Event,User,Source IP,Source Port,Destination IP,Destination Port Failed Login,bjones,192.168.50.100,1024,10.100.24.25,22 Successful Login,nlabadie,192.168.64.76,1743,10.100.24.25,110 Privilege Escalation,bjones,192.168.50.100,1028,10.100.1.100,23

1. First create a parser that matches all relevant values using the previous patterns:

.*?\,.*?\,\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\,\d{1,5}\,\d{1,3}\.\d{1,3}

\.\d{1,3}\.\d{1,3}\,\d{1,5}

2. Next place capture groups around each value:

(.*?)\,(.*?)\,(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\,(\d{1,5})\,(\d{1,3}

\.\d{1,3}\.\d{1,3}\.\d{1,3})\,(\d{1,5})

3. Map out the field that each capture group is mapped to, incrementing the value as you

move from left to right:

1 = Event, 2 = User, 3 = Source IP, 4 = Source Port, 5 = Destination

IP, 6 = Destination Port

4. Now include the values in the LSX, mapping the capture group to the relevant event:

5. The LSX can then be uploaded and the events mapped.

Multiple Log Formats in a Single Log Source

Occasionally you will have multiple log formats included in a single log source. For example:

May 20 17:15:50 kernel: DROP IN=vlan2 OUT= MAC= SRC=67.149.62.133

DST=239.255.255.250 PROTO=UDP SPT=1900 DPT=1900

May 20 17:16:26 dropbear[22331]: password auth succeeded for 'root' from

192.168.50.80:3364

May 20 17:16:28 dropbear[22331]: exit after auth (root): Exited normally

</br> May 20 17:16:14 dropbear[22331]: bad password attempt for 'root' from

192.168.50.80:3364

There are two distinct log formats: one pertaining to firewall events, and one for authentication

events. It is necessary to write multiple patterns for parsing all of the events. You can specify the

order in which they should be parsed in the LSX. Typically, the more frequent events are parsed

first, followed by the less frequent events. You can have as many patterns as required to parse

out all of the events. The order variable is utilized to determine what order the patterns are

matched in.

We can see there are multiple formats for the following fields: EventName, SourceIP,

SourcePort and UserName. Separate patterns are written to parse each unique log type. Both of

the patterns are referenced when assigning the value to the normalized fields. The following

screenshot illustrates this:

Another example is included below that illustrates 4 unique event patterns: