Transaction Mining for Deeper Machine Data Intelligence

Ariel Smoliar

Phone registrations failures over specific period

Tracking transactions in payment processing platform

Tracking a renewal or new signup transaction

E-commerce: typical user session, anomalous checkout transactions, catching drop off in checkout

Tracking users on-boarding process

Attribution modeling - Determining the origin of a user action

How Sumo Logic handles a search query and on-boarding of new users

Analyzing Related Sequences of Logs - Use Cases

2

The new capability provides tools to analyze related sequences of logs

Two main modes of operation: unordered and ordered transaction analysis

Several result type view:

– Unordered analysis by transaction, states (and filtering)

– Ordered analysis by flow (and drill-down from the graph)

Transaction (operator) Capability

3

The operator requires the following components:

– Transaction IDs (Session ID, IP, user name, email, etc.) to group related messages together

– States mapping from the logs

Transaction Operator - Required Components

4

transaction on ip

transaction on userid, usersessionid

transaction on sessionid

transaction on location, part

Transaction Operator - Transaction IDs (examples)

5

| transaction on sessionid

with "Starting session *" as init,

with "Initiating countdown *" as countdown_start,

with "Countdown reached *" as countdown_done,

with "Launch *” as launch

_sourceCategory=ecom "/login" OR "/checkout”

| parse regex "(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"

| parse regex "GET (?<url>[^\" ]+)"

| where url matches "/login" or url matches "/checkout*"

| parse regex field=url "^(?:/checkout)?/(?<step>[A-Za-z0-9_]+)"

| transaction on ip

with states login, cart, checkout, shipping_method, billing, review, progress, confirmation in step

Transaction Operator - Mapping States (examples)

6

Queries are constrained by a time window

Some transactions may be cut off if they occur near the edges of the window

Filter the transactions by using the fringe argument

Transaction Operator - fringe cut-off

7

Not taking into account the ordering of the messages within a transaction

Covering many of the use cases

Unordered Analysis

8

Results for Unordered Analysis (1/3)

9

by transactions - counts the number of times a transaction hits a state

Transactions can be filtered by using where states="___110”

Threshold (on count) for a state can be added, with the thresh argument with "…" thresh=2 as

Aggregates other than count can be specified using the showing clause, the first aggregate definition applies globally, additional aggregates may relate to a specific state. To count, use the function sum(“1”)

Results for Unordered Analysis (2/3)

10

by states - number transactions with specific states combination

Results for Unordered Analysis (3/3)

11

by logs - shows the actual logs for the transactions that satisfy the filter, where statues=“101_1110”

Monitoring transition between (two distinct) states

Which transitions does a transaction go through

Number of transactions between transitions

Latency between transitions

Supports the Sankey diagram (new chart type)

Ordered Analysis

12

Results for Ordered Analysis

13

by flow - The default aggregate between states is count, but users can add other aggregates (max(latency) or avg(latency))

Sankey diagram is used to visualize the magnitude of flow between states in ordered analysis

New chart icon in the Search page, enabled only for the relevant syntax (otherwise grayed out)

Sankey Diagram - A New Chart Type

14

Sankey Diagram - Sumo’s Site

15

Hovering over the state box exposes inbound and outbound flow

Sankey Diagram - UI Features (1/3)

16

Sankey Diagram - UI Features (2/3)

17

Hovering over the link exposes the count and flow direction

Try to drag the state boxes vertically

Sankey Diagram - UI Features (3/3)

18

Clicking on a link/edge between two states will launch a new search showing only the relevant result for the transition

Sankey Diagram - Drilldown from the graph!

19

Sankey Diagram - Specified Topology

20

E-commerce website