Graph Learning in Financial Networks

Graph Learning in Financial Networks

Jiaxuan YouStanford University

Collaboration with Tianyu Du, Fan-yun Sun, Jure Leskovec

Financial Networks

§ Financial Networks: Describe financial entities and their connections

Jiaxuan You, Stanford University 2Image credit: The Political Economy of Global Finance: A Network Model Image credit: https://dailyblockchain.github.io/

International banking• Nodes: Countries• Edges: Capital flows

Bitcoin transactions• Nodes: BTC wallets• Edges: Transactions

Graph Learning in Financial Networks

§ Goal: A graph learning framework for financial networks§ Applications: Fraud detection, Anti-money laundering, Anomaly detection§ Solution: Graph representation learning!

Jiaxuan You, Stanford University 3

Input: Financial networks

Node-level: Fraudsters, …

Edge-level: fraudulent/anomalous transactions, …

Output: Predictions

Subgraph-level: Money laundering subnetworks, …

Graph Neural Network

Why Graph Representation?

§ Transaction-based approach§ “On 01/03, Client 𝐴 sends Company 𝐵 $500”§ Build models based on transaction attributes§ Issues: ignore the context of a transaction

§ Graph-based approach§ Represent transactions as a dynamic graph§ Predictions are made based on the entire graph§ Benefits:

§ Represents a transaction with a broader context§ Requires fewer feature-engineering


$400, 01/05

bankclient company

$100, 01/01

$500, 01/03

$200, 01/02

$200, 01/02

$100, 01/06

Overview of This Talk

ROLAND: GNN for Financial

Networks

Results & Insights from

ROLAND

Application: Anomaly Detection

5Jiaxuan You, Stanford UniversityGraph Neural Networks

bankclient company

NN

NN

NN

NN

NN

NN

$400, 01/05

ROLAND: GNN for Financial Networks

6

bankclient company

$100, 01/01

$500, 01/03

$200, 01/02

$200, 01/02

Will there be a transaction? YesWhat is the amount? $500When will it happen? 01/03…Is it a fraud? NoIs it money laundering? Yes…

Graph Neural NetworksFinancial networks Learning Signals & Application

$100, 01/06

bankclient company

Self-supervised(from raw data)

Supervised(from external sources)

bankclient company

NN

NN

NN

NN

NN

NN

?

§ ROLAND framework:§ Transform financial networks as GNN

computational graphs§ Learning from diverse signals

Jiaxuan You, Stanford University

J. You, T. Du, J. Leskovec. ROLAND: Graph Neural Networks for Dynamic Graphs, in submission

node embeddings

ROLAND Model: From Static to Dynamic GNN


GNN Layer 1

Graph !

Pred "

GNN Layer 2

#(")

#($)

a) A static GNN with modern architectural design options

Skip connections

LayerconnectivityGNN Layer

GNN Layer

GNN Layer

Intra-layer Design: 4 dims

MLP Layer

MLP Layer

Messagepassinglayers

Post-processlayers

MLP Layer

MLP Layer

Pre-processlayers

Linear

BatchNorm

Dropout

Activation

Attention

Aggregation

Transformation

GNN Layer

GNN Layer

GNN Layer

MLP Layer

MLP Layer

Post-processlayers

MLP Layer

MLP Layer

Pre-processlayers

GNN Layer

GNN Layer

GNN Layer

Linear

BatchNorm

Dropout

Activation

Attention

Aggregation

Aggregation

Attention

Activation

Dropout

BatchNorm

LinearGNN Layer 1

Snapshot !%&"

Pred "%

GNN Layer 2

Embedding update

Embedding update

#$%&"(")

#%&"(")

#$%&"($)

#%&"($)

GNN Layer 1

Snapshot !%

Pred "%'"

GNN Layer 2

Embedding update

Embedding update

#$%(")

#%(")

#$%($)

#%($)

#%&"(")

#%&"($)

b) Extend static GNNs to dynamic GNNs

……

Hierarchical Node state #% = {#%" , … , #%(()}

Question: Can we adapt a SOTA static GNN to dynamic prediction tasks?

t+1t+2 t+3

This transaction is fraudulent

SOTAStatic GNN

ROLAND Model: From Static to Dynamic GNN


§ Idea: Recurrently update node embeddings at each layer

§ Introduce a new module to a static GNN:

§ Benefits:§ Simple and effective§ Benefit from the SOTA designs of a static GNN

Static GNN Dynamic GNN

Embedding update

Input:§ Previous embeddings from the

the same layer§ Current embeddings from the

previous layerOutput: Updated embeddings

GNN Layer 1

Graph !

Pred "

GNN Layer 2

GNN Layer 1

Snapshot !!"#

Pred "!

GNN Layer 2

Embedding update

Embedding update

GNN Layer 1

Snapshot !!

Pred "!$#

GNN Layer 2

Embedding update

Embedding update

……

ROLAND Training: Efficient Training

§ Incremental training: § Only keep these in GPU

§ GNN Model 𝐺𝑁𝑁!§ Historical node states 𝐻!"#§ Incoming new graph snapshot 𝐺!§ Efficient and work well in practice

§ Meta training:§ Train a meta-GNN that can quickly adapt

to new data§ Benefits: ROLAND does not need to be

frequently retrained9Jiaxuan You, Stanford University

Snapshot!!"#

"!

!##!"#

Evaluate

!##!

Train

Time !

"$!

State%!

Snapshot!!

"!$#

!##!

Evaluate

!##!$#

Train

"$!$#

…

Time ! + #

…

Live-update evaluation pipeline

Only this part in GPU



Networks


ROLAND



bankclient company

NN

NN

NN

NN

NN

NN

ROLAND Implementation: GraphGym

§ GraphGym offers a standardized pipeline for graph learning§ Design Space of Graph Neural Networks (NeurIPS 2020 spotlight)§ Now officially part of PyG 2.0! (Stay tuned for my next talk :)

11

General GNN design space

Clean interface for comparing designs


J. You, R. Ying, J. Leskovec. Design Space of Graph Neural Networks, NeurIPS 2020

§ We implement the financial prediction pipeline with GraphGym§ We adopt many optimal design choices from GraphGym

https://arxiv.org/pdf/2011.08843.pdf

An Example Financial Dataset


§ Transactions between companies captured by a bank clearing system§ 336 million transactions, 6 million accounts, 6 years of daily data

§ Task setting: Supervised learning§ Unbalanced labels: 2% of all the transactions are fraudulent§ Random dataset split: 80% training, 10% validation, 10% testing§ Metric: AUC on test set (random guess has AUC 0.5, higher is better)

§ Best non-deep learning baseline§ AUC: ~0.8

§ Simple GNN implementation using out-of-the-box PyG§ AUC: 0.81

§ ROLAND framework works much better!§ AUC: 0.94

Task 1: Classify Fraud Transactions

13Jiaxuan You, Stanford University

Task 2: Forecast Future Transactions

§ Task setting: Self-supervised/Unsupervised learning§ Rolling prediction: On each day, use all the historical information to predict the

transactions on the next day§ Metric: Mean reciprocal ranking (MMR) of ground-truth future transactions


ROLAND significantly outperforms SOTA baselines

Analysis: ROLAND’s Performance over Time


ROLAND maintains a robust performance over transaction pattern changes

# of retraining epochs

Almost no retraining needed!

Test MRRRecall@1

(week)day of the year

Transaction amount

Analysis: Interpreting ROLAND’s Predictions

§ Question: What kind of transactions the model paid attention to?

§ Method: Inspect edges' attention scores of the last GNN layer.§ Select edges with low/high attention scores.§ Get the feature value distribution of these

edges


Domestic -> Abroad

Non-financial àHousehold

Higher attention to large amount transactions



Networks


ROLAND



bankclient company

NN

NN

NN

NN

NN

NN

§ Goal: Detect anomalous transactions without external supervision§ Combine Graph Representation Learning with Outlier Detection


18

Input: Financial networks

node embeddings

edge embeddings


Self-supervised objectives

19

Real world Indicators (from baselgovernance.org)

Objectives for GNN

investment funds sent to countries of concern Predict country

Large cash deposits into (company) accounts Predict amount

unusual subject of transaction Predict subject of transaction(link prediction)

high volume of transactions within a short period predict acculumated amount

multiple individuals sending funds to the one beneficiary predict # transactions


“Indicators and patterns of money laundering or terrorist financing”

https://baselgovernance.org/sites/default/files/2019-04/indicators_and_patterns_of_money_laundering_or_terrorist_financing.pdf

Objectives for Anomaly Detection

20

t+1t+2 t+3

Self-supervised Objectives● Objective 1: predict amount● Objective 2: predict recipient● Objective 3: predict country● …

dim

|N|

Outlier Detection Objectives


§ Solution: ROLAND framework + diverse objectives

Quantitative Evaluation for Anomaly Detection

● Task: Detect anomalous transactions without supervision● Goal: Simulate real-world anomaly detection use cases○ Corrupt the dataset to see if ROLAND can extract them○ Report precision/recall of corrupted edges detected by ROLAND

○ Ways of corrupting the dataset:■ Amount corruption (~1% transactions)■ Country corruption (~1% transactions)■ Recipient corruption (~1% transactions) → Rewire !

21

A B

C D

A B

C D


Quantitative Evaluation for Anomaly Detection+14.7% Precision +14.2% Recall


Deep learning baseline:

Rule-based baseline: +172% Precision +171% Recall


Normal accountsPotential anomalous accounts



Networks


ROLAND


24Jiaxuan You, Stanford University

Graph Neural Networks

bankclient company

NN

NN

NN

NN

NN

NN

Resources: ROLAND (in submission)Design Space for Graph Neural NetworksLearn more about GNN: PyG.org

https://arxiv.org/abs/2011.08843

http://pyg.org/

Documents

Graph Learning in Financial Networks