Siamese Neural Network - Kangwoncs.kangwon.ac.kr/~parkce/seminar/Siamese Neural Network.pdf · 2016-06-17 · I. ntelligent . S. oftware. Lab. 2. Intro. S2Net • Siamese Neural Network(S2Net)

Intelligent Software Lab.

Siamese Neural Network

박천음

강원대학교


Intelligent Software Lab. 2

Intro. S2Net

• Siamese Neural Network(S2Net)• 입력 text들을 concept vector로 표현하기 위함에 기반

• 즉, similarity를 위해 가중치가 부여된 vector로 표현

• linear projection // metric learning approach

• Model parameters• 𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼 𝐼𝐼𝑝𝑝𝑝𝑝𝑝𝑝 𝑝𝑝𝑝𝑝𝑟𝑟 𝐼𝐼𝑡𝑡𝑝𝑝𝑡𝑡 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝, 𝐼𝐼𝑡𝑡𝑡𝑝𝑝𝑝𝑝 𝑙𝑙𝑝𝑝𝑙𝑙𝑡𝑡𝑙𝑙𝑙𝑙 , 𝑙𝑙𝑝𝑝𝑙𝑙𝑡𝑡𝑙𝑙: 𝑙𝑙𝑝𝑝𝑡𝑡𝑝𝑝𝑙𝑙𝑝𝑝𝑝𝑝 𝑣𝑣𝑝𝑝 𝐼𝐼𝑣𝑣𝐼𝐼

• 각 𝑣𝑣𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝𝑙𝑙 → 𝑙𝑙𝑝𝑝𝑡𝑡 . : min 𝑙𝑙𝑣𝑣𝑙𝑙𝑙𝑙(. )

Input Output

Originaltermvector(one-hot)

Projectedconceptvector

Model parameters (weights) e.g. word embedding


Learning concept vector

𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼: 𝐼𝐼𝑟𝑟

𝐻𝐻𝑝𝑝𝐻𝐻𝐻𝐻𝑡𝑡𝐼𝐼: 𝐼𝐼𝑟𝑟𝑡

Originalterm vector(one-hot)

<0, 1, 0, 0, .. 0, 1>_(dx1)

Projectedconcept vector

𝑉𝑉𝑝𝑝: < ⋯ >

…

…

Model parameters

…

…

𝑙𝑙𝑝𝑝𝑡𝑡(𝑉𝑉𝑝𝑝,𝑉𝑉𝑞𝑞)

𝑉𝑉𝑞𝑞: < ⋯ >

𝑂𝑂𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼

vector z

𝑝𝑝0,0, … , 𝑝𝑝0,𝑘𝑘𝑝𝑝1,0, … , 𝑝𝑝1,𝑘𝑘

…𝑝𝑝𝑑𝑑,0, … , 𝑝𝑝𝑑𝑑,𝑘𝑘


Model design

• 𝑓𝑓𝑡𝑡𝑡𝑡𝐻𝐻𝑓𝑓𝑣𝑣𝑝𝑝𝑟𝑟𝑝𝑝𝑝𝑝𝐻𝐻• NN: 𝐼𝐼𝑡𝑡𝐼𝐼𝑗𝑗 = ∑𝑖𝑖=0𝑑𝑑 𝑟𝑟𝑖𝑖𝑗𝑗𝑥𝑥𝑖𝑖• S2Net: 𝐼𝐼𝑟𝑟′ 𝑣𝑣𝑗𝑗 = ∑𝑡𝑡𝑖𝑖∈𝑉𝑉 𝛼𝛼𝑖𝑖𝑗𝑗 � 𝐼𝐼𝑟𝑟(𝐼𝐼𝑖𝑖)

• 𝛼𝛼𝑖𝑖𝑗𝑗 ∈ 𝐴𝐴

• 𝑝𝑝𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝑝𝑝𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼 𝑓𝑓𝐼𝐼𝐼𝐼𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼• sigmoid (non-linear)

• 𝑙𝑙𝑣𝑣𝑙𝑙𝑣𝑣𝑡𝑡 𝐼𝐼𝑟𝑟′(. )• 𝑓𝑓 = 𝐼𝐼𝑡𝑡𝑝𝑝𝑡𝑡 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝐼𝐼𝑟𝑟, 𝑓𝑓 = 𝑉𝑉, 𝐻𝐻 × 1 𝑝𝑝𝑝𝑝𝑟𝑟 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝• 𝐴𝐴 = 𝛼𝛼𝑖𝑖𝑗𝑗 𝑑𝑑×𝑘𝑘

, 𝛼𝛼𝑖𝑖𝑗𝑗 ∈ 𝐴𝐴 ( weights)

• 𝑔𝑔 = 𝐴𝐴𝑇𝑇𝑓𝑓 ∴ 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝑔𝑔𝑘𝑘×1( projected concept vector)

• 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴(𝑓𝑓𝑝𝑝, 𝑓𝑓𝑞𝑞)


Loss function

• using the cosine similarity• 𝐼𝐼𝑡𝑡𝑝𝑝𝑡𝑡 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝐼𝐼𝑝𝑝𝑝𝑝𝑝𝑝 𝑓𝑓𝑝𝑝, 𝑓𝑓𝑞𝑞• 𝑣𝑣𝑣𝑣𝑝𝑝𝑝𝑝𝑡𝑡𝑙𝑙𝐼𝐼𝑣𝑣𝐼𝐼𝐻𝐻𝑝𝑝𝐼𝐼𝑔𝑔 𝑣𝑣𝑣𝑣𝐼𝐼𝑣𝑣𝑡𝑡𝐼𝐼𝐼𝐼 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝𝑙𝑙 𝑣𝑣𝑣𝑣𝑙𝑙𝑝𝑝𝐼𝐼𝑡𝑡 𝑣𝑣𝑝𝑝𝑙𝑙𝐼𝐼𝑡𝑡 𝑔𝑔𝑝𝑝,𝑔𝑔𝑞𝑞

• 𝑔𝑔𝑝𝑝 = 𝐴𝐴𝑇𝑇𝑓𝑓𝑝𝑝, 𝑔𝑔𝑞𝑞 = 𝐴𝐴𝑇𝑇𝑓𝑓𝑞𝑞

∴ 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝, 𝑓𝑓𝑞𝑞 =𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑞𝑞

𝑔𝑔𝑝𝑝 � 𝑔𝑔𝑞𝑞

• 𝑇𝑇𝑝𝑝𝐼𝐼𝑡𝑡 𝑙𝑙𝑝𝑝𝑙𝑙𝑡𝑡𝑙𝑙 = 𝑦𝑦𝑝𝑝𝑞𝑞

• 𝑙𝑙𝑣𝑣𝑙𝑙𝑙𝑙 𝑓𝑓𝐼𝐼𝐼𝐼𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼(𝑀𝑀𝑀𝑀𝑀𝑀) = 12𝑦𝑦𝑝𝑝𝑞𝑞 − 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝, 𝑓𝑓𝑞𝑞

2

• query에 가까운 text object를 선택하기 위해 MSE 사용

요런방법도있음(기본적인방법)


Training procedure

• ∆: 𝑙𝑙𝑝𝑝𝑡𝑡𝑝𝑝𝑙𝑙𝑝𝑝𝑝𝑝𝑝𝑝𝐼𝐼𝑦𝑦 𝑙𝑙𝑣𝑣𝑣𝑣𝑝𝑝𝑡𝑡• 𝐼𝐼𝑡𝑡𝑝𝑝𝑡𝑡 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝑓𝑓𝑝𝑝1,𝑓𝑓𝑞𝑞1 , (𝑓𝑓𝑝𝑝2,𝑓𝑓𝑞𝑞2)를 비교• 첫 번째 term vector가 더 high similarity

• 식으로 표현하면 ∆ = 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝1,𝑓𝑓𝑞𝑞1 − 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝2,𝑓𝑓𝑞𝑞2• ∆를 logistic loss로 변환

• [0 or 1]로 loss 값을 반올림

∴ 𝐿𝐿 ∆;𝐴𝐴 = log(1 + exp −𝛾𝛾∆ )• ∆값이커질수록 loss값이작아짐• 𝛾𝛾는 cosine function때문에 ∆를 𝛾𝛾만큼확대시킴 (∆𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑡𝑡: [-2, 2])• 따라서 prediction error에도움됨• 𝛾𝛾는경험상크게 (= 10)

• 여기서 𝐴𝐴를최적화

본논문에서의loss function

(타겟, high score) (출력)


Gradient derivation

• 𝐿𝐿 ∆;𝐴𝐴 에서 𝐴𝐴 최적화• 𝜕𝜕𝐿𝐿 ∆;𝐴𝐴

𝜕𝜕𝐴𝐴= −𝛾𝛾

1+exp(−𝛾𝛾∆)𝜕𝜕∆𝜕𝜕𝐴𝐴

• 𝜕𝜕∆𝜕𝜕𝐴𝐴

= 𝜕𝜕𝜕𝜕𝐴𝐴𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝1,𝑓𝑓𝑞𝑞1 − 𝜕𝜕

𝜕𝜕𝐴𝐴𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝2,𝑓𝑓𝑞𝑞2

• 𝜕𝜕𝜕𝜕𝐴𝐴𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝, 𝑓𝑓𝑞𝑞 = 𝜕𝜕

𝜕𝜕𝐴𝐴cos 𝑔𝑔𝑝𝑝,𝑔𝑔𝑞𝑞

• cos 𝑔𝑔𝑝𝑝,𝑔𝑔𝑞𝑞 =𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑞𝑞

𝑔𝑔𝑝𝑝 � 𝑔𝑔𝑞𝑞

• 𝛻𝛻𝐴𝐴𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑞𝑞 = 𝛻𝛻𝐴𝐴𝐴𝐴𝑇𝑇𝑓𝑓𝑝𝑝 𝑔𝑔𝑞𝑞 + 𝛻𝛻𝐴𝐴𝐴𝐴𝑇𝑇𝑓𝑓𝑞𝑞 𝑔𝑔𝑝𝑝 = 𝑓𝑓𝑝𝑝𝑔𝑔𝑞𝑞𝑇𝑇 + 𝑓𝑓𝑞𝑞𝑔𝑔𝑝𝑝𝑇𝑇

• 𝛻𝛻 1𝑔𝑔𝑝𝑝

= 𝛻𝛻𝐴𝐴 𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑝𝑝−12

= −12𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑝𝑝

−23𝛻𝛻𝐴𝐴 𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑝𝑝 = − 𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑝𝑝−23𝑓𝑓𝑝𝑝𝑔𝑔𝑝𝑝𝑇𝑇

• 𝛻𝛻 1𝑔𝑔𝑞𝑞

=− 𝑔𝑔𝑞𝑞𝑇𝑇𝑔𝑔𝑞𝑞−23𝑓𝑓𝑞𝑞𝑔𝑔𝑞𝑞𝑇𝑇

𝐿𝐿 ∆;𝐴𝐴 = log(1 + exp −𝛾𝛾∆ )

∆ = 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝1,𝑓𝑓𝑞𝑞1 − 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝2,𝑓𝑓𝑞𝑞2𝑔𝑔𝑝𝑝 = 𝐴𝐴𝑇𝑇𝑓𝑓𝑝𝑝, 𝑔𝑔𝑞𝑞 = 𝐴𝐴𝑇𝑇𝑓𝑓𝑞𝑞

1

2 31

2

3


S2Net example: max()

• Input sentences• Sen1: what movies johnny depp is in?

• Sen2: what movies does johnny depp play in?

• Sen3: who has been married to julia roberts?

• Bag of words(size 16):

• Term vector:


S2Net example: max() cont`

• Weights:(size: 2 x 16)

• 𝑊𝑊𝑇𝑇

0.005 0.00150.1 0.0750.12 0.0050.0015 0.020.1 0.050.1 0.050.1 0.050.0005 0.20.003 0.20.1 0.0050.1 0.080 0.0750 0.090.2 0.0760.005 0.01250.065 0.01

𝑥𝑥𝑖𝑖

𝑥𝑥 𝑟𝑟



• 𝑦𝑦 = 𝑓𝑓(𝑟𝑟𝑇𝑇𝑥𝑥), (𝑣𝑣𝑣𝑣𝐼𝐼𝑣𝑣𝑡𝑡𝐼𝐼𝐼𝐼 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝑦𝑦2×1)sigmoid

• 𝑙𝑙𝑣𝑣𝑣𝑣𝑝𝑝𝑡𝑡 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑1, 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑2 ,

𝑥𝑥𝑖𝑖

𝑦𝑦𝑖𝑖

𝑦𝑦𝑖𝑖

𝑓𝑓(𝐼𝐼𝑡𝑡𝐼𝐼𝑗𝑗)

max(0, 1 − 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑1 + 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑2)

𝑦𝑦𝑖𝑖

𝑦𝑦𝑖𝑖𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑1

𝑦𝑦𝑖𝑖

𝑦𝑦𝑖𝑖𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑2

score(.)



• Back prop.• 𝜕𝜕𝑙𝑙𝑝𝑝𝑡𝑡 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑1, 𝑧𝑧𝑑𝑑𝑑𝑑𝑑𝑑2

• gradient update

• 𝑓𝑓𝑓 𝐼𝐼𝑡𝑡𝐼𝐼 = 𝜕𝜕𝑙𝑙𝑝𝑝𝑔𝑔𝑡𝑡𝑣𝑣𝑝𝑝𝐻𝐻 .

• gradient update

logistic sigmoid

𝑓𝑓 𝐼𝐼𝑡𝑡𝐼𝐼 =1

1 + exp −𝐼𝐼𝑡𝑡𝐼𝐼𝜕𝜕𝑓𝑓 𝐼𝐼𝑡𝑡𝐼𝐼𝜕𝜕𝐼𝐼𝑡𝑡𝐼𝐼 = 𝑓𝑓 𝐼𝐼𝑡𝑡𝐼𝐼 1− 𝑓𝑓 𝐼𝐼𝑡𝑡𝐼𝐼

𝜕𝜕𝑡𝑡𝑝𝑝𝑥𝑥 0,𝐼𝐼𝑡𝑡𝐼𝐼 =−1 𝑝𝑝𝑓𝑓 1 − 𝑙𝑙 + 𝑙𝑙𝑑𝑑 > 00 𝑣𝑣𝐼𝐼𝑡𝑡𝑡𝑝𝑝𝑟𝑟𝑝𝑝𝑙𝑙𝑡𝑡



S2Net example: cosine_sim()





• Term vector:


S2Net example cosine_sim() cont`

• Weights:(size: d x k)





0.005 0.00150.1 0.0750.12 0.0050.0015 0.020.1 0.050.1 0.050.1 0.050.0005 0.20.003 0.20.1 0.0050.1 0.080 0.0750 0.090.2 0.0760.005 0.01250.065 0.01

Term vector:



• Term vectors:

• 𝐴𝐴𝑇𝑇:0.005, 0.1, 0.12, 0.0015, 0.1, 0.1, 0.1, 0.0005, 0.003, 0.1, 0.1, 0, 0, 0.2, 0.005, 0.0650.0015, 0.075, 0.005, 0.02, 0.05, 0.05, 0.05, 0.2, 0.2, 0.005, 0.08, 0.075, 0.09, 0.076, 0.0125, 0.01



• 𝑔𝑔 = 𝐴𝐴𝑇𝑇𝑓𝑓 (𝑣𝑣𝑣𝑣𝐼𝐼𝑣𝑣𝑡𝑡𝐼𝐼𝐼𝐼 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝑔𝑔𝑘𝑘×1)

• 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝,𝑓𝑓𝑞𝑞 = 𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑞𝑞𝑔𝑔𝑝𝑝 � 𝑔𝑔𝑞𝑞

, (cosine similarity)

(𝑝𝑝𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝑝𝑝𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼 𝑓𝑓𝐼𝐼𝐼𝐼𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼)



• 𝑌𝑌 𝑙𝑙𝑝𝑝𝑙𝑙𝑡𝑡𝑙𝑙

• 𝑙𝑙𝑣𝑣𝑙𝑙𝑙𝑙 𝑓𝑓𝐼𝐼𝐼𝐼𝑣𝑣𝐼𝐼𝑝𝑝𝑣𝑣𝐼𝐼(𝑀𝑀𝑀𝑀𝑀𝑀) = 12𝑦𝑦𝑝𝑝𝑞𝑞 − 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝,𝑓𝑓𝑞𝑞

2

Or

0



S2Net example3 cosine_sim() activation





• Term vector:


S2Net example3 cosine_sim() activation`

• Weights:(size: d x k)





0.05 0.050.05 0.0750.1 0.1250.02 0.0750.1 0.050.005 0.050.005 0.050.08 0.20.125 0.01250.1 0.050.1 0.0250.125 0.0750.115 0.0750.005 0.050.01 0.01550.01 0.022

Term vector:



• Input vector:

• 𝐴𝐴𝑇𝑇:0.05, 0.05, 0.1, 0.02, 0.1, 0.005, 0.005, 0.08, 0.125, 0.1, 0.1, 0.125, 0.115, 0.005, 0.01, 0.010.05, 0.075, 0.125, 0.075, 0.05, 0.05, 0.05, 0.2, 0.0125, 0.05, 0.025, 0.075, 0.075, 0.05, 0.0155, 0.022



• 𝑔𝑔 = 𝐴𝐴𝑇𝑇𝑓𝑓 (𝑣𝑣𝑣𝑣𝐼𝐼𝑣𝑣𝑡𝑡𝐼𝐼𝐼𝐼 𝑣𝑣𝑡𝑡𝑣𝑣𝐼𝐼𝑣𝑣𝑝𝑝 𝑔𝑔𝑘𝑘×1)

• 𝑙𝑙𝑝𝑝𝑡𝑡𝐴𝐴 𝑓𝑓𝑝𝑝,𝑓𝑓𝑞𝑞 = 𝑔𝑔𝑝𝑝𝑇𝑇𝑔𝑔𝑞𝑞𝑔𝑔𝑝𝑝 � 𝑔𝑔𝑞𝑞

, (cosine similarity)



• Y label

Documents

Siamese Neural Network - Kangwoncs.kangwon.ac.kr/~parkce/seminar/Siamese Neural Network.pdf · 2016-06-17 · I. ntelligent . S. oftware. Lab. 2. Intro. S2Net • Siamese Neural Network(S2Net)