• Home

  • Technology

  • Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network
15
Komachi Lab M1 Ryosuke Miyazaki 2015/10/02 Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network ACL-IJCNLP 2015 Qiao Qian, Bo Tian, Minlie Huang, Yang Liu, Xuan Zhu and Xiaoyan Zhu ACL 2015 reading group ※このスライド内の図は論文内のものです

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Embed Size (px)

Citation preview

Page 1: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

M1 Ryosuke Miyazaki

2015/10/02

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network ACL-IJCNLP 2015 Qiao Qian, Bo Tian, Minlie Huang, Yang Liu, Xuan Zhu and Xiaoyan Zhu

ACL 2015 reading group

※このスライド内の図は論文内のものです

Page 2: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

RNN✤ ２つの子ベクトルから親ベクトルを構成!

✤ 木構造に従って再帰的に文全体のベクトルを構成

2

Page 3: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Abstract

✤ VPを構成するcomposition functionと

NPを構成するcomposition functionは 本当に一緒？タグごとに違うのでは？!

✤ 句構造タグも構成に利用しよう!

✤ Tag Guided RNNとTag Embedded RNN/RNTNを提案

3

Page 4: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Tag Guided RNN✤ 親ノードのタグ毎にcomposition functionを変える!✤ すべてのタグを区別すると頻度の低いタグに関するパラメータが学習しきれない

4

➡Top k 個とその他で

k+1 の composition function

Page 5: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Tag Embedded RNN✤ 親のタグが同じでも子のタグが違う場合

(VP→V NP と VP→V Advは同じcomposition？)!

✤ 子ノードのタグ情報も使いたいけど

Tag Guidedでやるとcomposition functionがk × k個もパラメータが多すぎる!

➡ タグをベクトルに埋め込んでスパースネス解消

(1000回未満の出現頻度のタグは無視)

5

Page 6: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Tag Embedded RNN

6

Page 7: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Tag Embedded RNTN

7

✤ RNTN(Recursive Neural Tensor Network)と同様!

✤ テンソルTを使って, 子ベクトルの相互作用を考慮する

Page 8: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Learning✤ 今回はSentiment Analysis (分類問題)のタスクで学習

8

✤ 誤差関数はCross Entropy Error

Page 9: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Experiments Settings

✤ Sentiment Tree Bankを使用

すべての部分木に極性ラベルがついている (5値)

(train: 8544, dev: 1101, test:2210)!

✤ Stanford Parserを使って句構造タグを付与(Sentiment Tree BankもStanford Parserを使ってるが

ver違いによる木構造の違いでtrain:74, dev:11,test11 の文を無視)

9

Page 10: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Experiments Settings

✤ 単語ベクトルはword2vecでpre-train(100,000 movie reviews)

RNN:d=25, RNTN: d=20!

✤ その他のベクトルは[-0.01, 0.01]の一様分布からサンプル!

✤ 構成されたフレーズベクトルにはベクトルの正規化を施す

10

Page 11: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Experiments Settings✤ ハイパーパラメータ

batch size 30, momentum 0.9, L2 reg 0.0001, Learning rate 0.005

11

TG-RNNのタグの数k: 6 TE-RNNのタグの次元数 de: 8, TE-RNTN : 6

Page 12: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Results

12

✤ Fine-grainedではDRNNの次によい!

✤ RNN → TG-RNN (+3.8)!

✤ RNN → TE-RNN (+4.8)!

✤ RNTN → TE-RNTN (+3.2)

Page 13: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Model Size

13

✤ DRNNよりも少ない!✤ RNN, RNTNより少ないのは、次元dの設定が小さいから

Page 14: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Tag Vectors Analysis

✤ 目的関数にはタグの分類の気持ちは入っていないのに似たタグは似たベクトルになってる!

✤ 恐らく似たタグは似た役割を担うから

14

Page 15: Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Komachi Lab

Conclusion

✤ 句構造タグを利用した

新しいモデルTG-RNN, TE-RNN/RNTNを考案!

✤ 有用であるだけでなく便利である

(比較的良いパフォーマンスと少ないパラメータ)

15


Improving the Compositionality of Word Embeddings · 2020-05-08 · Improving the Compositionality of Word Embeddings We present an in-depth analysis of four popular word embeddings

Improving the Compositionality of Word Embeddings · 2020-05-08 · Improving the Compositionality of Word Embeddings We present an in-depth analysis of four popular word embeddings

Documents
Active Learning through Adversarial Exploration in ... · The typical NCE [5] approach in tasks such as word embeddings[18], order embeddings[27], and knowledge graph embeddings can

Active Learning through Adversarial Exploration in ... · The typical NCE [5] approach in tasks such as word embeddings[18], order embeddings[27], and knowledge graph embeddings can

Documents
Recursive SQL from a Performance Perspective - gsefr.org Recursive SQL From a Performance... · Recursive SQL from a Performance Perspective. 2 Agenda • What Is Recursive SQL? •

Recursive SQL from a Performance Perspective - gsefr.org Recursive SQL From a Performance... · Recursive SQL from a Performance Perspective. 2 Agenda • What Is Recursive SQL? •

Documents
Dependency-Based Word Embeddings

Dependency-Based Word Embeddings

Documents
Learnable PINs: Cross-Modal Embeddings for …for character labelling in a TV drama. This again evaluates the embeddings on unseen-unheard identities. 2RelatedWork Cross-modal embeddings:

Learnable PINs: Cross-Modal Embeddings for …for character labelling in a TV drama. This again evaluates the embeddings on unseen-unheard identities. 2RelatedWork Cross-modal embeddings:

Documents
PRIMITIVE RECURSIVE VECTOR SEQUENCES, POLYNOMIAL …people.math.carleton.ca/~sartaj/mythesis.pdf · PRIMITIVE RECURSIVE VECTOR SEQUENCES, POLYNOMIAL SYSTEMS ... RECURSIVE VECTOR SEQUENCES,

PRIMITIVE RECURSIVE VECTOR SEQUENCES, POLYNOMIAL …people.math.carleton.ca/~sartaj/mythesis.pdf · PRIMITIVE RECURSIVE VECTOR SEQUENCES, POLYNOMIAL SYSTEMS ... RECURSIVE VECTOR SEQUENCES,

Documents
Convolutional Graph Embeddings - cs.ubc.ca

Convolutional Graph Embeddings - cs.ubc.ca

Documents
Iterated elementary embeddings and the model … elementary embeddings and the model ... elementary embeddings can be found in ... similar requirement that the elementary submodel

Iterated elementary embeddings and the model … elementary embeddings and the model ... elementary embeddings can be found in ... similar requirement that the elementary submodel

Documents
4th Workshop on Semantic Deep Learning …Deep Structured Semantic Models - Graph Representations of Word Embeddings - Multimodal Networks - Tag extraction using end-to-end memory

4th Workshop on Semantic Deep Learning …Deep Structured Semantic Models - Graph Representations of Word Embeddings - Multimodal Networks - Tag extraction using end-to-end memory

Documents
Word Embeddings - w4nderlustw4nderlu.st/content/4-teaching/3-word-embeddings/... · Word Embeddings: hot trend in NLP ... Different languages use different ... Zellig Harris, Mathematical

Word Embeddings - w4nderlustw4nderlu.st/content/4-teaching/3-word-embeddings/... · Word Embeddings: hot trend in NLP ... Different languages use different ... Zellig Harris, Mathematical

Documents
Learning Word Embeddings from Tagging Data: A ...ceur-ws.org/Vol-1917/paper32.pdfWord Embeddings. The concept of word embeddings, i.e., word representations in low dimensional vector

Learning Word Embeddings from Tagging Data: A ...ceur-ws.org/Vol-1917/paper32.pdfWord Embeddings. The concept of word embeddings, i.e., word representations in low dimensional vector

Documents
Recursive Static Route - cisco.com · Recursive Static Route TheRecursiveStaticRoutefeatureenablesyoutoinstallarecursivestaticrouteintotheRoutingInformation …

Recursive Static Route - cisco.com · Recursive Static Route TheRecursiveStaticRoutefeatureenablesyoutoinstallarecursivestaticrouteintotheRoutingInformation …

Documents
Other Approaches to Computability Church’s Thesis · Recursive Functions Turing Machine Church’s Thesis Primitive Recursive Function Partial Recursive Function Primitive Recursive

Other Approaches to Computability Church’s Thesis · Recursive Functions Turing Machine Church’s Thesis Primitive Recursive Function Partial Recursive Function Primitive Recursive

Documents
Chapter 11 – Recursion Recursive Processes Writing a Recursive Method A Recursive Factorial Method Comparison of Recursive and Iterative Solutions Recursive

Chapter 11 – Recursion Recursive Processes Writing a Recursive Method A Recursive Factorial Method Comparison of Recursive and Iterative Solutions Recursive

Documents
Factorization and Embeddings

Factorization and Embeddings

Documents
Dynamic Word Embeddings - arxiv.org

Dynamic Word Embeddings - arxiv.org

Documents
Word Embeddings - cocoxu.github.io

Word Embeddings - cocoxu.github.io

Documents
Four different groups of recursive tests Forward recursive tests

Four different groups of recursive tests Forward recursive tests

Documents
Recursive weighted median filters admitting negative ...arce/Research_files/2-Recursive... · Recursive Weighted Median Filters Admitting ... RECURSIVE WEIGHTED MEDIAN FILTERS ADMITTING

Recursive weighted median filters admitting negative ...arce/Research_files/2-Recursive... · Recursive Weighted Median Filters Admitting ... RECURSIVE WEIGHTED MEDIAN FILTERS ADMITTING

Documents
Self embeddings of Bedford-McMullen carpetsmath.huji.ac.il/~mhochman/preprints/affine-embeddings-of-carpets.pdf · Self embeddings of Bedford-McMullen carpets Amir Algom and Michael

Self embeddings of Bedford-McMullen carpetsmath.huji.ac.il/~mhochman/preprints/affine-embeddings-of-carpets.pdf · Self embeddings of Bedford-McMullen carpets Amir Algom and Michael

Documents
Recursive Algorithms Recursive procedure

Recursive Algorithms Recursive procedure

Documents
8.5 Using Recursive Rules with Sequences - Big Ideas Math · Evaluate recursive rules for sequences. Write recursive rules for sequences. Translate between recursive and explicit

8.5 Using Recursive Rules with Sequences - Big Ideas Math · Evaluate recursive rules for sequences. Write recursive rules for sequences. Translate between recursive and explicit

Documents
from embeddings Supporting online material for: Embeddings

from embeddings Supporting online material for: Embeddings

Documents
Recursive Definitions and Structural Induction 1 Recursive

Recursive Definitions and Structural Induction 1 Recursive

Documents
Recursive Neural Networks - Technion – Israel this assignment” Recursive Neural Networks for Structure Predicon Recursive Neural Networks for Structure Predicon A standard Recursive

Recursive Neural Networks - Technion – Israel this assignment” Recursive Neural Networks for Structure Predicon Recursive Neural Networks for Structure Predicon A standard Recursive

Documents
Recursive Static Route - Cisco · Information About Recursive Static Route How to Install Recursive Static Route Installing Recursive Static Routes in a VRF

Recursive Static Route - Cisco · Information About Recursive Static Route How to Install Recursive Static Route Installing Recursive Static Routes in a VRF

Documents
Recursive Dynamic CS: Recursive Recovery of Sparse Signal ...home.engineering.iastate.edu/~namrata/TSP_RecRecon_Paper_4.pdf · Recursive Dynamic CS: Recursive Recovery of Sparse Signal

Recursive Dynamic CS: Recursive Recovery of Sparse Signal ...home.engineering.iastate.edu/~namrata/TSP_RecRecon_Paper_4.pdf · Recursive Dynamic CS: Recursive Recovery of Sparse Signal

Documents
Vector Semantics And Embeddings

Vector Semantics And Embeddings

Documents
Machine Learning & Data Mining CS/CNS/EE 155 Lecture 14: Embeddings 1Lecture 14: Embeddings

Machine Learning & Data Mining CS/CNS/EE 155 Lecture 14: Embeddings 1Lecture 14: Embeddings

Documents
NodeSketch: Highly-Efficient Graph Embeddings via Recursive … · 2020-05-18 · NodeSketch: Highly-Efficient Graph Embeddings via Recursive Sketching Dingqi Yang1, Paolo Rosso1,

NodeSketch: Highly-Efficient Graph Embeddings via Recursive … · 2020-05-18 · NodeSketch: Highly-Efficient Graph Embeddings via Recursive Sketching Dingqi Yang1, Paolo Rosso1,

Documents