= MatrixTree(scoreij)

§Tree analysis:

• Yang Liu and Mirella Lapata. 2018. ”Learning Structured Text Representations”. Transactions of the Association for Computational Linguistics, 6:63–75.

• design changes and bug fix are made to L&L code and results are also confirmed without these modifications; see paper for details.

• Code and data at https://github.com/elisaf/structured

05

1015202530

Full

-biLSTM

-biLSTM

, +1perc

-biLSTM

, +4perc

parsed RST

Tree

Hei

ght

012345

Yelp

DebatesWQ

WQTC

WSJSO

Tree

Hei

ght

§ 14%

4% 3% 3% 0%048

1216

Full

-biLSTM

-biLSTM

, +1perc

-biLSTM

, +4perc

parsed RST

% V

acuo

us T

rees

73%

38% 42%

14%

100%

020406080

100

Yelp

DebatesWQ

WQTC

WSJSO

% V

acuo

us T

rees

§

Removing biLSTM + adding 4 levels of children percolation yields similar performance as Full model.

b. percolate context from children of subtreesn levels down (+nperc)

§

Adding structured attention helps only WQTC. On Yelp, WQ, WSJSO, there is no difference. On Debates, the attention hurts.

Bett

er

Bett

erProblem: Although RST trees are linguistically defined and motivated, they are hard to exploit because annotations are scarce and limited by genre.

Do the induced trees learn discourse?

Induced Tree: (same Yelp review as above)

RST Dependency Tree:

Wor

seBe

tter

On the upside they have a lot of classes.

ELABORATION

Yelp Rating […]They offer it at a great price.

however the classes aren’t early enough.

The big downside […]

Evaluating Discourse in Structured Text Representations

Elisa Ferracane, Greg Durrett, Jessy Li, Katrin Erk elisa@ferracane.com, gdurrett@cs.utexas.edu, jessy@utexas.edu, katrin.erk@mail.utexas.edu

Structured Text Representations Induced Trees

Experiments

Yelp uuu, sterne, star, rating, deduct, 0, editDebates oppose, republican, majority, thankWQ valley, mp3, firm, capital, universal

RT

31 2

CONTRAST

ELABORATION

Structured text representations such as trees generated by Rhetorical Structure Theory (RST) are helpful for NLP end tasks including sentiment analysis.

s1 s2 st

structured attention

compose

y

max

Solution…? Liu & Lapata (2018) train on text classification tasks and use structured attention to induce dependency trees over the text, akin to RST discourse dependency trees.

Does structured attention help? Can we learn better structure?

a. remove biLSTM over sentences (-biLSTM)

4

biLSTM over sentences

discourse

semantic

d1 d2 dt

e1 e2 et

e1! e2

! et!

Structured Attention

scoreij = bilinear(di, dj)

context for possible immediate children of parent i

updatedsemantic

updated semantic vector

RT 1 2 3

parent= RT1 2 3child=

12

3

4

RT

85 6 7 91 2 3

Model

-4

-2

0

2

4

Yelp Debates WQ WQTC WSJSO

Accu

racy

Diff

eren

ce

Accuracy Gain with structured attention

Setup: Train 4x with different random seeds and report mean (see paper for std dev, max)

A

s1 s2 st

Vacuous tree: Flat, uninformative discourse structure. Root is one sentence at beginning (or end) of text, and all other sentences are children.

Best model produces less vacuous and deepertrees, but these are still far from ‘gold’ parsed RST discourse dependency trees.

Dataset Label TaskYelp 1-5 Review sentiment Debates 0/1 Vote predictionWriting Quality (WQ) 0/1 Good writingWQ Topic-Controlled (WQTC)

0/1 Good writing (topic-controlled)

WSJ Sentence Ordering (WSJSO)

- Sentence order discrimination

Generally, no. It only significantly helps one task.

No. The model focuses on lexical cues.

§

Tree statistics:

Root sentence analysis:

Performance gain with structured attention:

Top PPMI words in root sentence are indicative of label: rating or sentiment (Yelp), stance or politeness (Debates), topic (WQ).

Model modifications to increase reliance on structure:

A No. Model changes induce better trees, but are still far from discourse.

§ Performance:

§ Tree statistics:

✘ ✘

A

embed sentences

parent

12

3

1 2 3child=

matrix tree

…can be interpreted as edge marginals of dependency structure

Q Q Q

Trees are very shallow; WQTC is the least shallow.

Vacuous trees are frequent, except in WQTC.

+1perc

+2perc

81.1

77.877.1

81.7

74

78

82

Full -biLSTM -biLSTM,+1perc

-biLSTM,+4perc

Accu

racy

Bett

er

Bett

er

= RT

di

ei

ci =nX

k=1

aikek<latexit sha1_base64="6bhtbcxLqnBL6ziJQr/FOU0spDE=">AAACAnicdVDLSsNAFJ3UV62vqCtxM1gEVyGpim4KRTcuK9gHtDFMptN2yMwkzEyEEoIbf8WNC0Xc+hXu/BunD8HngQuHc+7l3nvChFGlXffdKszNLywuFZdLK6tr6xv25lZTxanEpIFjFst2iBRhVJCGppqRdiIJ4iEjrTA6H/utGyIVjcWVHiXE52ggaJ9ipI0U2Ds4oNWuSnmQRVUvvxYoyGiUkyAK7LLrVI5dA/ibeI47QRnMUA/st24vxiknQmOGlOp4bqL9DElNMSN5qZsqkiAcoQHpGCoQJ8rPJi/kcN8oPdiPpSmh4UT9OpEhrtSIh6aTIz1UP72x+JfXSXX/1M+oSFJNBJ4u6qcM6hiO84A9KgnWbGQIwpKaWyEeIomwNqmVTAifn8L/SbPieIdO5fKoXDubxVEEu2APHAAPnIAauAB10AAY3IJ78AierDvrwXq2XqatBWs2sw2+wXr9AJQyl44=</latexit>

e0i = tanh(Wr[ei, ci])<latexit sha1_base64="gxsyfbUlk+hWh/uoDmtmeO3tCUM=">AAACAHicdVDLSsNAFJ3UV62vqAsXbgaLWEFKUhXdCEU3LivYB6QhTKa3duhkEmYmQgnd+CtuXCji1s9w5984fQg+D1w4nHMv994TJpwp7TjvVm5mdm5+Ib9YWFpeWV2z1zcaKk4lhTqNeSxbIVHAmYC6ZppDK5FAopBDM+xfjPzmLUjFYnGtBwn4EbkRrMso0UYK7C3YC9iZJqJXagbSg4Ad0ID5+4FddMqVY8cA/yZu2RmjiKaoBfZbuxPTNAKhKSdKea6TaD8jUjPKYVhopwoSQvvkBjxDBYlA+dn4gSHeNUoHd2NpSmg8Vr9OZCRSahCFpjMiuqd+eiPxL89LdffUz5hIUg2CThZ1U451jEdp4A6TQDUfGEKoZOZWTHtEEqpNZgUTwuen+H/SqJTdw3Ll6qhYPZ/GkUfbaAeVkItOUBVdohqqI4qG6B49oifrznqwnq2XSWvOms5som+wXj8AWIOVmw==</latexit>

a12<latexit sha1_base64="yC+VtJgn7RYyni5ztIBy2ugLTqo=">AAAB7XicdVDJSgNBEK2JW4xb1KOXxiB4GmbGiB6DXjxGMAskQ+jp9CRterqH7h4hDPkHLx4U8er/ePNv7CyC64OCx3tVVNWLUs608bx3p7C0vLK6VlwvbWxube+Ud/eaWmaK0AaRXKp2hDXlTNCGYYbTdqooTiJOW9Hocuq37qjSTIobM05pmOCBYDEj2FipiXu5H0x65YrnBqeeBfpNfNeboQIL1Hvlt25fkiyhwhCOte74XmrCHCvDCKeTUjfTNMVkhAe0Y6nACdVhPrt2go6s0kexVLaEQTP160SOE63HSWQ7E2yG+qc3Ff/yOpmJz8OciTQzVJD5ojjjyEg0fR31maLE8LElmChmb0VkiBUmxgZUsiF8for+J83A9U/c4LpaqV0s4ijCARzCMfhwBjW4gjo0gMAt3MMjPDnSeXCenZd5a8FZzOzDNzivHz9Hjuo=</latexit>

a13<latexit sha1_base64="rRXMTZK+1S5MoH2KPAFbgpvR2T4=">AAAB7XicdVDLSgNBEOyNrxhfUY9eBoPgadlNFD0GvXiMYB6QLGF2MpuMmZ1ZZmaFsOQfvHhQxKv/482/cTaJ4LOgoajqprsrTDjTxvPencLS8srqWnG9tLG5tb1T3t1raZkqQptEcqk6IdaUM0GbhhlOO4miOA45bYfjy9xv31GlmRQ3ZpLQIMZDwSJGsLFSC/czvzbtlyueWz31LNBv4rveDBVYoNEvv/UGkqQxFYZwrHXX9xITZFgZRjidlnqppgkmYzykXUsFjqkOstm1U3RklQGKpLIlDJqpXycyHGs9iUPbGWMz0j+9XPzL66YmOg8yJpLUUEHmi6KUIyNR/joaMEWJ4RNLMFHM3orICCtMjA2oZEP4/BT9T1pV16+51euTSv1iEUcRDuAQjsGHM6jDFTSgCQRu4R4e4cmRzoPz7LzMWwvOYmYfvsF5/QBAzI7r</latexit>

aij<latexit sha1_base64="TFdJkW0yRCqwMkcQJ7NbfedU2aI=">AAAB7XicdVDLSgNBEOyNrxhfUY9eBoPgadmNih6DXjxGMA9IljA7mU0mmZ1ZZmaFsOQfvHhQxKv/482/cTaJ4LOgoajqprsrTDjTxvPencLS8srqWnG9tLG5tb1T3t1rapkqQhtEcqnaIdaUM0EbhhlO24miOA45bYXjq9xv3VGlmRS3ZpLQIMYDwSJGsLFSE/cyNpr2yhXPrZ55Fug38V1vhgosUO+V37p9SdKYCkM41rrje4kJMqwMI5xOS91U0wSTMR7QjqUCx1QH2ezaKTqySh9FUtkSBs3UrxMZjrWexKHtjLEZ6p9eLv7ldVITXQQZE0lqqCDzRVHKkZEofx31maLE8IklmChmb0VkiBUmxgZUsiF8for+J82q65+41ZvTSu1yEUcRDuAQjsGHc6jBNdShAQRGcA+P8ORI58F5dl7mrQVnMbMP3+C8fgDpr49a</latexit>