Problems in Current Text Simpliﬁcation Research · Problems in Current Text Simpliﬁcation...

Problems in Current Text Simplification Research

TACL paper @ EMNLP Sep-20-2015

Wei XuChris Callison-Burch

Courtney Napoles

UPennUPenn JHU

What is Text Simplification

Wei Xu, Chris Callison-‐Burch, Courtney Napoles. “Problems in Current Text Simplifica@on Research: New Data Can Help” TACL (2015)

Applesauce is a puree made of apples.INPUT

Applesauce is a soft paste. OUT-1

Applesauce is a paste. It is made of apples.OUT-2

• for children, disabled, non-native speakers … • for other NLP tasks (MT, summarization …)

paraphrasing

paraphrasing deletion

paraphrasing deletion splitting!!

Goal of Text Simplification

grammaticality

grammaticality meaning preservation

grammaticality meaning preservation simplicity

Human EvaluationApplesauce is a puree made of apples.INPUT

Human Evaluation

(no reliable automatic evaluation yet)

Brief History of Sentence Simplification

rule-based

1997 Chandrasekar & Srinivas 1999 Dras (PhD thesis)2000 Carroll, Minnen, Pearce, Canning, Devlin 2002 Canning (PhD thesis)2004 Siddharthan (PhD thesis)2010 Zhu, Bernhard, Gurevych2011 Woodsend & Lapata2011 Coster & Kauchak2012 Wubben, van den Bosch, Krahmer2014 Narayan & Gardent2014 Siddharthan (Survey)2014 Angrosh, Nomoto, Siddharthan2014 Narayan (PhD thesis)Now Xu, Callison-Burch, Napoles (Opinion)

rule-based

Parallel Wikipedia Corpus

rule-based

machine translation

rule-based

machine translation

rule-based

machine translation

Problems in Simplification Research

• State-of-the-art evaluation is suboptimal. But we have been doing this in the past 5 years*.

• Simple Wikipedia data dominated in the past 5 years. But its quality was taken for granted. It limits the scope of research.

* (Angrosh et al. 2014) tried comprehension quizWei Xu, Chris Callison-‐Burch, Courtney Napoles. “Problems in Current Text Simplifica@on Research: New Data Can Help” TACL (2015)

Breakthrough on Sea

Why this is important?

• better understanding

• better review

• more diverse research

• better data and evaluation

• better model

wind direction

a straight pathupwind

a zigzag pathupwind

wind direction

Breakthrough on Sea

Why this is important?⎯⎯

• better review

• better model

wind direction

a zigzag pathupwind

wind direction

SimplificationBreakthrough on Sea

Why this is important?⎯⎯

• better review

• better model

wind direction

a zigzag pathupwind

wind direction

—— Štajner, Béchara, Saggion (2015)

“Recently, there have been several attempts ataddressing the text simplification task as a monolingualtranslation problem … However, they did not try to seekreasons for the success or the failure of their systems.”

—— Štajner, Béchara, Saggion (2015)

“Recently, there have been several attempts ataddressing the text simplification task as a monolingualtranslation problem … However, they did not try to seekreasons for the success or the failure of their systems.”

“state-of-the-art” competition not easy to doWHY DID THIS HAPPEN?

Opinion #1

Current evaluation doesn’t tell us what’s going on.

System Comparabilitygrammaticality

meaning preservation

simplicity

paraphrasing

deletion

splitting!!

not easy to measure

sub-systems evaluation criteria

simplicity

paraphrasing

deletion

splitting!!

not easy to measureWe need more controlled evaluation:

simplicity

paraphrasing

deletion

splitting!!

not easy to measureWe need more controlled evaluation:- evaluate sub-tasks separately

simplicity

paraphrasing

deletion

splitting!!

not easy to measureWe need more controlled evaluation:- evaluate sub-tasks separately- target specific audience (e.g. 10-12 year old)

Opinion #2

Simple Wikipedia isnot that simple

—— Advaith Siddharthan (2014 Survey)

“Specific questions that need addressing are :

… we need to better understand the quality of SimpleEnglish Wikipedia, a resource that has been used totrain many SMT based simplification systems…”

—— Advaith Siddharthan (2014 Survey)

“Specific questions that need addressing are :

… we need to better understand the quality of SimpleEnglish Wikipedia, a resource that has been used totrain many SMT based simplification systems…”

We quantitively and systematically answer this quest.WHAT’S NEW?

Quality of Parallel Wikipedia Corpus*

alignment error

not simpler

real simplification

alignment error

not simpler

real simplification

Inaccuracy in Parallel Wikipedia Corpus*

not simpler

real simplification

(two sentences have different meaning)

alignment error

not simpler

real simplification

Inaccuracy in Parallel Wikipedia Corpus*

not simpler

real simplification

Best automatic sentence alignment gets about 0.7 F1 score (Hwang et al. 2015)

alignment error

not simpler

real simplification

Inadequacy in Parallel Wikipedia Corpus*

alignment error

not simpler

real simplification

Sentences can have similar meaning but not simplification

alignment error

not simpler

real simplification (aligned and simpler)

alignment error real simplification (aligned and simpler)?

alignment error real simplification (aligned and simpler)

Some sentences are simpler by only one word while the rest of sentence is still complex

Issues with Parallel Wikipedia Corpus

• suboptimal for estimating “translation” probabilities

• suboptimal for developing automatic metrics

• suboptimal for tuning MT system

• unsuitable for document-level simplification

Opinion #3

New data can help

Newsela Dataset

every article at 5 levels of simplification written by trained editors, comes with comprehension quizzes

Newsela

33%50%

alignment error

not simpler

real simplification

alignment errornot simpler real

simplification

Wikipedia*

manual inspection of aligned sentence pairs

NewselaWikipedia*

degree of paraphrasing

deletion +

paraphrase

paraphrase only

deletion only

paraphrase only

deletion onlydeletion

+paraphrase

Good simplification needs more paraphrasing.

NewselaWikipedia*

sentence length (#words) see syntax analysis in the paper

Normal Simple

Good simplification could be much shorter.

NewselaWikipedia*

71,340

23,7716,669

Normal Simple

19,849

19,197583

Normal Simple

48% reduction18% reduction

(total 2.6 million tokens) (total 1.3 million tokens)

vocabulary size (#unique words)

“chimp”

“chimpanzee”

Good simplification uses a much smaller vocabulary.

NewselaWikipedia*

most significantly reduced words (weighted log-odds-ratio analysis w/ informative Dirichlet prior)

commune,

northernnorthwestern

southwesternfootballer

"—of

whichas

percentincludingdirector

Good simplification reduces certain function word usage.

NewselaWikipedia*

most significantly reduced words see syntax analysis in the paper

0 2,000 4,000 6,000 8,000

NormalSimple

0 750 1500 2250 3000

approximately

0 125 250 375 500

approximately

0 10 20 30 40

Postal officials recently tried to … … , which could … .…Postal officials recently tried to … … . That could … .…

NewselaWikipedia*

0 2,000 4,000 6,000 8,000

NormalSimple

0 750 1500 2250 3000

approximately

0 125 250 375 500

approximately

0 10 20 30 40

NewselaWikipedia*

0 2,000 4,000 6,000 8,000

NormalSimple

0 750 1500 2250 3000

approximately

0 125 250 375 500

approximately

0 10 20 30 40

NewselaWikipedia*

document compression ratio (simple/normal) see discourse analysis in the paper

0.00 0.25 0.50 0.75 1.00 1.250.00 0.25 0.50 0.75 1.00 1.25

3.19% 57.28%

Wikipedia is not suitable for full-document simplification.

Current evaluation doesn’t tell us what’s going on.Opinion #1

Simple Wikipedia is not that simple.Opinion #2

New data can help.Opinion #3

My Suggestions

My Suggestions• to reviewers:

- be open-minded to papers that may not follow previous evaluation setup, may not outperform the “state-of-the-art” on Wikipedia

- be sympathetic towards papers specially on data construction*, data analysis* and automatic evaluation metrics

* (Pellow & Maxine, 2014 HCOMP; Marcelo & Specia, 2014 PITR)

- be sympathetic towards papers specially on data construction*, data analysis* and automatic evaluation metrics

- read our paper

* (Pellow & Maxine, 2014 HCOMP; Marcelo & Specia, 2014 PITR)

My Suggestions

My Suggestions• to researchers:

- consider working on text simplification (“pre-BLEU age”)

- improve evaluation

- make your system replicable

- read our paper

Thank you

Sponsor: NSF

Newsela data are available at h5ps://newsela.com/data/

Questions? Opinions?

Back Up

NewselaWikipedia*

simple cue

complex conjunc1ons

change of discourse connectives (odds-ratio)Wei Xu, Chris Callison-‐Burch, Courtney Napoles. “Problems in Current Text Simplifica@on Research: New Data Can Help” TACL (2015)

Reasons of Quality Issues in Parallel Wikipedia Corpus

• The Simple Wikipedia was created by volunteers with no specific objective;

• Articles in Simple Wikipedia do not necessarily map Normal Wikipedia;

• As an encyclopedia, Wikipedia contains extremely difficulty words and sentences.

Newsela DatasetOriginal

Slightly more fourth-graders nationwide are reading proficiently compared with a decade ago, but only a third of them are now reading well, according to a new report.

Simple-1Fourth-graders in most states are better readers than they were a decade ago. But only a third of them actually are able to read well, according to a new report.

Simple-2Fourth-graders in most states are better readers than they were a decade ago. But only a third of them actually are able to read well, according to a new report.

Simple-3 Most fourth-graders are better readers than they were 10 years ago. But few of them can actually read well.

Simple-4 Fourth-graders are better readers than 10 years ago. But few of them read well.

Newsela Dataset1,130 news articles

Time: 2013 January ~ 2015 March

Source: Chicago Tribune, Seattle Times, LA Times, The Baltimore Sun

Original: 56k sentences — Simple: 64k sentences

Problems in Current Text Simpliﬁcation Research · Problems in Current Text Simpliﬁcation...

Documents

Generalized View-Dependent Simpliﬁcation

Automatic dynamics simpliﬁcation in Fast Multipole Method ...webpages.iust.ac.ir/mozayani/Papers-pdf/razavi-supercomputing2012.pdf · Automatic dynamics simpliﬁcation in Fast

Radical Simpliﬁcation through Polyglot and Poly-paradigm … · 2013-05-20 · Radical Simpliﬁcation through Polyglot and Poly-paradigm Programming Dean Wampler dean@objectmentor.com

Towards simplification of hydrologic modeling ... · 4656 S. L. Markstrom et al.: Towards simplification of hydrologic modeling: identification of dominant processes Figure 1

Impact of pipes networks simpliﬁcation on water hammer

Learning and Generating Paraphrases From Twitter and Beyondmt-class.org/penn/slides/guest-lecture-wei-xu.pdf · 2016. 10. 21. · Research Overview TACL 15 !! NAACL 15!! TACL 14 !!

Topology Simpliﬁcation for Polygonal Virtual Environments · Topology Simpliﬁcation for Polygonal Virtual Environments Jihad El-Sana Amitabh Varshney Department of Computer Science

TACL 2011, the Fifth International Conference on Topology, …luigi.santocanale/tacl2011/... · 2011-07-14 · Preface TACL 2011, the 5th Conference on Topology, Algebra, and Categories

Vector and Tensor Field Topology Simpliﬁcation on ... · Vector and Tensor Field Topology Simpliﬁcation on Irregular Grids Xavier Tricoche, Gerik Scheuermann, Hans Hagen, and

TACL LID Camp 2009: Revival

TACL Reference Manual - NonStopToolsnonstoptools.com/manuals/Tacl-Reference.pdf · HP NonStop TACL Reference Manual Abstract This publication describes the syntax and use of the HP

Semantics, Analysis and Simpliﬁcation of DMN Decision Tablesdumas/pubs/DMNSimplify.pdf · Semantics, Analysis and Simpliﬁcation of DMN Decision Tables Diego Calvanesea, Marlon

Topology-based Simpliﬁcation for Feature Extraction from ...graphics.cs.ucdavis.edu/~hamann... · on topological simpliﬁcation of scalar functions. In the case of a real-valued

Model Simpliﬁcation via Super Elements using the …GeoandEnvironmentalEngineering ChairforComputationinEngineering Prof. Dr. rer. nat. ErnstRank Model Simpliﬁcation via Super

Simpliﬁcation of mass spring models - WorldComp …worldcomp-proceedings.com/proc/p2012/CGV4298.pdf · Simpliﬁcation of mass spring models ... Incorporating such objects in a

Problems in Current Text Simpliﬁcation Researchxwe/publications/EMNLP_2015_TACL_simplificat… · Problems in Current Text Simpliﬁcation Research TACL paper @ EMNLP Sep-20-2015

Volume and Complexity Bounded Simpliﬁcation of Solid Model

Topological Semantics of Modal Logiclogica.dmi.unisa.it/tacl/tacl2011/slides/109.pdf · TACL’2011 - Marseille, July 26, 2011. Topological Semantics of Modal Logic David Gabelaia

A Web-Based Medical Text Simpliﬁcation Tool

Transformation of Formal Models...2012/02/16 · Examples normal forms +simplification of proofs +more efficient construction of parsers elimination of erasing rules +simplification