Embed Size (px)
Citation preview
SUCCESS WITH STYLE:
USING WRITING STYLE TO
PREDICT THE SUCCESS OF NOVELS
Vikas Ashok, Song Feng, Yejin Choi
PREDICTING THE
SUCCESS OF NOVELS
PREDICTING THE
SUCCESS OF NOVELS
• Paul Harding’s “Tinkers”
• Pulitzer winner!
• J.K. Rowling’s “Harry Potter
and the Philosopher’s Stone”
• Sold over 450 millions copies.
Can we predict the success of literary work based on writing style??
Can we predict the success of literary work based on writing style??
What are the secret elements in successful books??
“That is to say, in either instance you insist on what you want by a process of making other people thoroughly uncomfortable till you get your way—unless they happen to be stronger than you! And incidentally a good many innocent folk who have nothing to do with the matter get badly hurt in the fray.”
― Margaret Pedler, The Moon out of Reach
“Gossip is never fatal until it is denied. Gossip goes on about every human being alive and about all the dead that are alive enough to be remembered, and yet almost never does any harm until some defender makes a controversy. Gossip's a nasty thing, but it's sickly, and if people of good intentions will let it entirely alone, it will die, ninety-nine times out of a hundred.”
― Booth Tarkington, The Magnificent Ambersons
“That is to say, in either instance you insist on what you want by a process of making other people thoroughly uncomfortable till you get your way—unless they happen to be stronger than you! And incidentally a good many innocent folk who have nothing to do with the matter get badly hurt in the fray.”
― Margaret Pedler, The Moon out of Reach
“Gossip is never fatal until it is denied. Gossip goes on
about every human being alive and about all the dead that
are alive enough to be remembered, and yet almost never
does any harm until some defender makes a controversy
….”
― Booth Tarkington, The Magnificent Ambersons
“Gossip is never fatal until it is denied. Gossip goes on about every human being alive and about all the dead that are alive enough to be remembered, and yet almost never does any harm until some defender makes a controversy. Gossip's a nasty thing, but it's sickly, and if people of good intentions will let it entirely alone, it will die, ninety-nine times out of a hundred.”
― Booth Tarkington, The Magnificent Ambersons “That is to say, in either instance you insist on what you want by a process of making other people thoroughly uncomfortable till you get your way—unless they happen to be stronger than you! And incidentally a good many innocent folk who have nothing to do with the matter get badly hurt in the fray.”
― Margaret Pedler, The Moon out of Reach
“That is to say, in either instance you insist on what you want by a process of making other people thoroughly uncomfortable till you get your way—unless they happen to be stronger than you! And incidentally a good many innocent folk who have nothing to do with the matter get badly hurt in the fray.”
― Margaret Pedler, The Moon out of Reach
“Gossip is never fatal until it is denied. Gossip goes on about every human being alive and about all the dead that are alive enough to be remembered, and yet almost never does any harm until some defender makes a controversy. Gossip's a nasty thing, but it's sickly, and if people of good intentions will let it entirely alone, it will die, ninety-nine times out of a hundred.”
― Booth Tarkington, The Magnificent Ambersons
“That is to say, in either instance you insist on what you want by a process of making other people thoroughly uncomfortable till you get your way—unless they happen to be stronger than you! And incidentally a good many innocent folk who have nothing to do with the matter get badly hurt in the fray.”
― Margaret Pedler, The Moon out of Reach
“Gossip is never fatal until it is denied. Gossip goes on about every human being alive and about all the dead that are alive enough to be remembered, and yet almost never does any harm until some defender makes a controversy. Gossip's a nasty thing, but it's sickly, and if people of good intentions will let it entirely alone, it will die, ninety-nine times out of a hundred.”
― Booth Tarkington, The Magnificent Ambersons
HOW TO
DEFINE SUCCESS
HOW TO
QUANTIFY SUCCESS
POPULARITY & QUALITY
POPULARITY & QUALITY
Downloaded Downloads
2013-10-10 143540
last 7 days 1099579
last 30 days 4443895
POPULARITY & QUALITY
Downloaded Downloads
2013-10-10 143540
last 7 days 1099579
last 30 days 4443895
Project Gutenberg
offers over 40,000 free ebooks.
Title, author, genre, download count.
NOVEL DATA
Project Gutenberg
offers over 40,000 free ebooks.
Title, author, genre, download count.
NOVEL DATA
More
successful
Less
successful
Project Gutenberg
offers over 40,000 free ebooks.
Title, author, genre, download count.
50 books per class, 8 genres.
NOVEL DATA
Adventure
Fiction
Historical
Love
Mystery
Poetry
Sci-fi
Short Story
Project Gutenberg
offers over 40,000 free ebooks.
Title, author, genre, download count.
50 books per class, 8 genres.
<=2 books per author.
NOVEL DATA
Authorship attribution
Adventure
Fiction
Historical
Love
Mystery
Poetry
Sci-fi
Short Story
PREDICTING “SUCCESS”
Lexical Choices
unigrams / bigrams
Word Categories
POS tags
Constituents
Phrasal & clausal tags
Grammatical Rules
CFG rules
STYLISTIC FEATURES
Lexical Choices
unigrams / bigrams
Word Categories
POS tags
Constituents
Phrasal & clausal tags
Grammatical Rules
CFG rules
STYLISTIC FEATURES
Lexical Choices
unigrams / bigrams
Word Categories
POS tags
Constituents
Phrasal (e.g., ADJP, WHNP ) & clausal tags (e.g., S, SBAR )
Grammatical Rules
CFG rules
STYLISTIC FEATURES
Lexical Choices
unigrams / bigrams
Word Categories
POS tags
Constituents
Phrasal & clausal tags
Grammatical Rules
CFG rules (e.g. NP^VP NP PP , SBAR S WHNP )
STYLISTIC FEATURES
Setup
Feature encoding: tf-idf
80% training, 20 % testing
5-fold cross validation
LIBLINEAR (Fan et al., 2008) with L2-regulization
EXPERIMENTS
84
75 75
61
82
76 77 78
Adventure Mysterious Fiction History Love Poetry Sci-fi Short story
PREDICTION:
BEST PERFORMANCES
Accuracy (%)
84
75 75
61
82
76 77 78
Adventure Mysterious Fiction History Love Poetry Sci-fi Short story
PREDICTION:
AVERAGE ACCURACY
Average accuracy: 77.2%
Task : discriminating well-written texts (published
work).
Features : widely different writings styles.
Training : limited size.
Testing : only the books by previously unseen authors
THIS IS SURPRISING
BECAUSE…
SECRET ELEMENTS IN
SUCCESSFUL NOVELS
ANALYSIS OF SUCCESSFUL WRITING STYLES
Insights on lexical choices
Distribution of constituents
Connection to sentiment / connotation
Connection to readability
Insights on lexical choices
Distribution of constituents
Connection to sentiment / connotation
Connection to readability
ANALYSIS OF SUCCESSFUL WRITING STYLES
84
75 75
82
76 77 78
Adventure Mysterious Fiction Love Poetry Sci-fi Short story
Accuracy (%)
BEST PERFORMANCES
BY UNIGRAMS
Unigrams
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
risk, worse, slaves, hard,
murdered, bruised, heavy,
prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
risk, worse, slaves, hard,
murdered, bruised, heavy,
prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
risk, worse, slaves, hard,
murdered, bruised, heavy,
prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
risk, worse, slaves, hard,
murdered, bruised, heavy,
prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
risk, worse, slaves, hard,
murdered, bruised, heavy,
prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
More Successful Less Successful
Report /
Quote
said, words, says Emotional /
Action Verbs
want, went, took, jump,
promise, cry, shout, glare
Thinking
Verbs
recognized, remembered Negative
Words
never, risk, worse, slaves,
hard, murdered, bruised,
heavy, prison
Connectives and, which, though, that, as,
but, where, what, whom,
since, whenever
Extreme
Words
never, very, breathless,
sacred, slightest, absolutely,
perfectly
Prepositions up, into, out, after, in, within Love Related desires, affairs
Negation not Location room, beach, bay, hills, avenue,
boat, door
Self-Reference I, me , my Body Parts face, arm, body, skins
Adventure Novels
ANALYSIS OF SUCCESSFUL WRITING STYLES
Insights on lexical choices
Distribution of constituents
Connection to sentiment / connotation
Connection to readability
Distribution of
POS tags (Koppel and Schler (2003))
phrasal tags
clausal tags
DISTRIBUTION OF
CONSTITUENTS
Distribution of
POS tags (Koppel and Schler (2003))
phrasal tags
clausal tags
DISTRIBUTION OF
CONSTITUENTS
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
NP PP VP CONJP QP UCP WHADJP
ADJP ADVP FRAG INTJ WHNP WHPP
Adventure
Mystery
Fiction
History
Love
Poetry
Sci-Fi
Story
Distribution of Phrasal Tags
journalism style
(Douglas and Broussard 2000)
ANALYSIS OF SUCCESSFUL WRITING STYLES
Insights on lexical choices
Distribution of constituents
Connection to sentiment / connotation
Connection to readability
CONNECTION TO
SENTIMENT
0
5
10
15
20
25
30
35
Sentiment words (%) over most discriminative unigrams
Less Successful
More Successful
ANALYSIS OF SUCCESSFUL WRITING STYLES
Insights on lexical choices
Distribution of constituents
Connection to sentiment / connotation
Connection to readability
READABILITY
High Readability
Low Readability
More Successful
Less Successful
READABILITY
More Successful
Less Successful
(Pitler and Nenkova (2008))
Increased use of VP
High Readability
Low Readability
READABILITY
More Successful
Less Successful
Increased use of VP
(Pitler and Nenkova (2008))
High Readability
Low Readability
READABILITY
More Successful
Less Successful
Increased use of VP
(Pitler and Nenkova (2008))
High Readability
Low Readability
READABILITY
High Readability
Low Readability
More Successful
Less Successful
METRIC More Successful Less Successful
FOG index 9.88 9.80
Flesch index 87.48 87.64
?
READABILITY
High Readability
Low Readability
More Successful
Less Successful
METRIC More Successful Less Successful
FOG index 9.88 9.80
Flesch index 87.48 87.64
LITERATURE BEYOND
PROJECT GUTENBERG
LITERATURE BEYOND
PROJECT GUTENBERG
Amazon Best Sellers Rank
Rank lower than 200,000
More Successful Less Successful
PREDICTION
Amazon Best Sellers Rank
Rank lower than 200,000
#Download (400) #Download (400) Train
Test
More Successful Less Successful
(10) (4)
Unigram-feature based
Kl-divergence based
Distribution of phrasal tags of PCFG parse trees
CLASSIFIERS
Unigram-feature based
Kl-divergence based
Distribution of phrasal tags of PCFG parse trees
CLASSIFIERS
Book KL Unigram
More Successful 8/10 10/10
Less Successful 3/4 3/4
“The old man and the sea”
- Ernest Hemingway
✖ ✔
“The lost symbol”
- Dan Brown
✔ ✔
PREDICTION RESULTS
Book KL Unigram
More Successful 8/10 10/10
Less Successful 3/4 3/4
“The old man and the sea”
- Ernest Hemingway
✖ ✔
“The lost symbol”
- Dan Brown
✔ ✔
PREDICTION RESULTS
Book KL Unigram
More Successful 8/10 10/10
Less Successful 3/4 3/4
“The old man and the sea”
- Ernest Hemingway
✖ ✔
“The lost symbol”
- Dan Brown
PREDICTION RESULTS
CASE STUDY - I:
ERNEST HEMINGWAY
• Signature style: minimalism
• 70% simple sentences.
Book KL Unigram
More Successful 8/10 10/10
Less Successful 3/4 3/4
“The old man and the sea”
- Ernest Hemingway
✖ ✔
“The lost symbol”
- Dan Brown
PREDICTION RESULTS
CASE STUDY - II:
DAN BROWN
Commercial success vs. literary critics
Book PDKL Unigram
More Successful 7/10 10/10
Less Successful 3/4 3/4
“The old man and the sea”
- Ernest Hemingway
✖ ✔
“The lost symbol”
- Dan Brown
LITERATURE BEYOND
PROJECT GUTENBERG
PREDICTING SUCCESS OF
MOVIE SCRIPTS
Movie script dataset (Danescu-Niculescu-Mizil and Lee (2011))
Quantifying success: IMDb ratings
More successful : >= 8
Less successful: <= 5.5
11 Genres, 15 movies per class.
PREDICTING SUCCESS OF
MOVIE SCRIPTS
PERFORMANCES
76.7
89.3 86.7
80
Adventure Fantacy Romanance Thriller
Accuracy (%)
RELATED WORK
Literary styles of significant authors
Ellegard (1962), McGann (1998)
Predicting success among academic papers
Sawyer et al. (2008), Bergsma et al. (2012)
Text quality and readability
Louis et al. (2012) , Pitler and Nenkova (2008), Schwarm and
Ostendorf (2005)
RELATED WORK
Study the literary styles of
a few significant authors.
Literary styles of significant authors
Ellegard (1962), McGann (1998)
Predicting success among academic papers
Sawyer et al. (2008), Bergsma et al. (2012)
Text quality and readability
Louis et al. (2012) , Pitler and Nenkova (2008), Schwarm and
Ostendorf (2005)
RELATED WORK
Study on scientific writing
style (non-fiction).
Literary styles of significant authors
Ellegard (1962), McGann (1998)
Predicting success among academic papers
Sawyer et al. (2008), Bergsma et al. (2012)
Text quality and readability
Louis et al. (2012) , Pitler and Nenkova (2008), Schwarm and
Ostendorf (2005)
RELATED WORK
First quantitative study on predicting the success of
literary works based on writing styles.
Insights on successful writings
stylistic syntactic elements of successful writing.
connections between successful writing styles and readability.
links between sentiment / connotation and the literary success.
CONCLUSION
THANK YOU!
![Learning Styles and Variables in Predicting Attitude ... PUH New formated.pdf · Learning Styles and Variables in Predicting Attitude towards Learning . ... [19] . In a study by Honey](https://img.pdfslide.net/doc/110x75/5f07e47e7e708231d41f47b4/learning-styles-and-variables-in-predicting-attitude-puh-new-learning-styles.jpg)
