Proceedings of Discourse Relation Parsing and Treebanking (DISRPT2019), pages 163–168Minneapolis, MN, June 6, 2019. c©2019 Association for Computational Linguistics

163

Applying Rhetorical Structure Theory to Student Essays for ProvidingAutomated Writing Feedback

Shiyan Jiang, Kexin Yang, Chandrakumari Suvarna,Pooja Casula, Mingtong Zhang, Carolyn Penstein Rose

School of Computer ScienceCarnegie Mellon University

Pittsburgh, PA{shiyanj,kexiny, csuvarna,

pcasula, mingtonz, cprose,}@andrew.cmu.edu

Abstract

We present a package of annotation resources,including annotation guideline, flowchart, andan Intelligent Tutoring System for training hu-man annotators. These resources can be usedto apply Rhetorical Structure Theory (RST) toessays written by students in K-12 schools.Furthermore, we highlight the great potentialof using RST to provide automated feedbackfor improving writing quality across genres.

1 Introduction

Recent work in automated essay scoring focuseson local features of writing, often simply topredict grades, though sometimes to offer feed-back (Burstein et al., 2003; Wilson et al., 2017).Our focus is specifically at the rhetorical struc-ture level. Structural writing feedback is de-signed for helping writers to develop a clear struc-ture in which sentences and paragraphs are well-organized (Huang et al., 2017). Researchers havemade much progress in providing feedback for en-hancing writing structure with the development ofintelligent writing systems, such as Writing Men-tor (Madnani et al., 2018) and Writing Pal (Roscoeand McNamara, 2013). However, structural writ-ing feedback generated from existing systems iseither locally situated in individual sentences ornot specific enough for students to take actions.This paper presents how RST can be used toprovide global structural feedback for improvingwriting quality and discusses future work aboutproviding automated writing feedback with deeplearning technology. Our contributions are 1) pre-senting RST annotation resources that can be usedto annotate student essays and 2) highlighting thehuge potential of using RST annotation for pro-viding automated writing feedback in K-12 edu-cation.

2 Creating an Annotated RST Corpus ofStudent Writing

Though there is an existing data source annotatedwith RST (Carlson et al., 2002), for our effort werequired a corpus of student writing that was an-notated with RST. We obtained a student writingcorpus through our partnership with TurnItIn.com.Here we describe the data we received, our effortto develop a coding manual for RST applied to thisdata for our purposes, and the resulting coded cor-pus.

2.1 Source data

Our data is drawn from a set of 137 student es-says from Revision Assistant (Woods et al., 2017),which is an automated writing feedback systemdeveloped by TurnItIn.com. Of the 137 essays, 58are from two genres (i.e., analysis and argumen-tative writing) and were the primary focus of oureffort to design and develop resources to supportour annotation effort, including a fine-grained an-notation flowchart, guideline, and an intelligent tu-toring system (ITS) for training human annotators.As a test of generality, we analyzed the remain-ing 79 essays, which were randomly sampled fromfour genres (i.e., analysis, argumentative, histori-cal analysis, and informative writing).

2.2 Goal of annotation

The goal of annotation is to represent an essayin a rhetorical structure tree whose leaves are El-ementary Discourse Units (EDUs) (Stede et al.,2017). In the tree, EDUs and spans of text are con-nected with rhetorical relations (explained in sec-tion 2.3). We assume a well-structured essay willhave meaningful relations connecting the portions.When meaningful relations connecting EDUs orspans cannot be identified, the assumption is thata revision of structure is needed. The goal of

164

our envisioned automatically generated feedbackis to point out these opportunities for improvementthrough restructuring to students.

More specifically, a span is formed by EDUsconnected with rhetorical relations and usually in-cludes multiple EDUs. For example, Figure 1 rep-resents a tree that includes six EDUs (28-33) andfour spans (span 29-31, 28-31, 32-33, and 28-33).In some cases, a single EDU is a span when thereare no EDUs connecting with it.

Figure 1: Example of RST annotation with rstWeb

Notice that the EDUs of text at the leaf nodesare mostly single sentences. We segment essayswith sentences to represent essays with higherlevel structure and trigger structural feedback. Weused individual sentences as EDUs to providewriting feedback between sentences and para-graphs. Namely, each EDU is a complete sen-tence, which could be indicated by a full stop, ex-clamation, or question mark. However, studentsmight use several sentences in prompt sources orpunctuation in the wrong way. In these two sce-narios, the criteria of punctuation cannot be usedto segment essays into EDUs. Instead, we treatedcontinuous sentences in prompt sources as oneEDU and segmented essays based on correct punc-tuation.

2.3 Adaptation of RST for our Data

Though our goal was to retain as much of the spiritof RST as possible, we adjusted its definitions andscope in order to tailor it for our data. We couldnot share the dataset due to privacy issues. Instead,we clearly demonstrate how to adapt RST for an-notating student essays in this section. Annota-tors need to identify rhetorical relations betweenthree kinds of units: EDUs, spans, and paragraphs.These relations can be divided into two categories:Nucleus-Satellite (N-S) relation and multi-nuclearrelation. N-S relation represents the relation be-

Combine Eliminate ChangeConjunction Condition BackgroundSequence Unless JustifyList Purpose Preparation

Disjunction Summary

Table 1: Adaptation of RST relations.

tween units that are not equally important whilemulti-nuclear relation represents the relation be-tween equally important units. We developed aguideline for annotators to understand the defini-tion and examples of these relations.

Mann and Thompson (1987) defined 23 rhetor-ical relations and the set of relations has been aug-mented with eight more relations. We combined,eliminated, and made minor changes (e.g., divid-ing one relation into multiple ones) to some re-lations for the purpose of providing meaningfulwriting feedback (see Table 1).

Specifically, we use Conjunction to representthe relations of Conjunction, Sequence, and List.These three relations all represent sentences beingconjoined to serve a common purpose, we com-bined them because it is doubtful that distinguish-ing the nuance between these homogeneous rela-tions will have much benefit for triggering writingfeedback. In addition, we eliminated the relationsof Condition, Unless, Purpose, and Disjunction asthey rarely occurred between sentences.

Furthermore, based on the characteristics of stu-dent essays, we made minor changes to the re-lations of Background, Justify, Preparation, andSummary. We divided the relation of Back-ground into two relations, namely the relations ofBackground-1 and Background-2. Background-1 refers to the relation that describes two closelyconnected units in which one of them includespronouns (e.g., it or them) pointing at somethingmentioned in the other one. This makes it nec-essary to present one unit for readers to under-stand the other unit. Background-2 refers to therelation that describes two loosely related units inwhich one unit increases the reader’s ability to un-derstand the other one. We made this distinctionbecause these two relations are very frequentlyseen in students’ essays, yet they can potentiallyprompt for different writing feedback.

In terms of the relation of Justify, we used thecommon scenario of two units being inseparable(i.e., one unit is a question and the other unit is the

165

answer) to identify it. This differs from the rela-tion of Solutionhood as it refers to a pair of answerand question, instead of problem and solution.

In addition, we extended the definition of therelation of Preparation. Our definition of Prepa-ration includes the common scenario of one unitbeing the generic description and the other unit be-ing the detailed description. For instance, one unitis: “I have three reasons to be vegetarian”, and theother unit is: “First, it is healthier, second, it pro-tects animals, and the third is that it saves the earthfrom global warming.” This type of sentence pairsfit the definition of Preparation which describesthat the reader’s comprehending the Satellite in-creases the reader’s readiness to accept the writer’sright to present the Nucleus.

For the relation of Summary, we only looked atthe paragraph level granularity. One unit being thesummary of parts of a paragraph is not useful forproviding feedback and not much different fromthe relation of Restatement, while one unit sum-marizing all other units in a paragraph could indi-cate a high-quality student writing. Therefore, weonly considered cases where one unit summarizesa whole paragraph for providing feedback.

While these changes may seem arbitrary, wefind it necessary to make these changes duringour annotation process to reduce confusion, in-crease inter-rater reliability and identify relationsthat can reveal the structure of student essays andtrigger meaningful writing feedback. Specifically,the first and second author independently anno-tated all essays. Any inconsistencies were dis-cussed and resolved resulting in 100% agreement.

2.4 Annotation process

The structure of the coding manual is driven by theprocess we advocate to human annotators and wefollowed a top-down annotation strategy (Iruskietaet al., 2014). Overall, the annotation process ismeant to consist of five steps:

First step: Segment an essay into EDUs. Thisstep is explained in subsection 2.2.

Second step: Identify central claims in eachparagraph. In this step, annotators should first readthe whole essay and understand its line of argu-mentation. Then annotators should identify EDUsthat are central claims in each paragraph. Identi-fying central claims is useful for deciding whethertwo units are equally important in the third step.

Third step: Identify rhetorical relations be-tween EDUs. In this step, annotators can userstWeb, a tool for RST annotation developed byZeldes (2016), to decide the relations betweenadjacent EDUs in each paragraph from left toright. Specifically, annotators should first deter-mine whether two adjacent EDUs are equally im-portant. The more important EDU is a Nucleuswhile the other EDU is a Satellite. To identifywhether two EDUs are equally important, annota-tors can use the flowchart in Figure 2. Then anno-tators should follow a flowchart (Jiang et al., 2019)to identify the relation. The order of relations inthe flowchart is based on the ease they can be ex-cluded. Namely, the easier it is to decide whetherone relation applies or not, the earlier it appearsin the flowchart. If no relation can be used to de-scribe the relation, then the left EDU is the end ofa span. A span is formed by EDUs connected withrhetorical relations, as described in subsection 2.2.

Figure 2: Flowchart for identifying importance

Fourth step: Identify rhetorical relations be-tween spans. In this step, annotators should iden-tify relations between spans within each paragraphfrom left to right. When identifying relations be-tween two spans, annotators should use the sameflowchart in the third step to determine relationsbetween the Nucleus of two spans. If no relationexists between spans, annotators should use Jointto build the paragraph into a tree.

Fifth step: Identify rhetorical relations betweenparagraphs. Annotators should identify relationsbetween paragraphs from left to right. Similar tothe fourth step, annotators should determine therelation between the Nucleus of two paragraphs.If any of the two paragraphs contain the relationof Joint, it indicates that spans in the paragraph do

166

not have strong relations. In this case, the relationof Joint should be used to connect two paragraphs.

2.5 Practical RST Intelligent Tutor

Based on the flowchart and guideline that makeup our coding manual, we developed an Intelli-gent Tutoring System (ITS) to help novice annota-tors learn RST annotation efficiently. We built thePractical RST Intelligent Tutor using CognitiveTutor Authoring Tools (CTAT), an authoring toolfor ITS (Aleven et al., 2009). This tutor (Figure3, access upon request) is hosted on an open plat-form TutorShop that provides free access for re-search or public use. As shown in Figure 3, anno-tators are first presented with three RST relationsincluding their definitions, examples of sentencepairs, conjunction phrases, and groups. Conjunc-tion phrases refer to connection words or phrasesthat can naturally conjoin two spans. For exam-ple, “because” can be used to connect two spansthat indicate the relation of Cause. Groups referto the categories of relations: progressive, supple-mentary, conjunct, repeating, contrast or no rela-tion. These categories represent a higher level ofRST relations. Annotators are then guided to iden-tify the relation of a given sentence pair, and arescaffolded with step by step procedures and hintsto complete the task.

To develop the system, we conducted tworounds of cognitive task analysis (Crandall et al.,2006), respectively with five subjects who had noprior experience in RST and three subjects withexperience in RST. After analyzing think-alouddata from the first round, we found that noviceannotators regularly referred back to the defini-tion of RST relations, compared given sentencepairs with provided examples, and inserted con-junction phrases between annotation units to seewhether it made sense logically. Based on thesefindings, we developed an initial intelligent tutor-ing system. We further ran a pilot study involv-ing target users with background or experience inRST. These users further provided feedback onboth the interface and instructional design. Werefined our tutor accordingly with additional fea-tures of arranging problems from easy to hard,adjusted granularity of step-loop, and used moreharmonious visual design. This intelligent tutoralso takes advantage of the Bayesian KnowledgeTracing algorithm (Baker et al., 2008) developedat Carnegie Mellon University to provide adap-

Figure 3: Interface of Practical RST Intelligent Tutor

tive problem selection, which can assist learnersto achieve mastery in four knowledge components(i.e. identifying groups, conjunction phrases, nu-clearity, and relations) about identifying RST rela-tions (Koedinger et al., 2012).

3 From RST Analysis to WritingFeedback

Here we explain the potential of using RST forproviding structural writing feedback across gen-res and for specific genres.

RST can be used to provide writing feedbackfor enhancing coherence across genres. Coher-ence refers to how sentences in an essay are con-nected and how an essay is organized. RST couldbe used to provide actionable writing feedback forincreasing the level of coherence that traditionalautomated coherence scores were deemed insuffi-cient to realize. Specifically, the relation of Jointindicates a low level of coherence. As an exam-ple, Figure 1 is an annotation of one paragraphfrom student writing. This paragraph includes twospans (i.e., span 28-31 and span 32-33) that are notconnected clearly. In span 28-31, the writer listedthree benefits of joining a club. In span 32-33, the

167

writer might intend to encourage people to jointhe club while the intention is not clear as thereis no mention of joining the club. The RST treehad the potential of giving more concrete contextfor low-level coherence and in this way, studentscould identify where they can make revisions forclearer structure.

In terms of providing feedback in specific gen-res, the combination of relations can indicate high-quality writing. For example, presenting and ana-lyzing evidence is an indication of high-quality ar-gumentative writing (Gleason, 1999). Researchershave made much effort in predicting whether thereis evidence in student writing and pointed out theneed for future studies in examining how evidencewas used to show the soundness and strength ofarguments. RST can be used to meet the needwith predicting the combination of relations, suchas the combination of evidence and interpretationor the combination of evidence and evaluation.

Furthermore, RST is valuable to provide writ-ing feedback in analysis writing. Making com-parisons is a common structure of well-organizedanalysis writing. It’s easy to identify sentences in-volving comparison locally. However, identifyingthe whole structure of making comparisons in anessay remains to be a challenging automation task.RST has the potential to address the challenge byillustrating a global comparative structure with therelation of Contrast, Antithesis, or Concession.

4 Conclusion

We take full advantage of RST in providingstructural feedback for enhancing writing qualityacross genres. Currently, based on the work fromLi et al. (2014), we are building an RST parser thatcan generate RST trees to represent student essaysautomatically with deep learning techniques. Inthe future, we plan to build the work from Fiaccoet al. (2019) to generate RST trees more accu-rately and efficiently. Our long term goal is to em-bed these techniques in a writing tutor like Revi-sion Assistant and conduct large-scale classroomstudies to evaluate the effect of RST trees in writ-ing instruction.

Acknowledgments

This work was funded in part by NSF grant#1443068 and a Schmidt foundation postdoc fel-lowship. We thank our collaborators at Tur-nItIn.com for providing student essays. We also

thank Vincent Aleven and Jonathan Sewall fordesign suggestions and guidance in building thepractical RST intelligent tutor.

ReferencesVincent Aleven, Bruce McLaren, Jonathan Sewall, and

Kenneth R. Koedinger. 2009. Example-tracing tu-tors: A new paradigm for intelligent tutoring sys-tems. International Journal of Artificial Intelligencein Education, 19(2):105.

Ryan S. Baker, Albert T. Corbett, and Vincent Aleven.2008. More accurate student modeling through con-textual estimation of slip and guess probabilities inBayesian Knowledge Tracing. In International con-ference on intelligent tutoring systems, pages 406–415.

Jill Burstein, Daniel Marcu, and Kevin Knight. 2003.Finding the write stuff: automatic identification ofdiscourse structure in student essays. IEEE Intelli-gent Systems, 18:32–39.

Lynn Carlson, Daniel Marcu, and Mary EllenOkurowski. 2002. RST Discourse Treebank. Lin-guistic Data Consortium, University of Pennsylva-nia.

Beth Crandall, Gary Klein, and Robert R. Hoffman.2006. Working Minds: A Practitioner’s Guide toCognitive Task Analysis. A Bradford Book.

James Fiacco, Elena Cotos, and Carolyn Rose. 2019.Towards enabling feedback on rhetorical structurewith neural sequence models. In Proceedings of the9th International Conference on Learning Analyticsand Knowledge, pages 301–319.

Mary M . Gleason. 1999. The role of evidence in argu-mentative writing. Reading and Writing Quarterly,15(1):81–106.

Yi-Ching Huang, Jiunn-Chia Huang, Hao-ChuanWang, and Jane Yung jen Hsu. 2017. SupportingESL writing by prompting crowdsourced structuralfeedback. In Proceedings of the Fifth AAAI Confer-ence on Human Computation and Crowdsourcing,pages 71–78.

Mikel Iruskieta, Arantza Daz de Ilarraza, and MikelLersundi. 2014. The annotation of the central unitin rhetorical structure trees: A key step in annotat-ing rhetorical relations. In Proceedings of COLING2014, the 25th International Conference on Compu-tational Linguistics: Technical Papers, pages 466–475.

Shiyan Jiang, Kexin Yang, Chandrakumari Suvarna,and Carolyn Ros. 2019. Guideline and Flowchartfor Rhetorical Structure Theory Annotation. Tech-nical report, Carnegie Mellon University, School ofComputer Science.

168

Kenneth R. Koedinger, Albert T. Corbett, andCharles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning.Cognitive Science, 36:757–798.

Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recur-sive deep models for discourse parsing. In Proceed-ings of the 2014 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages2061–2069.

Nitin Madnani, Jill Burstein, Norbert Elliot,Beata Beigman Klebanov, Diane Napolitano,Slava Andreyev, and Maxwell Schwartz. 2018.Writing mentor: Self-regulated writing feedback forstruggling writers. In Proceedings of the 27th Inter-national Conference on Computational Linguistics:System Demonstrations, pages 113–117.

William C. Mann and Sandra A. Thompson. 1987.Rhetorical Structure Theory: A Theory of Text Or-ganization. University of Southern California, In-formation Sciences Institute.

Rod Roscoe and Danielle McNamara. 2013. Writingpal: Feasibility of an intelligent writing strategy tu-tor in the high school classroom. Journal of Educa-tional Psychology, 105(4):1010–1025.

Manfred Stede, Maite Taboada, and Debopam Das.2017. Annotation guidelines for rhetorical structure.Version 1.0.

Joshua Wilson, Rod Roscoe, and Yusra Ahmed. 2017.Automated formative writing assessment using alevels of language framework. Assessing Writing,34:16–36.

Bronwyn Woods, David Adamson, Shayne Miel, andElijah Mayfield. 2017. Formative essay feedbackusing predictive scoring models. In Proceedings ofthe 23rd ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, pages2071–2080.

Amir Zeldes. 2016. rstWeb-a browser-based annota-tion interface for Rhetorical Structure Theory anddiscourse relations. In Proceedings of the 2016 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Demonstra-tions, pages 1–5.