Promoting Argument-Counterargument Integration in Students' Writing

This article was downloaded by: [UTSA Libraries]On: 06 October 2014, At: 14:10Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

The Journal of ExperimentalEducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/vjxe20

Promoting Argument-Counterargument Integration inStudents' WritingE. Michael Nussbaum a & Gregory Schraw aa University of Nevada, Las VegasPublished online: 07 Aug 2010.

To cite this article: E. Michael Nussbaum & Gregory Schraw (2007) Promoting Argument-Counterargument Integration in Students' Writing, The Journal of Experimental Education, 76:1,59-92, DOI: 10.3200/JEXE.76.1.59-92

To link to this article: http://dx.doi.org/10.3200/JEXE.76.1.59-92

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoeveras to the accuracy, completeness, or suitability for any purpose of the Content. Anyopinions and views expressed in this publication are the opinions and views of theauthors, and are not the views of or endorsed by Taylor & Francis. The accuracy ofthe Content should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for any losses,actions, claims, proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly in connection with, inrelation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms& Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/loi/vjxe20

http://www.tandfonline.com/action/showCitFormats?doi=10.3200/JEXE.76.1.59-92

http://dx.doi.org/10.3200/JEXE.76.1.59-92

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

59

Promoting Argument–Counterargument Integration in Students’ Writing

E. MICHAEL NUSSBAUMGREGORY SCHRAWUniversity of Nevada, Las Vegas

ABSTRACT. It is important, when writing opinion essays, for students to consider and integrate both arguments and counterarguments to develop a final conclusion. In this article, the authors explored the effect of criteria instruction and a graphic organizer to promote integration of arguments and counterarguments. The researchers randomly assigned 84 participants from an undergraduate educational psychology course to 1 of 4 conditions: training only, organizer only, combined, and control. The graphic organizer resulted in more refutations of counterarguments. However, criteria instruc-tion resulted in better integration of argument and counterargument (with stronger rebuttals and more balanced reasoning). The authors discussed how the 2 interventions may have activated somewhat different argumenta-tion schema in students.

Keywords: argument, argumentation, critical thinking, graphic organizers, writing

EFFECTIVE ARGUMENTATION SKILLS are essential for good communica-tion. A good argument is one that is organized, elaborated, and supported by evidence or personal experience (Perloff, 2003). It is important for educators to help students develop effective argumentation skills because research suggests that these skills are often poor (Applebee, Langer, Mullis, Latham, & Gentile,

The Journal of Experimental Education, 2007, 76(1), 59–92 Copyright © 2007 Heldref Publications

Address correspondence to: E. Michael Nussbaum, Department of Educational Psychology, Box 453003, University of Nevada, Las Vegas, 4505 Maryland Parkway, Las Vegas, NV 89154-3003, USA. E-mail: [email protected]

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014

60 The Journal of Experimental Education

1994; Kuhn, 1991). Researchers have found that good argumentation skills pro-mote learning and conceptual change in science (Alexopoulou & Driver, 1996; Baker, 1999), understanding of mathematical concepts (Lampert, Rittenhouse, & Crumbaugh, 1996; Schwarz, Neuman, & Biezuner, 2000), better mathematical problem solving (Vye et al., 1997), and comprehension of issues in history and social studies (De La Paz, 2005; Wiley & Voss, 1999). Argumentation can also enhance intrinsic motivation and problem-solving performance (Chinn, 2006).

Counterargumentation

Good arguments typically have multiple sides, thus presenting students with opportunities to engage in deeper processing. Deeper processing is more likely to occur when students consider counterarguments to their own (or others’) ar-guments and integrate their arguments and the counterarguments into an overall final position. However, students often do not generate any counterarguments (Perkins, Farady, & Bushey, 1991). Lack of counterarguments occurs in writing (Knudson, 1992; Leitão, 2003; Stapleton, 2001) and oral and Internet discus-sions (Koschmann, 2003). There are several possible reasons for these findings, including high cognitive load (Coirier, Andriessen, & Chanquoy, 1999) and a desire to maintain cognitive consistency (Simon & Holyoak, 2002). Furthermore, students often do not realize that considering and rebutting an opposing side often increases the persuasiveness of their arguments (Nussbaum & Kardash, 2005), per a number of empirical studies (O’Keefe, 1999).

Integration of Argument–Counterargument

Effective argumentation involves not only considering counterarguments but also evaluating, weighing, and combining the arguments and counterarguments into support for an overall final position. In this article, we call this process argument–counterargument integration. We make an analogy between such integration and a jigsaw puzzle. Thus, argument–counterargument integration is like putting the pieces together, although in some cases one has to turn over (i.e., negate) pieces. We propose that students can use three strategies to integrate arguments with counterarguments. Consider the argument that “candy should be banned from school because it makes kids overly active and hurts concentration” versus the counterargument that “children should sometimes have candy in school because it makes them more satisfied and ready to learn.” By a final overall argu-ment—which addresses both the argument and the counterargument—one might try to rebut the counterargument (e.g., by arguing that there are other things, such as additional recess or physical education, that can satisfy children more or serve as rewards). We call this a refutation strategy. Alternatively, one might develop a final conclusion that represents a compromise between the argument and the

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014

Nussbaum & Schraw 61

counterargument (e.g., arguing that children should only be allowed to eat candy during the last hour of the school day, so that any resulting hyperactivity will not distract them from learning). We call this second strategy synthesizing because it develops an in-between position or solution that combines the merits of both sides. A third strategy, weighing, is to argue that the weight of evidence on one side of the issue is stronger than that on the other or that the advantages of a solution outweigh the disadvantages.

We use the term integration as an umbrella term to refer to all these strate-gies, because in each case, the writer is attending to both sides of an issue while developing a final position. With a refutation strategy, one integrates rebuttals into one’s justifications for a position, and the rebuttal implicitly acknowledges and responds to the counterargument. With a synthesis strategy, the final position (argument) explicitly integrates both sides. In a weighing strategy, the author and the audience consider the two sides simultaneously.

Argument–counterargument integration is important for several reasons. First, the process requires students not only to make arguments, but also to examine how arguments (including counterarguments) are related. Therefore, the process involves students in elaborating and organizing their thinking, which facilitate learning (Mayer, 2003).

Second, argument–counterargument integration makes written arguments logi-cally stronger (Voss, 2001). In a meta-analysis of 107 studies on audience response to persuasive messages, O’Keefe (1999) found refutational two-sided messages to be the most credible and persuasive (except when the messages related to adver-tising). Refutational two-sided messages seem more credible because the writer appears more knowledgeable and less biased. They are more persuasive because the writer addresses and refutes some of the reader’s potential objections. Although some researchers have suggested that a writer should not present counterarguments when anticipating that the audience will agree with the position being advocated (Hovland, Lumsdaine, & Sheffield, 1949), O’Keefe found little empirical support for that suggestion.

Third, argument–counterargument integration is important because it is a central aspect of critical thinking. Researchers have often viewed sophisticated thinking as dialogic (Bakhtin, 1981; Lotman, 1988; Vygotsky, 1962), involving active open-minded thinking (Baron, 1988) that considers counterarguments (Johnson, 2002; Kuhn, 2005). For example, in promoting dialogic thinking, Paul (1986) argued that all thinking takes place in a frame of reference, but the only way to evaluate the soundness of the frame is to compare various frames against one another dialecti-cally. In Paul’s view, most thinking is monological, but with effort, individuals can learn to think dialogically, evaluating and integrating various frames. We intend our notion of argument–counterargument integration to give more specificity to the notion of dialogic thought so that researchers can study it in empirical and theoretical terms.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Argument–counterargument integration requires one’s attention to what phi-losophers call the macrostructure of arguments (Freeman, 1991). This level of analysis stands in contrast to microstructural analysis, which examines the struc-ture of specific arguments (usually by identifying argument parts; e.g., conclu-sions, warrants, and backing; see Toulmin, 1958). Although many researchers in education have examined students’ arguments in terms of microstructure by using the Toulmin model (Bell, 2002; Chambliss & Murphy, 2002; Erduran, Simon, & Osborne, 2004; Kelly, Drucker, & Chen, 1998; Yeh, 1998), macrostructure has received less attention, especially among North American researchers. Means and Voss (1996) examined three types of argument structures and found that the use of the most complex one (involving counterarguments) was positively correlated with general ability. Kuhn (1991) examined causal arguments in the context of clinical interviews and found that few participants were able to make valid counterarguments and that even fewer were able to make integrative rebut-tals, which address both sides of an argument. Leitão (2000) examined replies to counterarguments in the context of everyday discourse (specifically university department meetings and weekly meetings of a community group) and found examples of integrative replies—defined as responses that agree in part with the counterarguments—but did not quantify their frequency. Overall, researchers have suggested that argument–counterargument integration is a difficult process. They have linked developmental trends in integration ability to growth in work-ing-memory capacity and the acquisition of verbal reasoning structures that one acquires from reading and observation (Means & Voss) and to improvements in metacognitive abilities and epistemological understanding (Kuhn, 1991, 2001). In particular, the move to dialogic thinking requires one to overcome egocen-trism (Paul, 1986) and to think critically about one’s theories, not just with one’s theories (Kuhn, 2005).

However, none of these researchers specifically examined written discourse. We know that writing imposes high cognitive demands related to planning, translating, and revising of written text (Mayer, 2003). However, writing allows individuals to externalize and reflect on their thoughts, lowering load on working memory. Bernardi and Antolini (1996) examined the integrative complexity of written argumentative texts that pupils in grades 3, 5, 7, and 11 produced, and those researchers found that integration improved with grade level.

In the personality and cognitive-styles literature, researchers have also mea-sured individuals’ integrative complexity (Suedfeld, Tetlock, & Streufert, 1992), which is the degree to which people make distinctions (differentiation) in oral or written discourse and conceptual connections between these distinctions (inte-gration). Researchers have typically assessed integrative complexity by analyz-ing either archival data or by having participants write essays (Baker-Brown et al., 1992; Liht, Suedfeld, & Krawczyk, 2005). Hunsberger, Lea, Pancer, Pratt, and McKenzie (1992) found that levels of this construct can be increased with

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


prompting, but those researchers had more success with enhancing differentia-tion than integration per se.

In this study, we did not use integrative complexity as a construct because it requires special training to administer the measure. Nevertheless, we were concerned with how one can facilitate argument–counterargument integration. However, before addressing this specific question, we will review some of the literature on the training of students in argumentation skills.

Research on Instruction in Argumentation

One question that we address in this article is whether exposing students to the qualities of a good argument can be effective in enhancing both (a) the gen-eration of counterarguments and (b) argument–counterargument integration. Re-searchers have previously investigated the training of students in argumentation (Nussbaum, Sinatra, & Poliquin, in press). Some of the researchers conducted their investigations on explicit instruction in argumentation in the context of oral or electronic discussions (e.g., Kuhn, 2005; Sprod, 1998), typically involving the teaching of students to respect and consider alternative points of view (Marttunen & Laurinen, 2001; Mercer, 1996; Waggoner, Chinn, Yi, & Anderson, 1995), to consider and weigh evidence (Anderson, Howe, Soden, Halliday, & Low, 2001; Driver, Newton, & Osborne, 2000; Herrenkohl & Guerra, 1998), and to gener-ate arguments by looking for problems, solutions, and consequences (Rancer, Whitecap, Kosberg, & Avtgis, 1997). Few empirical studies have focused specifi-cally on training students to write argumentative text, although such training is a component of most high school curricula. These studies have generally focused on training students to provide evidence for their assertions (Bensley & Haynes, 1995; De La Paz & Graham, 2002; Yeh, 1998) with generally positive results.

In addition, Fulkerson (1996b) noted that the field of English composition emphasizes teaching students the Toulmin model of argument (that arguments have claims, grounds, warrants linking grounds to claims, and rebuttals), but that only anecdotal evidence has supported the effectiveness of teaching the model on the quality of students’ essays. At the junior-college level, McCleary (as cited in Fulkerson) conducted a study in English-composition classes compar-ing teaching the Toulmin model with teaching deductive logic, but he found no statistically significant differences between the experimental and control groups (he used 15 course sections in total). This may be because applying the Toulmin model (and especially the concept of warrants) may be complex and awkward (Stein & Albro, 2001) unless one applies extensive scaffolding (Yeh, 1998; see also Knudson, 1994, for another study, involving grades 3–12 students, in which direct instruction using the Toulmin model was not effective).

However, Ferretti, MacArthur, and Dowdy (2000) found that instructing stu-dents to set a subgoal to generate counterarguments and rebuttals did improve the

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


overall persuasiveness of essays, specifically among sixth graders, but not among fourth graders (see also Dolz, 1996). The training resulted in essays that were longer and more persuasive. This effect presumably occurred because setting goals and subgoals helps students with self-regulation during writing, especially during planning. However, one limitation of the study was that the researchers did not specifically measure and report the number of counterarguments (or their development).

Nussbaum and Kardash (2005) built on Ferretti, MacArthur, and Dowdy’s (2000) work to evaluate the effect of various goal instructions—which are brief statements that instructors add to the end of a writing prompt that specify what the student should try to achieve—on college students’ written argumentation. Results indicated that a general goal to persuade the reader decreased the num-ber of counterarguments considered, with students believing that considering counterarguments (even if rebutted) would reduce the persuasiveness of their essays (see also Santos & Santos, 1999). However, students whom researchers asked to consider and rebut counterarguments increased counterargument gen-eration when compared with a control group. Therefore goal instructions must be specific. Ferretti et al. called their goal instruction a subgoal because it helped students to achieve the larger, implicit goal of persuading the reader.

Nussbaum and Kardash (2005) also found that knowledge of the issue enhanced counterargumentation. In particular, when those researchers gave students a text outlining potential lines of argumentation and counterargumenta-tion, the number and quality of counterarguments increased (as long as students were also not asked to persuade and did not have strong prior beliefs on the issue). Students did not merely reiterate the counterarguments in the text of their essays but also selected and elaborated on the most important ones and then developed rebuttals.

Nussbaum and Kardash (2005) concluded that providing students with specific goal instructions and requiring them to read a text before writing their essays in-creased the consideration of counterarguments. However, those researchers noted several limitations to their findings. First, the counterarguments that students considered were not necessarily well developed. Second, their study (as well as other studies in this area) did not examine students’ use of integration strategies other than rebuttals, such as synthesis and weighing, and the particular goal in-structions may have discouraged students from using these other strategies.

One goal of our study was to see if a slightly expanded intervention (and revi-sion of the coding system) could remedy these problems. The attractiveness of goal instructions as an intervention is that they are extremely easy for researchers and teachers to implement. However, we felt that a slightly expanded interven-tion would be more cost-effective. Our purpose was to develop a brief, relatively efficient intervention that improved the quality of students’ argumentative writ-ing. We did not want to privilege one integration strategy over another.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Our revised intervention was to expose students, through examples, to the normative criteria of what makes a good argument, including the consideration and integration of counterarguments. These normative criteria would not only trigger planning activity but could also serve as goals and subgoals that could help structure students’ planning prior to writing. Models and worked examples are ef-fective instructional tools in other domains (Atkinson, Derry, Renkl, & Wortham, 2000), especially when instructors emphasize goals and subgoals of various solu-tion steps (Catrambone, 1998). We did not know whether brief descriptions and examples would be sufficient to induce better written argumentation. However, researchers have found brief goal instructions have moderate effects (Nussbaum, 2005; Nussbaum & Kardash, 2005). We hypothesized that unlike goal instructions, normative criteria that do not specifically focus students on a refutation strategy would encourage students to engage in more balanced reasoning, in which stu-dents explore both sides of an issue in greater depth. Furthermore, a refutation strategy reflects an adversarial form of argumentation and is less consistent with current trends to foster collaborative reasoning and argumentation (Andriessen, Baker, & Suthers, 2003; Mallin & Anderson, 2000). The latter fosters greater search of the problem space (Keefer, Zeitz, & Resnick, 2000).

Research on Graphic Organizers and Argumentation

In addition to providing verbal instruction on the criteria of a good argument, we evaluated the effect of visual aides that can simultaneously convey these crite-ria and scaffold students’ planning activities. Several researchers have suggested that when expert writers compose essays, an argumentation schema—which is a typical pattern for organizing arguments—guides them (Ferretti et al., 2000; Hidi & Hildyard, 1983; Reznitskaya et al., 2001). A claim-support schema ap-pears to develop first (Chambliss & Murphy, 2002) followed for some writers by an argument–counterargument–rebuttal schema (Ferretti et al.) or a pro–con schema (Piolat, Roussey, & Gombert, 1999). According to Piolat et al., these more complex schemata tend to develop in adolescence. However, according to the aforementioned research, most students do not necessarily use them consis-tently without prompting, even in college.

This circumstance suggests that a graphic organizer, visually displaying a two-sided structure, would help students to activate, strengthen, and refine their existing schemata or to develop new ones. A graphic organizer could also help to scaffold the students’ ability to integrate arguments and counterarguments. Although verbal instructions to generate arguments and counterarguments have also been modestly successful (Nussbaum & Kardash, 2005), they do not neces-sarily lower the load on working memory that is necessary to coordinate and inte-grate arguments and counterarguments, whereas graphic organizers can. Graphic organizers have been successful in improving reading comprehension among

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


college students for this reason (Moore & Readence, 1984) and in promoting cognitive organization more than does just taking notes or making an outline (Robinson & Kiewra, 1995).

A body of research also exists regarding computerized visual displays that people use to scaffold argumentation (Bell, 2002). For example, in the Belvedere system (Toth, Suthers, & Lesgold, 2002), students collaboratively constructed argumentation diagrams regarding scientific investigations. Students (ninth grad-ers) considered more evidence opposing their conclusions when constructing ar-gumentation maps, but that effect did not transfer to prose (final reports on their investigations), perhaps because of the cognitive load involved in their working with two different computer interfaces. (Cho and Jonassen [2002] reported some transfer effects of the Belvedere system on written essays among college students but did not report specific figures regarding the effects specifically on counter-arguments and rebuttals.)

Although argumentation diagramming focuses attention on how specific claims and pieces of evidence relate to one another, it is not clear that argument diagramming alone is sufficient to trigger a more global argumentation schema that can guide students during writing activities. Also, argumentation diagram-ming does not typically prompt for argument–counterargument integration or synthesis. Schwarz and Glassner (2003) developed an interface with a more global, pro–con structure, but they have not used it to support essay writing.

Our goal was to examine the effects of both a graphic organizer and criteria instruction (with examples) on the integration of arguments with counterargu-ments in argumentative writing. We proposed that students would become at-tuned to the importance of considering counterarguments through either explicit instruction or a graphic organizer that helped them to order their thoughts. We examined the effects of two interventions that we hypothesized would affect in-tegration: (a) instruction on the criteria of a good argument (including argument– counterargument integration) and (b) the use of a graphic organizer that allowed students to visually organize arguments, counterarguments, and their integra-tion. We also examined the interaction of these two interventions. As we noted earlier, our purpose was to develop brief, relatively efficient interventions that would improve the development and integration of students’ arguments and counterarguments. We did not know a priori to what extent brief interventions would suffice to achieve this aim or what aspects of students’ argumentation would be most affected. However, we hypothesized (a) that both would have positive effects on the degree of argument–counterargument integration and would help structure students’ planning activities and (b) that the graphic orga-nizer would also help to reduce cognitive load. Furthermore, we felt that argu-ment criteria training would facilitate the use of integration strategies other than (or in addition to) refutation because we could provide examples during training of the various strategies.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Method

Participants

Participants were 84 undergraduates who had enrolled in educational psychol-ogy courses and who participated in this study to satisfy a course requirement (88% were seeking a teaching credential). The students were juniors (61%), seniors (23%), sophomores (14%), and freshmen (2%). Most of the participants were majoring in a program on elementary education (35%), secondary educa-tion (18%), or some other type of education (25%). Most of the others were majoring in psychology (8%), were majoring in science or the humanities (7%), or had not declared their majors. In terms of ethnicity, 66% indicated White, 11% indicated Asian or Pacific Islander, 6% indicated African American, 6% indi-cated Hispanic, and 11% indicated other. In terms of gender, 80% were women, and 20% were men. Participants’ ages ranged from 18–55 years, with an average age of 25 years. Their average grade point average was 3.13.

Design and Materials

We manipulated two independent variables: (a) a graphic organizer and (b) in-struction on the criteria of a good argument. We randomly assigned the participants to each of the four conditions: control, criteria instruction only, graphic organizer only, and the graphic organizer combined with instruction. In the following subsec-tions, we will describe the materials that we used in each of the conditions.

Criteria instruction materials. The criteria instruction materials included primarily a handout that listed and described the criteria of a good argument. The criteria included (a) having a clear position, (b) providing adequate supporting reasons, (c) presenting counterarguments, (d) considering both arguments and counter-arguments in developing the final conclusion (i.e., argument–counterargument integration), and (e) organization. Appendix A shows the actual handout. We also developed a verbal script, which we read to the participants, along with several overheads. We intended the script and overheads to help walk the participants through the handout. We explained and applied the criteria to issues that were cur-rent at the time (the war in Iraq and whether the singers Janet Jackson and Justin Timberlake should be fined for exposing Jackson’s breast during a Super Bowl halftime broadcast). Appendix B shows the actual script.

Graphic organizer. We developed a graphic organizer so that participants could plan arguments and counterarguments, supporting reasons, and a final conclusion in preparation for writing their essays. The organizer’s dimensions were 16 × 10 in. The large size afforded students plenty of room to write. Figure 1 shows the actual organizer (not to scale). We constructed the organizer so that participants

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


would complete both arguments and counterarguments. We placed ovals for reasons below the arguments and counterarguments to indicate that the reasons supported the arguments and counterarguments. (Although we expected that many participants might include some reasons in the argument ovals—technically at least one reason needs to be included to form an argument—we wished to encourage participants to include secondary reasons, or supporting evidence, for the primary reasons, thus deepening and developing the argument). At the top of the graph was an oval that we labeled Final Conclusion. The purpose of this oval was to help students to weigh the relative strengths of the arguments and counterarguments in developing a final conclusion, to develop rebuttals of one side, or to synthesize the arguments and counterarguments to develop an in-between position. We placed the final conclusion oval on top of the graph to indicate that it was supported by the arguments and counterarguments. We drew upward arrows from the arguments and counterarguments to the final conclusion to indicate that the final conclusion should flow from the arguments and counterarguments.

To indicate to students the order in which they should complete the ovals, and more generally how they should complete the organizer, we also provided them with a short, one-page handout that we titled “Explanation of Worksheet.” The handout presented the following text:

1. Argument: Place an initial position on the issue here. Write reasons that sup-port that position in the ovals underneath.

2. Counterargument: Place a counterposition here (for example, an objection to the initial position or a different position on the issue). Write reasons that sup-port the counterargument in the ovals underneath.

3. Final conclusion: After considering both the argument and counterargument, write your final conclusion here, along with any additional supporting reason-ing that you may have.

FIGURE 1. Graphic organizer for developing arguments, counterarguments, and a final conclusion on an issue.

Final Conclusion

CounterargumentArgument

Reason ReasonReasonReasonReasonReason

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


NOTE—If you would like, you may make additional ovals on the worksheet to further elaborate your arguments.

Writing prompt and text. For writing their essays, we gave participants the fol-lowing writing prompt:

Please write an essay expressing your opinion on the following question, “Does watching television cause children to become more violent?”

You will have thirty minutes to work on your essay. Feel free to first spend a few minutes organizing your thoughts and even making an outline. Don’t worry too much about spelling, grammar, punctuation, or writing style.

We used the same prompt as we did in several previous studies (Nussbaum, 2005; Nussbaum & Kardash, 2005) to enable some cross-study comparisons.

In addition, prior to writing their essays, we provided participants with a text that outlined some of the arguments and counterarguments that they might consider. The text (508 words long)—which was also used in the Nussbaum and Kardash (2005) study—summarized four arguments on one side of the issue and a similar number on the opposite side, generally alternating between pro and con arguments. Nussbaum and Kardash found that provision of a text stimulated students’ thinking and improved the quality of their arguments. In this study, we provided this text to all students to ascertain how they would try to integrate the arguments on both sides, if at all.

Procedure

After the students completed informed consent forms and a short demograph-ics survey, we showed all participants an overhead defining an argument. In particular, it stated, “Arguments are stands you take on an issue, supported by reasons and evidence.” We then read the following script:

In this study, we are going to ask you to write an essay where you make arguments. Arguments are stands you take on an issue, supported by reasons and evidence. For example, you could argue that the war in Iraq was justified in order to eliminate weapons of mass destruction. Alternatively, you could argue that the war in Iraq was not justified, because there was little evidence of such weapons. As another example of an issue you could form arguments about, consider the recent controversy with Janet Jackson and Justine Timberlake at the Super Bowl. [Show picture of perform-ers.] One might make an argument that the performers should be fined for indecent exposure. The reason one might give to support this claim is that a fine is necessary to deter performers from ever trying this again.

The script gave examples of arguments and counterarguments but did not elaborate on the differences between a good argument and a weak argument. Then, for the argument criteria instruction conditions, we provided a lesson on the argument criteria (see Appendix B, Steps A and B) with the handout

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


on argument criteria. Then, for all conditions, we gave students 5 min to read the text on television violence. (The researcher explained that students were to write an essay on a new topic but first would read a text on some of the argu-ments that they could use, although they could also think of their own.) Students then handed back the text. Next, we provided students in the graphic organizer conditions with 5 min to complete the graphic organizer (as well as to read the accompanying instructions). Then, students handed back the organizers. They did not have access to the organizers while writing their essays because we were interested in the effect of the organizers on students’ encoding and elaboration of arguments and did not want to confound this with retrieval effects. Last, we provided students with the writing prompt, five sheets of paper, and 30 min to write their essays.

Coding

We coded the essays by employing the same system that Nussbaum and Kardash (2005) used. It was similar to systems that some other argumentation researchers (cf. Ferretti et al., 2000; Kuhn, 1991; Page-Voth & Graham, 1999) have used. According to Inch and Warnick (2002), there are two types of concep-tual models for analyzing argumentation: (a) the standard model (which analyzes how various claims are structured to create arguments, counterarguments, and re-buttals; see Beardsley, 1950) and (b) Toulmin-based (1958) models, which seek to further categorize supporting claims, including implicit claims, into grounds and warrants (which link grounds and claims). Because Toulmin-based models introduce a layer of complexity that is often unnecessary for analytic purposes (Fulkerson, 1996a, 1996b) and that complicates coding, we based our coding on the standard model.

The basic unit of analysis was the idea unit, the number of different ideas that participants expressed in the text (Chafe, 1985). First, we identified each student’s position on the issue (i.e., “TV causes violence,” “TV does not cause violence,” or “It depends”), which we termed the final claim. Second, we ascer-tained what reasons the student presented as supporting the final claim, which we called supporting claims. Third, we examined whether the student presented any reasons (or evidence) for these claims, which we termed supporting reasons. Fourth, we coded whether any participants presented counterclaims that refuted one of their other claims and whether there were any supporting reasons for the counterclaims. Last, we ascertained whether the essay contained any rebuttals, that is, claims that refuted a counterclaim, and whether the essay provided any reasons that supported the rebuttals. We also created a new category that we had not used in our previous studies: reservations, defined as brief qualifications serving as rebuttals to short or implicit counterarguments. (That new category is one that we did take from the Toulmin, 1958, model). Table 1 (which we adapted

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


from Nussbaum & Kardash, 2005) provides more information on the nature of these categories and presents the coding rules that we used.

Next, we coded, on a 4-point scale, the degree to which each essay integrated arguments and counterarguments. The specific points on the scale follow:

TABLE 1. Argumentation Coding Categories and Rules

Category Description

Final claim An opinion or conclusion on the main question, specifically whether (a) TV causes violence, (b) TV does not cause violence, (c) it depends on the situation (contingent), or (d) no position is adopted or no essay is contradictory. The overall thrust of the essay is used in determining the final claim.Supporting claim A claim that supports the final claim. Applicable scoring rules: • May include personal examples if relevant to TV. • Solutions to a problem can only count if part of a contingent argument (as in the previous example). • May include contingencies (related to “it depends” arguments) relating to when the final claim does apply. • Count a maximum of one supporting claim per sentence.Counterclaim A claim that refutes another claim or gives an opposing reason to the final claim. • May include contingencies that indicate when the final claim does not apply. For example: “It depends” (final claim). “If the child watches violent programs unsupervised, he or she may lash out” (supporting claim). “But if parents explain the difference between reality and fantasy, the child may not imitate what he or she sees” (counterclaim, because it qualifies final claim that TV may cause violence by specifying conditions in which it will not).Rebuttal A claim that refutes a counterclaim by demonstrating that the counterargument (a) is invalid, (b) lacks as much force or correctness as the original argument, or (c) rests on a false assumption. Restatement of a supporting claim is not counted as a rebuttal.Supporting reason A separate idea or example that supports another claim in a line of reasoning (i.e., backing for a supporting claim or a counterargument or rebuttal). Otherwise, the same rules governing supporting claims apply. An explicit warrant that links a supporting claim to a final claim is also counted as a supporting reason.Reservation A short qualification made in passing (no more than one sentence) that responds to a potential objection or exception.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


4. Exceptional. Essay was balanced, with integrative closing paragraph. It may weigh evidence on two sides.

3. Well integrated. Essay develops substantial counterarguments and rebuttals or a substantial it-depends argument.

2. Slightly integrated. Essay has (a) a minor it-depends argument, (b) a minor reservation, or (c) different conclusions stated at the beginning and end of the essay.

1. Unintegrated. Essay discusses only one side of the issue or has no final con-clusion.

For each essay with any amount of integration (score ≥ 2), we also noted the in-tegration strategy that the participant used: rebutting, weighing, or synthesizing.

Third, we scored each essay with a holistic scoring rubric that rated the de-velopment of each essay. A developed essay is one with more explanation and support than an undeveloped one, reflects better and more coherent organization, and takes counterarguments into account. Although the frequency scores also captured some of these dimensions, a holistic approach allows researchers to judge how the participant put together and integrated various factors. Ferretti et al. (2000) found training to affect ratings of the overall quality and persuasiveness of arguments, and we used a holistic rubric that Nussbaum and Kardash (2005) adapted from Ferretti et al.’s study. Appendix C shows a copy of the rubric.

Two raters independently scored the entire corpus of transcripts. The raters met periodically to discuss and resolve disagreements. The procedure of double scor-ing all the essays (rather than just a sample) makes interrater reliability statistics somewhat less relevant because we did not use the raters’ independent scores in the final analysis. Rather, we used the scores after discussion, and these scores were more reliable. Nevertheless, we calculated interrater agreement during scoring to determine if any categories were especially problematic or if some categories should be combined. For example, agreement on reservations and sup-porting reasons for rebuttals was especially low (.35 and .02), so we dropped these categories from the analysis. Also, on reliability grounds, we decided to combine supporting claims with reasons supporting those claims and to combine counter-claims with their supporting reasons. After we made these adjustments, overall interrater agreement was 82% before discussion and 100% after discussion.

Statistical Analysis

We conducted a preliminary (univariate) analysis of the data to inform our choice of statistical methods. Table 2 shows the means and standard deviations of the outcome variables. On average, each essay contained about four different supporting claims and one counterclaim. Rebuttals were rare (M = 0.27, SD = 0.55) and only about one third of the essays contained any sort of rebuttal. The

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


median holistic score was 4.00 (M = 4.04, SD = 1.14), indicating that, on average, participants only partially developed the essays (essays gave some elaboration but not enough to convince the readers, and some inconsistencies or problems with organization and clarity may have existed).

The distributions of most of the argument components were skewed (skew = 1.01 for counterclaims and 1.90 for rebuttals). This circumstance is not surpris-ing because these variables reflect counts of many students with scores of zero. Such variables typically follow a Poisson distribution rather than a normal distri-bution. The Poisson distribution is an upwardly skewed distribution with many zero counts. Such a distribution makes sense with this type of argumentation data because many students do not usually include counterarguments or rebuttals in their essays (but, as we will indicate, our interventions substantially increased the rate). Table 2 shows substantial variance in the argument component data: If researchers square the standard deviations, the variances are near to or greater than the means in most cases.

We performed a Kolmogorov-Smirnov test (which is a nonparametric test measuring deviations from an ideal distribution), and Table 2 shows that the actual distributions of the argument components differed significantly from the normal distribution but not from the Poisson distribution. For these reasons, we analyzed these variables by using Poisson regression, as Le (1998) and Rabash et al. (2000) described. We considered the other measures (synthesis and holistic scores) as ordinal, and so for those measures, we used ordinal regression.

The independent variables in our model were the presence or absence of argu-ment criteria instruction, the presence or absence of the graphic organizer, and

TABLE 2. Descriptive Statistics of Outcome Variables (N = 84)

Kolmogorov-Smirnov z

Null hypothesis–distribution is:Outcome M SD Normal Poisson

Argument components Supporting claimsa 3.71 1.82 1.73** 0.41 Counterclaimsa 0.96 1.10 2.40** 0.65 Rebuttals 0.27 0.55 4.27** 0.15Holistic 4.04b 1.14b ––c ––c

Integration 1.81 0.72 ––c ––c

aIncludes reasons supporting the claims and counterclaims. bOrdinal statistic: Md = 4; Interquartile range = 1. cTest not applicable to ordinal variables.**p < .01.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


the interaction of the two (Instruction × Graphic Organizer). The design was a 2 × 2 factorial, but we did not use an ANOVA because ANOVAs assume the use of interval-level variables.

Poisson regression. Poisson regression is a more valid and powerful technique for analyzing count data than are techniques based on the normal curve (t tests and ANOVA; see Le, 1998; Nussbaum, Elsadat, & Khago, in press). Poisson re-gression is based on the following exponential model specifying the relationship between independent and dependent variables:

E(Yi) = eβo + βiXi,

where βiXi reflects the various independent variables in the model: β1X1 = ar-gument criteria instruction, β2X2 = graphic organizer, and β3X1X2 = interaction effect. The model ensures that Yi (the count) is not less than 0. The technique also (a) assumes that in the population, the dependent variable (Yi) follows a Poisson distribution at each level of the independent variables and (b) chooses parameter estimates that would most likely produce the observed data. The parameter esti-mate, exp(βi) in βiXi, is a ratio (risk ratio) that reflects the percentage increase in Yi for a one-unit increase in Xi. (So, for example, β = 0.26 = exp(β) = e0.26 = 1.30 and reflects a 30% increase in Y for a one unit increase in X).

Ordinal regression. In contrast to the argument components, which were fre-quency counts, the holistic score was not a count but on an ordinal scale from 1 to 7 (range was from 2 to 6). Specific anchor points were 2 (minimally devel-oped), 4 (partially developed), and 6 (well developed). We considered the scale as ordinal because there was no basis for assuming that the distances between points on the scale were the same. For this reason, we used ordinal regression (Agresti, 2002) to analyze the holistic data. We also considered the data for the argument–counterargument as ordinal and so analyzed them with ordinal regres-sion as well. Ordinal regression is similar to a logistic regression for each point on the scale (e.g., 2, 3, 4, 5), if the logistic curves all have the same slope so that researchers can calculate an overall slope. The slope measures the percentage increase in the odds that Yi will be one unit higher if Xi increases by one.

Results

Effects of Conditions

Argument components. Table 3 shows the regression coefficients for the vari-ous conditions. We fit the model in two steps. In Step 1, we fit a full, saturated model. We did this to check for interactions between the conditions. In Step 2,

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014

Nussbaum & Schraw 75T

AB

LE

3. A

rgum

ent

Com

pone

nt R

egre

ssio

n C

oeff

icie

nts

and

Eff

ect

Size

s by

Con

diti

on

C

ondi

tion

term

C

ontr

ol

Arg

umen

t

(int

erce

pt

crite

ria

G

raph

icO

utco

me

term

) tr

aini

ng

orga

nize

r In

tera

ctio

n

Supp

ortin

g cl

aim

sa

Ste

p 1

1.21

**

0.05

0.

17

–0.0

4

% in

crea

se in

pro

babi

lity

—

5 18

–4

Ste

p 2

1.22

**

0.03

0.

15

—

% in

crea

se in

pro

babi

lity

3

16C

ount

ercl

aim

sa

Ste

p 1b

–0.6

5*c

0.95

**

0.77

* –1

.12*

%

incr

ease

in p

roba

bilit

y —

15

9 11

7 –6

8R

ebut

tals

S

tep

1 –2

.04**

c 0.

30

1.10

–0

.33

%

incr

ease

in p

roba

bilit

y

35

200

–28

Ste

p 2

–1.9

2**c

0.06

0.

94*

—

% in

crea

se in

pro

babi

lity

—

6 15

7

Not

e. In

tera

ctio

n te

rm d

elet

ed in

Ste

p 2

if n

ot s

tatis

tical

ly s

igni

fica

nt in

Ste

p 1.

Est

imat

es b

ased

on

Pois

son

regr

essi

ons.

Cal

cula

tion

of p

erce

ntag

e in

crea

se

in p

roba

bilit

y ba

sed

on ta

king

the

antil

og o

f th

e re

gres

sion

coe

ffic

ient

(w

hich

pro

duce

s an

est

imat

e of

how

the

ratio

of

prob

abili

ties

chan

ges

from

add

ing

each

inte

rven

tion)

and

mea

suri

ng th

e di

stan

ce fr

om 1

.0, w

hich

refl

ects

no

chan

ge. F

or e

xam

ple,

giv

en a

n es

timat

e of

0.9

5 fo

r cou

nter

clai

ms,

the

risk

ratio

is

e0.

95 =

2.5

9, w

hich

, sub

trac

ting

1.0,

ref

lect

s a

159%

incr

ease

.a I

nclu

des

reas

ons

supp

ortin

g th

e cl

aim

s an

d co

unte

rcla

ims.

b Ste

p 2

was

not

com

plet

ed b

ecau

se th

e in

tera

ctio

n te

rm in

Ste

p 1

was

sig

nifi

cant

. c A n

ega-

tive

inte

rcep

t te

rm r

efle

cts

an e

stim

ated

ave

rage

cou

nt i

n th

e co

ntro

l co

nditi

on l

ess

than

e (

i.e.,

2.72

). F

or e

xam

ple,

–0.

65 f

or c

ount

ercl

aim

s im

plie

s an

es

timat

ed a

vera

ge c

ount

of

e–0.6

5 , o

r .5

2 pe

r st

uden

t.* p

< .0

5. **

p <

.01.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


we dropped the interaction term if it was not statistically significant. Including an interaction term in a regression equation precludes the testing for main ef-fects, because with an interaction term, the equation only predicts individual cell means and not marginal averages of one factor (regardless of the level of the other). We were theoretically interested in testing for main effects because we predicted that both interventions would have independent effects. However, we tested for interactions first because, as most readers know, researchers cannot interpret main effects in the presence of interactions.

Table 3 shows no statistically significant effects on the generation of supporting claims. However, argument criteria instruction had a strong main effect on the gen-eration of counterclaims, B = 0.95, χ2(1) = 7.16, p < .01. The regression coefficient is a measure of effect size and can be interpreted—as shown in Table 3—as reflect-ing a 159% increase in the chance of generating counterarguments in comparison of the argument criteria conditions with the noncriteria conditions. The table presents percentage increases because that is what researches calculate in Poisson regressions (see note to Table 3). However, because percentage increases can be high when the base is low, it is also important for researchers to consider mean values, which Table 4 shows. The average number of counterclaims with only argument criteria instruc-tion was 1.35 (SD = 1.37), whereas it was only 0.52 (SD = 0.73) in the control condi-tion, yielding a difference of 0.83 or about one additional counterclaim per essay. We argue that this is practically important because developing a point in an essay requires time and effort of the writer, so that including an additional counterargument in an essay is not insignificant. Furthermore, getting students to consider any counterargu-ments in their essays is part of the problem that we are addressing.

Provision of the graphic organizer also had an effect on counterclaims. Table 3 shows that it produced an estimated increase in counterclaims of 117%, B = 0.77, χ2(1) = 4.91, p < .05; the mean difference was 0.61. A pair-wise comparison of the regression coefficients showed that there was not a statistically significant dif-ference between the graphic and criteria conditions, χ2(2) = 0.39, p = .82, so we cannot say with confidence that one intervention was stronger than the other.

Although both interventions had somewhat comparable effects on frequency of counterarguments, Table 3 indicates a negative interaction when we combined the graphic organizer with the argument criteria instruction, B = –1.12, χ2(1) = 5.87, p < .05. Table 4 shows the average number of counterclaims for the com-bined condition was 0.95 (SD = 0.87), which was still significantly higher than the control condition (M = 0.52, SD = 0.73). The 68% decrease that Table 3 shows is not relative to the control condition, but relative to the average count (2.92) that we would expect if there had been no interaction. The mean in the combined condition was still 0.42 higher than that in the control condition.

For rebuttals, only the graphic organizer had an effect, B = 0.94, χ2(1) = 4.03, p < .05, reflecting an estimated 157% increase in the frequency of rebuttals. The regression coefficient (Step 2) reflected a positive main effect of the graphic

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


organizer. Table 4 shows that the mean counts increased from 0.15 to 0.39. This increase means that approximately every other essay contained a rebuttal in the organizer condition, whereas almost none did in the nonorganizer conditions.

It is possible that the above result may represent a slight distortion by inclusion of participants who made no counterarguments. Those participants could make no rebuttals—because there was nothing to rebut—and so could not have been affected by the interventions. However, excluding these participants from the analysis made no substantive difference in our findings, B = 0.86, χ2(1) = 4.28, p < .05.

Integration score. Recall that because the integration scale was ordinal, we used ordinal regression to analyze the results. The model’s predicted values did not differ significantly from the actual values, Pearson χ2(4) = 3.62, p = .46.

Although the graphic organizer increased argument components (i.e., rebut-tals), instruction increased the overall integration score. In particular, there was a statistically significant main effect of criteria instruction on the integration score, B = 0.51, Common odds ratio = 1.67, Wald z = 3.92, p < .05. Scores ranged from 1 to 3 because participants gave no scores of 4. As Table 5 shows, the percentage of students receiving low scores of 1 dropped from 43.5% to 28.9% with crite-ria instruction. Without criteria instruction, only 10.9% of the students received scores of 3; whereas with instruction, 26.4% did so, indicating that criteria in-struction had a positive effect on integration. The second part of Table 5 indicates that for cases with some integration (scores ≥ 2), there was a trend for criteria instruction to increase the synthesis of in-between responses, but this trend was not statistically significant, p = .21, Fisher’s exact test. The majority (60%) of

TABLE 4. Mean Number of Counterclaims and Rebuttals, by Condition

Argument criteria condition

Argument criteria No argumentOrganizer condition training criteria training Total

Graphic organizer Counterclaim M 0.95 1.13 1.04 Rebuttals M 0.38 0.39 0.39No graphic organizer Counterclaims M 1.35 0.52 0.94 Rebuttals M 0.18 0.13 0.15Total Counterclaims M 1.15 0.83 0.99 Rebuttals M 0.28 0.26 0.27

Note. M = mean.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


those receiving a score of 3 in the criteria condition gave a response reflecting synthesis whereas only 20% of them did in the noncriteria conditions, but again the trend was not quite significant, p = .10, Fisher’s exact test. However, re-searchers should note that the criteria instruction that we provided to participants addressed all three strategies for integrating arguments with counterarguments, not just synthesis.

Overall, there was significantly more integration with criteria instruction, and participants used a mixture of different types of integration strategies. However, across conditions, the percentage of participants using a refutation strategy was low (only 4%). This, in part, reflects the fact that the scoring rubric for the in-tegration score required the counterarguments and rebuttals to be substantially developed and central to the essay for us to consider integration to be occurring with a refutation strategy. Thus, our criteria here were stricter than just that of the presence of a rebuttal.

In contrast to criteria instruction, which enhanced integration, the graphic or-ganizer did not have a statistically significant main effect on the integration score, B = 0.07, Wald z = 3.92, p > .05, or an interaction effect with criteria instruction. This result was surprising because we designed the graphic organizer to scaffold integration. Recall that integration can involve a number of different strategies: (a) synthesizing an in-between position, (b) rebutting counterarguments, and (c) explaining why one side of the issue is stronger than the other. It is interesting that use of the graphic organizer significantly reduced the number of in-between responses from 30.0% to 6.8%, B = –1.86, Wald z = 7.00, p < .01. Students ap-

TABLE 5. Integration Scores and Nature of Integration With and Without Criteria Training

Argument criteria No argumentIntegration training (%) criteria training (%)

Integration score 3-Well integrated 26.4 10.9 2-Slightly integrated 44.7 45.7 1-Unintegrated 28.9 43.5Total 100.0 100.0Nature of integration One side stronger 58.3 68.0 Refutes other side 4.2 4.0 In between and it depends on positions 37.5 28.0Total 100.0 100.0

Note. Percentages for nature of integration exclude unintegrated cases.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


peared to mainly pick sides when using the graphic organizer, rather than devel-oping in-between positions. We confirmed this interpretation when we examined the graphic organizers that the students completed. Focusing on the final-conclu-sion section, we coded whether each was balanced (reflecting both sides of the issue), unbalanced, or incomplete. We found that 63% of the final conclusions were unbalanced, and, of these, 74% simply reiterated one of the argument or counterargument boxes.

Holistic score. We also used ordinal regression to analyze the holistic scores. All terms were statistically significant. The model’s predicted values did not dif-fer significantly from the actual values, Pearson χ2(9) = 3.79, p = .93. Criteria instruction had a positive effect on the holistic score, B = 0.87, Wald z = 5.40, p < .05; as did the graphic organizer, B = 0.93, Wald z = 6.86, p < .01. As with counterclaims, there was a statistically significant negative interaction between the two conditions, B = –1.04, Wald z = 4.09, p < .05. The negative interaction again reflected the fact that the observed scores in the combined condition were lower than what we would expect if the conditions were to combine additively.

Analysis of the sample proportions showed that whereas 17% of the control group received Level 5 scores, 41% of the criteria group, 36% of the organizer group, and 52% of the combined group received Level 5 scores. Proportions of Level 6 scores were also higher in the individual treatment conditions compared with the control group (6% for criteria instruction and 9% for graphic organizer versus 0% in the control). That was not the case for the combined condition: No one received a Level 6 score in either the control condition or the combined con-dition. (No one received a 7 in any condition.) That fact—fewer Level 6 scores in the combined condition than in the individual treatment conditions—contributed to the negative interaction that was associated with the combined condition. Ac-cording to the scoring rubric, the major differences between Levels 5 and 6 were that essays in the latter had better developed counterarguments, indicating that counterarguments were less well developed in the combined condition.

Discussion

In this study, our goal was to examine the effects of both a graphic organizer and criteria instruction on participants’ integration of arguments with counterarguments in argumentative writing. Both interventions increased the number of counterarguments and produced higher holistic quality scores. Students using graphic organizers also included more rebuttals to counterarguments in their essays, but their overall integration scores were not higher, suggesting that participants’ consideration of the opposing side was limited. In contrast, criteria instruction resulted in higher integration scores than did the graphic organizer. In particular, although students used a variety of strategies to obtain

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


integration, criteria instruction seemed to foster more in-between arguments than did the organizer. Last, the interactions between the two treatments were negative on two outcome measures: counterargument frequency and the ho-listic score. In the following subsections, we will discuss the implications of these findings.

Effect of Criteria Instruction

In this study, criteria instruction consisted of explicit instruction on the qualities of a good argument and an extended example. These qualities included develop-ing a clear position, providing adequate supporting reasons, presenting counterarguments, and considering both arguments and counterarguments when de-veloping the final conclusion. Prior researchers have not much emphasized argument–counterargument integration, our last step. There is reason to believe that it is cognitively demanding because students must simultaneously consider, coordinate, and integrate in working memory various issues (Halford & McCred-den, 1998). Therefore, it is impressive that at least some of the students achieved better integration with brief instruction. The intervention may have been effective because it provided students with a set of goals and subgoals to structure their planning.

Our results are consistent with the finding of Zohar and Nemet (2002) that—in the context of a unit on genetics—explicit instruction on the quali-ties of a good argument improved the complexity of written arguments. One limitation of their study was that discussion activities, which can also affect students’ written argumentation (Reznitskaya et al., 2001), were confounded with explicit instruction. Also, Zohar and Nemet’s study did not involve opinion essays (rather, students completed written worksheets that answered specific questions pertaining to moral dilemmas). Our study, which used an experimental methodology, shows that explicit instruction that researchers target to the qualities of a good argument can affect essay writing. Further-more, we extend Zoher and Nemet’s work, specifically to argument–counter-argument integration.

Effect of the Graphic Organizer

The graphic organizer enhanced argument–counterargument integration mostly by increasing the generation of rebuttals. We believe that the graphic organizer may have invoked a more adversarial argumentation schema than did the first intervention. In adversarial argumentation (e.g., a debate), the debater trys to win an argument by defeating the opposing arguments (Pollock, 1987), specifically through rebuttals. The debater mentions counterarguments in passing, if at all, because the debater does not wish to make the opposing arguments more persua-

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


sive. In this study, use of the graphic organizer appeared to encourage students to pick one side and defend it by including some rebuttals in their essays. We were initially surprised that our graphic organizer would have this effect because we designed it so that students would think about and support both sides of the issue and then develop a final position. The graphic organizer encouraged participants’ application of several alternative strategies to develop their final conclusion, and the brief written instructions accompanying the graphic organizer mentioned three such strategies. Nevertheless, the instructions were brief and not accompa-nied by any examples or verbal explanation (as in the criteria instruction condi-tion). Our data suggest that the graphic organizer tended to cue an adversarial refutation strategy more than the other strategies.

One possible explanation for this finding is that students may have had more experience with the refutation strategy. Some—if not many—students likely had received instruction in argumentative writing before participation in our study, even if that instruction was just minimal instruction in writing a five paragraph essay (Nystrand & Gamoran, 1991). Instructors usually cast that kind of instruc-tion as persuasive writing rather than analytical writing. In other words, instruc-tors expect students to develop and defend a particular point of view, rather than explore different points of view and then come to a conclusion (Kroll, 2000). The graphic organizer may have cued students to generate counterarguments and rebuttals. A second possible explanation is that some students did not follow the directions and completed the “final conclusion” box first rather than last, in effect starting out by picking a side. Committing themselves initially to a side may have then biased them to use a refutation strategy rather than the other types of inte-gration strategies. A third possible explanation is that participants’ filling out one side of the graphic organizer (the “arguments”) caused cognitive interference in the recall and production of counterarguments, creating impoverished and easily rebutted counterarguments. This possibility may have made a refutation strategy more attractive and students’ reasoning generally unbalanced.

Evidence for the third alternative comes from Hoch (1984), who had individu-als generate arguments and counterarguments for why a future event may or may not occur. Individuals differed in whether Hoch asked them to generate pro or con reasons first but always produced more reasons on the side that they were asked to produce first, and they produced those reasons more quickly. As Nickerson (2004) pointed out, generation of the first list tended to interfere with the second. The interference declined as the influence of the second list increased with a time delay (3 min) between the generation of the two lists. These findings suggest that graphic organizers, by systematically focusing students’ attention on one side of the issue before the other, may cause reasoning to become unbalanced because of interference effects. Although it would make the use of graphic organizers more cumbersome, instructors could instruct students to wait a few minutes between completing arguments and completing counterarguments to reduce interference.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Effect of Combining the Treatments

A final question that this study addressed was about the effect of combining the two treatments. They both had positive main effects: Criteria instruction had such effects on overall integration, and the graphic organizer had such effects on rebuttals. However, combining the two treatments produced a negative interaction on both the frequency of counterarguments and the holistic quality scores. We can only speculate about the reasons for the negative interactions, but cognitive load may be a factor. In the combined condition, students had to try to remember and apply a list of criteria constituting a good argument and complete the graphic orga-nizer. This situation may have reduced the amount of cognitive resources available in working memory for elaborating on counterarguments. In partial support of this explanation, students elaborated in the graphic organizers less in the combined condition than in the organizer-only condition. In addition, the holistic scores were lower on the part of the scale that, according to our rubric, was particularly related to the elaboration of counterarguments. Cognitive load from following two sets of directions may have reduced students’ ability to elaborate. Another possible expla-nation is that because the two treatments may have activated somewhat different argumentation schemata, interference between the schemata may have resulted in greater load or otherwise reduced the amount of elaboration.

Validity Issues

One variable that could have affected our results was time. In particular, because we provided participants in the graphic-organizer condition with an ad-ditional 5 min to think about the issue—while completing the organizer—those participants may have thought of additional arguments. Thus, although we pro-vided students in the other conditions with 30 min, we provided those in the or-ganizer conditions with 35 min. However, we do not think that this was a serious confound because students rarely used all 30 min to prepare their essays. Mean writing time was 25 min (we asked students to record the time when they stopped working). In addition, we did not provide participants in the criteria training condition with any additional time, so time would not be a factor influencing our results there.

Although both interventions were effective, were they any more effective than just telling students to focus on both sides of the issue? Astleitner, Nussbaum, and Hofer (2007) tested this possibility in another study by using the same dis-cussion prompt, specifically in a study using a discussion prompt that instructed students to think critically about both sides of the issue. However, Astleitner et al. found no significant effects for such prompting. Furthermore, there is no reason to believe that such an intervention would cause students specifically to integrate arguments with counterarguments, as in the present study.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Several factors may have limited the external validity of our results. For ex-ample, in all the conditions, we used a dual-positional text that outlined potential lines of argument on both sides of the issue. In a previous study, Nussbaum and Kardash (2005) used such text as an independent variable and found dual-positional text to improve the holistic quality of the essays and the amount of counterargumentation. So in this study, we had all participants read the text. The rationale was that we did not want one of our interventions to become ineffec-tive because participants could not think of counterarguments. Participants did not appear to just cite the arguments from the text from rote. Rather, the text ap-peared to stimulate their thinking. To check this possibility, we selected five es-says at random and classified the assertions as new (an assertion not appearing in the text), elaborated (an extended thought or description of personal experience that the text addressed and could have triggered), and rote. We judged only 6% of the assertions as rote, and even with these, participants put the ideas in their own words. Of the remaining assertions, 55% were new and 39% were elaborated. The data suggest that the dual-positional text stimulated students’ thinking. We do not claim that participants necessarily processed all of the issues in the text, although those who received our intervention may have engaged with the coun-terarguments more. However, we do not know how effective our intervention would have been in the absence of dual-positional text.

Another limitation to external validity was that all the participants were col-lege students. Although both interventions increased the holistic quality of the essays, these results might not hold for younger students. Gleason (1999) used a planning sheet that was similar to our graphic organizer with middle and high school students and found only weak effects. Future researchers need to examine the efficacy of these interventions for different grade levels.

In addition, our participants were predominantly women. There is a small tendency for women to be more affiliative in interpersonal interactions (Kyratzis, 2001) but not necessarily to disagree any less than do men (Leman, Ahmed, & Ozarow, 2005). Furthermore, a meta-analysis by Goldsmith and Fulfs (1999) found that gender differences in communication styles are weak, inconsistent, and often statistically swamped by other factors (e.g., personality, geographic upbringing, culture, and situational dynamics). Others specifically studying argumentation skills have also failed to find strong gender differences (Kuhn, 1991; Weinstock & Cronin, 2003). Nevertheless, additional research may be nec-essary to determine to what extent, if any, gender may have affected the present findings. For example, does one gender tend to use specific integration strate-gies more than the other? Researchers might predict that men would be more adversarial and therefore would use a refutational strategy more. Nevertheless, in our sample (which favored women), participants used a refutational strategy extensively, especially with the graphic organizer, and that circumstance does not support the gender hypothesis.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Future researchers might also examine how the aforementioned treatments might interact with other individual difference variables, such as prior knowledge levels. Thus, Golanics and Nussbaum (in press) explored interactions between some argumentation interventions and three individual-level variables (certainty, interest, and prior knowledge) and found significant interactions with knowl-edge. In addition, Wiley (2005) found students with low prior knowledge levels to have a more difficult time in processing both sides of a dual-positional text. Because of these findings, future researchers should examine whether students’ knowledge levels moderate the effectiveness of graphic organizers and argument criteria training. Future researchers should also examine the long-term effects of the interventions. Specifically, would the same effects occur a week or month past any training and when a graphic organizer is no longer accessible. We are currently conducting research on these issues.

Conclusions

Overall, for fostering argument–counterargument integration, students’ re-ceiving instruction on the criteria of a good argument was superior to using the graphic organizer. Although the graphic organizer did increase rebuttals, it did not result in particularly balanced reasoning. Nevertheless, graphic organizers may be useful in teaching students persuasive writing, in which the writer tries to argue for a particular point of view (Yeh, 1998). However, some people have criticized this form of argumentation because it limits the amount of elabora-tion that occurs (see Keefer et al., 2000; Mercer, 1996). For persuasive writing, students may not explore the problem as deeply because their goal is not to explore but only to win an argument. In addition, this study shows that students’ receiving explicit instruction in argument–counterargument integration is more effective than their using a graphic organizer.

We are currently experimenting with different designs for graphic organiz-ers that may improve their effectiveness. Specific ideas that we are examining include (a) placing the final-conclusion box at the bottom, with downward point-ing arrows, (b) eliminating support boxes (to simplify the design), and (c) add-ing integrative prompts above the integration box (e.g., “Is there a compromise or creative solution, and why?”; “Is one side stronger than the other?”). Future researchers will need to determine whether these features will promote integra-tion strategies beyond refutation or whether refutation is generally an inherent feature of graphic organizers because of interference effects. Paradoxically, use of a less structured intervention (e.g., our criteria instruction condition) might produce more balanced reasoning because (a) students are given a more explicit goal, that of producing balanced and integrated arguments, and (b) they are not systematically focused on one side before the other (e.g., Wiley, 2005, found the latter circumstance to make a significant difference in text comprehension). The

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


current study is significant in raising the possibility that pro–con graphic organiz-ers might inherently encourage a refutation strategy, but future researchers need to test this possibility.

It is also important to recognize that relatively explicit instruction can be ef-fective in improving the balance and complexity of students’ arguments. This finding confirms the findings of Zohar and Nemet (2002) that brief explicit in-struction was effective in improving students’ arguments in the domain of genet-ics, and it extends their work to essay writing and to argument–counterargument integration. It is interesting that those researchers also found large and immediate positive effects on students’ oral arguments: The increases were dramatic, and it seemed unlikely that students’ argumentation skills were produced by instruc-tion from scratch. Instead, those researchers concluded that students initially had some implicit argumentation skills but that explicit instruction helped them to refine those skills and learn conditional knowledge regarding where those skills can be productively used (see also Chinn, 2006; Kuhn & Udell, 2003). That pos-sibility may also have been the case in the present study.

ACKNOWLEDGMENT

The authors presented a previous version of this article at the annual meeting of the American Educational Research Association in San Francisco, CA, April 2006. The authors thank Yun Peng for assisting with data collection, Michael Foy and Deborah Obara for helping with data coding, and Ordene Edwards for editorial assistance.

REFERENCES

Agresti, A. (2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley-Interscience.Alexopoulou, E., & Driver, R. (1996). Small-group discussion in physics: Peer interaction modes in

pairs and fours. Journal of Research in Science Teaching, 33, 1099–1114.Anderson, T., Howe, C., Soden, R., Halliday, J., & Low, J. (2001). Peer interaction and the learning

of critical thinking skills in further education students. Instructional Science, 29, 1–32.Andriessen, J., Baker, M., & Suthers, D. (2003). Argumentation, computer support, and the educa-

tional context of confronting cognitions. In J. Andriessen, M. Baker, & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 1–25). Boston: Kluwer.

Applebee, A. N., Langer, J. A., Mullis, I. V. S., Latham, A. S., & Gentile, C. A. (1994). NAEP 1992: Writing report card. Washington, DC: U.S. Government Printing Office.

Astleitner, H., Nussbaum, E. M., & Hofer, M. (2007, August). Stimulating the generation of coun-terarguments during writing. Paper presented at the 12th Biennial Conference for Research on Learning and Instruction, Budapest, Hungary.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: In-structional principles from the worked examples research. Review of Educational Research, 70, 181–214.

Baker, M. (1999). Argumentation and constructive interaction. In G. Rijlaarsdam & E. Espéret (Se-ries Eds.) & J. Andriessen & P. Coirier (Vol. Eds.), Foundations of argumentative text processing (pp. 179–202). Amsterdam: Amsterdam University Press.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Baker-Brown, G., Ballard, E., Bluck, S., Vries, B. D., Suedfeld, P., & Tetlock, P. E. (1992). The con-ceptual/integrative complexity scoring manual. In C. P. Smith (Ed.), Motivation and personality: Handbook of thematic content analysis (pp. 401–418). New York: Cambridge University Press.

Bakhtin, M. M. (1981). The dialogic imagination: Four essays by M. M. Bakhtin. Austin: University of Texas Press.

Baron, J. (1988). Thinking and deciding. New York: Cambridge University Press.Beardsley, M. (1950). Practical logic. New York: Prentice.Bell, P. (2002). Using argument map representations to make thinking visible for individuals and

groups. In T. Koschmann, R. Hall, & N. Miyake (Eds.), CSCL 2: Carrying forward the conversa-tion (pp. 449–485). Mahwah, NJ: Erlbaum.

Bensley, D. A., & Haynes, C. (1995). The acquisition of general purpose strategic knowledge for argumentation. Teaching of Psychology, 22, 41–45.

Bernardi, B., & Antolini, E. (1996). Structural differences in the production of written arguments. Argumentation, 10, 175–196.

Catrambone, R. (1998). The subgoal learning model: Creating better examples so that students can solve novel problems. Journal of Experimental Psychology: General, 127, 355–376.

Chafe, W. (1985). Linguistic differences produced by differences between speaking and writing. In D. Olson, N. Torrance, & A. Hildyard (Eds.), Literacy, language, and learning: The nature and consequences of reading and writing (pp. 105–122). New York: Cambridge University Press.

Chambliss, M. J., & Murphy, P. K. (2002). Fourth and fifth graders representing the argument struc-ture in written texts. Discourse Processes, 34, 91–115.

Chinn, C. A. (2006). Learning to argue. In A. M. O’Donnell, C. E. Hmelo-Silver, & G. Erkens (Eds.), Collaborative learning, reasoning, and technology (pp. 355–383). Mahwah, NJ: Erlbaum.

Cho, K. L., & Jonassen, D. H. (2002). The effect of argumentation scaffolds on argumentation and problem solving. Educational Technology: Research and Development, 50, 5–22.

Coirier, P., Andriessen, J., & Chanquoy, L. (1999). From planning to translating: The specificity of argumentative writing. In G. Rijlaarsdam & E. Espéret (Series Eds.) & J. Andriessen & P. Coirier (Vol. Eds.), Foundations of argumentative text processing (pp. 1–28). Amsterdam: Amsterdam University Press.

De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in cul-turally and academically diverse middle school classrooms. Journal of Educational Psychology, 97, 139–156.

De La Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94, 687–698.

Dolz, J. (1996). Learning argumentative capacities: A study of the effects of a systematic and intensive teaching of argumentative discourse in 11–12 year old children. Argumentation, 10, 227–251.

Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in the classrooms. Science Education, 4, 287–312.

Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education, 88, 915–933.

Ferretti, R. P., MacArthur, C. A., & Dowdy, N. S. (2000). The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normal achieving peers. Journal of Educational Psychology, 92, 694–702.

Freeman, J. B. (1991). Dialectics and the macrostructure of argument. New York: Foris.Fulkerson, R. (1996a). Teaching the argument in writing. Urbana, IL: National Council of Teachers

of English.Fulkerson, R. (1996b). The Toulmin model of argument and the teaching of composition. In B.

Emmel, P. Resch, & D. Tenney (Eds.), Argument revisited, argument redefined: Negotiating mean-ing in the composition classroom (pp. 45–72). Thousand Oaks, CA: Sage.

Gleason, M. M. (1999). The role of evidence in argumentative writing. Reading & Writing Quarterly, 14, 81–106.

Golanics, J. D., & Nussbaum, E. M. (in press). Enhancing online collaborative argumentation through question elaboration and goal instructions. Journal of Computer Assisted Learning.

Goldsmith, D. J., & Fulfs, P. A. (1999). You just don’t have the evidence: An analysis of claims and

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


evidence in Deborah Tannen’s “You just don’t understand.” In M. E. Roloff (Ed.), Communication Yearbook (Vol. 22, pp. 1–49). Thousand Oaks, CA: Sage.

Halford, G. S., & McCredden, J. E. (1998). Cognitive science questions for cognitive development: The concepts of learning, analogy, and capacity. Learning and Instruction, 8, 289–308.

Herrenkohl, L. R., & Guerra, M. R. (1998). Participant structures, scientific discourse, and student engagement in fourth grade. Cognition and Instruction, 16, 431–473.

Hidi, S. E., & Hildyard, A. (1983). The comparison of oral and written productions in two discourse types. Discourse Processes, 6, 91–105.

Hoch, S. J. (1984). Availability and inference in predictive judgment. Journal of Experimental Psy-chology: Learning, Memory, and Cognition, 10, 649–662.

Hovland, C. I., Lumsdaine, A. A., & Sheffield, F. D. (1949). Experiments on mass communication. Princeton, NJ: Princeton University Press.

Hunsberger, B., Lea, J., Pancer, S. M., Pratt, M., & McKenzie, B. (1992). Making life complicated: Prompting the use of integrative complex thinking. Journal of Personality, 60, 95–114.

Inch, E. S., & Warnick, B. (2002). Critical thinking and communication: The use of reason in argu-ment (4th ed.). Boston: Allyn & Bacon.

Johnson, R. H. (2002). Manifest rationality: A pragmatic theory of argument. Mahwah, NJ: Erl-baum.

Keefer, M. W., Zeitz, C. M., & Resnick, L. B. (2000). Judging the quality of peer-led student dia-logues. Cognition and Instruction, 18, 53–81.

Kelly, G. J., Drucker, S., & Chen, C. (1998). Students’ reasoning about electricity: Combining per-formance assessments with argumentation analysis. International Journal of Science Education, 20, 849–871.

Knudson, R. E. (1992). The development of written argumentation: An analysis and comparison of argumentative writing at four grade levels. Child Study Journal, 22, 167–184.

Knudson, R. E. (1994). An analysis of persuasive discourse: Learning how to take a stand. Discourse Processes, 18, 211–230.

Koschmann, T. (2003). CSCL, argumentation, and Deweyan inquiry: Argumentation is learning. In J. Andriessen, M. Baker, & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 261–269). Boston: Kluwer.

Kroll, B. M. (2000). Broadening the repertoire: Alternative to the argumentative edge. Composition Studies, 28, 11–27.

Kuhn, D. (1991). The skills of argument. New York: Cambridge University Press.Kuhn, D. (2001). How do people know? Psychological Science, 12, 1–8.Kuhn, D. (2005). Education for thinking. Cambridge, MA: Harvard University Press.Kuhn, D., & Udell, W. (2003). The development of argumentation skills. Child Development, 74,

1245–1260.Kyratzis, A. (2001). Children’s gender indexing in language: From the separate worlds hypothesis

to considerations of culture, context, and power. Research on Language and Social Interaction, 34, 1–13.

Lampert, M. L., Rittenhouse, P., & Crumbaugh, C. (1996). Agreeing to disagree: Developing sociable mathematical discourse. In D. R. Olson & N. Torrance (Eds.), Handbook of human development in education (pp. 731–764). Cambridge, MA: Blackwell.

Le, C. T. (1998). Applied categorical data analysis. New York: Wiley.Leitão, S. (2000). The potential of argument in knowledge building. Human Development, 43,

332–360.Leitão, S. (2003). Evaluating and selecting counterarguments. Written Communication, 20, 269–306.Leman, P. J., Ahmed, S., & Ozarow, L. (2005). Gender, gender relations, and the social dynamics of

children’s conversations. Developmental Psychology, 41, 64–74.Liht, J., Suedfeld, P., & Krawczyk, A. (2005). Integrative complexity in face-to-face negotiations

between the Chiapas guerillas and the Mexican government. Political Psychology, 26, 543–552.Lotman, Y. M. (1988). Text within a text. Soviet Psychology, 26(3), 25–45.Mallin, I., & Anderson, K. V. (2000). Inviting constructive argument. Argumentation and Advocacy,

36, 120–133.Marttunen, M., & Laurinen, L. (2001). Learning of argumentation skills in networked and face-to-

face environments. Instructional Science, 29, 127–153.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Mayer, R. E. (2003). Learning and instruction. Upper Saddle River, NJ: Merrill/Prentice Hall.Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among chil-

dren of different grade, ability, and knowledge levels. Cognition and Instruction, 14, 139–178.Mercer, N. (1996). The quality of talk in children’s collaborative activity in the classroom. Learning

and Instruction, 6, 359–377.Moore, D. W., & Readence, J. E. (1984). A quantitative and qualitative review of graphic organizer

research. The Journal of Educational Research, 78, 11–17.Nickerson, R. S. (2004). Cognition and chance: The psychology of probabilistic reasoning. Mahwah,

NJ: Erlbaum.Nussbaum, E. M. (2005). The effect of goal instructions and need for cognition on interactive argu-

mentation. Contemporary Educational Psychology, 30, 286–313.Nussbaum, E. M., Elsadat, S., & Khago, A. H. (in press). Best practices in analyzing count data: Poisson

regression. In J. Osborne (Ed.), Best practices in quantitative methods. Thousand Oaks, CA: Sage.Nussbaum, E. M., & Kardash, C. M. (2005). The effects of goal instructions and text on the genera-

tion of counterarguments during writing. Journal of Educational Psychology, 97, 157–169.Nussbaum, E. M., Sinatra, G. M., & Poliquin, A. (in press). The role of epistemic beliefs and scien-

tific argumentation in science learning. International Journal of Science Education.Nystrand, M., & Gamoran, A. (1991). Instructional discourse, student engagement, and literature

achievement. Research in the Teaching of English, 25, 261–290.O’Keefe, D. J. (1999). How to handle opposing arguments in persuasive messages: A meta-analytic

review of the effects of one-sided and two-sided messages. In M. E. Roloff (Ed.), Communication yearbook (Vol. 22, pp. 209–249). Thousand Oaks, CA: Sage.

Page-Voth, V., & Graham, S. (1999). Effects of goal setting and strategy use on the writing perfor-mance and self-efficacy of students with writing and learning problems. Journal of Educational Psychology, 91, 230–240.

Paul, R. W. (1986). Dialogical thinking: Critical thought essential to the acquisition of rational knowledge and passions. In J. B. Baron & R. J. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 127–148). New York: Freeman.

Perkins, D. N., Farady, M., & Bushey, B. (1991). Everyday reasoning and the roots of intelligence. In J. F. Voss, D. N. Perkins, & J. W. Segal (Eds.), Informal reasoning and education (pp. 83–106). Hillsdale, NJ: Erlbaum.

Perloff, R. M. (2003). The dynamics of persuasion (2nd ed.). Mahwah, NJ: Erlbaum.Piolat, A., Roussey, J.Y., & Gombert, A. (1999). The development of argumentative schema in writ-

ing. In G. Rijlaarsdam & E. Espéret (Series Eds.) & J. Andriessen & P. Coirier (Vol. Eds.), Founda-tions of argumentative text processing (pp. 117–135). Amsterdam: Amsterdam University Press.

Pollock, J. L. (1987). Defeasible reasoning. Cognitive Science, 11, 481–518.Rabash, J., Browne, W., Goldstein, H., Yang, M., Plewis, I., Healy, M., et al. (2000). A user’s guide to

MLwiN. London: Centre for Multilevel Modelling, Institute of Education, University of London.Rancer, A. S., Whitecap, V. G., Kosberg, R. L., & Avtgis, T. A. (1997). Testing the efficacy of a

communication training program to increase argumentativeness and argumentative behavior in adolescents. Communication Education, 46, 273–286.

Reznitskaya, A., Anderson, R. C., McNurlen, B., Ngyuen-Jahiel, K., Archodidou, A., & Kim, S. (2001). Influence of oral discussion on written argument. Discourse Processes, 32, 155–175.

Robinson, D. H., & Kiewra, K. A. (1995). Visual argument: Graphic organizers are superior to out-lines in improving learning from text. Journal of Educational Psychology, 87, 455–467.

Santos, C. M. M., & Santos, S. L. (1999). Good argument, content and contextual dimensions. In G. Rijlaarsdam & E. Espéret (Series Eds.) & J. Andriessen & P. Coirier (Vol. Eds.), Foundations of argumentative text processing (pp. 75–95). Amsterdam: Amsterdam University Press.

Schwarz, B. B., & Glassner, A. (2003). The blind and the paralytic: Supporting argumentation in everyday and scientific issues. In J. Andriessen, M. Baker, & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 227–260). Boston: Kluwer.

Schwarz, B. B., Neuman, Y., & Biezuner, S. (2000). Two wrongs may make a right: If they argue together! Cognition and Instruction, 18, 461–494.

Simon, D., & Holyoak, K. J. (2002). Structural dynamics of cognition: From consistency theories to constraint satisfaction. Personality and Social Psychology, 6, 283–294.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


Sprod, T, (1998). “I can change your opinion on that”: Social constructivist whole class discussions and their effects on scientific reasoning. Research in Science Education, 28, 463–480.

Stapleton, P. (2001). Assessing critical thinking in the writing of Japanese university students. Written Communication, 18, 506–548.

Stein, N. L., & Albro, E. R. (2001). The origin and nature of arguments: Studies in conflict under-standing, emotion, and negotiation. Discourse Processes, 32, 113–133.

Suedfeld, P., Tetlock, P. E., & Streufert, S. (1992). Conceptual/integrative complexity. In C. P. Smith (Ed.), Motivation and personality: Handbook of thematic content analysis (pp. 393–400). New York: Cambridge University Press.

Toth, E. E., Suthers, D. D., & Lesgold, A. M. (2002). “Mapping to know”: The effects of representa-tional guidance and reflective assessment on scientific inquiry. Science Education, 86(2), 264–286.

Toulmin, S. (1958). The uses of argument. New York: Cambridge University Press.Voss, J. F. (2001). Argumentation in psychology: Background comments. Discourse Processes, 32,

89–111.Vye, N. J., Goldman, S. R., Voss, J. F., Hmelo, C., Williams, S., & Cognition and Technology Group

at Vanderbilt. (1997). Complex mathematical problem solving by individuals and dyads. Cognition and Instruction, 15, 435–484.

Vygotsky, L. S. (1962). Thought and language. Cambridge, MA: MIT Press.Waggoner, M., Chinn, C., Yi, H., & Anderson, R. C. (1995). Collaborative reasoning about stories.

Language Arts, 72, 582–589.Weinstock, M., & Cronin, M. A. (2003). The everyday production of knowledge: Individual differences in

epistemological understanding and juror reasoning skill. Applied Cognitive Psychology, 17, 161–181.Wiley, J. (2005). A fair and balanced look at the news: What affects memory for controversial argu-

ments? Journal of Memory and Language, 53, 95–109.Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote

understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311.Yeh, S. S. (1998). Empowering education: Teaching argumentative writing to cultural minority

middle school students. Research in the Teaching of English, 33, 49–83.Zohar, A., & Nemet, F. (2002). Fostering students’ knowledge and argumentation skills through

dilemmas in human genetics. Journal of Research in Science Teaching, 39, 35–62.

APPENDIX A

Handout: Criteria for a Good Argument

Good written arguments have certain criteria. When you write your essays, consider these criteria:

1. Clear position. Do you clearly state a position on the issue?2. Adequate supporting reasons. Do you provide multiple reasons to support your position, as many as necessary to make a sufficient case? Are your supporting reasons actually true and relevant to your conclusion?3. Counterargumentation. Have you fairly presented alternative points of view and counterarguments (and supporting reasons)? Have you tried to think of the most compelling counterarguments?4. Final conclusion. Do you consider both the argument(s) and counterargument(s) when developing your final conclusion? For example: • Explain why one side is stronger or more convincing than the other • Refute one side • Develop a final position that is in between two sides5. Organization. Is the essay well organized with different issues addressed separately?

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014

APPENDIX B

Argument Criteria Training Protocol

In this study, we are going to ask you to write an essay in which you make arguments. Arguments are stands you take on an issue, supported by reasons and evidence. For ex-ample, you could argue that the war in Iraq was justified to eliminate weapons of mass destruction. Then again, you could argue that the war in Iraq was not justified because there was little evidence of such weapons. As another example of an issue you could form arguments about . . .

A. Present overhead (picture of Janet Jackson and Justin Timberlake at Super Bowl, with Janet covering her breast with hand).Read script: “ . . . the recent controversy with Janet Jackson and Justin Timber-lake at the Super Bowl. One might make an argument that the performers should be fined for indecent exposure. The reason one might give to support this claim is that a fine is necessary to deter performers from doing this again. Now, please take a look at the next sheet in your packet titled ‘Criteria for a Good Argument.’This handout shows the criteria that a good argument should possess. For example, a good argument should have a clear position. Applied to the Janet Jackson–Justin Timberlake controversy, just saying that “something should be done” is not a very clear position. A clear position would be something like, “Janet and Justin should NOT be fined.” The second criterion on your handout relates to whether you provide adequate supporting reasons to make your case. In particular, do you provide multiple reasons to support your position, as many as are necessary to make a sufficient case? For example, some of the reasons you could present to support your position that ‘Janet and Justin should NOT be fined’ might include: (a) it was an accident, (b) even if it was not an acci-dent, fines will not deter rich people, who already make millions of dollars per year, from doing these things, and (c) there are many more offensive situations on TV than what was seen at the Super Bowl. The handout also asks, ‘Are your supporting reasons actually true and relevant to your conclusion?’ For example, was it really true that the whole incident was an accident?

Number 3 on the handout refers to counterargumentation, which is opposing viewpoints. Have you fairly presented alternative points of view and counterargu-ments (along with supporting reasons)? Have you tried to think of the most com-pelling counterarguments? For example, the recent edition of US Weekly (hold up edition) suggests some reasons why it probably was not an accident and that this was all a publicity stunt. The supporting reasons are, first, Janet Jackson’s people had previously hinted that her performance would be memorable and the wardrobe would be important. Second, she was releasing a new single the next day. Third, although Justin said that he ripped off too much, the fact that the ‘cups came off with no threads was a give away that it was rigged.’ So the counterargument is that it was all a publicity stunt and not an accident.

Another counterargument that you could make is that even if it was an acci-dent, the performers should still be fined as an incentive to be more careful, so it does not happen again. Another is that even though the performers are rich, if you make the fine big enough, it will then deter them (or others) from trying anything like this again.”

Appendix continues


Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


APPENDIX B (continued)

Criteria for a Good ArgumentB. Present overhead and read script: “Now, once you have considered both argu-

ments and counterarguments, you need to develop a final conclusion that con-siders both sides. Number 4 on your handout lists different ways of doing this [show overhead of #4]. For example, you can refute one side or explain why one side is more convincing than the other. With Janet Jackson, for example, the clincher for you might be that ‘even if it was an accident, they should still be fined so they’ll be more careful the next time.’ That really makes all the other issues irrelevant. Another strategy is to develop a final position that is in between the two sides. For example, your final conclusion could be that there should be a hefty fine but only if an independent investigation by the FCC shows that the incident was intentional.

The final criterion listed on your handout is organization. A good argument should be organized and easy to follow, rather than jumping around from point to point in a disorganized way.”

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014


APPENDIX C

Holistic Scoring Rubric

Based on rubrics used by Ferretti et al. (2000) and Nussbaum and Kardash (2005).

0. Response to topic. Paper responds to the topic in some way but does not pro-vide an opinion on the issue.

1. Undeveloped opinion. Paper states an opinion but no reasons are given to sup-port the opinion, the reasons given are unrelated to or inconsistent with the opinion, or the reasons are incoherent.

2. Minimally developed. Paper states a clear opinion and gives one or two rea-sons to support the opinion but the reasons are not explained or supported in any coherent way. The reasons may be of limited plausibility and inconsisten-cies may be present.

3. Between the standards for 2 and 4. (For example, there may be one developed reason and one undeveloped reason.)

4. Partially developed. Paper states an opinion and gives reasons to support the opinion, plus some explanation or elaboration of the reasons. The reasons are generally plausible although not enough information is provided to convince a reader. There may be some inconsistencies, irrelevant information, or prob-lems with organization and clarity. (More poorly organized than level 5).

5. Between the standards for 4 and 6. Could have counterclaims but are not de-veloped.

6. Well developed. Paper states a clear opinion and gives reasons to support the opinion. The reasons are explained clearly and elaborated by using informa-tion that could be convincing. Should mention opposing opinion. The essay is generally well organized and may include a concluding statement.

7. Elaborated and addressed opposition. Meets the criteria for previous level. In addition, the paper deals with the opposing opinions either with refutation, alternative solutions, or explaining why one side is more convincing than the other. Overall, the essay is positive. The paper is free of inconsistencies and irrelevancies that would weaken the argument. Counterarguments presented are strong and elaborated or multiple counterarguments are presented.

Dow

nloa

ded

by [

UT

SA L

ibra

ries

] at

14:

10 0

6 O

ctob

er 2

014

Documents

Promoting Argument-Counterargument Integration in Students' Writing