Modeling Reasoning Processes of Experts and Novices in Argument Diagramming Task: Sequential analysis of

Diagramming Behavior and Think-aloud Data

Paper submitted to Educational Technology, Research & Development

By Haeyoung Kim & Allan Jeong Abstract This study identified the reasoning processes used by experts and novices to analyze a complex argument and operationally defined observed processes in terms of sequential patterns in expert and novices’ actions. Using qualitative and quantitative analysis, this study found that: 1) both experts and novices implemented a five-phased process to construct their argument diagrams; 2) experts used more of a backward approach whereas novices used more of a forward approach, with experts creating more accurate argument diagrams because the backwards approach has been found in previous studies to be more effective when working with unfamiliar problems centered around a specific goals; 3) both groups used the breadth-first over the depth-first approach. Although the findings are post-hoc and correlational in nature, the findings and methods used in this study produced evidence that were consistent with prior research and evidence to suggest that some approaches (if taught and used by students) can be more effective than others when used to analyze and better understand complex arguments.

Introduction and Related Work Critical thinking is considered to be an essential skill in higher education and professional areas, yet, recent research shows that many college students fail to develop critical thinking skills and effectively use the skills (Davies, 2011; Reimold, Slifstein, Heinz, Mueller-Schauenburg, & Bares, 2006; Gold & Holman, 2002; Kuhn, 1991). Davies (2013) points out that employers are more likely to hire students with strong critical thinking skills than students with weak critical thinking skills despite their superior grades and content knowledge. This trend may reflect the ever changing dynamic nature and complexity of today’s real-world problems. As a result, teaching and improving students’ critical thinking is a paramount goal in higher education. To address students’ deficiencies in critical thinking, argument analysis is one method used in higher education to teach critical thinking across many, if not most, disciplines (e.g., education, philosophy, psychology, economics, and political science) because argumentation is an essential part of the scientific and problem-solving process. Bensely (2010) defines argument analysis as a process of “evaluating evidence, drawing appropriate conclusions along with other skills, such as distinguishing arguments from non-arguments and finding assumptions” (p. 49). In argument analysis, students identify the functional roles of each proposition (i.e., conclusion, premise, co-premise, counterargument), analyze the hierarchical relationships among propositions across major and minor claims (i.e., levels of premise), and evaluate the quality and validity of a given argument. It is a process that can be used to help identify flaws and evaluate the truth value of stated arguments, which ultimately can help one draw more well-reasoned conclusions and make better decisions. As a result, the skill of analyzing arguments is an important component of critical thinking and hence is a skill that college students should develop (Harrell, 2008). However, the structure of arguments can be and are often complex and ill-defined. Analyzing the structure of complex arguments requires students to perform cognitive operations that are complex and multi-step. Cañas et. al.’s (2003) found that students in general use a five-step process to complete a concept map: 1) identify the goal or question; 2) identify and list concepts associated with goal; 3) order the concepts from top to bottom on the map; 4) add and label links; and 5) review and correct map. These processes of argument analysis require significant attention, memory, and cognitive effort and are likely to produce heavy cognitive load that inhibits performance and learning (Harrell, 2007; van Bruggen, Kirschner, & Jochems, 2002). To address these challenges, argument diagramming software has been developed to help students draw visual diagrams to scaffold the process of mapping out hierarchical relationships between major and minor claims (van den Braak, Oostendorp, Prakken & Vreeswijk, 2006). Using this approach, students do not have to rely solely on memorizing multiple series’ of propositions in

1

the form of verbal representations. In other words, the use of diagramming tools enables students to allocate more working memory capacity to interpret the text, identify the functional elements of the text (claims, supports, objections, counter-arguments, etc.), and analyze the nature and quality of the hierarchical relationships between claims. As a result, using argument diagrams helps to reduce cognitive load when analyzing complex arguments (Harrell, 2007; van Bruggen, Kirschner, & Jochems, 2002). In spite of reductions in cognitive load when using argument diagrams, the efficacy of constructing argument diagrams is still in question. van den Braak et al.’s (2006) critical review noted that most, if not all, prior studies on argument diagramming assessed students’ argumentation skills based primarily on the evaluation of the final product – students’ argument diagrams. Furthermore, prior studies have focused primarily on determining the effects of specific interventions on students’ understanding of arguments. Although some studies have found the construction of argument diagrams to improve understanding, van den Braak et al. (2006) noted that these studies still reported high variance in the level of understanding achieved by students that constructed argument diagrams. Moreover, these studies did not examine how the interventions affected the processes students use to construct an argument diagram (Kuhn & Udell, 2003) and how resulting changes in processes in turn affected the quality of students’ argument diagrams. The emphasis on evaluating the final maps alone has not advanced our understanding of the processes students use to construct their argument diagrams, and which processes are most effective in improving and reducing the variance in learning outcomes. Achieving a deeper understanding of the processes students use to diagram arguments may help us to understand why and when particular tools work and do not work. However, little research has been conducted to explicitly model and identify the processes that improve versus hinder students’ analysis and understanding of complex arguments. With the goal of identifying the processes used by students, Author (2010) developed the jMAP software application to automatically log and codify the sequences of mechanical actions students perform while constructing complex diagrams. This type of data can be sequentially analyzed and used to visualize, reveal, and identify the mental reasoning processes used to produce high and low quality argument diagrams. For example, Author (2014) found that high performers were more likely to perform certain action sequences than low performers. High performers not only deleted links three times more often than low performers, high performers were also likely to follow a link deletion by adding a new link (delete new link). This action sequence is at a certain level an indication that high performers were correcting errors produced by leaping to conclusions (when AC and BC should be changed to ABC). However, the underlying reasoning process that accounts for and is associated with this specific action sequences was not determined in the study. As a result, more qualitative research is needed to reveal the processes of reasoning underlying the diagramming actions and action sequences in order to fully explain and understand differences in students’ analysis and diagrams. Identifying both the reasoning and diagramming processes that help and hinder understanding will contribute to further development and refinements in instructional interventions and diagramming tools. One approach to identifying the reasoning processes that produce more versus less accurate diagrams is to compare the processes used by subject-matter experts to the processes used by novices. Cognitive research has shown that experts use different cognitive processes than novices to produce superior performances on tasks (Livingston & Borko, 1989; Norman, 2005). For example, experienced programmers have been found to use more of a top-down and a breadth-first than novice students when writing complex software programs (Jefferies, Turner, Polson, & Atwood, 1981). Experts also engage in problem solving using a more holistic approach whereas novices tend to localize and focus on specific fragments of the problem (Chi , Glaser, & Rees, 1982). Experts have also been found to use both forward/inductive and backward/deductive approaches to solve problems (Frensch & Funke, 1995). However, experts tend to use a forward (or bottom-up) approach when working with familiar problems and use a backward (or top-down) approach when working with unfamiliar problems (Crespo, Torres, & Recio, 2004; Smith, 2012; Sweller, Clark, & Kirschner, 2011). Because the backward approach has been identified by Sharma, Tiwari, and Kelkar (2012) to be a more efficient process than the forward approach (in terms of the number of operations needed to successfully identify a rule or relationship between two propositions), the gains in efficiency with the backward approach might serve to compensate for the additional time and effort needed to solve unfamiliar problems. At the same time, however, the forward approach (although less efficient) enables people to make judgments with higher degrees of confidence and lesser uncertainty (Tversky & Kahneman, 1977). What should be noted however is that certain processes are more effective regardless of the level of expertise. The backward approach is better suited to solving problems centered around specific goals and outcomes (Rubin & Ragusa, 1992) whereas a forward approach is better suited

2

for planning, making predictions, and increasing control (Sharma, Tiwari, & Kelkar, 2012). At this time, no prior research has determined whether experts are more likely to use a depth-first or breadth first approach, and which of the two approaches is more effective. Research questions Using think-aloud protocol and the jMAP diagramming software, this study observed, coded, and identified the reasoning and mapping processes used by experts and novices to analyze a complex argument. The coded data was analyzed to identify sequential patterns in the actions experts and novices perform while diagramming arguments in order to determine the mapping actions (and the reasoning processes that are indicated by the mapping actions) that help to produce high versus low understanding of complex arguments. In addition, qualitative analysis was conducted to identify high level strategies used by the participants to help interpret and understand the processes identified in the sequential analysis. As a result, this study addressed the following questions:

1. What processes do experts and novices use to diagram complex arguments? 2. What differences exist in the processes used by experts versus novices?

Method

Research Design Using a mixed method design, video recordings of verbal protocols were used to collect data on reasoning behaviors (diagramming processes) and operations (cognitive processes). The jMAP software was used to capture the expert and novices’ diagramming behaviors. At the same time, think-aloud protocol was used to elicit the participant’s verbal explanations of cognitive underlying reasoning processes behind their diagramming behaviors. Grounded theory approach was then used to generate a coding scheme to categorize observed diagramming behaviors and participants verbal descriptions of their thoughts as they constructed their argument diagrams. Sequential patterns in the diagramming processes and cognitive processes performed by experts and novices were identified by sequentially analyzing the coded data. Lastly, qualitative analysis was used to help determine to what extent the action sequences identified with sequential analysis can serve as indicators of global level reasoning processes. As a result, qualitative findings were used to evaluate the value of using sequential analysis as a method for assessing and diagnosing students’ reasoning processes. Participants Ten participants were recruited from a large university located at southeast in the USA in spring of 2014: five experts in argumentation and five graduate students. Nielson (1994) recommended that a 4 ±1 sample size is sufficient for studies employing think-aloud protocol. The participants were recruited via an email and graduate level class visit using a purposeful and convenience sampling. The participants in the expert group were all instructors that have taught argumentation and/or have had formal training in argumentation. The novices were all graduate students with no previous formal training in argumentation.

Procedures

Each participant completed a survey to determine demographic information and prior experience with argument diagrams. This information was used to verify and classify each participant as an expert or novice. Each participant was presented instructions on how to use jMAP to construct an argument map by viewing a 2-minute video demonstration on how to move and re-position nodes, insert and delete directional arrows linking nodes, re-route arrows between different nodes, and change the color of arrows (black for supporting premise, red for opposing premise). The participants then completed a practice exercise to construct a mini argument map consisting of five nodes (claims about the importance of critical thinking in college students) within a period of 10 minutes. During the practice session, participants practiced talking aloud to familiarize themselves with the talk-aloud protocol.

3

Figure 3.1. jMAP screen at the start of the argument diagramming task. Next, each participant was instructed to read a 6-page article (Clark, 2002) describing six e-learning design principles with supporting empirical evidence. Next, they were presented with a jMAP screen (Figure 3.1) populated with 14 claims and one main claim (all of which were drawn directly from the article) and instructed to construct an argument diagram to reveal the logical structure between claims and conclusion. After reading the article, the participants constructed their argument diagrams while talking aloud to report what they were thinking while performing the task. The verbal reports and the actions they performed on screen were recorded using a video screen-capture software program. When a participant did not talk over a period of 5 or more seconds, they were prompted to ‘keep talking’. The researcher provided assistance only when participants experienced technical difficulties with the mechanics of using the software. When the participants completed their diagrams, one-on-one interviews (spanning 30 minutes or less) were conducted starting with open-ended questions followed by seven structured questions intended to elicit details about the strategies, processes, and difficulties they experienced during the task. Data collection Data were collected from four sources: final argument diagrams, diagramming behaviors logged in jMAP, video-audio recordings of the each participant’s jMAP screen and verbal reports, and retrospective one-on-one interviews. jMAP logged each action performed on an argument diagram (Table 3.1) into an MS-Excel spreadsheet in chronological order with a time stamp and sequence number. The spreadsheet for each participant was used to record the transcribed verbal reports alongside with each recorded action at their time of occurrence. Verbal reports and actions occurring at the same time were assigned the same sequence number. The final argument diagrams of each participant and the instructor’s diagram (Figure 3.2) were imported into jMAP to assess the quality of each participant’s diagram in relation to the instructor’s diagram. The one-on-one interviews were conducted to ask the participants to describe the processes and strategies they used while constructing their argument diagrams.

4

Figure 3.2. Instructor’s argument diagram used as the criterion diagram Demographic Information The novice group consisted of one male and four female Master-level graduate students ranging in age from 24- to 54-years-old from four different departments in Counseling, Information Studies, Criminal Justice, and Education (Table 1). One novice reported previous coursework in reasoning and argumentation. The remaining four novices had taken no formal argumentation/reasoning courses. In the expert group were five male instructors that have taught argumentation/reasoning in the Philosophy and the Instructional Systems and Learning Technology departments with ages ranging from 29- to 54-years. The average years of teaching experience among the experts was 12 years, ranging from 2 to 25 years. With regard to familiarity with the content of the arguments (Table 2), five novices reported having little prior knowledge of the six e-learning multimedia principles. Two of the novices were familiar with one or two of the principles. Four of the experts reported that they had never heard of the principles while one had some familiarity with all six principles.

Table 1 Demographic information of the participants

ID Gender Age Major Profession (yrs.)

Argumentation courses/experiences

Argument mapping/tool experiences

Novi01 Female 26 Career Counseling

Graduate student

1 course taken No / No

Novi02 Female 24 Mental Health Counseling

Graduate student

0 No / No

Novi03 Female 24 Library and information science

Graduate student

0 No / No

Novi04 Male 24 Library and information science

Graduate student

0 No / No

Novi05 Female 54 Performance Improvement and Human Resource Development

Graduate student

0 No / No

5

Exp01 Male 42 Criminal Justice Professor (8 yrs.)

Teaching argumentation/reasoning courses

Yes / No

Exp02 Male 32 Philosophy Professor (24 yrs.)

Teaching argumentation/reasoning courses

Yes / No

Exp03 Male 52 Philosophy Postdoc researcher & Instructor (2 yrs.)

Teaching argumentation/reasoning courses

No / No

Exp04 Male 54 Philosophy Professor (25 yrs.)

Teaching formal reasoning courses

No / No

Exp05 Male 29 Philosophy Doctoral Candidate & Instructor (2 yrs.)

Teaching argumentation/reasoning courses

Yes / No

Table 2 Participant's perceived content familiarity and time spent on tasks

ID Familiarity with content

(0 to 12)

Time spent on reading (minutes)

Time spent on argument map (minutes)

Map Score

Novi01 3 6:24 21:03 14

Novi02 0 3:09 06:41 7

Novi03 2 5:50 35:07 11

Novi04 0 4:41 12:14 5.5

Novi05 0 5:41 32:29 9

Exp01 12 5:10 21:06 19

Exp02 0 4:14 29:10 16

Exp03 0 3:57 34:10 31

Exp05 0 4:32 18:17 26

Perceived content familiarity and time spent on tasks The novices and experts spent 5 minutes 9 seconds and 4 minutes 25 seconds on average, respectively, reading the article on e-learning principles. The novices took an average of 21 minutes to complete the argument map, while the experts took an average of 26 minutes. Data collected from one expert was omitted from analysis because the expert displayed a fair amount of discomfort and misunderstood the purpose and expectations of the argument diagramming task. The expert’s background was in formal reasoning and as a result, this expert did not understand what to do with the nodes presented on the jMAP screen. The expert also found the jMap software to be unfamiliar and frustration with the software limited the expert’s ability to perform the task. Assessment of argument diagrams The argument diagrams were assessed across five criteria (Table 3) by assigning one point for correctly: 1) identifying the main conclusion, 2) identifying the lowest level or root claims (claims with no child claims), 3) linking two adjacent claims branching from the root claims, 3) linking three consecutive claims branching from the root claims, and 4) linking four consecutive claims branching from the root claims. In addition, 1 point was given for correctly identifying the relationships between two claims, .5 points were deducted for inserting arrows pointing in the wrong direction, and.5 points deducted for each

6

arrow with an incorrect valence (positive or negative). The scores for the argument diagrams ranged from a low of 5.5 points to as high of 31 points, with all four experts scoring higher than the five novices. A close qualitative analysis of the argument diagrams revealed that the experts committed some fallacies in their reasoning as did the novices. However, the experts overall tended to make fewer leaps to conclusion, associations with irrelevant claims, reversed causation, and single cause fallacies. Table 3 Participants' argument diagram scores

ID

Correctly identified the main conclusion (1)

# of root causes correctly identified (8)

# of correct 1st order chain (10)

# of correct 2nd order chain (10)

# of correct 3rd order chain (10)

# of correct 4th order chain (10)

Total score (49)

Exp03 1 8 7 5 5 5 31 Exp05 1 8 5 4 4 4 26 Exp01 1 7 5 5 1 0 19 Exp02 1 8 5 2 0 0 16 Novi01 0 6 4 4 0 0 14 Novi03 1 7 1 1 1 0 11 Novi05 1 7 (5*) 2 (1*) 1 1 0 9 Novi02 1 5 1 0 0 0 7 Novi04 0 3 3 (1*) 0 0 0 5.5 Note: The values with asterisks indicate the number of chains correctly identified in both direction and valence. The values in parentheses presented in the first row indicate the maximum possible score per criterion. Data analysis The video recordings in conjunction with the retroactive one-on-on interviews (where the participants’ described the processes they used to construct their argument diagrams) were reviewed to identify common macro-level processes and strategies that were used by the experts and novices. In an effort to validate the findings resulting from the qualitative analysis, the participants’ mapping actions performed on computer screen as well as the participants’ verbal/cognitive actions were sequentially analyzed to identify sequential patterns in action sequences that could be used to operationally define and validate the strategies and processes identified from the qualitative analysis. To identify these sequential patterns, the participants’ mapping behaviors and verbal reports were transcribed and analyzed to identify an initial set of coding categories for classifying both mapping and verbal/cognitive actions. Through iterative examination of the video recordings, the categories emerged progressively from the analysis of one participant to the next participant, with new categories added to the coding scheme when new actions could not be assigned to any existing categories. A second coder was then trained to code the videos of one expert and one novice with the initial coding scheme. Inter-rater reliability between the second coder and the researcher was tested, disagreements were addressed, and further revisions were made to the coding scheme to resolve the disagreements. Using the revised and final coding scheme (Table 4), the two coders coded the videos of another novice and another expert. The Cohen’s kappa inter-rater reliability for coding the novice and expert data was .78 and .97, respectively, indicating substantial agreement between the raters. The final coding scheme was then used by the researcher to code the video recordings of the remaining experts and novices. The coded data was then sequentially analyzed (Bakeman & Gottman, 1997) using the Discussion Analysis Tool or DAT (Author, 2012) to produce transitional state diagrams that provide a graphical representation of: 1) the probabilities in which one given action was immediately followed by another given action (using arrows with densities that are proportional to the observed probabilities); and 2) the probabilities that were significantly higher than the expected probability based on z-score tests at p < .05 (using black-colored over gray-colored arrows). One transitional state diagram was produced using the coded data from the experts only, and a second state diagram was produced using coded data from the novices only. A side by side comparison of the two state diagrams was then conducted to identify the patterns in action sequences that were unique to experts and unique to novices, and to determine which of

7

the unique patterns corresponded with and validated the strategies and processes (identified from the qualitative analysis) found to be unique among experts and unique among the novices. More details on this analysis are presented later. Table 4 Coding scheme used to classify verbal and mapping actions

Code Meaning Examples in context RC Read a claim Read a claim - Encode; understanding of meaning

IMC Identify the main conclusion Identified the main conclusion

IL Identify a level of a claim Identify a claim's level and position it to top/bottom or right/left

PN* Position a node Move/position a node

IA Identify an association Noticed some associations (without saying Cause-effect). E.g., N3 and N6 are related.

IACE

Identify a cause-effect association Verbally state that N3 is feeded into N6. E.g, I'm connecting from N3 to N6 since N3 will help N6

ITC Interpret a claim by his or her own words

INEG

Identify a negative relationship Verbally state a negative relationship between two claims

ID Identify dependencies between claims

Specify dependencies/commons between claims

IID Identify independencies between claims

Specify that there are reasons to support the same claim but in different reasons.

IIR Identify irrelevant claims to the main argument

E.g., I think this is a different issue.

REASO

Provide a reason for an association

Explicitly state a reason why there is a relationship

MC* Make a connection (connect a link between two nodes)

Add a link and connect two nodes

REVIE

Review the flow of reasoning

RERRO

Recognize a reasoning error

DL* Delete a link, detach a link Delete a link --> disconnect the relationship, or reserved the direction

Note: The symbol * indicates codes identifying mapping behaviors.

8

Table 5 Observed frequencies for each code observed in each participant

Note: The symbol * indicates the map behavioral codes. Underlined values indicate that average frequencies between the novices and experts were significantly based on t-tests at alpha level .05.

Table 5 shows the frequencies of each code per participant and the average frequencies by group. Because sequential analysis requires an observed frequency of 5 or more per action sequence to conduct z-score tests to determine which probabilities are significantly higher than the expected probabilities by chance alone, six of the codes with observed frequencies of 4 or less (IMC, IL, ID, IID, IIR, INEG) that were generally associated with the action of identifying relationships were collapsed into one single category (IA). Recognizing errors (RERRO) was subsumed under review (REVIE). Repositioning nodes (PN) was subsumed under deleting links (DL) because the deletion of links was almost always followed by the action of repositioning a node. In the end, a total of eight codes (identified in bold type in Tables 4 and 5) were used to conduct the sequential analysis.

Main Findings

The qualitative analysis of the video recordings and retrospective interviews revealed that both experts and novices constructed their argument diagrams using actions that can be broken down into five phases (Table 1). Performed within as well as across these phases were four specific strategies that were used by the participants’ to construct the argument diagrams. Two of the strategies were used by and closely associated with the experts (holistic approach, backward approach) while one strategy (forward approach) was used by and mostly associated with the novices. The fourth strategy was used in large part by both the experts and novices (breadth-first approach) to identify relationships and the links from two or more minor premises to one specific super- or sub-ordinate premise. Five-phase process Both experts and novices overall performed actions that fell into five distinct phases –actions used from start to end to complete their argument diagrams. As shown in Table 5, the experts on average

Expert Group Novice Group

Codes 1 2 3 5 Sum Ave. 1 2 3 4 5 Sum Ave. RC 34 92 51 47 224 56.00 63 23 91 16 66 259 51.80 IA 13 20 19 13 65 16.25 9 20 25 12 18 84 16.80 IACE 14 11 30 22 77 19.25 14 0 9 3 8 34 6.80 PN* 34 41 32 32 139 34.75 17 28 18 15 37 115 23.00 REASO 2 0 6 3 11 2.75 10 3 11 3 2 29 5.80 MC* 20 13 47 17 97 24.25 18 15 20 19 28 100 20.00 REVIE 20 21 3 17 61 15.25 2 2 19 1 7 31 6.20 DL* 7 0 17 2 26 6.5 3 0 4 2 10 19 3.80 IMC 1 2 1 2 6 1.50 2 0 2 5 1 10 2.00 IL 11 5 7 1 24 6.00 0 0 1 0 1 2 0.4 ITC 1 2 8 2 13 3.25 0 0 2 0 0 2 0.40 INEG 0 6 2 2 10 2.50 3 7 1 0 6 17 3.40 ID 0 2 0 2 4 1.00 0 0 0 3 2 5 1 IID 0 3 1 1 5 1.25 0 0 0 0 0 0 0 IIR 2 6 5 1 14 3.5 0 0 0 0 0 0 0 RERRO 3 2 4 2 9 2.75 2 1 4 0 6 13 2.60

9

performed all of the various actions across all phases at higher frequencies than the novices. In table 6 are the five action phases illustrated with quotations from the verbal reports. Table 6 Description of the five phases of actions used to complete an argument diagram.

Phases Indicators from verbal reports & interviews (1) Scan claims

EXPERT: “So first I'll just read all my claims on the side.” NOVICE: “So, I'm just going to read through all of these reasonings and see where I'll begin, how I'll form the map. So I'm just going to read them out loud.”

(2) Identify conclusion

EXPERT: "I think generally, what I always try to do is find whatever I take to be the ultimate conclusion first.” NOVICE: “All right, okay I'm just thinking-- okay yeah that's-- that'll be main one, main point.”

(3) Structure the claims & insert links

EXPERT: “As I was reading all the reasons, I noticed various similarities. So we're excluding text, we're excluding gratuitous visuals, we're excluding gratuitous sounds. I knew that all of those would typically go up towards the top and then go to support something about decreasing load on some sort of memory, so that helped me at least get that structure set…. I think that I tried to not put any links in until I felt fairly confident that I wanted them there.”(holistic approach) NOVICE: “All right, okay I'm just thinking--that'll be main one, main point…I'm going to begin making a map. I'm going to pick number 11, use of multimedia increases learning. I'm going to bring this over to the section and I'm going to …. Okay, let's see. Helps encode into long term memory. I think that definitely relates to-- That's a good reason to have to use multimedia in e-learning…I'm going to add the arrow to connect to use of multimedia increases learning.” (atomistic approach) “I'm trying to figure out... what other examples I can attach to main point, use of multimedia increases learning… I'm going to take this one, exclude gratuitous text…So I'm going to add the arrow to this one and connect [to main conclusion]”. (breadth-first approach)

(4) Review linkages between claims and conclusion

EXPERT: “Well, there's this arrow here. I'm not sure if that should go there."

(5) Correct errors in links due to fallacies in the reasoning

EXPERT: “If I did put a link in, the only reason why I wanted to delete it was to put something else in between as a separate step.”

Holistic vs. atomistic approach The experts used a more holistic approach than novices to analyze the inter-relationships between the claims. This approach often started with categorizing the claims into groups by moving related claims into close proximity to one another without inserting any links between the related claims (see expert’s verbal protocol presented in phase 3 Table 6). This action was then followed with an analysis of the overall level of each claim (major premise, minor premise, so forth) in relation to the conclusion. For example: “I started working with the conclusion and then ended up going back to the topics we use further out reasons, which helped me to see what they were supporting and then draw the connection in there together.” Once the general structure was put in place, the experts inserted the links between the claims and while doing so, they presented verbal explanations as to how and why the claims were linked. In contrast, the novices exhibited a more atomistic approach than the experts by focusing on identifying specific relationships between one claim to another specific claim and immediately inserting links between related claims (see novice’s verbal report presented in phase 3 Table 6).

10

To quantitatively measure and verify this observed difference in the way the experts (positioning nodes into groups without inserting links) and novices (positioning a node and immediately inserting a link) approached the diagramming task, the coded actions were sequentially analyzed to produce two transitional state diagrams (Figure XX) to reveal sequential patterns in the action sequences performed by the experts and by the novices. By tracing paths through the black-colored arrows within each of the state diagrams, the left state diagram shows that the experts exhibited the tendency to read claims, then identify their association, position the claims accordingly, and resume reading/scanning other claims. In a separate series of actions (after positioning related claims in close proximity to one another), the state diagram reveals that the experts has the tendency to identify the causal relationship between two claims, and then inserted a link between the claims along with explaining the reasons the two claims are linked. In contrast, the state diagram on the right shows that the novices identified the association between claims, then positioned the claims accordingly, identified further associations between claims, inserted a link between the claims, explicitly stated the relationship between the linked claims, and then presented the reasons for linking the claims. As a result, the state diagrams provide quantitative data to support the qualitative finding that the experts used a more holistic approach than the novices.

Note: Black-colored lines identify probabilities that were significantly higher than expected based on z-score tests at p < .05. The density of each arrow is proportional to its observed probability. Values presented within a node identify the observed frequency of the specific action. White circles denote mapping actions, gray nodes depict verbal cognitive operations.

Figure 1. Transitional diagrams of the processes of the experts and novices.

Backward vs. forward approach The experts exhibited a higher tendency to use the backward approach than the novices - using the process of scanning for and identifying a sub-ordinate claim that is associated with a specific super-ordinate claim. For example: “I started working with the conclusion and then ended up going back to the topics we use further out reasons, which helped me to see what they were supporting.” In contrast, the novices showed more of the tendency to use a forward approach than the experts, working upward by scanning for and identifying super-ordinate claims that can be linked to a specific sub-ordinate claim. This finding was based on close analysis of the video recordings - counting the number of times each participant’s scanned for and identified a sub-ordinate claim that could be linked to a specific super-ordinate claim (backward approach) and vice versa (forward approach). The backwards approach was used by the experts and novices 84 and 39 times, respectively, whereas the forward approach was used by the experts and novices 27 and 46 times, respectively. The relationship between group and approach was statistically significant, χ2(1, N = 196) = 18.281, p < .001.

11

Breadth vs. depth-first approach Both the experts and novices exhibited a strong tendency to use a breadth-first approach - consecutively scanning for and identifying two or more sub-ordinate claims associated with a specific super-ordinate claim (when using a backward approach), and consecutively scanning for two or more super-ordinate claims that is associated with a specific sub-ordinate claim (when using a forward approach). In contrast, the depth-first approach (e.g., associating/linking XYZ; or ZYX) was used infrequently by the experts and novices. The analysis of the video recordings revealed that experts and novices performed the breadth-first search a total of 85 and 46 times, respectively. In contrast, the experts and novices performed the depth-first search a total of 10 and 3 times, respectively. The relationship between group and approach was not statistically significant, χ2(1, N = 144) = .321, p < .571. As a result, no significant difference was found in the relative frequency in which experts and novices used the breadth-first versus depth-first approach.

Discussion

The purpose of this study was to identify the reasoning processes used by experts and novices to analyze a complex argument and to operationally define the observed processes in terms of sequential patterns in the actions of experts and novices. Using both qualitative and quantitative analysis, this study found both experts and novices implemented a five-phased process to construct their argument diagrams: 1) scan claims; 2) identify conclusion; 3) structure the claims and insert links; 4) review linkages between claims and conclusion; and 5) correct errors in links due to fallacies in the reasoning. This five-phase process is consistent with mapping processes observed and reported by Cañas et. al. (2003). As for the differences between expert and novice, the experts in this study were found to use a more holistic approach whereas the novices tended to use an atomistic approach. This finding is consistent with prior research that show experts engage in problem solving using a more holistic approach whereas novices tend to localize and focus on specific fragments of the problem (Chi, Glaser, & Rees, 1982). In addition, this study found that the experts and novices showed a tendency to use more of a backward and forward approach, respectively – a finding that is consistent with the experts’ use of a backward approach when designing and programming computers (Jeffries et al., 1981). The experts in this study were able to produce more accurate causal maps using the backwards approach because: 1) a backwards approach has been found to be more effective when working with unfamiliar problems and when knowledge is limited (Crespo, Torres, & Recio, 2004; Smith, 2012; Sweller, Clark, & Kirschner, 2011) and when the problem situation is centered around specific goals and outcomes (Rubin & Ragusa, 1992; Sharma, Tiwari, & Kelkar, 2012); and 2) the topic of the argument in this study was not familiar to the participants and the analysis was centered around a given conclusion/claim. Finally, both groups used the breadth-first approach over the depth-first approach when they analyzed claims. As a result, this finding provides preliminary evidence not yet reported in prior research to suggest that neither approach is more effective than the other when creating diagrams centered and stemming from a specific goal, outcome, and/or conclusion. Overall, the significance of this study is that further evidence was produced to show that specific processes of reasoning that may and may help (holistic and backward approaches) and may not help (breadth first vs. depth-first approaches) produce more accurate and deeper understanding of complex arguments and perhaps understanding of complex problems in general. Although many of the main findings in this study were consistent with prior research, the direct causal relationship between the holistic and the backward approaches and accuracy in analysis and understanding cannot be fully validated with the findings from this study given the correlational nature of this study. Instructional and design implications The findings in this study suggest that instructors should encourage students to analyze arguments using a more holistic approach by applying the following procedure: 1) use an iterative process of positioning nodes using a breadth-first and backward approach to explore the associations between claims; and 2) insert links between nodes once the hierarchical relationships have been thoroughly examined. Mapping software can be designed and further refined to scaffold some of these processes. For example, the function for inserting links can be disabled early in the mapping process so that students focus first on identifying the main conclusion and exploring the possible relationships in the process of positioning and re-positioning claims on the screen. The system can count the number of times each claim has been moved

12

and/or re-positioned to indicate to what extent the student has actively explored the hierarchical structure between the claims and conclusion. Once students have stepped through these early phases of the diagramming process, the system can enable the student to execute a one-step function that automatically inserts links between all the claims based on each claim’s relative position and proximity to one another. Furthermore, future research can be conducted to refine the coding scheme developed in this study so that the sequential analysis of mapping actions can: 1) detect, measure, and monitor students’ use of top-down, bottom-up, depth first, and breadth first approaches; and 2) determine which reasoning processes tend to produce or not produce specific reasoning fallacies (e.g., leaping to conclusions, hasty generalizations). For example, the inclusion of a code that identifies the action of positioning one node below another node and the action of positioning a node above an existing node will make it possible to examine to what extent participants are using a top-down or bottom-up approach, respectively. Using this more elaborate coding scheme, it may be possible to use sequential analysis to identify a broader range of approaches that students are using and determine how these particular approaches affect the quality of students’ argument maps. Ultimately, the mapping processes found to be associated with specific strategies can be formalized and used by systems to monitor, diagnose and provide students’ with the immediate feedback needed to learn the complex skill of analyzing arguments. Limitations and directions for future research Although the findings in this study are consistent with prior research and provide insights into the processes used by experts and novices to map out and analyze complex arguments, the findings are not conclusive. This study was primarily qualitative in nature, conducted using more of an exploratory design with small sample size. As a result, the cause-effect relationship between the observed processes and map accuracy cannot be verified using the data from this study. To determine the cause-effect relationships between the observed processes and performance scores, different versions of argument mapping software can be developed to guide users through different reasoning processes (e.g., top-down versus bottom-up, holistic vs. atomistic, breadth vs. depth-first) to test and compare the effects of specific reasoning processes on map accuracy. Furthermore, the jMAP software and coding scheme will need further refinement so that sequential analysis can be used to model and validate not only the holistic/atomistic process, but also the breath/depth and top/bottom processes to better identify a broader range of weaknesses and/or flaws in students’ reasoning processes and to deliver more targeted feedback to assist students in mapping out complex arguments. In addition, the participants in this study were presented with claims that were extracted in advance by the researcher (not by the participants). It is not known to what extent this particular characteristic of the task affected the processes used by the experts and novices. As a result, this study’s findings may not be generalizable to contexts when participants themselves extract and identify the claims as they reflect on the antecedents and outcomes associated with each specific claim. Future studies are needed to identify whether the processes differ when students are versus are not given the claims in advance. Other potential limitations of this study are the differences in participants’ age (novices were younger on average than the experts) and gender (with novices mostly female and experts all male). As a result, future studies might test for potential differences in processes based on age and gender. Furthermore, the novices were not able to verbalize their thought processes as well as the experts, and it cannot be determined at this time whether this difficulty can be attributed to the novices’ ability to apply reasoning skills and to what extent this affected the fidelity of the data collected from the novices. Lastly, the use of sequential analysis required a minimum cell frequency of 5 for each action sequence in order to determine with at a level of acceptable reliability the existence of an underlying pattern. Along the same lines, the critical p-value of .05 was used in the sequential analysis due to the exploratory nature of this study. However, a more conservative p value along with a larger sample size can more accurately identify differences between the experts versus novices in future studies. Conclusion The goal of higher education is to train students to think critically and to apply the skills of reasoning to accurately analyze, understand, and evaluate complex arguments. The findings in this study demonstrate that graduate students’ reasoning skills can vary in terms of the strategies they employ (compared to processes used by experts) and the level of understanding that can be achieved by employing specific strategies. As a result, educators need to focus efforts on identifying students’ current reasoning skills and teaching more advanced reasoning skills, in addition to teaching knowledge and content specific to their

13

disciplines. One way to identify the target reasoning skills to be taught to students is to engage in some reverse engineering by first identifying and modeling the reasoning processes used by experts. Once these processes have been adequately identified and validated, further studies are needed to test the effects of the strategies in controlled experimental studies. Once tested, students reasoning skills can be improved by providing students with guidance on how to perform the empirically tested processes. One approach to modeling and testing the processes, as demonstrated in this study, is to implement specific processes within argument mapping software with built-in functions that can diagnose and recognize students’ behavioral patterns and provide adaptive feedback to teach the processes and to improve learning and performances. Although more evidence and studies are needed to fully identify, flesh out, and better understand the reasoning processes of experts, this study provides insights into how people reason when asked to break down and analyze complex arguments by using a computerized mapping tool and sequential analysis to model and operationally define the processes of argument analysis.

14

References Bakeman, R., & Gottman, J.M. (1997). Observing Interaction: An Introduction to Sequential Analysis.

Cambridge: Cambridge University Press.

Cañas A. J. (2003). A Summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. Technical report: Pensacola, FL, 108 p.

Chi, M. T., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (pp. 7-77). Hillsdale, NJ: Erlbaum.

Crespo, K.E., Torres, J. E., & Recio, M. E. (2004). Reasoning process characteristics in the diagnostic skills of beginner, competent, and expert dentists. Journal of Dental Education, 68(12), 1235-1244.

Davies, M. (2011). Introduction to the special issue on critical thinking in higher education. Higher Education Research & Development, 30(3), 255-260.

Frensch, P. A., & Funke, J. (1995). Definitions, traditions, and a general framework for understanding complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 3-26). Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc.

Gold, J., Holman, D., & Thorpe, R. (2002). The role of argument analysis and story telling in facilitating critical thinking. Management Learning, 33(3), 371-388.

Harrell, M. (2007). Using argument diagramming software to teach critical thinking skills. Proceedings of the 5th International Conference on Education and Information Systems, Technologies and Applications. July, Orlando. 2007

Harrell, M. (2008). No Computer Program Required: Even Pencil-and-Paper Argument Mapping Improves Critical-Thinking Skills. Teaching Philosophy, 31(4).

Jeffries, R., Turner, A.A., Polson, P., & Atwood, M.E. (1981). The processes involved in designing software. In J.R. Anderson (Eds.), Cognitive Skills and Their Acquisition. Hillsdale, N.J.: Erlbaum, 225-283.

Author (2010). xxxxx

Author (2014). xxxxxx

Kuhn, D. (1991). The Skills of Argument. Cambridge: Cambridge University Press.

Kuhn, D., & Udell, W. (2003). The development of argument skills. Child Development, 74(5), 1245-1260.

Livingston, C., & Borko, H. (1989). Expert-novice differences in teaching: A cognitive analysis and implications for teacher education. Journal of Teacher Education, 40(4), 36-42.

Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical Education, 39, 418-427

Reimold, M., Slifstein, M., Heinz, A., Mueller-Schauenburg, W., & Bares, R. (2006). Effect of spatial smoothing on t-maps: arguments for going back from t-maps to masked contrast images. Journal of Cerebral Blood Flow and Metabolism, 26(6). doi: 10.1038/sj.jcbfm.9600231

Rubin, R.S., & Ragusa, J.M. (1992). Developing expert systems for small business: An application for selecting a legal form of organization. Journal of Small Business Strategy, 3(2), 51-61.

Sharma, T., Tiwari, N., & Kelkar, D. (2012). Study of difference between forward and backward reasoning. International Journal of Emerging Technology and Advanced Engineering, 2(10), 271-273.

Smith, M. U. (Ed.). (2012). Toward a unified theory of problem solving: Views from the content domains: Routledge.

15

Sweller, J., Clark, R. E., & Kirschner, P. A. (2011). Teaching general problem solving does not lead to mathematical skills or knowledge. EMS Newsletter, 41-42.

van Bruggen, J.M., Kirschner, P.A., & Jochems, W. (2002). External representation of argumentation in CSCL and the management of cognitive load. Learning and Instruction, 12(1), 121-138.

van den Braak, S. W., Oostendorp, H. V., Prakken, H., & Vreeswijk, G. A. (2008, January). A critical review of argument visualization tools: Do users become better reasoners? In Workshop Notes of the ECAI-06 Workshop on Computational Models of Natural Argument (pp. 67-75).

16