Upload
vankien
View
214
Download
0
Embed Size (px)
Citation preview
FLORIDA STATE UNIVERSITY
COLLEGE OF EDUCATION
MODELING THE REASONING PROCESSES IN EXPERTS AND NOVICES’ ARGUMENT
DIAGRAMMING TASK: SEQUENTIAL ANALYSIS OF DIAGRAMMING BEHAVIOR
AND THINK-ALOUD DATA
By
HAE YOUNG KIM
A Dissertation submitted to the
Department of Educational Psychology and Learning Systems
in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy
Degree Awarded:
Spring Semester, 2015
ii
Hae Young Kim defended this dissertation on April 13, 2015.
The members of the supervisory committee were:
Allan Jeong
Professor Directing Dissertation
Michael Kaschak
University Representative
Valerie Shute
Committee Member
Vanessa Dennen
Committee Member
The Graduate School has verified and approved the above-named committee members, and
certifies that the dissertation has been approved in accordance with university requirements.
iii
I dedicate this work to my mother, Jeong Ja Ma, who loves and supports me in my life, and to
my children, Aileen Yerin and Daniel Seojin.
iv
ACKNOWLEDGMENTS
First of all, acknowledgment goes to my parents, Jeong Ja Ma and Myung Jun Kim, who
supported me through difficult times in my doctoral studies. As a mother of two children, I was
able to spend time with them in the USA because of my mother’s help. When I gave birth to my
first child, my mom held my hand during my c-section. She took care of me and my two babies
and helped me to perform my dual roles as mother and student. My father always says good
things about me ever since I was a kid. His positive belief and strong support led me to pursue
my dream and to achieve my goals. Seeing my mom’s tears as expression of joy and pride, I
realized again how much love I have received from my mom. Without my family’s love and
support, I could not have finished my doctoral studies.
Also, I would like to give many thanks to my advisor, Dr. Allan Jeong. Dr. Jeong is a
patient and considerate professor. Whenever I fell down and lost motivation, he provided me
new insights to help me overcome emotional obstacles. Without his advising, I would not have
been able to complete my dissertation. I deeply appreciate his knowledge and helpful advising.
My committee, Dr.Valerie Shute and Dr.Vanessa Dennen are great instructors and I
learned a lot from their courses. Dr. Shute is an excellent researcher and life mentor and I
always think that I wish I could live like her. Also, Dr. Dennen provided me various teaching
experiences and her support really helped to make me to complete my PhD without financial
difficulty and to prepare for my future career. My previous committee Dr. Jonathan Adams
provided new insights on my research so that I could see my project from a different point of
view. Lastly, Dr.Michael Kaschak, as my new committee, willingly took my request to be my
outside committee and helped me get through the final defense. Without their support and
advising, I could not have finished my dissertation.
v
In addition, I would like to thank Dr. Woon Jee Lee, my friend and an excellent
researcher. She helped me to analyze my data and contributed to the analysis. Also, thank you
to Karen Hand for volunteering her excellent editing skills to help me improve the quality of
writing in my dissertation. Lastly, I would like to thank my former advisors who inspired me to
be an educational researcher, Dr. Chul-Hwan Lee and Dr. Seon-Gwan Han. They showed their
strong belief in me and supported me in my future studies.
I have to mention that I had a great opportunity to learn Educational Statistics and
Measurement at FSU. Especially, Dr. Betsy Becker, Dr. Yangyun Yang, and Dr. Russell
Almond who taught me statistics and helped me to earn a Master of Science in Educational
Statistics and Measurement. This was a really outstanding privilege that I had since I had great
professors at FSU.
Lastly, I also thank to Dr. Danielle Sherdan as my former boss. She has the warmest heart
and positive attitudes toward people and life. Her consistent support helped me financially and
emotionally to go through my doctoral life.
And Mary Kate McKee is a wonderful person who cares every student in Instructional
Systems and Learning Technologies. I am very lucky to have all these amazing people in my
life.
Now, I’m closing my acknowledgment as imaging myself to hold my future students’
hands on their graduation day. I would like to share my experiences with my students and
inspire them to pursue and achieve their goals. In future, I hope I can be the one of the amazing
mentors and teachers whom l listed above.
vi
TABLE OF CONTENTS
List of Tables ...................................................................................................................................x
List of Figures ............................................................................................................................... xii
Acknowledgments........................................................................................................................... iv
Abstract ......................................................................................................................................... xiv
CHAPTER 1: INTRODUCTION ................................................................................................... 1
Lack of Critical Thinking in Higher Education ........................................................................................ 1
Teaching Critical Thinking with Argument Analysis............................................................................... 2
Supporting Argument Analysis with Diagramming Tools ....................................................................... 3
Need for Identifying Reasoning Processes Used to Diagram Arguments ................................................ 6
Comparing Experts versus Novices to Identify Reasoning Processes that Produce More Accurate
Maps ........................................................................................................................................................ 8
What Argument Diagramming Can and Cannot Reveal About the Process ............................................ 9
Goals of the Study .................................................................................................................................. 12
Significance of this Study ....................................................................................................................... 13
CHAPTER 2: LITERATURE REVIEW ...................................................................................... 15
Overview ................................................................................................................................................ 15
Critical Thinking and Argument Analysis .............................................................................................. 16
Argument Structure ............................................................................................................................ 18
Problems in Real-World Arguments .................................................................................................. 21
Approach to Argument Analysis ........................................................................................................ 22
Argument Mapping to Enhance Argument Analysis ......................................................................... 23
Cautions in Using Argument-Mapping Software ............................................................................... 29
Reasoning Processes ............................................................................................................................... 30
Effects of Content and Context on Reasoning Processes ................................................................... 33
Reasoning in Arguments .................................................................................................................... 34
Informal Arguments in Everyday Life ............................................................................................... 35
Common Reasoning Fallacies ............................................................................................................ 36
Research on Argument Diagramming Processes ............................................................................... 41
Expertise Research ................................................................................................................................. 43
vii
Characteristics of Expertise ................................................................................................................ 44
Definition of Expertise ....................................................................................................................... 45
Expertise in Reasoning Process .......................................................................................................... 48
Literature Review for Methodologies ..................................................................................................... 55
Verbal Reports ................................................................................................................................... 55
Think-Aloud Protocol ........................................................................................................................ 56
Limitations of Concurrent Think-aloud Protocol. .............................................................................. 57
CHAPTER 3: METHOD .............................................................................................................. 59
Overview ................................................................................................................................................ 59
Research Design and Approach .............................................................................................................. 60
Participants ............................................................................................................................................. 60
Settings and Technology ........................................................................................................................ 61
Procedures .............................................................................................................................................. 61
Introductory Session ........................................................................................................................... 61
Training Session ................................................................................................................................. 62
Tasks .................................................................................................................................................. 63
Data Collection ....................................................................................................................................... 66
Assessing the Quality of Argument Diagrams ................................................................................... 66
Coding the Video and Audio Data.......................................................................................................... 70
Data Analysis .......................................................................................................................................... 71
Assumptions for Sequential Analysis ................................................................................................ 74
Limitation of Sequential Analysis ...................................................................................................... 80
Scoring Students’ Argument Diagrams .............................................................................................. 81
CHAPTER 4: RESULTS .............................................................................................................. 84
Introduction ............................................................................................................................................ 84
Demographic Information ...................................................................................................................... 84
Research Question 1: What Reasoning Processes Do Experts and Novices Perform when Diagramming
a Complex Argument? ............................................................................................................................ 88
Sequential Patterns in Experts’ Actions ............................................................................................. 89
Sequential Analysis of Novice Actions .............................................................................................. 93
Transitional State Diagrams of Patterns in Action Sequences of Expert vs. Novice ......................... 95
viii
Research Question 2: What Differences Exist in the Reasoning Processes Used by Experts versus
Novices? ............................................................................................................................................... 101
Similarities in the Reasoning Processes of Experts and Novices ..................................................... 101
Differences In the Reasoning Processes of Experts and Novices .................................................... 102
Research Question 3: Which Processes Might Help Produce Diagrams of High versus Low
Accuracy? ............................................................................................................................................. 104
Primary Findings .............................................................................................................................. 104
Post-hoc Analysis ............................................................................................................................. 104
Frequencies and Transitional Probabilities between Action Sequences .......................................... 104
Sequential Patterns Exhibited by the Two High versus Low Performers ........................................ 109
Sequential Patterns Exhibited by the High Performers .................................................................... 109
Sequential Patterns Exhibited by the Low Performers ..................................................................... 109
Qualitative Findings ............................................................................................................................. 112
Coding Processes.............................................................................................................................. 112
Five Global Processes Used for Diagramming an Argument Map .................................................. 120
Types of Reasoning by Experts and Novices ................................................................................... 129
Experts Use of Two Strategies to Analyze and Construct an Argument Map ................................. 130
Experts Also Committed Reasoning Fallacies ................................................................................. 132
Summary of Main Qualitative Findings ........................................................................................... 137
Outliers and Limitations ................................................................................................................... 139
CHAPTER 5: DISCUSSION ...................................................................................................... 141
Research Question 1. What Reasoning Processes Do Experts and Novices Perform When
Diagramming a Complex Argument? ................................................................................................... 141
Research Question 2: What Differences Exist in the Reasoning Processes Used by Experts versus
Novices? ............................................................................................................................................... 145
Research Question 3: Which Processes Might Help Produce Diagrams of High versus Low
Accuracy? ............................................................................................................................................. 146
Qualitative Findings and Discussion .................................................................................................... 149
Instructional and Software Design Implications ................................................................................... 153
Limitations of the Study ....................................................................................................................... 156
Limitations of Sequential Analysis ....................................................................................................... 158
Directions for Future Research ............................................................................................................. 159
ix
Conclusion ............................................................................................................................................ 161
APPENDIX A ............................................................................................................................. 162
PARTICIPANT’S PROFILE SURVEY............................................................................................... 162
APPENDIX B ............................................................................................................................. 164
SUMMARY OF SIX E-LEARNING PRINCIPLES ........................................................................... 164
APPENDIX C ............................................................................................................................. 170
RETROSPECTIVE INTERVIEW QUESTIONS ................................................................................ 170
APPENDIX D ............................................................................................................................. 171
IRB APPROVAL MEMO .................................................................................................................... 171
APPENDIX E ............................................................................................................................. 172
PARTICIPANT CONSENT FORM .................................................................................................... 172
APPENDIX F.............................................................................................................................. 173
INVITATION EMAIL FOR THE SECOND CODER ........................................................................ 173
APPENDIX G ............................................................................................................................. 174
CODING PROCEDURES FOR THE SECOND CODER ................................................................... 174
APPENDIX H ............................................................................................................................. 177
EXAMPLES OF CODING RESULTS FOR MAPPING BEHAVIOR AND VERBAL REPORT .... 177
APPENDIX I .............................................................................................................................. 199
INTERVIEW TRANSCRIPT RESULTS ............................................................................................ 199
REFERENCES ........................................................................................................................... 205
BIOGRAPHIC SKETCH............................................................................................................ 213
x
LIST OF TABLES
2.1 The Seven Steps in Argument Analysis. Adapted from Scriven (1976, P. 39) .................22
2.2 Characteristics of Two Cognitive Systems in Dual-Processing Theories. Adapted from
Evans (2008, p. 257 ...........................................................................................................31
2.3 Common Informal Reasoning Fallacies. Adapted from Ricco (2007, p. 460). .................37
2.4 Psychological Characteristics That Facilitate and Limit Expertise. Summarized from Chi
(2006) .................................................................................................................................46
2.5 The Proficiency Scales. Adapted from Hoffman (1996, p. 84-85) ....................................47
2.6 Expert and Novice Comparisons on Reasoning Processes ................................................54
3.1 Codes Assigned to Each Action Students Perform in jMAP Software .............................67
3.2 A Generic 2 by 2 Contingency Table ................................................................................77
3.3 An Example of 2 by 2 Contingency Table for Post-hoc Analysis .....................................78
4.1 Demographic Information of the Participants....................................................................85
4.2 Participant’s Perceived Content Familiarity and Time Spent on Tasks ............................86
4.3 Participants’ Argument Map Scores in Details ..................................................................88
4.4 Frequency Matrix of Experts’ Reasoning Processes .........................................................90
4.5 Transitional Probabilities with Associated Z-scores of Sequential Reasoning Processes in
Expert Group ......................................................................................................................91
4.6 Frequency Matrix of Novice Group’s Reasoning Process .................................................93
4.7 Transitional Probability with Associated Z-scores of Sequential Reasoning Processes in
Novice Group .....................................................................................................................94
4.8 Summary of Mapping Processes Observed in Experts and Novices ...............................101
4.9 Frequency Matrix for High Performer Group ..................................................................106
4.10 Transitional Matrix and Z-scores for High Performer Group ..........................................106
4.11 Frequency Matrix for Low Performer Group ..................................................................107
xi
4.12 Transitional Matrix and Z-scores for Low Performer Group...........................................107
4.13 Summary of Reasoning Processes Observed in the Two High and Low Performers ......109
4.14 The Initial Codes Emerged from Verbal and Mapping Action Data ...............................113
4.15 Modified Coding Scheme for Verbal and Mapping Action Data ....................................115
4.16 An Example of Coding Sheet (Exerted from Novice 2) ..................................................116
4.17 Frequencies of Each Code Observed in Individual Participants ......................................118
4.18 The Final Codes Combining Think-aloud and Mapping Behaviors for Sequential
Analysis............................................................................................................................119
4.19 Observation Summaries of Novices’ Argument Mapping Processes ..............................124
4.20 Observation Summaries of Experts’ Argument Mapping Processes ...............................127
4.21 Chi-square Test for the Frequencies across Participants .................................................129
4.22 Contingency Table for Reasoning Styles Used by Expert and Novice Groups ...............130
4.23 Frequencies of Reasoning Fallacies Observed in Final Argument Maps ........................133
4.24 An Example of the Leap to the Conclusion Fallacy (Expert 3’s mapping and verbal action
script) ...............................................................................................................................134
4.25 An Example of the Leap to the Conclusion Fallacy (Expert 5’s mapping and verbal action
script) ...............................................................................................................................134
4.26 Summary of the Quantitative and Qualitative Findings by Research Question ..............140
5.1 Mapping Actions as Indicators of the Map Construction and Reasoning Processes .......160
xii
LIST OF FIGURES
1.1 Important areas involved in argument diagramming .........................................................11
2.1 Toulmin’s model of arguments (Toulmin, 1958, p. 104) ..................................................19
2.2 An example of an argument structure (Rider & Thomason, 2008, p. 18) .........................20
2.3 An example of a tree diagram. Adapted from Scriven (1976, p. 39) .................................23
2.4 An example of a multi-reason argument (http://www.austhink.com) ...............................25
2.5 Correctly represented co-premises (Twardy, 2004, p. 6) ...................................................27
2.6 Incorrect representation of co-premises (Twardy, 2004, p. 7) ...........................................28
2.7 An example of the Rabbit Rule of argument (http://www.austhink.com) .........................28
2.8 An example of the Holding Hands Rule of argument mapping (http://www.austhink.com)29
2.9 Examples of induction (a) and deduction (b) problems (Heit, 2007, p. 3) ........................34
2.10 A hypothetical target argument diagram............................................................................38
2.11 Fallacy of hasty generalization (Missing ‘B’ node as a mediated factor)..........................39
2.12 Fallacy of begging the question (Circular reasoning) ........................................................39
2.13 Fallacy of single cause. ......................................................................................................40
2.14 Fallacy of irrelevance .........................................................................................................40
2.15 Fallacy of wrong direction (Reversed causation) ..............................................................41
2.16 Transitional state diagrams of action sequences (Jeong, 2014, p. 247) .............................42
2.17 A game tree (Chi et al., 1982, p. 13) ..................................................................................50
2.18 Chess trees presented by a novice and expert. Adapted from Ericsson (2006, p. 234) .....51
2.19 An example of the depth-first strategy in constructing an argument diagram ...................52
2.20 An example of the breadth-first strategy in constructing an argument diagram ................53
2.21 Memory model of cognitive system and the five information processes. Adapted from
van Someren et al. (1994, p. 19) ........................................................................................55
xiii
2.22 An illustration of concurrent think aloud protocol (Ericsson, 2006, p. 227) .....................57
3.1 The screen capture of a student jMAP with a default arrangement of nodes. ...................63
3.2 The overview of the procedure and data collection process. .............................................65
3.3 The criterion map. ..............................................................................................................69
3.4 An example of a transitional frequency matrix ..................................................................72
3.5 An example of a transitional probability matrix ................................................................73
3.6 An example of z-scores matrix ..........................................................................................74
3.7 A transitional diagram of Group A ....................................................................................76
3.8 A transitional diagram of Group B ....................................................................................76
3.9 The revised criterion map. .................................................................................................82
4.1 Transitional diagram of the expert group reasoning processes. .........................................92
4.2. The transitional diagram of the novice group’s reasoning processes. ...............................95
4.3. Comparisons of two transitional diagrams of the expert (left) and novice (right)’s
reasoning processes. ...........................................................................................................96
4.4. Bar graph to represent the participants’ final argument map scores. ...............................105
4.5. Transitional diagrams of the two high (left) and the two low performers (right) ............108
4.6. The diagram point at which Expert 3 committed the leaping to conclusions fallacy. .....135
xiv
ABSTRACT
A variety of software tools and guidelines have been developed to help students diagram,
analyze, and better understand complex arguments. However, little or no empirical evidence
exists to validate whether the processes embedded within existing tools and guidelines are
processes that produce better argument diagrams. As a result, the purpose of this study was to
determine: 1) the mapping and reasoning processes used by experts and novices to analyze
complex arguments; 2) how the processes performed by experts versus novices differ; and 3)
based on the observed differences, identify the processes that facilitate and hinder more accurate
argument analysis. The verbal reasoning and argument-diagramming processes of four experts in
argumentation and five novices across four different graduate programs were recorded on video
as they constructed their argument maps using a diagramming software application called jMAP
and as they verbalized their thought processes in a think-aloud protocol/interview. Sequential
analysis was used to identify and differentiate the sequences of mapping actions used by experts
versus novices and the sequences of mapping actions that were used to produce the highest
versus lowest quality argument diagrams. The findings from this indicated that the experts’
processes for positioning, linking, and reviewing nodes produced more accurate maps than the
processes used by novices. Based on these findings, I discussed several possible interpretations
of the four experts’ reasoning processes in the context of argument diagramming tasks and in the
context of more global reasoning processes identified from a qualitative analysis of the video
recordings and verbal protocols. Lastly, I presented several educational implications with regard
to using the experts’ processes as a model for scaffolding and helping students better analyze and
evaluate complex arguments and for designing diagramming software.
1
CHAPTER 1
INTRODUCTION
Lack of Critical Thinking in Higher Education
The Partnership for 21st Century Skills (2009) claims that the U.S. education system should
prepare every student with critical thinking skills to enable students to be successful in their daily
lives. In fact, many educators have stressed the importance of critical thinking since the 20th
century and have claimed that promoting student critical thinking should be one of the most
essential goals in higher education (Davies, 2011; Harrell, 2011; McMillan, 1987). In addition to
having a deeper understanding of specific subject knowledge, undergraduate and graduate
students should be encouraged to develop critical thinking skills so that they are able to
effectively reason and accurately judge information and thinking (McMillan, 1987). Employers
have stressed that it is important that future employees possess strong critical thinking skills to
be successful in performing a variety of tasks (Davies, 2013; The Chronicle of Higher Education
& American Public Media’s Marketplace, 2012).
Even though there is some controversy over the definition of critical thinking, most
educators agree that reasoning, analyzing, judging and evaluating information are essential
components of critical thinking (Cosgrove, 2011; Harrell, 2011; The Partnership for 21st Century
Skills, 2009). For example, Kuhn (1991) claims that argument skills are fundamental
competencies for critical thinking. In everyday life, people face many arguments when making
important decisions or judgments. Some information may be incorrect and some arguments may
be based on faulty or inaccurate evidence. As a result, people are often unable to make the best
judgments when solving everyday problems. Similar to Kuhn’s definition of critical thinking,
Paul and Elder (2001) define critical thinking as “the art of analyzing and evaluating thinking
2
with a view to improving it” (p.2). Recently, the Partnership for 21st Century Skills (2009) and
Binkley et al. (2012) defined critical thinking as effective reasoning, systems thinking, and
judgments/decision-making skills (The Partnership for 21st Century skills, 2009). The
Partnership for 21st Century Skills (2009) identified the following sub-skills of critical thinking:
using an appropriate reasoning method based on the situation, analyzing interactions among
elements and outcomes in a complex system, analyzing evidence/claims/arguments, inferring,
drawing conclusions, and evaluating arguments and alternatives.
Although critical thinking is considered to be an essential skill in higher education and
professional areas, recent research shows that many college students fail to develop critical
thinking skills to the extent that they can effectively use the skills (Davies, 2011; Reimold,
Slifstein, Heinz, Mueller-Schauenburg, & Bares, 2006; Gold & Holman, 2002; Kuhn, 1991).
Davies (2013) points out that employers are more likely to hire students with strong critical
thinking skills than students with weak critical thinking skills despite their superior grades and
content knowledge. This trend may reflect the ever changing dynamic nature and complexity of
today’s real-world problems. As a result, teaching and improving students’ critical thinking is a
paramount goal in higher education.
Teaching Critical Thinking with Argument Analysis
To address students’ deficiencies in critical thinking, argument analysis is one method
that has been used in higher education to teach critical thinking across many, if not most,
disciplines (e.g., education, philosophy, psychology, economics, and political science) because
argumentation is an essential part of the scientific and problem-solving process. Bensely (2010)
defines argument analysis as a process of “evaluating evidence, drawing appropriate conclusions
along with other skills, such as distinguishing arguments from non-arguments and finding
3
assumptions” (p. 49). As a result, the skill of analyzing arguments is an important component of
critical thinking and hence is a skill that college students should develop (Harrell, 2008).
Specifically, argument analysis is the study of logical relationships among claims presented
in an argument (which can be mutually supporting or opposing opinions/claims) – an important
part of the process of reasoning through premises to reach a conclusion. In argument analysis,
students identify the functional roles of each proposition (i.e., conclusion, premise, co-premise,
counterargument), analyze the hierarchical relationships among propositions across major and
minor premises (i.e., levels of premise), and evaluate the quality and validity of a given
argument. It is a process that can be used to help identify flaws and evaluate the truth value of
stated arguments, which ultimately can help one draw more well-reasoned conclusions and make
better decisions.
Supporting Argument Analysis with Diagramming Tools
The structure of a presented argument can be complex and ill-defined. As a result,
argument analysis requires students to perform cognitive operations that are complex and multi-
step. For example, an argument analysis begins with the process of extracting the true intent or
major premise (or claim) presented in the text and distinguishing the major from the minor
premises – premises that are presented to establish the veracity of the major claim. Each minor
premise itself can be accompanied by a series or chain of premises presented to establish (and
sometimes, to challenge) the truth value of the minor premise. As a result, the analysis of an
argument requires one to flesh out the hierarchical relationships behind all major and minor
premises so that flaws in the lines of reasoning can be identified to determine the overall quality
and truth value of a given claim or conclusion. These processes of argument analysis require
significant attention, memory, and cognitive effort and are likely to produce heavy cognitive load
4
that can inhibit performance and learning (Harrell, 2007; van Bruggen, Kirschner, & Jochems,
2002). Another challenge that one faces when analyzing arguments is when minor premises or
assumptions are not explicitly stated, thus requiring students to infer the missing premise in order
to establish the logical relationship between given premises and a given conclusion (Ennis,
1982). Lastly, the individual’s biases, beliefs, and emotional states regarding a given topic can
affect the reasoning process and the quality of the final conclusion (Correia, 2011; Klaczynski,
2000) .
To address these challenges, argument diagramming software has been developed to help
students draw visual diagrams to scaffold the process of mapping out the hierarchical
relationships between major and minor premises (Braak, Oostendorp, Prakken & Vreeswijk,
2006). Diagramming software like Belvedere (De Neys, 2006), Rationale (van Gelder, 2007),
and jMAP (Jeong, 2010) enables students to draw, position, and link multiple nodes to create
diagrams that provide a visual and spatial means of mapping out and conveying complex
hierarchical relationships among premises. Using this approach, students do not have to rely
solely on memorizing multiple series’ of propositions in the form of verbal representations. As a
result, the use of argument diagrams can help to reduce cognitive load when analyzing complex
arguments (Harrell, 2007; van Bruggen, Kirschner, & Jochems, 2002). In other words, the use of
diagramming tools enables students to allocate more working memory capacity to interpret the
text, identify the functional elements of the text (premises, supports, objections, counter-
arguments, etc.), and analyze the nature and quality of the hierarchical relationships between
premises.
Recent studies on the impact of diagramming software on students’ argumentation skills in
higher education have reported positive effects (Harrell, 2011; Twardy, 2004; Bruggen,
5
Boshuizen, & Kirschner, 2003; van Gelder, 2002). For instance, Harrell (2011) found that
teaching argumentation with diagramming tools enhances college students’ critical thinking
skills. Prior to Harrell’s study, Easterday, Aleven, and Scheines (2007) examined the effects of
diagrams and diagramming tools on causal reasoning in college students. They found that by
having students create their own causal diagrams, students exhibited more complex cognitive
processes (combined comprehension, construction and interpretation) than students that simply
studied a given set of texts or causal diagrams. Like argument diagrams, causal diagrams enable
students to articulate and better understand the causal relationships between event-based
arguments and, as a result, help to improve students’ causal reasoning skills in the same manner
that argument diagrams help to improve students’ analysis of semantic relationships between
premises.
However, a critical review of research on the efficacy of using tools for diagramming and
visualizing arguments revealed that the majority of studies found no significant differences in
their effects on student learning (Braak et al., 2006). The review revealed that a large majority of
the research lacked validity due to problems in experimental design. As a result, the efficacy of
using such mapping tools is still in question. Also, the assessment of learning with students’
maps varied in terms of the nature of the tasks and subject matter, mapping techniques
(hierarchical, networking, etc.), and scoring rules employed by studies (Ruiz-Primo &
Shavelson, 1996). As a result, the overall effectiveness of these tools when used to teach
argumentation remains inconclusive.
6
Need for Identifying Reasoning Processes Used to Diagram Arguments
Braak et al.’s (2006) critical review noted that most, if not all, prior studies on argument
diagramming tools assessed students’ argumentation skills based on the evaluation of the final
product – students’ argument diagrams. The emphasis on evaluating the final maps alone has not
helped to advance our understanding of how students construct their argument diagrams, what
processes they use, and which of the processes are most effective. Likewise, prior research has
focused primarily on determining the effects of particular interventions on students’
understanding of arguments, not on how the interventions affect the processes students use to
construct an argument diagram (Kuhn & Udell, 2003) and how the resulting changes in these
processes in turn affect the final maps. Achieving a deeper understanding of the processes
students use to diagram arguments may help us to understand why and when particular tools
work and do not work. However, little research has been conducted to explicitly model and
distinguish the processes that improve versus hinder students’ analysis and understanding of
complex arguments.
In addition to the processes used to construct argument diagrams are the processes of
logical reasoning – another essential part of analyzing and evaluating arguments (Goel, Buchel,
Frith, & Dolan, 2000). Logical fallacies (e.g., leaping to conclusions, slippery slope, circular
arguments) and the processes used to identify and resolve these fallacies are illustrations of the
types of high-level reasoning processes that can occur when analyzing arguments. However, the
research that has examined reasoning processes provides a very limited picture of which and how
these particular processes are used to produce high quality versus low quality analysis of
arguments. For example, the reasoning processes used to analyze a syllogism has been a
frequent area of research (Evans, 2003; Johnson-Laird, 1999; Schaeken, 2000) within the field of
7
cognitive psychology. A syllogism is a logical form of an argument that includes three
propositions: two premises and one conclusion. Syllogistic reasoning is a form of deductive
reasoning in which a set of logical rules is applied to evaluate whether or not the conclusion is
true or false. However, arguments in the real world often involve the analysis of multiple,
complex, and/or incomplete syllogisms. Van Bruggen et al. (2003) describe the characteristics
of real world arguments as ‘ill-structured, incomplete, ambiguous, and not rule-based’.
Furthermore, the relationships among propositions are often probabilistic or conditional in nature
and not absolute in truth. As a result, the types of processes that help or hinder students’ ability
to successfully diagram complex arguments have not yet been thoroughly examined and
identified in prior research.
With the goal of identifying some of these processes, Jeong (2010) developed the jMAP
software application to automatically capture and codify the sequences of mechanical actions
students perform while constructing complex diagrams. This type of data can be sequentially
analyzed and potentially used to visualize, reveal, and identify the mental reasoning processes
used to produce high and low quality argument diagrams. For example, Jeong (2014) found that
high performers were more likely to perform certain action sequences than low performers.
High performers not only deleted links three times more often than low performers, high
performers were also likely to follow a link deletion by adding a new link (delete new link).
This particular action sequence can indicate a situation where high performers are correcting for
errors produced by leaping to conclusions (when AC and BC should be changed to
ABC). In addition, high performers not only deleted links more often than low performers,
they also re-routed existing links between nodes four times more often than low performers –
another action that can be used to correct for errors in an argument diagram. Given these
8
findings, Jeong concluded that the mechanical actions students perform on their diagrams may
serve as useful indicators of reasoning processes that produce more accurate maps and deeper
understanding of complex arguments (Jeong, 2014).
Although Jeong (2014) found sequential patterns that may help to explain the differences in
students’ map accuracy, it is still not clear what underlying reasoning processes are associated
with and account for specific diagramming actions and action sequences. As a result, more
qualitative research is needed to reveal the processes of reasoning underlying the diagramming
process in order to explain observed differences in students’ argument analysis and diagrams. In
other words, a qualitative approach is needed to identify and categorize both the mapping and
reasoning processes used by students to generate a tentative but explanatory theory about map
construction processes (Patton, 2001). Identifying both the reasoning and diagramming
processes that help and hinder students’ analysis and understanding will ultimately contribute to
further development and improvement of instructional interventions and diagramming tools.
Comparing Experts versus Novices to Identify Reasoning Processes that Produce
More Accurate Maps
The reasoning processes that lead to more accurate versus less accurate maps can be
identified by comparing the reasoning processes used by subject-matter experts to the processes
used by novices. Cognitive research has shown that experts use different cognitive processes
than novices to produce superior performances on tasks (Livingston & Borko, 1989; Norman,
2005). A study conducted by King, Wood, and Mines (1990) compared reasoning skills between
two groups – graduate and undergraduate students – and found that graduate students (expert
group) performed better in reasoning skills required for solving ill-defined complex problems
than undergraduate students (novice group). Even though their study does not explain how
9
graduate students’ reasoning processes differ from undergraduate students, the result indicates
that graduate students possess some, if not higher, levels of expertise in reasoning skills than
undergraduate students. However, King et al. (1990) pointed out that even advanced doctoral
students were unable to attain some of the higher levels of reasoning skills required to analyze
the most complex arguments. Their findings suggest that the development of reasoning skills
may be an ongoing and continual process as people engage in more academic cognitive tasks.
In the medical field, Norman (2005) found that experts in clinical diagnosis – whom were
found to possess better content knowledge, intuitive probabilistic skill, and experiential
knowledge than novices – make better use of memory and mental representation and clinical
diagnostic reasoning processes than novices. Likewise, experts in teaching have been found to
possess cognitive schemata that are more sophisticated, modifiable, interconnected and easily
accessible than those of novice teachers (Livingston & Borko, 1989). Livingston and Borko
(1989) concluded that some of these noted differences between expert and novice teachers can
affect the reasoning processes that are used and, in turn, affect the quality of outcomes in
constructing lesson plans. At this time, however, no studies have yet been conducted to
explicitly determine and model the reasoning processes used by experts versus novices.
What Argument Diagramming Can and Cannot Reveal About the Process
Observing the actions and action sequences students perform while constructing their
argument diagrams can potentially provide insights into the reasoning processes used to produce
more accurate argument diagrams (areas Y, X, and W in Figure 1.1) and less accurate argument
diagrams (areas R, Z, and M in Figure 1.1). By comparing sequential patterns in diagramming
behaviors used by novices versus experts, we can identify what types of action sequences
produce more versus less accurate diagrams as has been shown in Jeong’s (2014) study.
10
Although observations of diagramming processes may provide behavioral indicators of which
reasoning processes are being used by students to produce more accurate argument diagrams
(area X in Figure 1.1) and less accurate diagrams (area Z in Figure 1.1), the observed action
sequences student perform on their argument diagrams alone may not fully capture all of the
step-by-step reasoning processes that underlie each action students perform on their diagrams.
Furthermore, the diagramming actions students perform on an argument diagram may not be
representative of the internal reasoning processes that take place concurrently and/or between
diagramming actions to produce more accurate diagrams (area Y in Figure 1.1) versus less
accurate diagrams (area R in Figure 1.1).
For example, suppose a student creates an argument diagram containing three nodes: A,
B, and Claim. If the student correctly links B Claim and then correctly links A B, the
observer can surmise that the student has successfully used a backwards approach to identify the
major premise that supports the Claim, and then immediately moved on to identifying the minor
premises (A) that support the major premise B. In this case, this one-to-one correspondence
between diagramming processes and reasoning processes illustrates some of the possible
processes noted in area X of Figure 1.1. In contrast, a student that incorrectly links A Claim
and then links B Claim would reveal that the student has made a hasty generalization or is
leaping to conclusions (the belief that A leads to Claim when, in fact, A’s influence on the Claim
is moderated by B). This type of flaw in students’ reasoning process would be represented in
area Z of Figure 1.1 – an area that falls outside the Content Understanding circle. Area Y in
Figure 1.1 can represent, for example, a situation where the student correctly links B Claim,
then directs and redirects his/her eye gaze between A and Claim while making a mental
assessment of the possible linkage between A and Claim, then makes a determination as to
11
whether or not the Claim holds true if either A and B were not true, recognizes that the Claim
holds true even when A is not true, and finally, makes the decision not to link A to Claim. As a
result, area Y in Figure 1.1 represents some of the mental/internal (and not directly observable)
processes of reasoning that cannot be represented nor revealed by any set of observable actions
students perform on the diagram. Area W in Figure 1.1 can denote effective diagramming
actions that have no equivalent in terms of the mental processes of reasoning, such as moving
and positioning the Claim to the rightmost edge of the screen so that the sequencing of minor to
major premises to Claim flows visually from left to right (Jeong, 2014). In contrast, area M in
Figure 1.1 can denote ineffective diagramming actions that have no equivalent in terms of the
mental processes of reasoning, such as moving and positioning the Claim to the center of the
screen so that the links from minor to major premises to Claim flow haphazardly from left to
right, right to left, up to down, and down to up (Jeong, 2014).
Figure 1.1. Important areas involved in argument diagramming
To identify the types of reasoning and mapping processes denoted in all the aforementioned
six areas of Figure 1.1, this study examines diagramming actions that can be visually observed
Reasoning
Process (R)
X
ZY
Content
Understanding
(C)
Diagramming
Process (M) W
12
and performed by the subjects on diagrams displayed on the computer screen (areas W, M, X and
Z in Figure 1.1). Furthermore, this study examines the mental (and not directly observable)
processes that are performed by recording and analyzing the subjects’ verbal descriptions of the
mental processes they are performing (areas Y, R, X, and Z in Figure 1.1). The verbal
descriptions of the reasoning processes used by experts and novices are generated in this study
by using think-aloud protocol interviews. As a result, this study incorporates video recordings of
diagramming actions, verbal protocols, and retrospective interviews to identify in detail the
processes novices and experts use to analyze complex arguments.
Goals of the Study
Using think-aloud protocol and jMAP diagramming software, I will observe, code, and
identify the reasoning processes used by experts and novices to analyze a complex argument. I
will then analyze the coded data to identify sequential patterns in the actions experts and novices
perform while diagramming arguments in order to determine the mapping actions (and the
reasoning processes that are indicated by the mapping actions) that help to produce high versus
low understanding of complex arguments. Then I will use qualitative analysis to explore the
global processes that participants perform and to interpret the mapping processes identified with
the sequential analysis. As a result, this study addresses the following research questions:
1. What reasoning processes do experts and novices perform when diagramming a complex
argument?
2. What differences exist in the reasoning processes used by experts versus novices?
3. Which processes might help produce diagrams of high versus low accuracy?
13
Significance of this Study
Possessing good analytic skills is very important for graduate students given that these
skills help students to solve complex problems with multiple interrelated factors in real world
settings. Due to the tendency in prior research to overlook learning processes and to focus on
learning outcomes (Braak et al., 2006; Kuhn & Udell, 2003), studies on diagramming arguments
as a means of teaching argumentation skills have not yet examined the very processes that
students use while constructing argument diagrams. Other than the studies by Jeong (2012,
2014), few studies have yet identified the reasoning processes that help and/or hinder students’
ability to improve on their analysis and understanding of complex arguments while constructing
argument maps. The findings from this study provide preliminary insights into the types of
processes that can be promoted and discouraged when teaching students how to analyze and
diagram arguments. Identifying the unique processes used by experts (and not by novices) can
help us identify effective approaches to analyzing arguments and may provide helpful and
evidence-based guidelines and cognitive prompts (what you need to ask and think when you
identify the relationships between claims, etc.) to assist novice students in the argument analysis
process. Ultimately, the findings in this study will help researchers: a) better understand and
explain why particular interventions work or do not work in terms of how they affect the process,
and how in turn the process affects outcomes; and b) develop diagramming software that can
provide automated real-time process-oriented feedback to help students apply the appropriate
and most effective reasoning processes.
This study also helps to illustrate one approach to combining and integrating the use of
multiple data collection instruments (records of diagramming actions, think-aloud protocol,
retrospective interview) to identify, model, and better understand the processes of learning in
14
general. A separate look into the findings of this study from each data source will also help to
illustrate the possible shortcomings of using each data source alone to study and model complex
processes. For example, the findings from this study will illustrate some of the possible benefits
and shortcomings of using sequential analysis and related techniques to identify general patterns
in the processes used to achieve outcomes in complex learning tasks. With the increased level of
detail and specificity needed to identify, model, and better understand learning and problem-
solving from a process-oriented approach, this study can serve to help us understand the possible
limitations of and to improve on current methods used in the fledgling field of data mining and
learning analytics.
15
CHAPTER 2
LITERATURE REVIEW
Overview
To establish the justification, rationale, and theoretical and methodological framework for
this study, I present a review of the literature across four main topics: the use of argument
diagrams to support critical thinking and argument analysis; reasoning processes; differences in
reasoning processes between experts and novices; and think-aloud methods used to model
cognitive processes. First, I introduce the concept of arguments, its relationship to critical
thinking, the structures of complex arguments, and some of the prescribed procedures for
analyzing arguments. Next, I describe some computer-supported argument-diagramming tools
and argument-mapping rules used to enhance students’ argument diagramming and argument
analysis. Also, I present a review of studies that have examined diagramming processes and
point out what is lacking in current research about argument diagramming. Then I proceed to a
discussion of reasoning processes providing the two theoretical views of reasoning – problem
and process views – and dual-processing theories as the theoretical framework of this study. In
addition, I discuss different types of logical fallacies that people commit when analyzing
informal arguments and how some of the common reasoning fallacies can be identified in
students’ argument diagrams. To achieve a deeper understanding of reasoning processes that
produce high and low quality argument maps, comparisons between expert and novice reasoning
processes are reviewed to establish a methodological foundation for this study. Lastly, I discuss
a think-aloud method to model the experts’ and novices’ processes used to diagram and analyze
complex arguments.
16
As for the strategies that were used to search for the literature cited in this chapter, I used
ISI Web-of knowledge, Science Direct, and Google Scholar with the following (but not limited
to) search terms: ‘argument map*’, ‘reasoning or/and processes’, ‘experts and reasoning’,
‘expertise’, and ‘think-aloud protocol’. Among the citations that were listed in my search
results, I limited my search to journals in order to select articles that are from peer-reviewed
journals with high numbers of citations.
Critical Thinking and Argument Analysis
What is critical thinking? Siegel (1989) describes a critical thinker as a person who is
able to assess claims, make judgments, and reach a conclusion or take an action based on
reasons. “To be a critical thinker is to be appropriately moved by reasons” (Siegel, 1989, p.2).
Ennis (1989) defines critical thinking as “reasonable reflective thinking focused on deciding
what to believe or do” (p.4). Similar to Ennis’s assumption, Kuhn believes critical thinking is
related to thinking about thinking, a process he refers to as metacognition (1999). Practically
speaking, Ennis and Kuhn’s conception of critical thinking is rather broad in relation to Siegel’s
conception of critical thinking – too broad to apply to teaching and instruction. For this reason,
this study adopts Siegel’s definition of critical thinking by defining critical thinking as the ability
to evaluate claims (data), make judgments, and reach a conclusion based on reason.
Since the time of Socrates and the use of the Socratic method to develop questioning and
reasoning skills, critical thinking has been emphasized by most educators as one of the most
important educational goals in educational systems (Siegel, 1989). As social problems become
more complex and multifaceted, society requires people to possess critical thinking skills in
order to make good decisions and solve a variety of complex problems. In particular, higher
education programs have emphasized critical thinking (Davies, 2011; Harrell, 2011; McMillan,
17
1987) and there exists the expectation that schools should help elevate critical thinkers to higher
levels beyond simply being subject matter experts in their major field of study.
However, researchers have pointed out that our efforts in developing students’ critical
thinking in higher education have fallen short and have not adequately prepared students with the
level of critical thinking skills needed to address today’s complex problems. Despite the
consistent emphasis on critical thinking, higher education programs do not provide students with
sufficient experiences to successfully perform sophisticated high-level critical thinking
(Reimold, Slifstein, Heinz, Mueller-Schauenburg, & Bares, 2006). Davies (2011) criticizes the
fact that graduates are not equipped with the abilities required by employers, namely, the ability
to think critically and to reason and make judgments to resolve real-world problems in work
settings. Just as Kuhn (2007) pointed out the lack of argument skills in average adults, Gold and
Holman (2001), too, describe deficiencies in the ability of managers to analyze arguments,
understand perspectives which are different from their own, identify fallacies in arguments, and
establish and challenge the veracity of arguments with supporting or counter evidence. The lack
of these skills is prevalent, and hence the lack of development in critical thinking skills in college
and graduate-level education should increase educators’ attention on determining how to better
teach critical thinking across all disciplines.
To help students engage in critical thinking in higher education, argument analysis is one
instructional activity that has been used across many disciplines such as law, management,
economics, psychology, and philosophy (Bensley, Crowe, Bernhardt, Buckner, & Allman, 2010).
For example, Bensley et al. (2010) examined whether explicitly teaching argument analysis skills
enhanced students’ critical thinking in psychology courses at the college level. They found that
students with direct instruction in argument analysis had significant gains in critical thinking
18
compared to students with no argument analysis instruction. However, Bensley et al. (2010) did
not examine or test the efficacy of using specific techniques for teaching argument analysis
skills, nor did they identify the instructional materials used to teach argument analysis. Detailed
information about the procedures used to teach argument analysis is needed in order to produce
an operational definition of argument analysis so that further studies can be done to replicate
prior findings. Although Bensley et al. (2010) present a conceptual breakdown of the skills
associated with argument analysis (e.g., distinguishing arguments from non-arguments,
recognizing types of evidence and evaluating evidence, identifying assumptions in a text), the
majority of the skills they identified were low-order thinking processes (e.g., recognition,
identify). Furthermore, the argument task they examined in the study did not involve the
processes of analyzing the structure of an argument or the processes of identifying fallacies
within an argument. According to Harrell (2007), argument analysis must focus on: 1) the
logical structure of an argument; and 2) assessing the argument’s soundness (Harrell, 2007). The
logical structure of an argument shows how the conclusion is deduced from evidence and
premises, while the soundness of an argument deals with whether each claim is valid (Lim,
2011).
Argument Structure
Toulmin’s (1958) model of argumentation provides a graphical representation of the
fundamental structure of an argument (Figure 2.1). Toulmin’s model suggests that a good
analysis of an argument must take into consideration six important elements: data, claims,
warrants, qualifiers, rebuttals, and backing. Based on his book “The Uses of Argument” (1958),
data consists of facts and observations about a situation being discussed that leads to further
observations or facts, and ultimately, to the claim(s). Warrants connect claims and data by
19
making a rule of inference. A backing is a statement that supports the warrant. Qualifiers
(claims) provide a specific condition under which an argument holds true. A rebuttal, often
called a counter-argument or exception, is a statement indicating a situation where an argument
is not true.
Toulmin’s model of argumentation is a practical model that characterizes the nature of
arguments observed in everyday life (Taylor, 1971). However, Toulmin’s model has been
criticized for being useful only when there is a simple warrant in an argument. It is not practical
when used to analyze and break down more complex arguments that include multiple warrants
and that involve conflicts that emerge when applying particular rules of logic (Hitchcock &
Verheij, 2006). Although Toulmin’s model is used primarily to evaluate and determine the
veracity and accuracy of a single claim (Driver, Newton, & Osborne, 2000), researchers have
made attempts to extend and modify Toulmin’s model so that it can be used to analyze more
complex arguments. Despite its limitations, Toulmin’s model is the one that is most widely used
and is a model that is commonly used and embedded into the design of computerized argument
diagramming software (Hichcock & Verheij, 2006).
Figure 2.1. Toulmin's model of arguments (Toulmin, 1958, p. 104)
so
unless since
On account of
20
An argument consists in its most simple form of a conclusion and premise(s). A
conclusion is what the speaker/author argues. Premises are claims provided to support or oppose
the conclusion. The argument grows more complex when co-premises, objections, and rebuttals
are added to the argument. Co-premises are two or more premises that mutually support a given
conclusion or higher-order premise. An objection is a reason that challenges the veracity of
conclusions, premises and/or minor premises, whereas a rebuttal is an objection to an objection.
Figure 2.2 provides an illustration of a hierarchically structured argument that consists of the
elements described above (conclusion, premise and co-premises, objections, rebuttals).
Figure 2.2. An example of an argument structure (Rider & Thomason, 2008, p.18)
21
Problems in Real-World Arguments
In real-world settings, arguments are formulated, analyzed, and evaluated to resolve
social and scientific controversies and to generate solutions to complex problems (Taylor, 1996).
Contrary to formal arguments, which are well structured and complete, real-world problems and
arguments are ill-structured, incomplete, and complex (Jonassen, 1997). As a result, the
arguments presented to resolve conflicts and to solve such problems are also ill-structured and
complex. For example, the global warming argument is multifaceted in that it involves science,
economics, and health issues. Hence, one must consider all factors to make a good decision or
find a good solution. As arguments become more complex, understanding such arguments
becomes increasingly difficult.
Furthermore, real-world arguments are often presented in ways that make the arguments
difficult to analyze and evaluate. Sometimes, the conclusion in an argument is not stated
explicitly (Govier, 1987). In such cases, one must at times infer a speaker’s position and
conclusion based on the lines of reasoning and claims present within the argument (but only
when the claims and lines of reasoning are presented clearly). Likewise, the premises and/or
assumptions underlying the argument may be missing or poorly stated (Govier, 1987). In this
case, one must fill in the gaps by identifying the missing premises and assumptions, and if one is
unable to do this, an accurate evaluation of the argument may be very difficult if not impossible
to accomplish. However, the process of identifying missing premises (as well as the relationships
between multiple premises) can be facilitated by creating argument maps - visual diagrams that
are constructed during the process of mapping and flushing out the possible relationships
between stated premises and conclusions.
22
Approach to Argument Analysis
To facilitate the process of analyzing ill-structured, incomplete, and complex arguments,
Scriven (1976) prescribed a seven-step process (Table 2.1).
Table 2.1
The Seven Steps in Argument Analysis. Adapted from Scriven (1976, p. 39)
Step # Description of Each Step
Step 1. Clarification of Meaning (of the argument and of its components)
Step 2. Identification of Conclusions (stated and unstated)
Step 3. Portrayal of Structure
Step 4. Formulation of (Unstated) Assumptions (the ‘missing premises’)
Step 5. Criticism of
a. The Premises (given and ‘missing’)
b. The Inferences
Step 6. Introduction of Other Relevant Arguments
Step 7. Overall Evaluation of Argument in Light of Information Produced from steps 1
through 6.
The first step is to clarify the “meaning of the argument and of components” (1976, p.39).
In this step, students read arguments for comprehension, define terms when needed and identify
unstated premises, if any. The second step is to identify main and/or secondary conclusions.
After this step, students create the relational structure between premises and conclusions by
numbering each claim and linking premises and conclusions in a tree diagram (Figure 2.3). This
structure shows the hierarchical relationships between claims, and flows from top to bottom to
diagram the argument’s structure (with the main conclusion placed at the bottom of the diagram).
In this particular step, Scriven stressed the importance of finding ‘missing premises’ or ‘missing
assumptions’ to infer the relationships. He suggested the use of parentheses when there are
unstated premises or conclusions in the argument (Figure 2.3). The fourth step, which is the most
23
challenging step, is to formulate the unstated assumptions in the argument (Scriven, 1976). Once
this step is completed, students critique inferences and premises. In particular, Scriven (1976)
advised using counterexamples to criticize the soundness and reliability of inferences and
premises. In the sixth step, students reconceive the argument from an opposing view to find
different weights, directions, or conclusions among claims (Scriven, 1976). The seventh and last
step is to make a final decision on the quality of the argument based on the diagram produced
after completing steps 1 through 6.
Figure 2.3. An example of a tree diagram. Adapted from Scriven (1976, p. 42)
Note. 1 and 2 are claims (premises) that support 3 and 3 supports 4. However, 4 is an unstated conclusion
and as a result, it is placed in parentheses.
Argument Mapping to Enhance Argument Analysis
Argument mapping. Argument mapping is the process of “diagramming the structure
of argument, construed broadly to include any kind of argumentative activity such as reasoning,
inferences, debates, and cases” (van Gelder, 2013, p.1). As illustrated previously in Figure 2.2,
an argument map can display the structure of an argument using boxes, arrows, and colors to
reveal the relationships between premises, co-premises, objections, rebuttals, premises of
1
3
2
(4)
24
premises, and a major conclusion. Boxes contain a claim stated like a conclusion and the
premises. Lines and arrows represent the relationship between claims. The color of arrows or
boxes indicates whether a claim supports or opposes the upper premise or conclusion.
Visualizing an argument helps students enhance reasoning skills and critical thinking
(Harrell, 2007; Twardy, 2004; van Gelder, 2002a). Drawing an argument map can facilitate
argument analysis because of the following reasons: 1) the ease with which the boxes can be
visually scanned, moved, and positioned in relation to one another enables the student to
manipulate and explore the possible relationships between the boxes; 2) the cognitive load
placed on the learner while performing this complex process is reduced considerably by enabling
the learner to analyze abstract ideas using both visual and spatial representations (Harrell, 2007);
and as a result, 3) the student can allocate more cognitive resources to identify the relationships
between claims and premises, identify the structure of the argument, and assess the validity
and/or strength of the claims.
Argument-mapping software. To facilitate the process of mapping out arguments,
argument mapping software programs have been developed and tested to determine their effects
on students’ critical thinking/argument analysis skills. Reason!Able is a computerized argument-
mapping tool originally developed by van Gelder (2001) to help people understand informal
reasoning and identify the structure of an argument. The latest upgrade to the Reason!Able
software (van Gelder, 2002a) is the Rationale software application (van Gelder, 2007). Rationale
was designed purposely to support argument mapping. As a result, it provides a set of unique
functions to help people map out and identify the structure of a complex argument. For example,
it enables users to map out and delineate co-premises from multi-reason arguments. Co-premises
(see Figure 2.4) are premises that work together to support or oppose a contention as part of a
25
single reason. Unlike co-premises, a multi-reason argument (see Figure 2.3) has more than one
reason to support or oppose a contention, but its reasons are separate and independent of each
other. The Rationale software also provides mechanical rules such as the Rabbit rule and the
Holding Hands rule that assists users in finding missing premises and identifying premises in
relation to other premises or conclusions (Rider & Thomason, 2008; Twardy, 2004). The next
section – argument mapping conventions – provides a detailed description of these rules.
Figure 2.4. An example of a multi-reason argument. (www.austhink.com)
Van Gelder (2001) examined the effects of argument visualization on students’ reasoning
and critical thinking using the Reason!Able software. The study used the California Critical
Thinking Skills Test to measure critical thinking and compared CCTST scores between the
pretest and posttest following a 12-week period. As a result, van Gelder reported that students
who practiced argument visualization tasks using Reason!Able showed greater gain (.84 standard
deviation between the pretest and posttest) in the CCTST. In spite of the positive result in
CCTST scores, van Gelder did not use a control group in order to control other variables that
might affect the results (such as maturation, history, and test effects). In another study (Twardy,
2004), students who used the Reason!Able software to learn logic and to identify the structure of
26
arguments showed a significant improvement in their scores from the CCTST pretest and
posttest (effect size Cohen’s d =.7, indicating a large effect). Despite the positive effects of using
the Reason!Able software to foster critical thinking, neither study was conducted using a
controlled-experimental design. Furthermore, neither study provided explanations as to which
aspects of diagramming software and diagramming processes helped to enhance students’ critical
thinking and reasoning processes and improve the quality of students’ argument diagrams.
The mapping software used in this study is jMAP, developed by Jeong (2010, 2012) in
Microsoft™ Excel and designed specifically for drawing causal maps using nodes, links, and
arrows. Unlike the Rationale software, jMAP provides some unique features for sharing maps,
assessing maps, and mining the action sequences performed by students while constructing
causal maps. The mined data captured in jMAP can be used to model and identify mapping
processes used to produce more accurate argument diagrams (as is the purpose of this study).
First, jMAP allows users to download and upload maps and compare maps between students
and/or the instructor. Second, jMAP imports students’ maps, automatically grades them in
relation to the instructor’s map, and overlays the student map below the instructor’s map to
enable the instructor to see how and to what extent each student’s map matches the instructor’s
map. jMAP can also aggregate all student maps and superimpose the instructor’s map over the
group aggregate to evaluate overall group performance. Furthermore, jMAP chronologically
mines each student’s actions performed on the student’s map into an Excel spreadsheet, creating
data that can be used to identify sequential patterns in students’ behaviors by individual, group,
or experimental condition (Jeong, 2012). As a result, jMAP provides a means to identify unique
mapping processes that help students create low- vs. high-quality diagrams.
Argument mapping conventions and rules. With or without mapping software tools, the
process of mapping arguments often involves the use of certain mapping conventions to produce
27
argument maps that clearly communicate certain types of structures within an argument. Often,
two premises independently support an upper-level premise. This type of structure is illustrated
in Figure 2.4. In contrast, a co-premise consists of two premises that support the same upper-
level premise, but neither premise can support the upper-level premise without the other. This
mutual dependence between two premises is graphically represented using the convention
displayed in Figure 2.5. In contrast to the convention illustrated in Figure 2.5, Figure 2.6 shows
the argument map that is an incorrect representation of the relationship between two co-premises
and its conclusion.
To help students identify missing premises and assumptions in an argument, Rider and
Thomason (2008) introduced the Rabbit rule and the Holding Hands rule. For example, the
Rabbit rule (Figure 2.7) states that “every important term in the conclusion must appear at least
once (i.e., at least one premise) in each reason bearing on that conclusion” (Rider & Thomason,
2008, p.3).
Figure 2.5. Correctly represented co-premises (Twardy, 2004, p. 6)
28
Figure 2.6. Incorrect representation of co-premises (Twardy, 2004, p. 7)
As noted previously, missing premises are common in real-world arguments and
problems, and identifying them helps students understand arguments and assess the quality of
arguments. However, these types of mechanical rules do not guarantee the validity of claims
(Rider & Thomason, 2008) even though they can be helpful in prompting students to identify
missing premises and assumptions. Despite the usefulness of these rules when used to identify
missing premises, the application of these rules may not always be practical when analyzing
incomplete and complex arguments or evaluating the validity of arguments.
Figure 2.7. An example of the Rabbit Rule of argument mapping. (http://www.austhink.com)
29
Figure 2.8. An example of the Holding Hands Rule of argument mapping
(http://www.austhink.com)
Cautions in Using Argument-Mapping Software
At this time, the research on the efficacy of using argument-mapping tools (causal
mapping tools included) is very limited and inconclusive (Braak et al., 2006; Dwyer, Hogan, &
Stewart, 2010; Ruiz-Primo & Shavelson, 1996; Ruiz-Primo, 2004). There is no conclusive
evidence to show that argument-mapping software produces significant gains in learning and/or
develops students’ critical thinking skills. As mentioned before, using argument-mapping
software itself does not guarantee student learning and performance improvements (Rider &
Thomason, 2008). Argument mapping requires substantial cognitive effort that must draw on
students’ informal reasoning skills, understanding of the content knowledge and argument
mapping rules, and ability to apply the rules to construct an argument map. The main key to
using mapping tools to improve critical thinking skills is practice (Twardy, 2004; van Gelder,
2001). In the study conducted by Twardy (2004), students needed several weeks to learn how to
apply the Rabbit rule and Holding Hands rule to co-premises in multi-reason arguments.
30
Furthermore, Rider and Thomason (2008) suggest that critical thinking skills require repeated
practice with gradual increases in the complexity of arguments.
Despite the finding that learning the processes used to produce accurate argument
diagrams takes time and practice, no research at this time has been conducted to empirically test
and determine which map construction processes (and the reasoning processes that are reflected
in the map construction processes) produce more versus less accurate argument maps. Previous
studies on argument mapping have focused almost solely on assessing the quality of students’
final maps as a product of the mapping activity, and have tested the effects of argument-mapping
activities on students’ scores on traditional critical thinking tests (Braak et al., 2006).
Furthermore, the findings from these previous studies on the effectiveness of argument maps are
inconclusive due to flaws in experimental design, the measures used in the studies, and the small
numbers of empirical studies that have been conducted thus far. Without a deeper understanding
of underlying processes of reasoning used to construct an accurate argument map, it is difficult to
determine what aspects of a particular mapping program help or inhibit students’ ability to
produce high versus low-quality argument maps. Identifying and understanding the actual
processes students perform (which is the purpose of this study) while analyzing and mapping
arguments may help to explain when and why particular mapping programs work and do not
work.
Reasoning Processes
To explain reasoning processes used for argument diagramming, two theories from
psychology serve as the theoretical framework: The one-process theory and the dual-process
theory. The one-process theory presumes that, in general, one type of reasoning governs the
processes used to perform induction and deduction on reasoning (Johnson-Laird, 1983). In
31
contrast, the dual-process theory presumes that two different processes, system 1 and system 2,
underlie human reasoning processes (Evans & Over, 1996; Stanovich, 1999). In the dual-
processing theory, two different cognitive systems – called system 1 and system 2 – operate and
work competitively for control over the thinking process and outcomes (Evans, 2003; Strube,
1989). System 1, called a heuristic system, is associated with unconscious, implicit, belief-
based, less effortful, and automatic processes. In contrast, system 2, called an analytic system, is
associated with conscious, explicit, logical, effortful, and controlled processes. The differences
in the functional roles and characteristics of system 1 and 2 are summarized in Table 2.2.
Table 2.2
Characteristics of Two Cognitive Systems in Dual-Processing Theories. Adapted from Evans
(2008, p. 257)
System 1 (Heuristic) System 2 (Analytic)
Characteristics Unconscious
Implicit
Rapid
Automatic
Heuristic
Associative
Intuitive
Low effort
High capacity
Domain specific
Contextualized
Pragmatic/experience-based
Parallel
Conscious
Explicit
Slow
Controlled
Analytic/systematic
Rule-based
Reflective
High effort
Limited working memory capacity
Domain general
Abstract
Logical
Sequential
When people solve a reasoning problem, Evans (2006) presumes that system 1 processes
serve as the default reasoning process. Then system 2 processes modify the outcome produced
by system 1 processes (Thompson, 2010), but only under specific circumstances (e.g., when
32
there is sufficient time to revisit answers or to engage in metacognitive judgments, when given
certain deductive or inductive instructions). Also, content and context can play an important role
as to when system 1 is automatically activated because system 1 processes are largely reliant on
the retrieval of knowledge and experiences from memory that are relevant to the problem at hand
(Verschueren et al., 2005). Even though the two systems do not exactly correspond to induction
and deduction, the dual-process theories presume that inductive reasoning relies primarily on
system 1, which is fast, heuristic, and affected by context, whereas deductive reasoning is more
likely to rely on system 2, which is more deliberative, analytic and rule-based (Heit, 2007).
Although the dual-processing theory provides a useful framework to help us understand
the nature of some of the cognitive processes underlying reasoning, the constructs used to
differentiate system 1 from system 2 presented in Table 2.2 may not be mutually exclusive. In
addition, studies that have tested the dual-processing theories have produced either inconsistent
or inconclusive findings, which could largely be attributed to problems with the operational
definitions of each construct (Evans, 2008). Furthermore, studies have shown that one’s
heuristic response or analytic response can involve processes associated with either system 1 or
system 2 processes (De Neys, 2006). For example, an expert might appear to solve a logical task
using a rapid automatic process. However, this does not preclude the possibility that the expert is
putting forth high levels of conscious cognitive effort, given that the expert can also rely on his
or her prior experience and abundant practice to help complete the task rapidly. This suggests
that a particular outcome (e.g., rapid completion of task) is by no means a clear or certain
indication of the processes that are used to achieve that outcome. As a result, outcome measures
cannot and should not be used to determine what processes are being used (De Neys, 2006).
33
To address some of these issues, Evans (2008) suggests that the two systems should be
integrated into a single system by centering and tying these characteristics to the construct of
working memory and information processing processes. Working memory (WM) is considered
to be the executive controller of cognitive processes and, in particular, supports reasoning and
comprehension (Baddley & Hitch, 1974). Studies have found that people with high WM
capacity perform better than those with low WM capacities when reasoning tasks conflict with
their beliefs (Stanovich & West, 2000) and emotions (De Neys, 2006). These findings provide
some evidence (although at only a surface level) to support the hypotheses that the processes
used to analyze arguments can differ and can influence the quality and accuracy of one’s
argument analysis and understanding of complex arguments. The purpose of this study is to
break down the processes that people use when analyzing a complex argument into explicit,
more detailed, and discrete action sequences (and the cognitive operations associated with those
actions) to fully examine, model, and better understand the actual processes that support/inhibit
argument analysis.
Effects of Content and Context on Reasoning Processes
In addition to the effects of processes on argument analysis, psychologists have found
that content effects (Johnson-Laird, 2008), individual differences (i.e., intelligence; Evans, 2002;
Stanovich, 1999), and belief bias (Klauer, Musch, & Naumer, 2000) can influence human
reasoning process and outcomes (Evans, 2002). Although competence in reasoning skills is
believed to be independent of context or content (Evans, 2002), everyday reasoning in real life
involves “probabilistic, uncertain, approximate reasoning” (Heit, 2007). This type of reasoning is
inductive in nature, and hence is highly influenced by content and context. In fact, researchers
have found that people tend not to use deductive reasoning to solve every day reasoning
34
problems (Heit, 2007; Oasksford & Hahn, 2007) or to solve deductive reasoning problems
presented in controlled laboratory settings (Oaksford & Hahn, 2007). Instead, people generally
use inductive reasoning as a default reasoning process when constructing and evaluating
everyday life arguments (Heit, 2007; Oaskford & Hahn, 2007). At this time, however, most of
the research on reasoning has focused heavily on deductive reasoning (Evans, 2002; Schechter,
2013) and very limited research on inductive reasoning. As a result, this study will: 1) examine
inductive reasoning processes used by experts and novices to analyze claims that are more
probabilistic and uncertain in nature; and 2) measure participants’ prior knowledge of the content
in attempts to take differences in content knowledge into account when examining the argument
diagramming processes.
Reasoning in Arguments
Reasoning is the primary cognitive processes underlying the general processes used to
construct, analyze, and evaluate arguments (Shaw, 1996). Reasoning in general can be broadly
categorized as inductive and deductive reasoning. Defining induction and deduction depends on
the view that a researcher takes: the problem view or the process view (Heit, 2007). In the
problem view, “induction and deduction refer to particular types of reasoning problems” (Heit,
2007, p. 2). For example, if a reasoning problem goes from the general to the specific, the
problem is a deduction problem (Heit, 2007). On the other hand, if a reasoning problem goes
from the specific to the general, the problem is an induction problem (Heit, 2007). In Figure 2.9,
(a) and (b) are examples of inductive and deductive arguments, respectively.
(a) Dogs have hearts
All mammals have hearts.
(b) All mammals have hearts
Dogs have hearts
Figure 2.9. Examples of induction (a) and deduction (b) problems (Heit, 2007, p. 3)
35
From the process view of reasoning, “induction and deduction refer to psychological
processes” (Heit, 2007, p.2) and the question as to whether a problem is an induction or
deduction problem is irrelevant. Instead, the question of interest is which type of reasoning
processes (induction or deduction process) are used by people to arrive at a conclusion. In this
case, deductive reasoning is a psychological process that draws a valid conclusion based on the
given assumption that all premises are true (Johnson-Laird, 1999). In contrast, inductive
reasoning is a psychological process that infers a probability of conclusions to be plausible given
a set of premises, but this type of reasoning process does not guarantee the truth of its
conclusions even when all premises support or enable the conclusions (Schechter, 2013). In
summary, the goals of deductive and inductive reasoning are to arrive at an answer as to whether
or not a conclusion is true or false given the premises and to judge whether or not a conclusion is
strong/weak or plausible/implausible, respectively. The processes of inductive reasoning will be
examined in this particular study.
Informal Arguments in Everyday Life
Informal, “everyday life” arguments (such as those found in newspapers, advertisement
and academic journals) differ from formal arguments in terms of the structure of the arguments
and the rules used to evaluate the arguments. Shaw (1996) described two main characteristics of
informal arguments. First, informal arguments do not have an explicit formal argument structure
of premises and a conclusion. Instead, informal arguments include missing premises or unstated
assumptions and supporting and/or opposing premises. Second, informal arguments require
more inductive reasoning than deductive reasoning due to the fact that oftentimes formal logical
rules are not applicable to evaluating informal arguments. Informal arguments are ill-defined
36
and cannot be solved by deductive reasoning because the propositions cannot be posed as
discretely true or false. Instead, probabilistic judgment is more likely to be used to make
decisions of how likely it is for A to happen based on prior knowledge and experience. Since the
goal of informal arguments in real life is to make decisions, the criteria to evaluate informal
arguments are: how likely the premises and conclusion will be true, to what extent the inference
between premises and a conclusion is strong, to what extent the argument provides well-balanced
and relevant information for two opposing opinions of an issue (Shaw, 1996).
Common Reasoning Fallacies
When constructing and evaluating informal arguments, people tend to commit logical
errors, called informal reasoning fallacies. Table 2.3 presents a list of common informal
reasoning fallacies. Observing diagramming processes and examining think-aloud reports can
detect some of these informal reasoning fallacies. To date, no studies have examined the
reasoning processes that are used as people diagram arguments and reasoning fallacies they
produce in the process. Hence, this section will explain how informal reasoning fallacies can be
identified in terms of specific diagramming actions and action sequences.
Consider a hypothetical argument diagram displayed in Figure 2.10. If a student creates a
direct link between (DA) and (EA), such as in Figure 2.11, we can infer the student did not
recognize that the effects of D and E on A are mediated by B. As a result, the student made what
appears to be the error of ‘Hasty Generalization’ (Watson, 1999) or ‘jumping to a conclusion’.
According to Watson and Gordon (2009), hasty generalizations need to be treated as “an
underlying pattern of erroneous reasoning (p. 23)” associated with other fallacies such as arguing
with insufficient premises, argument from ignorance, and irrelevance.
37
Table 2.3
Common Informal Reasoning Fallacies. Adapted from Ricco (2007, p. 460)
Fallacy Definition Study
Anecdotal arguments Makes an inductive generalization on a
story or anecdote
Govier, 2010, p.277
Appeal to popularity Argues for a claim purely on the
grounds that other people or entities
(without any clear expertise in the
matter) accept it
Ricco, 2007, p.460
Argument from
ignorance
Maintains that since we do not have
evidence against some claim, the claim
must be true
Ricco, 2007, p.460
Biased Sample Argues with the sample which
misrepresents the population
Govier, 2010, p.275
False cause Argues that there is a correlation
between two things and then concludes,
on that basis, that cause and effect has
been shown
Ricco, 2007, p.460
Hasty Generalization
(Leaping to
conclusions)
Argues with the sample that is
inadequate to make an inference from
the sample to the population. Or,
making conclusions without a deep
consideration to reason throughout the
argument
Govier, 2010, p.276
Walton & Gordon,
2009
Irrelevance Attempts to support a claim by way of a
reason that is not relevant to the claim
Ricco, 2007, p.460
Begging the question
(Circular reasoning)
Assumes as a premise what it purports
to be proving. Seeks to support a
conclusion by appealing to that same
conclusion
Ricco, 2007, p.460
Single Cause
(Oversimplification)
Assumes that there is only one reason to
support the conclusion even though
multiple reasons work together to
support the conclusion
Slippery slope Claims that an innocent looking first
step will lead to bad consequences, but
does not provide reasons as to why or
how the one will lead to the other
Ricco, 2007, p.460
Wrong direction
(reverse causation)
The direction between cause and effect
is reversed
38
In another example illustrated in Figure 2.12, the two arrows drawn from A B and
from B A indicate that the student has committed the fallacy of “Circular Reasoning”. Other
fallacies that can be found in argument diagrams are illustrated in Figure 2.14, 2.15, and 2.16.
Given these examples, logical fallacies can potentially be identified by directly observing actions
that are performed on an argument diagram (with or without verbal reports). Identifying
informal reasoning fallacies, including the processes used to detect and resolve the fallacies, may
provide useful explanations and insights into the reasoning processes used by experts and
novices and how particular processes affect the quality of argument diagrams.
A
B C
D E
Figure 2.10. A hypothetical target argument diagram
39
A
C D E
B
Figure 2.11. Fallacy of hasty generalization (Leaping to conclusions)
Notes. The dotted red node B indicates that the student did not include the mediated node B in the
diagram.
A
B C
D
Figure 2.12. Fallacy of begging the question (Circular reasoning)
Notes. The dotted red box indicates the fallacy that a student makes in the diagram.
40
A
B C
D E
Figure 2.13. Fallacy of single cause
Notes. The red box indicates that the student did not include the node E as the cause of the node B in the diagram.
A
B C
D
E
Figure 2.14. Fallacy of irrelevance
Notes. The red dotted box indicates that the student added node E, which is irrelevant to node C.
41
Research on Argument Diagramming Processes
To study mapping processes, Jeong (2014) developed and used the jMAP software to
record the mechanical actions students perform while constructing causal maps (not argument
maps) and analyzed the data to identify patterns in action sequences. The behavioral patterns in
the mapping processes exhibited by students that produced high-quality maps were compared
with the patterns exhibited by students that produced low-quality maps in order to identify the
key processes that explain how students create more accurate maps.
Using the sequential analysis technique (Bakeman & Gottman, 1997), Jeong produced
two transitional state diagrams (see Figure 2.16) that convey the patterns observed in the action
sequences exhibited by high and low performers. A visual comparison of the state diagrams
reveals that high performers were more likely than lower performers to engage in two particular
action sequences – delete links add a new link and re-direct a link to point at a different node
add a new link – when drawing their causal maps. The findings suggest that these two action
A
D C
B
Figure 2.15. Fallacy of wrong direction (reversed causation)
Notes. The red dotted box indicates that the student misidentified the cause and effect thus inserted
the wrong direction between BD, instead DB (correct direction)
42
sequences are key processes used to create more accurate causal maps. Although the sample size
was small, Jeong’s study demonstrates a potential method for examining and modeling the
processes that help or hinder students’ ability to construct more accurate causal maps. As a
result, this study will use the same method to identify the processes students use when they
construct high versus poor-quality argument diagrams. To date, no research of this kind has
identified, modeled, and produced empirical evidence to support some of the prescribed
guidelines and processes for analyzing arguments with diagrams (Braak et al., 2006; Ruiz-Primo,
2004; Ruiz-Primo & Shavelson, 1996).
Figure 2.16. Transitional state diagrams of action sequences (Jeong, 2014, p. 247)
Note. Black and gray arrows identify probabilities that are and are not significantly greater than expected
based on z-score tests. Arrows are weighted in direct proportion to the observed transitional probability.
The first and second numerical value displayed in each node identifies the number of times the given
action was performed and the number of events that followed the given action. The size of the glow
emanating from each node conveys the number of times the given action was performed.
43
Expertise Research
Cognitive abilities and performances of experts have been studied across various
disciplines such as psychology, medicine, physics, education, sports, arts, and so on. In
particular, cognitive psychologists have tried to illuminate cognitive architectures of expertise
and to characterize experts’ cognitive abilities such as knowledge representation, memory bias,
and reasoning bias (Hoffman, 1996). These cognitive characteristics or components unique to
experts can be applied to the development of computational systems that model expert reasoning
(Hoffman, 1996: Farrington-Darby & Wilson, 2006) or to the development of instruction or
training that enables novices to acquire expertise (Crespo, Torres, & Recio, 2004).
Framing the research on expertise are two different views (absolute view and relative
view) – each with its own set of assumptions and definition of expertise. According to Chi
(2006), the absolute view in expertise focuses on outstanding or exceptional individuals in a
domain to identify their performances and underlying cognitive mechanisms of expertise. Since
the main focus is on the individual possessing an exceptional skill, the validity of these studies
relies heavily on the accurate identification of experts within any given domain. If an “expert”
examined in a study does not actually possess the characteristics that define expertise, the result
of a study is neither valid nor reliable. In contrast, the relative view focuses on making
comparisons between experts and novices to identify the unique characteristics of experts and to
measure relative levels of expertise between the two groups. The general goal of relative view
research is to use the findings to augment training or instruction for novices so they may attain
domain-expertise. Thus, uniformly defining experts and novices across studies may not be as
crucial to obtaining valid conclusions as in absolute view studies (Chi, 2006). In this study, a
44
relative view is taken because comparisons will be made between experts and novices on the
processes used to create argument diagrams.
Characteristics of Expertise
Though the assumptions of both viewpoints differ, research findings from both
viewpoints provide evidence to indicate that experts have cognitive characteristics that are
distinct from novices. Farrington-Darby and Wilson (2006) present the psychological
characteristics of experts: greater content knowledge and experiences within a domain, more
effective working memory and long-term memory, more detailed and elaborated mental structure
of knowledge, and more accurate and faster intuition and prediction than novices. Though most
studies show that experts perform better than novices, some expertise studies have produced
results to show that experts do not always perform better and sometimes perform worse than
novices. Chi (2006) points out that the majority of literature on expertise has tended to overlook
the characteristics that limit expertise. In addition to the positive characteristics of expertise, Chi
(2006) describes the negative characteristics that limit expertise (see Table 2.4). The most
salient negative characteristic of experts is in fact their high level of domain knowledge. Due to
their greater knowledge and mastery skills within their specialized field and their high level of
confidence in their own performance, experts tend to gloss over details, rely on contextual cues,
generate hypotheses in accordance with their domain expertise, and thus produce inaccurate
diagnoses, predictions, and judgments (Chi, 2006). Many of the characteristics listed in Table
2.4 may help explain some of the actions experts perform on their argument diagrams observed
in this study.
45
Definition of Expertise
The definition of ‘expert’ remains vague and differs across studies (Feldon, 2007;
Hoffman, 1996; Simon, 1980) and viewpoints (absolute or relative view). From the absolute
view, experts are defined in terms of individuals with quantitative evidence that exhibit
exceptional excellence in knowledge and skills within a domain (Chi, 2006). From the relative
view, experts are defined more liberally and are contextually operationalized because the relative
view focuses on ‘relative expertise’ in comparison to non-experts (novices) within a particular
situation/context/setting (Chi, 2006). Alternatively, Hoffman argues that experts and novices can
be defined along a developmental continuum of expertise. As shown in Table 2.5, Hoffman
(1996) provides the seven-proficiency scale of expertise in terms of cognitive developmental
levels of individuals. Based on the development continuum of expertise, Hoffman presents three
categories with which to define experts: cognitive developmental level, knowledge structure, and
reasoning processes (1996).
Experts can be defined as individuals who reach the highest cognitive developmental
level with “articulated, conceptual, principled understanding” and “accumulated skills based on
experience and practice” (Hoffman, 1996, p. 83). Experts’ performance and knowledge are
stable with consistent and deliberate practice and their intuitive judgment skills develop as they
accumulate experience (Hoffman, 1996). Secondly, experts can be defined in terms of
knowledge structure. Compared to novices, experts’ knowledge structures are more effective at
organizing and interconnecting information in meaningful ways and more effective at retrieving
relevant information based on situation or context (Hoffman 1996).
46
Table 2.4
Psychological Characteristics That Facilitate and Limit Expertise. Summarized from Chi
(2006).
Characteristics that facilitate expertise Characteristics that limit expertise
Generating the best solution
Experts excel in solving problems faster and
more accurately than non-experts
Domain-limited
Expertise is domain-limited. Experts do not
excel in recall for domain in which
they have no expertise.
Detection and recognition
Experts can detect features that novices cannot
and perceive the deep structure of a
problem
Overly confident
Experts can overestimate their abilities due
to over-confidence
Qualitative analyses
Experts spend a relatively great deal of time
analyzing a problem qualitatively
Glossing over
Experts recall fewer surfaces features and
overlook details than novices
Monitoring
Experts possess more accurate self-monitoring
skills to detect errors and judge their
own comprehension
Context-dependence within a domain
Experts rely on contextual cues and/or the
tacit when solving problems
Strategies
Experts are able to choose appropriate
strategies to solve problems
Inflexible
Experts have difficulties in adapting to
changes in problems or strategies
when solving problems
Opportunistic
Experts use resources if available
Cognitive efforts
Experts can retrieve relevant knowledge and
strategies with minimal cognitive effort
Inaccurate prediction, judgment, and advice
Experts are inaccurate when they predict the
novices’ performance
Bias and functional fixedness
Experts in medical fields tend to generate
hypotheses that corresponded to their
fields of expertise and thus can cause
bias
47
Table 2.5
The Proficiency Scales. Adopted from Hoffman (1996, p. 84-85)
Proficiency
scales
Description of scale
Naive One who is totally ignorant of a domain
Novice Someone who is new – a probationary member. There has been some minimal
exposure to the domain
Initiate A novice who has been through an initiation ceremony and has begun
introductory instruction
Apprentice One who is learning – a student undergoing a program of instruction beyond the
introductory level. Traditionally, the apprentice is immersed in the domain by
living with and assisting someone at a higher level. The length of an
apprenticeship depends on the domain, ranging from about one to 12 years in the
Craft Guilds.
Journeyman A person who can perform a day’s labor unsupervised, although working under
orders. An experienced and reliable worker, or one who has achieved a level of
competence. Despite high levels of motivation, it is possible to remain at this
proficiency level for life.
Expert The distinguished or brilliant journeyman, highly regarded by peers, whose
judgments are uncommonly accurate and reliable, whose performance shows
consummate skill and economy of effort, and who can deal effectively with
certain types of rare or “tough” cases. Also, an expert is one who has special
skills or knowledge derived from extensive experience with subdomains.
Master Traditionally, a master is any journeyman or expert who is also qualified to teach
those at a lower level. Traditionally, a master is one of an elite group of experts
whose judgments set the regulations, standards, or ideals. Also, a master can be
that expert who is regarded by the other experts as being “the” expert, or the
“real” expert, especially with regard to sub-domain knowledge.
Lastly, expertise can be defined in terms of reasoning processes. Research has shown
that experts use different reasoning processes than novices use when solving problems within
their domain of expertise. For example, Larkin (1983) examined how experienced physicists
(experts) and physics students (novices) solved mechanical problems and found that 1) experts
allocated more time to problem formulation than novices and 2) experts produced more detailed,
organized, and abstract representations of problems than novices. Given that the purpose of this
48
study is to examine the reasoning processes used by experts and novices to diagram the structural
relationships between conclusions and premises, the latter two categories, knowledge structure
and reasoning processes, are of particular relevance and interest in this study.
Expertise in Reasoning Process
Research shows that experts in various domains possess and use superior reasoning skills
that are distinct from those used by novices when solving problems within their domain of
expertise. For example, Schunn and Anderson (1999) examined how knowledge within the
domain of expertise affected scientific reasoning processes. They recruited three groups:
domain-experts in memory research, task-experts (researchers from another field in psychology),
and novices (undergraduates from various disciplines), and tested how each group designed an
experiment in cognitive psychology. The task involved domain specific knowledge of memory
research and general scientific reasoning skills to design an experiment - skills such as using
variables, interpreting data, and drawing conclusions. Schunn and Anderson (1999) found that a)
domain-experts outperformed task-experts and novices in domain-specific skills (selection of
useful variables, prediction of possible interaction between variables and violation rules), b)
domain and task-experts performed similarly in general reasoning skills (interpreting data,
drawing conclusions, and linking these conclusions to theories), and c) task-experts
outperformed novices in general reasoning skills but task-experts and novices performed poorly
in domain-specific skills. Schunn and Anderson’s (1999) findings suggest that experts possess
higher level of reasoning skills that are required for designing and analyzing research regardless
of their domain-specific knowledge. These findings suggest that expertise in reasoning
processes may exist independent of domain-specific knowledge and thus can be treated as a
domain-general skill. For this reason, this study assumes that the expert participants possess the
49
reasoning skills required to analyze a complex argument independent of their prior knowledge of
the domain/content embedded within the arguments they are asked to diagram.
What are the characteristics of experts’ reasoning processes? A study that examined the
types of reasoning processes involved in the medical field showed that experts tend to use
deductive reasoning processes when making a diagnostic decision based on presented evidence
or symptoms (Crespo, Torres, & Recio, 2004). Crespo et al. (2004) examined the reasoning
processes that characterized different developmental levels of expertise in dentistry. In their
study, three groups (expert, intermediate, and novice; Table 2.6) performed a dental diagnosis.
Crespo et al. (2004) found three different sequences of reasoning processes after analyzing
verbal protocol data: inductive, deductive, and a combination of backward and forward. In
particular, two out of five experts used deductive reasoning and three experts used a combination
of deductive and inductive reasoning to reach the diagnostic decision. In the intermediate and
novice groups, the deductive reasoning process was not observed during the task, and inductive
reasoning (used by 3 novices and 3 intermediate) and the combination of both types of reasoning
(used by 2 novices and 2 intermediate) were observed. Crespo et al. (2004) found the two
experts who used deductive reasoning provided a quick and accurate diagnosis compared to their
counterparts who used a combination of both types of reasoning. As a result, deduction, used by
experts, led to fast and accurate diagnosis while induction, used by novices and intermediates,
produced slower and inaccurate diagnosis. The differences in a sequence of reasoning process
affect the speed and accuracy of diagnostic decision-making. Based in part on these findings,
this study will determine to what extent experts in argumentation and critical thinking use
particular reasoning processes (e.g., top-down versus bottom up reasoning) in comparison to the
processes used by novices.
50
In addition to processes of deductive and inductive reasoning, Chi, Glaser and Rees
(1982) examined the reasoning processes behind the depth-first and breadth-first strategy used by
an expert-chess player while playing a game of chess. The reasoning processes that were
observed were best illustrated using a simple chess game tree shown in Figure 2.17.
Figure 2.17. A game tree (Chi et al., 1982, p. 13)
In the game tree, two searching strategies were observed: depth first search and width
first search. The study also found that expert chess players possess more complex chess game
trees in terms of their width and depth than novice players. In addition, Ericsson (2006) found a
similar result in that an expert chess player’s reasoning processes differ from those of novices
providing the chess trees (Figure 2.18).
51
Figure 2.18. Chess trees presented by a novice and expert. Adapted from Ericsson (2006, p.
234)
Like the chess playing processes, experts and novices in argumentation may use different
reasoning processes to construct an argument map using depth-first or breadth-first strategies.
With the depth-first strategy (Figure 2.19), people will start by finding the main conclusion and
the premise that supports the conclusion, then identify the next level premise to support the first-
level premise, and so on. In this strategy, people use convergent thinking to identify all possible
cause-effect relationships. On the other hand, in the breadth-first strategy (Figure 2.20), people
will start by finding the main conclusion and by identifying all possible premises that support the
main conclusion (such as divergent of thinking). Once they identify all first-level premises, then
they will move on to identify the next level (second-level) premises to support the first-level
premises.
Although the findings from the studies described above illustrate just some of the
possible types of reasoning processes used by experts across various domains, there is little
52
knowledge on how some of these, as well as other possible reasoning processes, are used by
experts and novices to analyze and map out complex arguments. As a result, the purpose of this
study is to observe experts and novices as they construct an argument diagram to determine if
some of the previously noted strategies are used more frequently among experts than novices.
Figure 2. 19. An example of the depth-first strategy in constructing an argument diagram.
Main
Conclusion
Premise 1a
Premise 2a
Premise 3a
Level 1
Level 2
Level 3
Depth-first
strategy in
constructing
an argument
diagram
53
Figure 2. 20. An example of the breadth-first strategy in constructing an argument diagram.
Main
Conclusion
Premise 1a Premise 2a Premise 3a Level 1
Level 2
Level 3
Breadth-first strategy in constructing an argument diagram
54
Table 2.6
Expert and Novice Comparisons on Reasoning Processes
Authors Domain/Definition Findings in Experts’ reasoning processes Findings in Novices’
reasoning processes
Larkin
(1983)
Mechanics problems solving processes
Experts (experienced physicist)
Novices (physics students)
Experts spend more time in the forming a conceptual
understanding of the problems
Generate conceptually rich and organized representations
Use ‘abstract’ representations that rely on deep knowledge
Gauge the difficulty of problems
Know the conditions for the use of particular knowledge and
procedures
Experts use ‘Thinking forward’ strategy
Tend to form ‘concrete’
‘superficial’ problem
representation
Novices use ‘thinking
backward’ strategy
Schunn &
Anderson
(1999)
Scientific reasoning processes
Domain Experts: memory researchers
Task Experts: non-memory researchers
Novices: Undergraduate students from
engineering, physical science, art
Domain-experts and task-experts performed better in domain
general skills than novices
Domain-experts performed better in domain specific skills
than task-experts and novices
Task-experts and novices performed poorly in domain
specific skills
Case,
Harrison
& Roskell
(2000)
Clinical reasoning processes
Experts: Experienced (senior) respiratory
physiotherapists
Novices: Inexperienced (junior)
physiotherapists
Generally logical and organized Logical approach not
always evident
Crespo,
Torres, &
Recio
(2004)
Dental clinical reasoning processes
Novices: junior students who had
demonstrated the mastery of foundation
knowledge by passing Dental Examination
Intermediate: recent graduates from the dental
school who were enrolled in a graduate
program
Experts: dentists who had clinically practiced
at least ten years as a general dentist.
Expert
Forward reasoning process
A combination of both
forward and backward
reasoning
Intermediate
Backward, a combination
of backward and forward
Novices
Backward, a combination of
backward and forward
55
Literature Review for Methodologies
Verbal Reports
Collecting and analyzing verbal data of one’s inner speech or thoughts during a task is
the most widely used method to identify and model cognitive processes. Verbal report methods
are based on the memory model of cognitive systems (Figure 2.21) which consists of five
cognitive processes: perception, retrieval, construction, storage, and verbalization and three
cognitive systems: sensory buffer, working memory and long term memory (Van Someren,
Barnard, & Sandberg, 1994). According to this cognitive model, “information that is active in
working memory is put into words. The output of this verbalization process is the spoken
protocol” (Van Someren et al., 1994. p. 20). The memory model assumes that verbalization is
one of the processes of working memory and provides a basic framework of verbal report
methods (Van Someren et al., 1994).
Several different verbal report methods exist in the
Figure 2.21. Memory model of cognitive systems and the five information processes (Adapted
from Van Someren et al., 1994, p. 19)
Sensory
Buffer
Working
Memory
Long-term
Memory
Protocols
Perception
Retrieval
Storage
Construction
Verbalization
56
literature: retrospection, introspection, questions and prompting, and concurrent think-aloud. For
retrospection, participants are asked after completing a task to explain verbally the sequence by
which they completed the task. This technique has been criticized due to tendencies for
participants to commit unintentional and/or intentional memory errors and to manipulate the
report of their thought processes to reflect what they later thought they should have done
(Ericsson, 2006; Van Someren et al., 1994). Alternatively, introspection allows participants to
choose the appropriate moment to report their thought processes during tasks. However, this
method is also vulnerable to memory errors or misinterpretation (Van Someren et al., 1994).
Questions and Prompting solicits specific information with targeted questions, such as ‘Why did
you do that?’ or ‘Why do you think that is the right way to solve the problem?’ However, this
method intrudes on participants’ thought processes and the questions themselves may influence
the thought process. Due to the limitations, disturbance of thought processes, memory errors,
and interpretations by participants, these three methods may be invalid measures for identifying
internal thought processes accurately.
Think-Aloud Protocol
Alternatively, a concurrent think-aloud technique asks participants to talk out loud and
verbalize their thoughts while engaging in a task. Ericsson and Simon, who developed Protocol
Analysis, explain that the action of speaking out loud thought processes out loud does not disturb
the actual thought processes (1993, 1998). Specifically, the content of working memory can be
verbalized without memory errors/loss when the verbalization of thoughts is done within 5 to 10
seconds right after the completion of a task (Ericsson, 2006). Ericsson further explains that the
validity of verbal data relies on time interval between the verbalization and its corresponding
thought and provides a model of concurrent think-aloud protocols (Figure 2.22). According to
57
Ericsson, a concurrent think-aloud protocol is a validated technique to capture accurate cognitive
processes without the three potential threats to the validity of measures (disturbance, memory
errors, interpretation by subject). Given these findings and given the limitations of using verbal
reports, think aloud protocol is used in this study to reveal the cognitive processes that are used
while constructing argument diagrams.
Figure 2.22. An illustration of concurrent think aloud protocol (Ericsson, 2006, p. 227)
Limitations of Concurrent Think-aloud Protocol
Think-aloud protocol has been shown to be a valid method for identifying cognitive
processes (Ericsson, 2006). However, one of its limitations is that the quality of the protocols can
be hindered by the participants’ language abilities and by the nature of the tasks (Van Someren et
al., 1994). Van Someren et al. explains that a think-aloud protocol may not fully capture the
cognitive processes if a participant is not able to report cognitive processes due to limitations in
verbal fluency and/or mental maturation (e.g., age). These factors can produce verbal reports
that: a) do not match or correspond to the cognitive processes that are actually being performed;
and b) do not accurately capture the cognitive processes that involve non-verbal information
58
(Van Someren et al., 1994). As a result, the participants in this study are selected according to
their cognitive maturation and language ability. Furthermore, the task in this study involves the
use and analysis of verbal rather than non-verbal information.
59
CHAPTER 3
METHOD
Overview
The purpose of this study is to explore the reasoning processes performed by experts and
novices while analyzing and diagramming a complex argument. Analyzing, and most of all,
identifying differences in the diagramming processes used by experts versus novices can help to
explain which and how particular reasoning processes are used to achieve a better understanding
of a complex argument. However, observing the diagramming processes themselves may not
fully reveal the actual reasoning processes that participants use while analyzing arguments. To
fully understand the reasoning processes, participants’ cognitive processes need to be identified
along with their diagramming processes. For this reason, participants’ cognitive processes
(called thought processes) were captured with the think-aloud protocol and a retrospective
interview. Using a qualitative approach, video recordings of the verbal protocol and interview
sessions were examined to develop a coding scheme that identifies the specific actions
performed during the process of diagramming a complex argument. The coding scheme was then
developed to code the actions of both experts and novices. The codes from each group were
sequentially analyzed to identify behavioral patterns that may reveal specific reasoning processes
used or not used by experts and by novices. As a result, this chapter describes in detail how the
research design, participants, tasks, materials, procedures, and data analysis addressed the
research questions.
60
Research Design and Approach
This study utilized a mixed method design. First, a qualitative method was used to
collect data on reasoning operations (cognitive processes) and reasoning behaviors (diagramming
processes). I used a general coding development strategy to generate coding schemes and
categories that explained unknown phenomena and diagramming processes. Sequential Analysis
was used to identify the sequential patterns demonstrated in a) diagramming processes performed
by experts and novices and b) cognitive processes used by two groups. To capture diagramming
and cognitive processes, I used jMAP software to capture experts and novices diagramming
behaviors and used in conjunction with a think-aloud protocol to capture their cognitive
processes to explain the underlying reasons behind their diagramming behaviors. Lastly,
qualitative analysis was used to help determine to what extent the action sequences identified
with sequential analysis can serve as indicators of global level reasoning processes. As a result, I
used qualitative findings to help evaluate the value of using sequential analysis as a method for
assessing and diagnosing students’ reasoning processes. At this time, research on diagramming
arguments is incomplete in the three aspects this study addressed: the precise processes that lead
to high quality argument diagrams; the processes that lead to low quality argument diagrams; and
the differences in experts’ and novices’ reasoning processes.
Participants
A total of ten participants were drawn from across departments at a large university
located at southeast in the USA: five experts in argumentation as the expert group and five
graduate students as the novice group. Nielson (1994) recommended that 4 ±1 sample sizes are
enough for studies employing think-aloud protocol. As for the sampling strategy, a purposeful
and convenience sampling will be used to recruit the experts and novices. The criteria to be
61
included in the expert group are 1) be an instructor in argumentation and/or 2) have formal
training in argumentation. The criteria for the novice group are 1) be a graduate student and 2)
do not have formal training in argumentation. I recruited the participants via an email and via
graduate level class visits. I completed the recruitment of participants and data collection
between May and September 2014.
Settings and Technology
For this study, the main lab was set up in the Education building at FSU with various
devices. In general, participants visited the lab to participate in the study. But in some cases, I
visited participants’ offices and installed the same devices that I used in the lab to conduct the
study. Each study was conducted in a one-on-one session. Diagram construction was performed
on a laptop with a mouse and keyboard. jMAP software was used by each participant to
construct an argument map and O-Cam recorded the computer screen and verbal protocol as
participants constructed their argument diagrams. The current time was displayed at the bottom
of the computer screen. A voice recorder with a microphone was used to capture participants’
voices.
Procedures
Introductory Session
Once the participants visited the lab for the study, the researcher explained the general
purpose of the study, the activity that participants should perform, and the data collection and
devices. The participants were asked to sign the consent form if they agreed to participate in the
study and to the use of the data. Then, participants completed a survey created by the researcher
to record participants’ demographic information and experience level – data the researcher used
to help classify participants as experts or novices (Appendix A). During this session, the
62
researcher also emphasized that pseudonyms would be used to protect participants’ privacy;
participants could withdraw from the study at any time without consequence; and, at the end of
the study, participants would receive a $25 gift card in appreciation of their time and effort.
Training Session
For training purposes, each participant was instructed by the researcher on how to use
jMAP to construct an argument map. The jMAP software is an Excel-based software tool that
enables students to draw argument diagrams by using and positioning nodes and arrows. The
jMAP software was selected over other argument diagramming software because jMAP
automatically records each action a student performs while constructing the argument diagram.
Most of all, jMAP records the time at which each action is performed and records it in a human
readable format in an Excel spreadsheet. In the training, the researcher provided a 2-minute
rudimentary video demonstration on how to use jMAP to move and re-position a node, how to
insert a directional arrow to link one node to another node, how to delete an arrow, how to detach
one end of an arrow from one node and re-attach it to a different node, and how to change the
color of an arrow from black (supporting premise) to red (opposing premise), or vice versa. The
video did not provide information on how to analyze an argument during the diagramming
process. To ensure that all participants were familiar with the basic mechanics of using jMAP,
the participants completed a mini argument map that consisted of five nodes (claims about the
importance of critical thinking in college students) as an exercise in jMAP to replicate the basic
diagramming tasks demonstrated by the training video and the researcher. While creating the
exercise map, the researcher asked the participant to talk aloud to share what she or he was
thinking and/or looking at, to familiarize the participants with the talk-aloud protocol.
63
Tasks
The participant’s task was to construct an argument diagram using jMAP. Fifteen
propositions from the article ‘Six principles for effective e-learning: What works and why’
(Clark, 2002) were placed in advance into fifteen nodes in jMAP. The participants analyzed the
claims presented in the fifteen nodes and constructed an argument diagram that showed the
argument structure between its various claims (i.e., the main conclusion and the premises). To
help the participants’ content understanding of the claims, the researcher provided a 6-page
printed copy of a summary of the six e-learning principles (Appendix B). The participants were
asked to read the summary before constructing an argument map and were advised not to use the
summary while constructing a diagram unless they did not understand the content of the nodes.
This constraint minimized the disruption of the participants’ reasoning processes. Figure 3.1
displays the default screen where students are presented with the jMAP task. Participants were
neither able to add/delete nor change the content inside each node.
Figure 3.1. The screen capture of a student jMAP with a default arrangement of nodes
64
In addition, participants were instructed to insert arrows to identify and convey the
logical connections between the premises and the claim. Red arrows were to be used to represent
a premise with an opposing relationship to the parent premise or claim, and black arrows were to
be used to represent a premise with a supporting relationship.
The participants talked aloud to verbally report what they were thinking while
constructing the argument diagram. The verbal reports were recorded along with the video
recording of the actions participants performed in jMAP while constructing the diagram. During
the session, if the researcher saw that the participant did not talk following a period of 5 or more
seconds, the researcher gave a prompt for the participant to report his or her current thoughts by
reminding the participant to ‘keep talking’ in order to facilitate the verbal reporting of thought
processes used while constructing the diagrams. During the argument diagramming task, the
researcher did not provide any hints or answer any questions on how to map out the relationships
between the premises and main claim. However, if a participant had difficulties using the jMAP
software, the researcher provided technical guidance.
Retrospective Interview Session
Following the completion of the argument mapping session, the researcher asked an
open-ended question about the diagramming processes and experiences in general. The
participants shared details about their diagramming strategies, processes, and any difficulties that
they experienced during the task. After the open-ended question, the participants were asked a
total of seven structured questions (Appendix C) to acquire more details about the processes
participants used to construct their argument diagrams. The participants looked at and referred
to their final diagram to help them answer the questions and recalled some of the processes they
65
used to construct the diagram. See notes on Figure 3.2 that provide a visual overview and
summary of all the tasks to be completed by each participant.
Figure 3.2. The overview of the procedure and data collection process
1. Introduction Session (10 minutes)
Inform the study
Fill in the ‘Profile’ and the consent form
2. Training Session (10 minutes)
Watch the tutorial and practice jMAP
software with feedback
Practice think-aloud protocol with prompt
3. Device Setting (5 minutes)
Check the video camera, voice recorder, and
screen-capture software
4. Task Session (40 min.)
Participants Researcher
Read the summary
Construct an
argument diagram
Talk aloud their
thinking process
Observe the
participant’s
mapping behavior
Prompt ‘talk
aloud’
5. Retrospective Interview Session (30 min.)
Start with the open-ended question
Interview using seven structured questions
z z
Participant’s profile
Consent forms
Screen capture software
used to record:
Actions performed on
argument diagram
Verbal reports from
talking aloud and from
responses to researcher
prompts
Record the interview
session
66
Data Collection
Data were collected using four sources: final argument maps, argument diagramming
behaviors captured and logged within the jMAP software, video recordings of the participants’
verbal reports captured while creating the argument diagrams, and a retrospective interview.
While students created their diagrams, jMAP captured and recorded, in real-time, the actions
performed on their argument diagrams (see Table 3.1). Each action was recorded into an MS-
Excel spreadsheet in chronological order along with its time and day of occurrence and event
sequence number starting from 1 to X. The spreadsheet containing the diagramming behaviors
for each participant then was used to record the transcribed verbal protocols at their time of
occurrence along with the recorded diagramming actions. Verbal actions and diagramming
actions that occurred at the same time were assigned the same sequence number. Once all verbal
protocols were entered into the spreadsheets, the spreadsheets were used to analyze, classify and
store the coded verbal protocols. After each participant completed an argument diagram, the final
argument maps were imported to the jMAP software (Instructor version) to assess the quality of
maps to the criterion map shown in Figure 3.3. Lastly, retrospective interview data were
transcribed.
Assessing the Quality of Argument Diagrams
Since this study focused on examining the reasoning processes used by experts and novices
as they analyze and diagram a complex argument, assessing the quality of argument diagrams as
a learning outcome was not the primary interest of this study. Rather, this measure played a role
as a secondary measure to describe, and most of all, validate observed differences in the mapping
processes used by selected experts versus novices that help to explain observed differences in the
quality of the argument diagrams. Although I assumed that expert-participants would produce
67
Table 3.1
Codes Assigned to Each Action Students Perform in jMAP Software
Final
codes Code Definition
LINK ADDR added new link pointing to the right
ADDL added new link pointing to the left
ADDU added new link pointing up
ADDD added new link pointing down
LK2 attached link to the affected node
RELINK RLK1 redirected the existing link to a new causal node
RLK2 redirected the existing link to a new affected node
- ULK1 detached the beginning tail of the link
- ULK2 detached the end of the link
ATTR ATT- changed link to color red to convey a negative or inverse
relationship
ATT+ changed link to the color black to convey a positive relationship
ATT2L changed link to low level of impact (not used in this study)
ATT2M changed link to moderate level of impact (not used in this study)
ATT2H changed link to high level of impact (not used in this study)
DEL DEL deleted the link
MOVE MS moved a node (which was the same node as the last moved node)
MDn moved node to the north of the previously moved node
MDne moved node to the NE of the previously moved node
MDe moved node to the East of the previously moved node
MDse moved node to the SE of the previously moved node
MDse moved node to the South of the previously moved node
MDsw moved node to the SW of the previously moved node
MDw moved node to the West of the previously moved node
MDnw moved node to the NW of the previously moved node
COMM COM added comment to link to explain how node influences affected
node
CREV revised the existing comment on the given link
68
high-quality argument maps, their superior performance was not always guaranteed. Thus, this
secondary measure informed us as to whether or not the selected experts performed better than
the novices in terms of the quality of argument diagrams. If the results showed the expert group
scored higher than the novice group on the quality of argument diagrams, the experts’ diagrams
were categorized as high quality maps while the novices’ diagrams would be categorized as low
quality maps. If the results showed no overall differences in quality between the experts and the
novices’ diagrams, this study: a) would have rank order the performance of all ten participants
from highest to lowest performer; and b) identified and compared the reasoning processes used
by the two lowest versus the two highest performers.
Each individual argument diagram was imported into jMAP to compare and score the
diagrams across three criteria. The jMAP software identifies, counts, and visually presents
missing links with gray arrows and matching links with dark green arrows when comparing a
participant map to a criterion map (see Figure 3.3) collaboratively constructed by a subject
matter expert in the ISLT program and me. The percentage of links in the participants’ maps that
are also in the criterion map determines the measure/level of accuracy in participants’ maps. As a
result, the first criterion used to measure accuracy is based on a percentage score so that
participants that indiscriminately insert links between every possible pair of nodes score lower
than participants that insert the links through genuine thought and deliberation. For the second
criteria, accuracy is measured in terms of the number of nodes that are correctly identified and
positioned at the bottom of the diagram as a root cause (a node with no arrows pointing into the
node, and with only arrows that point out from the node).
70
The third criterion is the number of chained links that directly stem out from each
correctly identified root cause (and only those that are correctly identified) up to the main
conclusion. The cumulative score across all three criteria are used to determine the overall
quality of each participant’s argument diagram. For example, if it is the case that
ABCDCONCLUSION, a diagram that contains A B C F CONCLUSION
receives 2 points and the diagram that contains A B C D F CONCLUSION would
receive 3 points, while the diagram that contains A C D CONCLUSION receives 0
points. This third criteria is perhaps the most important measure because it assesses how well the
participant is able to articulate the causal mechanism that explains how and why a particular root
cause (or multimedia principle) ultimately affects the learning outcome.
Coding the Video and Audio Data
Three different sources of data - diagramming behaviors (video), think-aloud data, and
interviews - were be transcribed and coded. jMAP was used to automatically log various types
of actions performed by participants while constructing a diagram. However, the validity of the
actions captured by jMAP has not yet been empirically tested. As a result, both the diagramming
actions and verbal reports captured in the video recordings were manually transcribed and then
coded to help ensure that the diagramming behaviors (and associated reasoning processes) that
were not recorded or recognized by jMAP were included in the data analysis. Through an
iterative and close examination of the video and audio recordings, initial coding schemes were
developed to identify and categorize the diagramming actions performed by each participant and
reasoning processes used by participants. After creating the initial coding scheme (creating a list
of defining codes for behaviors and verbal reports) for the video and audio data, a second coder
coded the same data in order to establish reliability of the data coding. The second coder was
71
trained by coding one expert and one novice’s video data to identify verbal utterances observed
in the video recordings. Although some researchers do not value the calculation of a numerical
inter-rater reliability on coding within qualitative research frameworks (Guba &Lincoln, 1994;
Madill, Jordan, & Shirley, 2000), generating an acceptable inter-rater reliability using Cohen’s
Kappa between two coders may contribute to minimizing possible errors and biases when
interpreting actions or texts in the given data.
After the second coder completed the coding task, I compared both coding results and
calculated the inter-rater reliability using Cohen’s Kappa coefficient (Cohen, 1960). According
to Cohen (1960), Kappa is a useful index of the agreement between two coders when: 1) the
units are independent; 2) the data are categorical/nominal; and 3) judges conduct their ratings
independently. In this study, the units of the video data set (diagramming behaviors) are nominal
scales and independent. Also, two coders conducted the coding process independently. In
practice, Kappa values from .41 to .60 indicate ‘moderate agreement,’ and values from .60 to .80
are substantial (Landis & Koch, 1977). While the reliability was less than .41, I discussed the
discrepancies in the coding results with the second coder and we came to a consensus by re-
coding the data that showed discrepancies until we reached an acceptable value (ranged from .41
to .60) of inter-rater reliability of the coding results. Commonalities and differences noted
between the codes determined the final categories and number of categories presented in the final
coding scheme.
Data Analysis
The coded data were sequentially analyzed (Bakeman & Gottman, 1997) within each
group (experts vs. novices) to identify overall patterns in the sequences of actions performed by
each group using the Discussion Analysis Tool or DAT (Jeong, 2012). First, all codes were
72
entered into the DAT software in chronological order for a given group (expert or novice). Then,
the DAT software created a frequency matrix (Figure 3.4), transitional probability matrix (Figure
3.7), and a transitional diagram that provides a visual means of identifying any sequential
patterns in the actions performed by the given group.
Figure 3.4. An example of a transitional frequency matrix
Format data for analysis. The coded data of diagramming behaviors and verbal reports
were separately copied into column 1 of an Excel worksheet. To differentiate a set of action
codes from one another, the number 1 was entered into the second column directly adjacent to
the first action performed and recorded in each participant’s data log. Next, all data from the
experts and novices were separately collated into a single column in Excel. The codes were then
collapsed into the eight major code categories (Read Claim, Identify Overall Association,
Position Node, Identify Cause-Effect Association, Make Connection, Provide Reason, Review
Nodes, Delete Link) so that overall patterns in the processes could better observed and identified
within each group.
Sequential analysis. The sequential data for one given group was copied from the Excel
worksheet and pasted into the Discussion Analysis Tool (DAT) software. The DAT program
73
was used to produce a frequency matrix (Figure 3.4) that displays how often a particular two-
event sequence (A A, A B, etc.) was observed in the given data set.
These transitional frequencies were used to generate a transitional probability matrix
(Figure 3.5). This matrix shows how likely one action is to follow another given action. The
green and red numbers in the cells identify which of the transitional probabilities are found to be
significantly higher or lower than the expected frequency, respectively, based on z-score tests at
alpha value < .05. The z statistic tests which transitional probabilities significantly differ from
their expected values. Each z-score is displayed in the z-score matrix (Figure 3.6) with green and
red colored z-scores representing values that are above and below the critical z-score value of
±1.96 at the alpha .05 level. Essentially, the z-scores are used to operationally define which
action sequences are determined to be sequential “patterns” in the processes participants use
while constructing their argument diagrams.
Figure 3.5. An example of a transitional probability matrix
74
Figure 3.6. An example of z-scores matrix
Assumptions for Sequential Analysis
Since the sequential analysis is based on the z-statistic, the assumptions for z-test should
be considered. In the study context, the z-test is used to identify which transitional probabilities
are different from the expected value. Each transitional probability for two-event sequences is
assumed to be the same (expected transitional probability will be the number of all possible pairs
divided by 100). Random sampling from the defined population is the first assumption. This
assumption is violated due to the nature of this study that involved convenience sampling within
targeted populations. The second assumption is the independence of the data. Sequential
analysis is a technique to identify possible dependencies among data patterns. However, the
series of diagramming behaviors are assumed to be independent of each other. Finally, the z-test
assumes the normality of data in the population. But, this normality assumption is robust to the
violation of the data set. Moreover, the z-test requires large samples and Bakeman and Gottman
(1997) suggest that each cell of marginal sum should be equal to or greater than 5 to use for data
analysis.
75
Identify action sequence patterns. To provide a clearer and Gestalt view of the patterns
identified in the transitional probability matrices for each group, the DAT software generated a
transitional state diagram using data from the expert group and a second state diagram using data
from the novice group. For example, the transitional state diagrams (Figures 3.7, 3.8) provide a
visual representation of the observed patterns. To determine whether there are differences
between the processes used by the experts versus the novices, the two state diagrams are placed
side by side for comparison. Of particular interest are the patterns (transitional probabilities
found to be significantly higher than the expected probability based on z-score tests with p < .05)
observed among the processes used by experts but not observed among novices. For example,
we can observe a total of three patterns, AB, BC, and CA in both Groups A and B
(Figures 3.7, 3.8). The pattern that is unique to Group A is AB while the pattern that is unique
to Group B is CA. The unique processes reveal what may be the desired target processes that
explain what it is that experts do to create more accurate argument diagrams. The undesirable
processes are also determined by identifying the patterns observed only in the state diagram for
the novices, but not observed among the experts. In summary, the following process was used to
compare the transitional state diagrams to reveal processes that could potentially help and/or
inhibit students’ ability to create more accurate argument diagrams:
1) Identify total number of patterns observed between the expert and novice groups.
2) Identify how many and which of these patterns are unique to the expert group.
3) Identify how many and which of these patterns are unique to the novice group.
4) Interpret the unique sets of patterns to discern any larger global that might reveal the key
processes used to create more accurate argument diagrams.
76
Figure 3.7. A transitional diagram of Group A
Notes* Black and gray colored arrows denote transitional probabilities that are and are not
significantly higher than expected, respectively, based on z-scores at p < .05
Post-hoc analysis: Identify action sequence patterns between groups. To determine if
the differences in the observed patterns (two-event sequences) between the expert and novice
groups are statistically significant, a phi-coefficient (f ) or Yule’s Q is used to measure the
strength of association between the group membership and a particular pattern (Bakeman,
Mcarthur, & Quera, 1996; Bakeman & Gottman, 1997; McComas et al., 2009). According to
Bakeman and Gottman (1997), a z-test or transitional probabilities should not be used to test for
significant differences between groups because: 1) the number of total tallies in the cells is not
the same across groups, and 2) the larger sample size of any one group inflates and produces a
Figure 3.8. A transitional diagram of Group B
77
larger z-score. Instead, the phi-coefficient can be computed using a 2 by 2 dimensional
contingency table (Table 3.2). For example, suppose a study is interested in determining
whether or not a particular two-event sequence (AB, A is a given event and B is the target
event in one group) is associated with a group membership (expert group and novice group). To
compute the phi-coefficients for each group’s two-event sequence (AB), the simple 2 by 2
dimensional table of the Expert Group can be represented as
Table 3.2
A Generic 2 by 2 Contingency Table
B ~B Total
A a b a + b
~A c d c + d
Total a + c b + d N
where A is a given behavior (lag 0), ~A represents not A, B is a target behavior (lag 1), ~ B
represents not B. The chi-squared ( c 2) statistic compares which cells (observed frequencies) are
significantly different from the expected frequencies. The chi-squared statistic tests whether two
variables, A behavior in lag 0 and B behavior in lag1, are independent and thus show no
sequential patterns in the sample. The c 2can be computed as (1)
c 2 =Oij -Eij( )
2
Eijj=1
c
åi=1
r
å (1)
where
Oij= the observed number of cases for a cell in the ith row and jth column
Eij = the expected number of cases for the same cell if the null hypothesis were true
åå = a double summation of the fraction across all rows (r) and columns (c)
78
To test the hypothesis of independence of two variables, the computed c 2 value and the
critical c 2value are compared. According to the chi-squared distribution table, the critical c 2
value for 2 by 2 is 3.84 at a=.05 with df (1). If the computed c 2value is greater than the
critical value 3.84, the hypothesis of independence of two variables is rejected and it can be
concluded that the two variables are dependent.
Although Table 3.2 is helpful to test whether or not AB pattern is sequential
(dependent) for each group, it does not show the magnitude of association of the sequential
pattern and group membership. This study uses Table 3.3 to compare the association of two
variables (AB sequential pattern and group membership) for the post-hoc analysis.
Table 3.3
An Example of a 2 by 2 Contingency Table for Post-hoc Analysis
Variable X
Variable Y
Expert 0
Novice 1
Total
A-B
pattern
Yes (1) a b a + b
No (0) c d c + d
Total a + c b + d N
The phi coefficient is computed using the following formula:
f =(ad -bc)
(a+b)(c+d)(a+ c)(b+ d)= c 2 / N
79
Based on the definition, we can have the phi coefficient for the Expert group of AB sequence.
Likewise, the phi coefficient for the Novice group of the AB sequence can be computed. The
range of the phi-coefficient is -1 to 1 with zero indicating no association. The result can be
interpreted using the rule of thumb of a phi coefficient: above .90 indicates an extremely strong
relationship, .70 to .89 indicates a strong relationship, .50 to .69 a moderate relationship, .30 to
.49 a low relationship, and below .30 a weak relationship (Pett, 1997). By comparing the two phi
coefficients, we can report and compare the magnitudes of association between the sequential
action and group membership. If we get the phi close to 1, Group 1 is strongly related to the A-B
pattern.
The null hypothesis is “the sequential behavior AB is not significantly associated with
the group (experts and novices).” To test the null hypothesis using the phi coefficient, two
variables XY should be dichotomous and observations should be independent and representing
frequencies (Pett, 1997). The data in this study may violate the independency assumptions of
observations. According to Bakeman and Gottman (1997), the violation of independency of
observations merely affects the result and the phi-coefficient can be used in the sequential
analysis to measure the strength of association between two variables.
Post-hoc analysis: Identify action sequences patterns between experts and between
novices. To examine the similarities and differences among experts and among novices, I
presented five experts’ state diagrams and five novices’ state diagrams side by side. Then, I
compared the experts’ state diagrams to determine to what extend experts use the same processes
using the same statistical analysis and tests described above. Also, I compared the novices’ state
diagrams to determine to what extent novices use the same processes.
80
Post-hoc analysis of retrospective interview data. Retrospective interview data was
analyzed to provide detailed explanations of specific sequential patterns used by individuals and
groups. The use of the interview helped the researcher to further identify and elaborate on
reasoning processes revealed in the sequential analysis and to achieve a deeper understanding of
how experts and novices analyze complex arguments. For each individual, I classified the
strategies that were used and any difficulties or problems that were experienced. Next, I
examined how the patterns identified in sequential analysis correspond to and are a product of
the strategies and difficulties reported in the retrospective interview.
Limitation of Sequential Analysis
As the participant’s time on task and number of actions performed increases in number,
the frequencies in the frequency matrices increase in number and so, too, will the corresponding
z-score. As a result, each participant’s time on task must be taken into consideration. Because it
is not possible to control the amount of time each participant needs to complete an argument
diagram, the alternative is to examine a fixed number of actions that the participants perform on
their argument diagrams from the beginning of the diagramming task. This fixed number can be
set to the least number of actions among the six participants used to complete the diagramming
task.
A second limitation of using sequential analysis is that the minimum number of tallies in
the frequency tables should be equal to or greater than five (Bakeman & Gottman, 1997). Thus,
the cells that possess values less than five should be treated as ‘missing data’ and excluded from
the data analysis.
81
Scoring Student’s Argument Diagrams
To assess each participant’s argument map, an associate professor in ISLT at FSU and I
collaboratively constructed the criterion map. Because the article that presented the arguments
(and was read by each participant in this study) included the domains of cognitive psychology
and multimedia learning design, an associate professor in cognitive psychology at FSU reviewed
the criterion map and pointed out the correlation between selective attention and cognitive load.
The new links that were added based on the cognitive psychologist’s feedback were found to be
in 100% agreement with the links produced in all the expert maps. This process was conducted
in order to help increase confidence in the validity of the criterion map (Figure 3.9).
To assess the accuracy of the argument maps, a sum total score across six criteria was
used to score each student’s argument map in relation to the criterion map (Figure 3.9). One
point was awarded when the student correctly identified the final conclusion among 15 claims.
To measure depth of understanding and the ability to identify the logical links from the lowest
premise all the way up through the mid-level premises and to the final conclusion, students
received one point if they correctly identified the correct link from the lowest level premise to a
mid-level premise. For each correctly identified link between the lowest level premise to a higher
level premise (AB), additional points were awarded if the students correctly identified the
correct link to the second-order premise (ABC), and for the third-order premise
(ABCD), and fourth-order premise (ABCDConclusion). No points were
awarded for any of the higher-order links unless all the previous lower-order links were correctly
identified.
83
Because the criterion map contained five hierarchical levels from the bottom premises to the
final conclusion, up to four points could be awarded for correctly identifying a four-level chain
of premises that linked the lowest-level premise all the way to the final conclusion. Finally, the
sixth criteria deducted a half point for each link with an incorrect direction and incorrect valence.
84
CHAPTER 4
RESULTS
Introduction
The purpose of this study was to explore the reasoning processes used by experts and
novices to analyze a complex argument in order to identify the reasoning processes that are
associated with and possibly used to achieve higher versus lower understanding of complex
arguments. To meet the goal of the research, this study addressed three research questions: 1)
What reasoning processes do experts and novices perform when diagramming a complex
argument? 2) What differences exist in the reasoning processes used by experts versus novices?
and 3) Which processes might help produce diagrams of high versus low accuracy? This chapter
begins with the demographic information of the participants and then presents the quantitative
analysis findings for each research question. After presenting the results from sequential
analysis, I present the qualitative analysis findings to help explain the reasoning processes
identified with sequential analysis and evaluate the value of using sequential analysis as a
method for assessing and diagnosing participants’ reasoning processes.
Demographic Information
The participants in this study were five graduate students and five instructors, all from a
higher education institution in the southeastern United States. The sample was purposefully
selected and assigned to two groups: novice and expert. The novice group included one male and
four female master-level graduate students ranging in age from 24- to 54-years-old from four
different departments in Counseling, Information Studies, and Education (Table 4.1). One
novice in this sample reported previous coursework in reasoning and argumentation, but four of
the novices had taken no formal argumentation/reasoning courses.
85
Table 4.1
Demographic Information of the Participants
ID Gender Age Major Profession (yrs.) Argumentation
courses/experiences
Argument
mapping/tool
experiences
Novi01 Female 26 Career Counseling Graduate student 1 course taken No / No
Novi02 Female 24 Mental Health
Counseling
Graduate student 0 No / No
Novi03 Female 24 Library and
information science
Graduate student 0 No / No
Novi04 Male 24 Library and
information science
Graduate student 0 No / No
Novi05 Female 54 Performance
Improvement and
Human Resource
Development
Graduate student 0 No / No
Exp01 Male 42 Criminal Justice Professor (8 yrs.) Teaching
argumentation/reasoning
courses
Yes / No
Exp02 Male 32 Philosophy Professor (24 yrs.) Teaching
argumentation/reasoning
courses
Yes / No
Exp03 Male 52 Philosophy Postdoc researcher &
Instructor (2 yrs.)
Teaching
argumentation/reasoning
courses
No / No
Exp04 Male 54 Philosophy Professor (25 yrs.) Teaching formal reasoning
courses
No / No
Exp05 Male 29 Philosophy Doctoral Candidate
& Instructor (2 yrs.)
Teaching
argumentation/reasoning
courses
Yes / No
86
For the expert group, five male instructors with experience teaching
argumentation/reasoning courses were recruited from the Philosophy and Instructional Systems
and Learning Technology departments. They ranged in age from 29- to 54-years-old. The
average years of teaching experience in the expert group was 12 years, with a range of 2 to 25
years. With regard to perceived content familiarity, all participants reported some degree of
familiarity with the six e-learning principles outlined by Clark (2002): Multimedia, Contiguity,
Modality, Redundancy, Coherence, and Personalization Principle (Seems like this needed a
reference). As reported in Table 4.2, five novices reported having little knowledge of these six
e-learning multimedia principles (two of novices shared that they were familiar with one or two
principles). Four of those in the expert group reported that they had never heard of the principles
and one had some familiarity with all six e-learning principles.
Table 4.2
Participant’s Perceived Content Familiarity and Time Spent on Tasks
ID Familiarity with
content
(0 to 12)
Time spent on reading
(minutes)
Time spent on argument
map (minutes)
Map
Score
Novi01 3 6:24 21:03 14
Novi02 0 3:09 06:41 7
Novi03 2 5:50 35:07 11
Novi04 0 4:41 12:14 5.5
Novi05 0 5:41 32:29 9
Exp01 12 5:10 21:06 19
Exp02 0 4:14 29:10 16
Exp03 0 3:57 34:10 31
Exp05 0 4:32 18:17 26
87
Perceived Content Familiarity and Time Spent on Tasks
While engaged in reading the article summarizing six e-learning principles in the training
period, the novice and expert groups spent 5 minutes 9 seconds and 4 minutes 25 on average,
respectively. The novices spent an average of 21 minutes to complete the argument map
whereas the experts spent an average of 26 minutes. Data collected from one of the experts was
omitted from analysis in this study. During the lab session, Expert 4 displayed a fair amount of
discomfort and misunderstanding of the argument map task. The background of Expert 4 was in
formal reasoning and as a result, he did not understand what to do with the nodes presented in
the jMAP screen. Also, he found the jMap software to be unfamiliar and frustration with the
software significantly limited his ability to perform the task. For this reason, the data from
Expert 4 was excluded from the final analysis in this study.
Expert and Novice Argument Diagrams
As shown in Table 4.3, a total of nine final argument maps were assessed in terms of five
criteria: 1) correctly identified main conclusion, 2) correctly identified fifth-level premises, 3)
correctly linked two-level chains from the lowest level premise within a chain, 3) correctly
linked three-level chains from the lowest level premise within a chain, and 4) correctly linked
four-level chains from the lowest level premise within a chain. In the maps of Novice 4 and
Novice 5, the arrow (Premise to Conclusion) was used inconsistently within the map. In order to
take into account each participant’s ability to identify the relationships between claims, I added 1
point for correctly identifying relationships between two claims, deducted .5 points for arrows
pointing in an incorrect direction, and deducted .5 points for each arrow with an incorrect
valence (positive or negative). The argument map scores ranged from as low of 5.5 points to as
high as 31 points, with all the experts performing better than the novices.
88
Table 4.3
Participants’ Argument Map Scores in Details
ID Correctly
identified the
main
conclusion
(1)
# of root
causes
correctly
identified
(8)
# of
correct
1st order
chain
(10)
# of
correct
2nd order
chain
(10)
# of
correct
3rd order
chain
(10)
# of
correct
4th order
chain
(10)
Total score
(49)
Exp03 1 8 7 5 5 5 31
Exp05 1 8 5 4 4 4 26
Exp01 1 7 5 5 1 0 19
Exp02 1 8 5 2 0 0 16
Novi01 0 6 4 4 0 0 14
Novi03 1 7 1 1 1 0 11
Novi05 1 7 (5*) 2 (*1) 1 1 0 9
Novi02 1 5 1 0 0 0 7
Novi04 0 3 3 (1*) 0 0 0 5.5
Note. The asterisk * indicates the number of chain with the use of incorrect direction or incorrect
valence. The number within () on each first column indicates the maximum score that
participants get per each criterion
Research Question 1
What Reasoning Processes Do Experts and Novices Perform when Diagramming a
Complex Argument?
To identify particular patterns in the mapping and reasoning processes of the experts and
novices, the data from each group was aggregated and sequential analyzed. The resulting
frequency matrix revealed how many times a particular action in a column tended to immediately
follow a given action in a row. Transitional probabilities and z-scores were then determined in
order to identify which sequential actions occurred at higher than expected frequencies. A
transitional state diagram of the action sequences produced by the experts and a transitional state
diagram of the action sequences produced by the novices were generated and placed side by side
to graphically convey the observed transitional probabilities found to be higher than the
89
expected, and to reveal more global and larger sequences of actions that depict the mapping
processes used by experts and by novices.
Sequential Patterns in Experts’ Actions
The observed frequency of each action sequence observed in the expert group is reported in
Table 4.4. The total number of two-event sequences observed was 746 among the four experts
whose data was included in the analysis. The two most frequently observed actions were Read
Claim (225 times) and Identify Association (141 times) and the two least frequent actions were
Provide Reason (11 times) and Delete Link (26 times).
Based on the values reported in the expert group’s frequency matrix (Table 4.4), the transitional
probabilities and z-scores in Table 4.5) and the resulting transitional state diagram in Figure 4.1
reveal a total of nine sequential “patterns” (action sequences that occurred at significantly higher
frequencies than expected): Read Claim Read Claim, Read Claim Identify Association,
Identify Association Position Node, Identify Cause-effect Association Reason, Identify
Cause-effect Association Make Connection, Position Node Read Claim, Make Connection
Identify Cause-effect Association, Review Review, and Delete link Delete Link. The
two most frequent actions followed by Read Claim were Read Claim (119 times) and Identify
Association (66 times) given the total of 225 actions that immediately followed Read Claim. As
a result, the transitional probabilities of RC RC and RC IA was .53 % and 29% with z-
scores of 9.08 and 4.43, respectively, at alpha level p < .05.
90
Table 4.4
Frequency Matrix of Experts’ Reasoning Processes
Note. Bold number indicates its significance at alpha level .05 with Z-score 1.94. Underline indicates that
the number is significantly lower than its expected number. The symbol * indicates mapping behavioral
codes.
RC
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
To
tal
Read Claim 119 66 12 24 0 2 1 1 225
Identify Associations 27 25 10 49 2 20 5 1 141
Identify Cause-effect association 4 7 8 5 5 45 2 2 78
Position Node* 44 20 6 21 3 9 3 1 107
Provide Reason 2 3 1 0 0 3 1 1 11
Make Connection* 18 12 33 6 0 17 7 4 97
Review 5 5 5 0 1 0 42 1 61
Delete Link* 2 3 3 2 0 1 0 15 26
Total 221 141 78 107 11 97 61 26 746
91
Table 4.5
Transitional Probabilities with Associated Z-scores of Sequential Reasoning Processes in Expert
Group
Note: * indicates the transitional probability is significant at alpha level .05 with z-score 1.94. Bold
number indicates that the number is significantly higher than its expected number at alpha level .05.
Underline indicates that the number is significantly lower than its expected number at alpha level .05.
RC
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
Read Claim TP .53* .29* .05 .11 .00 .01 .00 .00
Z 9.08 4.73 -3.03 -1.92 -2.20 -6.49 -5.09 -2.99
Identify Associations TP .19 .18 .07 .35* .01 .14 .04 .01
Z -2.96 -0.34 -1.41 7.76 -0.05 0.51 -2.20 -1.98
Identify Cause-effect association TP .05 .09 .10 .06 .06* .58* .03 .03
Z -5.03 -2.39 -0.08 -2.13 3.81 12.36 -1.92 -0.48
Position Node TP .41* .19 .06 .20 .03 .08 .03 .01
Z 2.77 -0.09 -1.79 1.66 1.22 -1.55 -2.21 -1.56
Provide Reason TP .18 .27 .09 .00 .00 .27 .09 .09
Z -0.85 0.70 -0.15 -1.37 -0.41 1.41 0.11 1.02
Make Connection TP .19 .12 .34* .06 .00 .18 .07 .04
Z -2.59 -1.79 8.10 -2.48 -1.30 1.40 -0.39 0.36
Review TP .08 .08 .08 .00 .02 .00 .71* .02
Z -3.73 -2.15 -0.53 -3.29 0.14 -3.10 18.35 -0.79
Delete Link TP .08 .12 .12 .08 .00 .04 .00 .58*
Z -2.51 -0.99 0.17 -0.99 -0.64 -1.42 -1.55 15.30
92
Figure 4.1. Transitional diagram of the expert group reasoning processes.
DeleteLink26
PositionNode107
Review61
.07
.06
.19
ReadClaim225
REASON11
Make
C onnection
97
.29
.53
07
15
IdentifyA ssociation
141
3504
.05 .09
.1
01
03
IdentifyCausality
78
.41 .19
.03
.08
.03
.27
.27
.09
.09
15
06
.18
.07
.08
.71
.02
58 0812
12
04
.18
.14
.58
.03
.06
.20
.18
.09
34.04
.08
.08
.02
.08
.00*
.00*
.00*
.00*
.00*
Note. Circles without shading indicate mapping actions and shaded circles indicate verbal actions. The
color black and grey of line identify significant transitional probabilities and non-significant probabilities,
respectively. The thickness of line indicates higher or lower transitional probability of the two action
sequence.
93
Sequential Analysis of Novice Actions
The observed frequencies of each sequential transition observed in the novice group are
presented in Table 4.6. The total number of two-event sequences observed was 716 among the
five novices. The two most frequent actions were Read Claim (260 times) and Identify
Association (126), and the two least frequent actions were Delete Link (19 times) and Provide
Reason (29 times). The transitional probabilities and z-score matrix in Table 4.7 and the
transitional diagrams in Figure 4.2 reveal the nine sequential patterns: Read ClaimRead Claim,
Identify AssociationPosition Node, Identify AssociationMake Connection, Identify Cause-
effect associationMake Connection, Position NodeIdentify Association, Make
ConnectionIdentify Cause-effect Association, Make ConnectionMake Connection, and
ReviewReview.
Table. 4. 6
Frequency Matrix of Novice Group’s Reasoning Process
RC
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
Tota
l
Read Claim 158 53 8 32 2 5 0 2 260
Identify associ. 18 13 4 50 7 30 1 2 126
Identify Cause-effect associ. 4 2 1 5 5 15 0 2 34
Position Node* 35 33 2 20 7 15 1 4 117
Provide Reason 7 2 4 3 1 7 3 2 29
Make Connection* 31 17 13 0 6 22 6 1 99
Review 1 5 0 2 0 0 20 2 31
Delete Link* 1 1 2 5 1 5 0 4 19
Total 255 126 34 117 29 99 31 19 715
94
Table. 4.7
Transitional Probability with Associated Z-scores of Sequential Reasoning Processes in Novice
Group
R
C
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
Read Claim TP .61 .20 .03 .12 .01 .02 .00 .01
Z 10.49 1.40 -1.62 -2.28 -3.39 -7.03 -4.33 -2.39
Identify
Associations
TP .14 .10 .03 .40 .06 .24 .01 .02
Z -5.52 -2.37 -0.92 7.81 0.94 3.58 -2.15 -0.82
Identify Cause-
effect Associ.
TP .12 .06 .03 .15 .15 .44 .00 .06
Z -3.01 -1.86 -0.52 -0.29 3.21 5.21 -1.28 1.19
Position Node* TP .30 .28 .02 .17 .06 .13 .01 .03
Z -1.48 3.24 -1.71 0.20 1.14 -0.38 -2.03 0.54
Provide Reason TP .24 .07 .14 .10 .03 .24 .10 .07
Z -1.35 -1.56 2.32 -0.91 -0.18 1.62 1.61 1.44
Make
Connection*
TP .32 .18 .14 .00 .06 .23 .06 .01
Z -0.80 -0.01 4.32 -4.68 1.15 2.73 0.97 -1.07
Review TP .03 .17 .00 .07 .00 .00 .67 .07
Z -3.80 -0.16 -1.26 -1.48 -1.15 -2.25 17.06 1.38
Delete Link* TP .05 .05 .11 .26 .05 .26 .00 .21
Z -2.82 -1.44 1.19 1.17 0.26 1.58 -0.94 5.03
Note. Bold number indicates its significance at alpha level .05 with Z-score 1.94. Underline indicates
that the number is significantly lower than its expected number. The symbol * indicates mapping
behavioral codes
95
Figure 4.2. The transitional diagram of the novice group’s reasoning processes
Note. The circle without shading indicates mapping actions whereas the shaded circle indicates verbal
actions. The color black and grey of line indicate the significant transitional probability and non-
significant probability respectively. The thickness of line indicates higher or lower transitional probability
of the two action sequence. The dotted black line indicates that the transition probability is significant but
the frequency is less than 5.
Transitional State Diagrams of Patterns in Action Sequences of Expert vs. Novice
Figure 4.3 displays the transitional state diagrams for the experts and novices. The
diagrams reveal chains three or more actions that reveal the reasoning processes used by experts
and novices. The side-by-side comparison of the two diagrams helps to identify where the
reasoning processes of experts and novices are similar and different.
DeleteLink19
PositionNode117
Review
30
03
15
ReadClaim260
REASON29
Make Connection
96
20
61
03
IdentifyA ssociation
125
40
06
01
12 06
03
15
06
IdentifyCausality
34
30 28
0613
02
07
10
03
24
07
18
23
17
07
21 0505
11
2601
01
10
14
24
02
44
02
17
01
24
1414
05
06
01
07
.00*
26
12
0210
32
03
67 05
.00*
.00*
96
Figure 4.3. Comparisons of two transitional diagrams of the expert (left) and novice (right)’s reasoning processes
Note: Black lines = probabilities significantly higher than expected; gray lines = probabilities neither significantly lower nor higher
than expected.
DeleteLink26
PositionNode107
Review61
.07
.06
.19
ReadClaim225
REASON11
Make
C onnection
97
.29
.53
07
15
IdentifyA ssociation
141
3504
.05 .09
.1
01
03
IdentifyCausality
78
.41 .19
.03
.08
.03
.27
.27
.09
.09
15
06
.18
.07
.08
.71
.02
58 0812
12
04
.18
.14
.58
.03
.06
.20
.18
.09
34.04
.08
.08
.02
.08
.00*
.00*
.00*
.00*
.00*
DeleteLink19
PositionNode117
Review
30
03
15
ReadClaim260
REASON29
Make Connection
96
20
61
03
IdentifyA ssociation
125
40
06
01
12 06
03
15
06
IdentifyCausality
34
30 28
0613
02
07
10
03
24
07
18
23
17
07
21 0505
11
2601
01
10
14
24
02
44
02
17
01
24
1414
05
06
01
07
.00*
26
12
0210
32
03
67 05
.00*
.00*
Expert Group (n=4) Novice Group (n=5)
97
Reasoning processes of experts. When examining the sequential patterns in terms of
larger chains of actions, the state diagrams in Figure 4.1 and 4.3 reveal primarily four processes
of reasoning used or exhibited by the experts:
1. Read Claim Read Claim Identify Association Position Node Read Claim:
When experts read the claims, they immediately followed that action by reading more
claims (53%) and then moved on to identifying the association between claims (29%).
After identifying an association between claims, the experts tended to position the node
into their map (35%) and then read claims (41%) as an immediate action followed by
positioning a node. These patterns, when examined together, reveal that the experts began
the map construction process by reading the claims and identifying the associations by
positioning associated nodes in close proximity to one another. The phi coefficient tests
revealed that only the Read Claim Identify Association action sequence was found to
be associated with the experts, = .10, p = .03
2. Identify Cause-effect Association Make Connection Identify Cause-effect
Association Reason: Once they identified a cause-effect association between two
claims, the experts tended to follow this action by connecting the two claims with a link
(58%). The experts showed the tendency to follow the addition of a link with identifying
the casual-effect association (34%) and then presenting an explanation for the causal
relationship (6%).
3. Review Review: After positioning and linking the nodes in the argument maps, the
observed transition from Review to Review (along with the qualitative observations)
suggests that the experts reviewed the chain of reasoning between claims by iteratively
reviewing one link and immediately followed that action by again reviewing the next link
98
up the chain of claims (71%). In other words, the experts reviewed the linkages between
nodes by examining links up the chain of linked nodes (e.g., 5th 4th, 4th 3rd, 3rd
2nd, 2nd Main conclusion). The phi-coefficient test on this paired-action sequence was
not found to be particularly associated with either the experts or the novices.
4. Delete link Delete link: The experts also tended to follow the deletion of a link
between two nodes by deleting links between two other nodes. This pattern may be an
indication of times when experts identified and corrected multiple errors found in the
linkages within a chain of linked nodes following the Review Review process (see
above). The phi coefficient tests revealed that the Delete Link Delete link action
sequence was associated with the experts, = .31, p <.001. Based on the qualitative
analysis, this action sequence was not believed to indicative of the experts’ reasoning
processes because this behavior was exhibited in only one of the four experts. One of the
four experts completed the process of linking the nodes in his argument map but then
realized that he did not understand the primary purpose of the task. As a result, this expert
individually deleted every link to begin the process all over again.
Reasoning processes of novices. When examining the sequential patterns in terms of
long chains of actions, the state diagrams in Figure 4.2 primarily reveals four processes of
reasoning used or exhibited by the experts:
1. Read claim Read claim: The novices tended to start reading claims and then follow
this action by reading other claims (51%). The phi-coefficient test on this paired-action
sequence was not found to be associated with the novices.
2. Identify AssociationPosition NodeIdentify AssociationMake Connection:
Once the novices identified an association of a claim of a node, they tended to follow this
99
action by positioning the node onto the map (40%). Once they positioned the node, the
novices were most likely to follow this section by identifying the association of another
claim (28%) and then inserted links to connection two claims (24%). None of the phi-
coefficient tests on each of the above paired-action sequences were found to be
associated to the novices.
3. Identify Cause-effect AssociationMake Connection Identify Cause-effect
AssociationReason: Like the expert group, the novices identified a cause-effect
association between claims, then inserted links between the claims to make the
connection (44%), then identified the cause-effect association between the claims (14%),
and the presented a reason for making the connection (15%).
4. Review Review: After positioning and linking the nodes in the argument maps, the
observed transition from Review to Review suggests that the novices also reviewed the
chain of reasoning between claims by iteratively reviewing one link and immediately
followed that action by again reviewing the next link up the chain of claims (71%).
However, the process of reviewing the linkages across a series of chained nodes was not
observed in the qualitative review of the video recordings of the novices, as discussed
later in this results section. As mentioned earlier, the phi-coefficient test on this paired-
action sequence was not found to be associated with neither the experts nor the novices.
5. Make ConnectionMake Connection: The novices tended make a connection
immediately after making a connection between nodes. This pattern indicates that the
novices laid out the nodes and then inserted multiple links between the nodes in an
iterative process. Based on the qualitative analysis of the video data, only one novice
positioned all the nodes first and then connected them with links. The phi-coefficient test
100
on this paired action sequence was not significant, indicating that this action sequence
was not associated with neither the novices nor experts.
101
Research Question 2
What Differences Exist in the Reasoning Processes Used by Experts versus Novices?
Table 4.8 provides a summary of the mapping processes identified in the patterns of action
sequences performed by the experts and novices. A comparison of the summary table reveals the
following similarities and differences in the mapping processes between the experts and novices.
Table 4.8
Summary of Mapping Processes Observed in Experts and Novices
Expert Reasoning Processes Novice Reasoning Processes
Read Claim Read Claim Identify Assoc
Position Node Read Claim
Read claim Read claim
Identify Assoc Position NodeIdentify
Assoc Make Connection
Identify Cause-effect Assoc Connect
Identify Cause-effect Assoc Reason
Identify Cause-effect Assoc Connect
Identify Cause-effect AssocReason
Review Review
Delete link Delete link** Make ConnectionMake connection
Note. The symbol ** indicates that the action pattern was the sole work of just one expert. The
symbol * indicates that the action pattern was the sole work of just one novice.
Similarities in the Reasoning Processes of Experts and Novices
1. Read ClaimRead Claim. When reading claims, both expert and novice group
members displayed the iterative process of Read ClaimRead Claim. This action
sequence has three different functions: 1) comprehension of claims; and 2) the search for
subordinate/antecedent claims. Most participants in the study started by reading all
claims to comprehend all the given claims. This iterative action sequence also appeared
102
when the expert and novice participants searched for a claim that was a subordinate or
antecedent claim to another claim on the map.
2. Identify Cause-effect Assoc Connect Identify Cause-effect Assoc Reason.
Both experts and novices used this action sequence to insert links to connect two claims.
Once connecting two claims, both experts and novices verbally provided some
explanation of the cause effect relationship between the two claims. For example, “I’m
connecting A to B” or, “A will help B.” Sometimes, they explained the cause-effect
association after making connections when they realized that they did not verbally
convey this action in their think-aloud report.
3. Review Review. Although this action sequence was observed in both experts and
novices, the experts tended to frequently review the links from bottom up after adding a
new node to the bottom of an existing chain of claims. Also, the experts used this
iterative process when finalizing their maps - reviewing all the chains of claims from root
level up to the main conclusion (e.g., ABCD). In contrast, the novices also
reviewed the link between two nodes (e.g., nodes AB) and then reviewed the link
between two other nodes (e.g., CD), but two nodes that are not chained to the two
linked nodes previously reviewed.
Differences in the Reasoning Processes Of Experts and Novices
1. To position nodes, experts exhibited the reasoning process of Read Claim Read Claim
Identify Assoc Position Node Read Claim where they initially postulated the
argument structure before actually connecting the claims with links. On the other hand,
103
the novices read claims and positioned a claim and then immediately inserted a link
between claims.
2. To insert links between nodes in the argument diagrams, novices exhibited the reasoning
process of Position NodeIdentify AssocMake Connection. This means that novices
tended to insert a link between two claims right when they identified the association
between the two claims and positioned one of the claims next to the associated claim. In
contrast, the experts identified the cause-effect relationship between the claims just prior
to inserting a link to connect the two claims.
104
Research Question 3
Which Processes Might Help Produce Diagrams of High versus Low Accuracy?
Primary Findings
The reasoning processes found to be unique to the experts (as identified in the previous
section) provide indications of the types of processes that may help students produce argument
diagrams of high accuracy. In contrast, the reasoning processes found to be unique to the novices
provide indications of the types of processes that may hinder students’ ability to produce more
accurate argument diagrams. The findings presented in the previous section suggest that two
particular processes that were unique to experts (one, to position nodes, and the other, to insert
links between nodes) can help to explain how students might produce more accurate argument
diagrams.
Post-hoc Analysis
To further determine whether the two reasoning processes described above, and perhaps
other reasoning processes, were unique to participants who produced argument diagrams of high
accuracy, the participants of this study were categorized into three groups: high, moderate, and
low scores (see Figure 4.4). To examine any reasoning processes associated with superior
performances in the argument map task, the actions of two highest performers and the two lowest
performers’ data sets were sequentially analyzed.
Frequencies and Transitional Probabilities between Action Sequences
Tables 4.9 and 4.11 present the observed frequencies of sequential actions observed in
the two highest performers and the two lowest performers. The total action sequences in the two
groups were 362 for the high performers and 182 for the low performers. Overall, the high
105
performers exhibited more than twice as many as the low performers in Reading Claims,
Identifying Cause-effect Association, Reviewing, and Deleting links. The corresponding
transitional probabilities and z-scores of each group are reported in Tables 4.10 and 4.12. Based
on these matrixes, nine sequential actions were identified in the high performer group and four
sequential actions were identified in the low performer group. Figure 4.5 presents the transitional
state diagrams produced from the frequency matrices to provide a graphical description of the
patterns of action sequences exhibited by the top two performers and the two lowest performers.
Figure 4.4. Bar graph to represent the participants’ final argument map scores.
106
Table 4.9
Frequencies Matrix for High Performer Group
R
C
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
To
tal
RC 44 31 9 12 0 2 0 1 99
IA 16 12 5 14 2 14 3 0 67
IACE 2 6 6 2 5 29 1 1 52
PN 15 8 3 2 2 2 0 0 32
REASO 2 2 1 0 0 3 0 1 9
MC 15 7 24 0 0 14 2 2 64
REVIE 1 0 3 0 0 0 14 1 20
DL 2 1 1 2 0 0 0 13 19
Total 97 67 52 32 9 64 20 19 362
Table 4.10
Transitional Matrix and Z-scores for High Performer Group
RC
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
RC TP .44 .31 .09 .12 .00 .02 .00 .01
Z 4.61 3.81 -1.78 1.33 -1.87 -4.82 -2.83 -2.23
IA TP .24 .18 .08 .21 .03 .21 .05 .00
Z -0.55 -0.10 -1.76 3.89 0.31 0.81 -0.40 -2.12
IACE TP .04 .12 .12 .04 .10 .56 .02 .02
Z -4.06 -1.42 -0.64 -1.38 3.55 7.75 -1.24 -1.17
PN TP .47 .25 .09 .06 .06 .06 .00 .00
Z 2.66 0.97 -0.85 -0.55 1.42 -1.79 -1.44 -1.40
REASO TP .22 .22 .11 .00 .00 .33 .00 .11
Z -0.32 0.28 -0.29 -0.95 -0.49 1.24 -0.74 0.79
MC TP .23 .11 .37 .00 .00 .22 .03 .03
Z -0.70 -1.74 5.79 -2.76 -1.41 0.95 -0.94 -0.85
REVIE TP .05 .00 .16 .00 .00 .00 .74 .05
Z -2.19 -2.14 0.17 -1.40 -0.72 -2.08 13.32 0.00
DL TP .11 .05 .05 .11 .00 .00 .00 .68
Z -1.66 -1.54 -1.17 0.26 -0.72 -2.08 -1.09 12.65
107
Table 4.11
Frequencies Matrix for Low Performer Group
R
C
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
To
tal
RC 14 18 0 7 0 1 0 0 40
IA 5 4 1 22 1 18 1 0 52
IACE 0 0 0 2 1 0 0 0 3
PN 9 17 0 9 2 5 0 1 43
REASO 1 1 0 1 0 3 0 0 6
MC 9 12 2 0 2 4 1 1 33
REVIE 0 0 0 2 0 0 1 0 3
DL 0 0 0 0 0 2 0 0 2
Total 38 52 3 43 6 33 3 2 182
Table 4.12
Transitional Matrix and Z-scores for Low Performer Group
RC
IA
IAC
E
PN
RE
AS
O
MC
RE
VIE
DL
RC
TP .35 .45 .00 .17 .00 .02 .00 .00
Z 2.44 2.55 -0.93 -1.07 -1.33 -2.93 -0.93 -0.76
IA TP .10 .08 .02 .42 .02 .35 .02 .00
Z -2.41 -4.00 0.17 3.69 -0.67 3.60 0.17 -0.91
IACE TP .00 .00 .00 .67 .33 .00 .00 .00
Z -0.90 -1.11 -0.23 1.75 2.92 -0.83 -0.23 -0.19
PN TP .21 .40 .00 .21 .05 .12 .00 .02
Z -0.03 1.77 -0.98 -0.52 0.55 -1.30 -0.98 0.87
REASO TP .17 .17 .00 .17 .00 .50 .00 .00
Z -0.27 -0.67 -0.32 -0.42 -0.46 2.04 -0.32 -0.26
MC TP .29 .39 .06 .00 .06 .13 .03 .03
Z 1.19 1.33 2.29 -3.43 1.06 -0.86 0.75 1.23
REVIE TP .00 .00 .00 .67 .00 .00 .33 .00
Z -0.90 -1.11 -0.23 1.75 -0.32 -0.83 4.32 -0.19
DL TP .00 .00 .00 .00 .00 1.00 .00 .00
Z -0.74 -0.91 -0.19 -0.80 -0.26 3.00 -0.19 -0.15
Note. Italics indicates that the number of the cell is less than five.
108
Figure 4.5. Transitional diagrams of the two high (left) and the two low performers (right).
DeleteLink19
PositionNode32
Review20
08
10
24
ReadClaim
99
REASON9
Make
C onnection
64
31
44
23 15
IdentifyA ssociation
67
2105
04 12
12
04
02
IdentifyCausality
52
.47 25
06
06
.06
22
33
11
11
22
03
74
05
68 115
5
18
21.56
.02
09
06
22
11
3703
05
16
.11
12
02
01
03
DeleteLink
2
PositionNode
43
Review
3
02
33
ReadClaim
40
REASON6
Make Connection
33
45
35
IdentifyA ssociation
52
42
02
02
12
67
IdentifyCausality
3
21 40
0512
02
17
17
03
50
39
13
17
100
8
10
3521
17
06
0
03
03
07
26
17
02
29
03
33
High Performers (n=2) Low Performers (n=2)
109
Sequential Patterns Exhibited by the Two High versus Low Performers
Table 4.13 provides a summary of the reasoning processes exhibited in the patterns of
action sequences performed by the two high performers and two low performers. A comparison
of the two highest and two lowest performers in the summary table reveals similarities and
differences in reasoning processes between the two groups.
Table 4.13
Summary of Reasoning Processes Observed in the Two High and Low Performers
High Performers Reasoning Processes Low Performers Reasoning Processes
Read Claim Read Claim Identify Assoc
Position Node Read Claim
Read claim Read claim Identify Assoc
Position Node
Identify Cause-effect Assoc Connect
Identify Cause-effect Assoc Reason
Read claim Read claim Identify Assoc
Make Connection
Review Review
Delete link Delete link**
Note. The symbol ** indicates that the action pattern was the sole work of just one of the four experts
Sequential Patterns Exhibited by the High Performers
The unique action sequences exhibited by the two high performers were exactly identical
to the reasoning processes observed across all four experts, as described in Table 4.8.
Sequential Patterns Exhibited by the Low Performers
For the two low performers’ mapping actions, close examination of the transitional right
state diagram in Figure 4.5 revealed two sequential patterns to produce only two reasoning
processes (in contrast to the five reasoning processes exhibited across all five novices).
110
1. Read ClaimRead claim Identify AssociationPosition Node: To position a
node, the low performers started by reading a claim and then immediately followed this
action by reading another claim (35%). After reading one of more claims, the low
performers tended to follow this action by identifying the association between the claims
(45%), then positioning the claim (42%). None of the pair-action sequences observed
within this process were found to be associated with the low performers based on the phi
coefficient tests.
2. Read ClaimRead claim Identify AssociationConnect: To insert a link between
two claims, the low performers started by reading a claim. This action was most likely
followed by reading another claim (45%). The low performers would follow reading the
claims by identifying associations between the claims, and then insert links to make the
connection (35%) between the claims. The phi coefficient tests on the action sequences
observed in this mapping process revealed that the action sequence Identify Association
Make Connection was associated with the low performers, =.23, p < .05.
Reasoning Processes that Helped or Hindered Production of Highly Accurate Diagrams
A review and comparison of the mapping processes presented in Table 4.13 reveals three
specific findings that help to explain how the experts were able to produce more accurate
argument diagrams, and insights that suggest what processes that students in general can use to
help them produce more accurate argument diagrams:
1. When positioning nodes in an argument diagram, both experts read claims, identified
associations, and then positioned the nodes in an iterative process while refraining
from inserting links between the associated nodes.
111
2. When inserting links between nodes, the experts used the process of identifying
cause-effect relationships between nodes (as opposed to identifying relationship in
general) before inserting a link between two nodes. In addition, the experts were able
to provide a reason or justification for inserting the link between the two nodes.
3. The experts engaged in an iterative and extended process of reviewing the links
between the series of chained nodes within their argument diagrams to identify
potential errors in links.
112
Qualitative Findings
Coding Processes
The participants’ mapping behaviors and verbal reports were transcribed and analyzed to
identify and construct the initial coding categories. Using the method of content analysis, I
developed the initial coding categories by operationally defining them and assigning each code
to the mapping action and think aloud data (Table 4.14). To develop meaningful categories that
explained specific actions in the reasoning processes, only the purposeful mapping actions and
verbal utterances related to the task were coded. In other words, the behavioral actions that were
not directly related to the reasoning processes, such as the action of moving nodes for purely
aesthetic reasons, were omitted from the analysis. Also, verbal reports that were not related to
the reasoning processes, such as asking technical questions (“How can I connect this arrow to
the red dot?”) or miscellaneous utterances (e.g., “Where are you from?” “I’m old to do this”)
were not included in the analysis. Some verbal data and map behaviors shared the same meaning
or function. For example, moving a node under/next to/above a previous identified node in the
map indicated that the participant identified some kind of association between the nodes. Some
participants verbally described associations as they moved and positioned a node next to another
node, whereas some participants simply moved a node without verbally describing or explaining
this action. By observing each participant’s behavioral and verbal data, the reasoning processes
used to analyze and structure the complex argument maps were identified for each individual
participant.
The set of coding categories emerged from the analysis of each individual participant’s
verbal protocol/data, with new categories added to the coding scheme when new actions could
not be assigned to any of the existing categories. For example, code IA (Identifying Association)
113
Table 4.14
Initial Codes Emerging from Verbal and Mapping Action Data
Think-Aloud Protocol jMAP Behavioral Data
Read Claims/comprehend claims (RC) Cannot be identified
Identify the main conclusion (IMC) Position node to top/bottom/right side
of map (PN)
Interpret claims (ITC) Cannot be identified
Identify the level of a claim (IL) Move a node under/next to/above to a
previously identified node
Identify an association between claims
(IA)
Move a node under/next to/above to a
previously identified node
Identify a cause-effect association
between claims (IACE)
Move a node under/next to/above to a
previously identified node
Identify the dependency (ID) The arrows are connected to the same
point
Identify Independency of two claims
(IID)
The arrows are connected to the
different points
Identify the irrelevant claim (IIR) The node is not connected or placed far
away from the map
Identify negative association (INEG) Change the link’s attribute
Make a cause-effect relationship Insert a Link and Connect two Nodes:
Make Connection (MC)
Review the chain of reasoning
(REVIE)
Cannot be identified
Recognize the reasoning Errors
(RERRO)
Cannot be identified
Correct the reasoning Errors Delete the existing Link (DL)
Reposition Nodes (PN)
ReLink the arrow (MC)
Provide Reasons (REASO) Cannot be coded
114
indicates that a participant verbally stated that there is some relationship between two claims.
Then, a new code was generated when a participant specifically identified the cause-effect
relationship between two claims (IACE) using terms like “if-then,” “because,” and “when.”
Using this coding process, a total of 13 reasoning processes emerged from the verbal
protocol/data: RC (Read a claim), IA (Identify Associations), IL (Identify a level of a claim),
IMC (Identify the main conclusion), IACE (Identify cause-effect relationship), ID (Identify the
dependency between claims), IID (Identify the independency between claims), IIR (Identify the
irrelevant claim) and INEG (Identify the negative relationship), ITC (Interpret/evaluate claims),
REVIE (Review the chain of the reasoning), RERRO (Recognize reasoning errors) and REASO
(Provide reason). Three behavioral codes were identified to describe the mechanical processes of
constructing the argument maps: PN (Position node), MC (Make a connection), and DL (Delete
link).
As shown in Table 4.14, some particular reasoning processes could be identified by either
observing the participants’ mapping behavior or the participants’ verbal reports. For example,
verbally describing the negative relationship between two claims (INEG) in think-aloud data was
equivalent to the act of changing the attributes of a link (CA) in mapping behaviors. To avoid
redundancy in the coded data, I recorded only one action code but not both. Some particular
reasoning processes, such as identifying an irrelevant claim, could not be identified by solely
observing the mapping behavior. Taking into consideration the redundancies in action codes, I
developed a new coding scheme (Table 4.15) and applied it to the participants’ mapping
behaviors and verbal data. I then assigned the codes to the behavioral actions and the verbal
reports as illustrated in Table 4.16.
115
Table 4.15
Modified Coding Scheme for Verbal and Mapping Action Data
Code Meaning Examples in context
RC read a claim Read a claim - Encode; understanding of meaning
IMC Identify the main conclusion Identified the main conclusion
IL Identify a level of a claim Identify a claim's level and position it to top/bottom or right/left
PN* Position a node Move/position a node
IA Identify an association Noticed some associations (without saying Cause-effect). E.g., N3 and
N6 are related.
IACE Identify a cause-effect association Verbally state that N3 is feeded into N6. E.g, I'm connecting from N3 to
N6 since N3 will help N6
ITC Interpret a claim by his or her own words
INEG Identify a negative relationship Verbally state a negative relationship between two claims
ID Identify dependencies between claims Specify dependencies/commons between claims
IID Identify independencies between claims Specify that there are reasons to support the same claim but in different
reasons.
IIR Identify irrelevant claims to the main
argument
E.g., I think this is a different issue.
REASO Provide a reason for an association Explicitly state a reason why there is a relationship
MC* Make a connection (connect a link between
two nodes)
Add a link and connect two nodes
REVIE Review the flow of reasoning
RERRO Recognize a reasoning error
DL* Delete a link, detach a link Delete a link --> disconnect the relationship, or reserved the direction
Note. The symbol * indicates mapping behavioral codes
116
Table 4.16
An Example of Coding Sheet (Exerted from Novice 2)
Time
frame
Map Behavior Behavior
Codes
Verbal
Codes Think aloud
0:00 Pointed N1 RC Okay. So, if you were scrolling web
browser for multiple graphics and text
description
IA Would be a supporting argument
0:14 First Move N1 to
the map
PN
0:16 Pointed N8 RC Exclude for gratuitous words
IA would be opposition,
0:18 First Move N8 to
the map
PN
0:19 pointed N13 RC and then, Exclude for gratuitous visuals
and text.
ID Well, you exclude them, then they work
together.
0:25 First Move N13 to
the map
PN
So that means that...Where's that arrow
thing?
0:29 Add a link on N13 There you are.
0:31 Detached the link
from N13 and
connect a link N8
to N13
MC So, you go here arrow. Where's the little
red box? Okay, and the little red box
here.
As for the limitations of coding processes, it was often difficult to interpret a particular
action across different contexts. Even though RC (Read claims) was defined in this study to be
the actions of verbally reading out a claim, the RC action also included “Read a Claim to encode
the meaning of the claim,” “Read claims to search for the claim that relates to the previously
identified claim,” and “Read claims to search for a position of the claim.” These nuances in the
RC actions were identified when the participant verbally reported what she/he was doing while
reading the claims (e.g., “I’m rereading these claims to find which claim is supporting the main
117
conclusion,” “I’m reading claims to find the main conclusion,” etc.). However, some
participants simply described the action they were performing (“I am linking these two claims”)
without explaining why they performed the particular action (e.g., I am linking these two claims
because A can only happen if B happens).
With regard to inter-coder reliability, an educational researcher with prior experience
examining mapping processes was invited to take on the role of second coder and was trained to
code the verbal data set. The researcher created a coding scheme (Table 4.17) and a training
document for the second coder. After that, the researcher randomly chose one expert and one
novice and sent the verbal transcripts to the second coder along with their video files. The
second coder learned the coding scheme and examples and independently coded the verbal data
while simultaneously watching the video. After the second rater coded this data independently,
both raters agreed that some codes needed to be combined or collapsed into a single code due to
their infrequent occurrence in the verbal data (Table 4.17). Table 4.17 showed the frequencies of
each code per participant and the average frequencies within each group. Since sequential
analysis requires more than 5 per action sequence to conduct a reliable z-score test, some codes
(IMC, IL, ID, IID, IIR, INEG) were collapsed into the IA code/category. In the end, a total of
eight codes (three codes from the mapping behaviors and five codes from the think aloud data)
were finalized and used to conduct the sequential analysis (Table 4.18). Using the finalized
coding scheme, the two coders reviewed and re-coded the videos and verbal transcripts of the
two selected participants. The researcher then computed Cohen’s kappa to assess the inter-coder
reliability on the coding results between two coders. The inter-rater reliability for the first data
set was .78, indicating substantial agreement between the two raters. For the second data set,
inter-rater reliability was .97, indicating almost perfect agreement between two raters.
118
Table 4.17
Frequencies of Each Code Observed in Individual Participants
Note. The symbol * indicates the map behavioral codes. Underline indicates that the number in the cell is significantly different from
the number in the other group (using t-test at alpha level .05)
Expert Group Novice Group
Codes 1 2 3 5 Sum Ave. 1 2 3 4 5 Sum Ave.
RC 34 92 51 47 224 56.00 63 23 91 16 66 259 51.80
IMC 1 2 1 2 6 1.50 2 0 2 5 1 10 2.00
PN* 34 41 32 32 139 34.75 17 28 18 15 37 115 23.00
IL 11 5 7 1 24 6.00 0 0 1 0 1 2 0.4
IA 13 20 19 13 65 16.25 9 20 25 12 18 84 16.80
IACE 14 11 30 22 77 19.25 14 0 9 3 8 34 6.80
ITC 1 2 8 2 13 3.25 0 0 2 0 0 2 0.40
INEG 0 6 2 2 10 2.50 3 7 1 0 6 17 3.40
ID 0 2 0 2 4 1.00 0 0 0 3 2 5 1
IID 0 3 1 1 5 1.25 0 0 0 0 0 0 0
IIR 2 6 5 1 14 3.5 0 0 0 0 0 0 0
REASO 2 0 6 3 11 2.75 10 3 11 3 2 29 5.80
MC* 20 13 47 17 97 24.25 18 15 20 19 28 100 20.00
REVIE 20 21 3 17 61 15.25 2 2 19 1 7 31 6.20
RERRO 3 2 4 2 9 2.75 2 1 4 0 6 13 2.60
DL* 7 0 17 2 26 6.5 3 0 4 2 10 19 3.80
119
Table 4.18
The Final Codes Combining Think-aloud and Mapping Behaviors for Sequential Analysis
Think-aloud data jMAP behavioral data
Code Definition Code Definition
RC Read Claims
(Encode and Comprehend claims
Search the targeted claim, search
a position for the claim)
IA
IACE
Identify associations of claims
Identify cause-effect associations
PN Move nodes, Position nodes,
Reposition nodes
REASO Provide reason/explanation about
the relationships
MC Add a link between two
nodes, Confirm the cause-
effect relationship between
nodes
REVIE Review the chain of reasoning
Recognize errors
DL Delete Links, Reposition (PN)
120
Five Global Processes Used for Diagramming an Argument Map
The qualitative analysis of the video recordings revealed similarities in the processes used
by participants at the global level as well as some individual differences in the specific action
sequences used to construct their argument maps. The specific mapping actions identified by the
qualitative analysis are presented first, followed by a qualitative description of global level
processes observed across all participants. Tables 4.19 and 4.20 present a summary of the
processes used by both experts and novices to construct the argument diagrams when examined
at a global level. Examination of such global actions across all the participants revealed five
global processes or the following five steps: 1) scan all fifteen claims; 2) find the final
conclusion; 3) structure the argument map; 4) review the logic of reasoning chains; 5), correct
reasoning errors. For some participants, all five of these operations were observed and
performed in this particular sequence.
Step1: Scan all claims. The first action exhibited by seven of the nine participants (four
experts and three novices) was to read all claims. In this step, participants focused on
comprehending or scanning the claims across the fifteen nodes to determine the nature of each
claim. The two other novices did not read all the claims. Instead they read one claim to three
claims and then positioned the node as an immediate action.
I'm just going to read through all of these reasons and see where I'll begin, how I'll form
the map. So I'm just going to read them out loud. (Novice 03)
The first thing I'm going to do here is just read these 15 items. (Expert 02)
Step 2. Identify the main conclusion. Following the scanning of claims, eight of the
nine participants (four novices and four experts) identified (or attempted to identify) which of the
fifteen nodes contained the main conclusion of the argument. Once identifying the conclusion,
they positioned it at a specific location on their screen (e.g., the top of the screen). Although
121
some participants could not identify the main conclusion in this step, they were able to find the
conclusion later on in the process of structuring the nodes in their argument map. One novice
was not able to correctly identify the main conclusion and as a result, the novice randomly chose
and positioned a node to begin the map construction process. However, this novice did
eventually identify the correct conclusion.
Step 3. Structure the arguments. As soon as the main conclusion was positioned, all
nine participants started the work of constructing their argument maps. Two approaches to
constructing the arguments were identified among the nine participants: structured and
unstructured approaches. The participants who used the structured approach provided verbal
explanations of how they were going about structuring the claims. For example, Expert 1 stated,
“I would use hierarchical structure at this time” and then placed the main conclusion at the top of
his map. Expert 2 stated, “What I want to do is I put these results to the right side and then I will
find the subordinate claims for each node.” After loosely categorizing the claims, the experts
started to structure the argument map by using top-down or bottom-up reasoning processes. For
example, Experts 1, 2, 3 and Novice 3 used top-down reasoning processes in which they started
their argument map by identifying the main conclusion and the major premise(s) to support the
main conclusion. Then, they identified the third-level claims that supported the second-level
claim and so on. On the other hand, Novice 5 identified the specific claims at the lowest level
(root cause claims) first and then identified more general claims using bottom-up reasoning
processes. In particular, Expert 5 and Novice 1 started by identifying the main conclusion and its
major conclusion using top-down reasoning then switched their strategies to identify all specific
claims at the lowest level and match their parental claims using bottom-up reasoning processes.
122
The participants who used an unstructured approach did not verbally state nor describe
the strategies they were using to construct their argument diagrams. Even though Novice 1
stated that she would place supporting claims to the left side of the map and opposing claims to
the right side, at the end, she mixed the supporting and opposing claims in the same place. In
addition, the participants who used unstructured approaches did not exhibit either top-down or
bottom-up reasoning processes. Novice 2 read the nodes in the order in which they were
displayed on screen and would move a particular node into the map based on the evaluation of
the claim as true or not true. In Novice 2’s map, every claim was directly linked to the main
conclusion. As a result, Novice 2 failed to identify and illustrate the complex hierarchical
relationships between the minor premises and conclusion. In addition, Novice 2’s map included
a circular loop between claims and a set of isolated or orphaned premises that were disconnected
from the main conclusion. In Novice 4’s map, there were three separate sets of argument maps
with two of these sets of premises revealing no connection or relevance to the main conclusion.
Step 4. Review the logical flow of chain of reasoning. Reviewing the logical flow
within the chains of linked claims occurred at different times during the task: in the middle of
map construction and after the map construction as a final review. In the expert group, all four
experts exhibited this behavior frequently while identifying and constructing the argument
structure. Using this iterative review process, experts found errors in the way claims were linked
and as a result, applied more elaborative reasoning to improve the structure of claims. At the
completion of the argument map, the experts reviewed the chain of linked premises using a
bottom-to-top or top-to-bottom approach. In contrast, the novice rarely performed this type of
review process (except on some occasions with Novice 3 and 5). In particular, Novice 2 and 4
did not exhibit any such behavior during the entire map construction process.
123
Step 5. Correct the reasoning errors. During the process of reviewing the argument
maps in step 4, participants identified errors in their reasoning (reversed direction, hasty of
generalizations). Once they detected such reasoning errors, eight of the nine participants made
corrections to their argument maps. In the expert group, 80% of the experts provided reasons or
explanations for the corrections or changes they made to their argument map. However, in the
final review phase, no corrections were made even though they discussed possible reasoning
errors. For example, while reviewing his final argument map, Expert 5 noticed a possible error
(reversed direction) in the link between two given nodes. Ignoring this error, expert 5 proceeded
to insert a mediated factor/node between the two nodes.
I'm not sure that ‘increase in selective attention’ is something that reduces the overall
cognitive load. It might be just that increasing the selective attention should help to
encode into long-term memory. But probably it would help encode in long-term memory
because you're reducing the cognitive load. So I think I'll leave that there. (Expert 05)
124
Table 4.19
Observation Summaries of Novices’ Argument Mapping Processes
Participant Descriptions of reasoning processes Final Argument map
Novice01 Scanned all fifteen nodes first
Identified the main conclusion and the
major promise (Top-down)
Identified the nodes at 4th level, 3rd
level, then 2nd level in a sequential
way (Used Bottom-up reasoning
process)
Changed the main conclusion from
N11 to N6
Breadth-first search (to identifying the
lowest premises and their parent
premise).
1st 2nd 4th 3rd
Novice02 Read a node and placed the node by
evaluating whether or not the claim
supports the main conclusion.
Used breadth first approach.
Fast and intuitive
Circular reasoning
1st 2nd
125
Table 4.19 - Continued
Participant Descriptions of reasoning processes Final Argument Map
Novice03 Scanned all fifteen nodes first then
started structuring the map by choosing
the final conclusion.
Used slow and analytic reasoning
process to identify the argument
Used top-down approach (Main
conclusion Major premise the 3rd
level premises) and breadth-first
approach.
Frequently reviewed the map (e.g.,
chains of reasoning)
1st 2nd 3rd 4th 5th
Novice04 Used a top-down and breadth-first
approach (from conclusions to the lower
levels).
Reversed direction errors
Created three separate argument maps
Failed to create a coherent argument
No final review
Refused to connect the three main
conclusions
1st 2nd, 1st2nd 3rd.
126
Table 4.19 - Continued
Participant Observation Final Argument Map
Novice05 Scanned all fifteen nodes first then started
structuring the map by choosing the final
conclusion.
Initially positioned all nodes first then
connected them without additional analysis.
While connecting them, Novice5 recognized
errors and reconstructed the whole argument
structure.
Used top-down and bottom-up approach
Reviewed the initial map and found many
reasoning errors and corrected them
Reversed direction errors (confused the main
conclusion with a premise)
1st 5th 4th 3rd 2nd
127
Table 4.20
Observation Summaries of Experts’ Argument Mapping Processes
Participant Observation Final Argument Map
Expert01 Scanned all fifteen nodes first then
started structuring the map by choosing
the final conclusion
Breadth-first
Top-down, general to specific
Cluster claims into two groups
(techniques vs. concepts)
Use depth-first when reviewing the
chain of reasoning
Identify the irrelevant claim
1st 3rd4th
2nd3rd
Expert02 Scanned all claims
Top-down reasoning process
Cluster claims into two groups
(methods vs. results)
Positioned nodes without linking
Used breadth-first search
Used forward (left to right) and depth-
first approach to review the final map.
Identify the irrelevant claim
1st 2nd 3rd 4th 5th
128
Table. 4.20 - Continued
Participant Observation Final Argument Map
Expert03 Scanned all fifteen nodes first then
started structuring the map by choosing
the final conclusion
Breadth-first search
Top-down reasoning
Experienced difficulties due to the
imperative tones used in claims
After resolving the misunderstanding,
the expert reconstructed his argument
map
Reviewed the whole chain of reasoning
when added a new node
Identify the irrelevant claim
1st 2nd 3rd 4th 5th
Expert05 Scanned all fifteen nodes first then
started structuring the map by choosing
the final conclusion
Switched from top-down to bottom-up
reasoning by analyzing all specific
claims first then matched them to their
inclusive claims
Breadth-first search
Reviewed the whole chain of reasoning
when adding a new node
Reviewed the map using bottom-up
(forward) reasoning as a final review
Identify the irrelevant claim
1st 2nd 5th 4th 3rd
129
Types of Reasoning by Experts and Novices
Table 4.21 summarizes the argument analysis processes used by each of the nine
participants to construct their argument maps. All experts and Novices 1, 3, and 5 exhibited all
five global steps in the map construction process. Novice 2 did not perform Step 1, Step 5, and a
final review of his/her map. Novice 4 read some (three claims only) but not all the claims before
starting the map construction process. Even though Novice 4 identified the main conclusion, it is
hard to say that he correctly identified the main conclusion of the argument because he drew
three separate maps—each with its own conclusion.
Table 4.21
Chi-square Test for Frequencies across Participants
Argument Analysis Novices Experts
Processes 01 02* 03 04 05 E01 E02 E03 E05
Step1. Scan nodes X X X X X X X X
Step2. Find the main
conclusion X X X X X X X X X
Step3. Structure the
map X X X X X X X X X
Top-down 5 0 27 7 0 17 27 40 0
Bottom-up 26 0 0 5 15 2 0 0 25
Total 31 0 27 12 15 19 27 40 25
Depth-first 3 0 0 0 0 2 0 0 8
Breath-first 17 0 13 3 13 13 25 31 16
Total # of
reasoning 20 0 13 3 13 15 25 31 24
Step4. Review X X X X X X X X X
Step5. Correct X X X X X X X
Note. The symbol * indicates that the participant’s reasoning process is unstructured and not clear enough
to categorize as either a top-down or bottom-up approach (neither depth-first nor breadth-first).
130
Table 4.22 showed the frequencies of the particular reasoning processes used by the
participants while structuring their maps. Generally speaking, Experts 1, 2, 3 and Novice 3 used
top-down reasoning processes whereas Expert 5 and Novices 1 and 5 used bottom-up reasoning
processes. As a post-hoc analysis, a chi-square test (Table 4.22) was performed to identify the
association between group membership (experts and novice) and types of reasoning (top-down
and bottom-up). The relationship between the two variables was statistically significant,
, , indicating that the experts in this study were more likely to
use top-down reasoning when analyzing arguments while the novices were more likely to use
bottom-up reasoning. The frequencies reported in Table 4.21 clearly show that both experts and
novices in this study tended to use the breadth-first approach over the depth-first approach.
Table 4.22
Contingency Table for Reasoning Styles Used by Expert and Novice Groups
Group
Top-down
reasoning
Bottom-up
reasoning Row Totals
Novice 39 46 85
Expert 84 27 111
Column Totals 123 73 196
Experts Use of Two Strategies to Analyze and Construct an Argument Map
Strategy 1: Categorize claims into groups. The experts identified the overall
characteristics of claims and clustered them into two groups – a reasoning process that was
unique to the expert group. For instance, Expert 1 recognized that the set of claims in fifteen
nodes could be categorized into two groups, principles and technical methods, to implement the
principles. He tried to distinguish from the broad concept and to particular techniques in order to
c 2(1,N =196) =18.281 p < .0001
131
identity the levels of each claim. Expert 2 recognized that the set of 15 claims could be
categorized into two sets of statements: methods and outcomes, with the methods used to achieve
the desired learning outcomes. Also, Expert 2 identified independencies or dependencies
between claims:
[10:05] The question is, do I switch, because that's the technique, not a concept?
Because these personalized-- if I start from the bottom, these are all techniques. That's a
technique - exclude gratuitous sound, remove word-for-word, audio narration and
exclude gratuitous text and exclude gratuitous visuals. So those are all techniques.
(Expert 1; categorized claims into “Concepts” and “Techniques”)
[3:40] Okay. So. The first thing to do is to move the..basically what we have here it looks
to me like..
What we have here are two sets of statements. We have one set of statement they've put
over here and these are the methods that you use to achieve--This one needs go over here
as well-- to achieve these results.
So… um…. Basically, what I'm doing now-- what you have over here are the methods
that you might use to achieve these results, which make learning easier.
(Expert 2; categorized claims into “Methods” and “Results”)
[1.59] It seems like some of the nodes tell you what benefits come from using
multimedia learning, and some of the nodes tell you ways to do better job at increasing
learning using multimedia. Those seemed to…some tell you worse ways and so on.
Those seemed like two different kind of things. One is supporting this conclusion and
another is, how to accomplish the conclusion or what are the goals stated in the
conclusion that use of multimedia increases learning. One is about best practices and one
is about expected benefits types of groups.
(Expert 3; categorized claims into “Best practices to achieve goals” and “Expected
benefits”)
Strategy 2: Elaborative questions to make inferences. While structuring the argument
map, the experts examined the overall level of each claim. They asked whether the claim is about
a general/broad concept or a specific example/technique and then positioned the claim near the
top or bottom of the screen. This overall structure was accomplished by identifying overall
associations or level of a claim relative to the main conclusion. The experts identified the claim’s
132
level relative to another claim in order to decide which claim was subordinate to the other.
Another important question the experts posed to themselves was how particular claims depended
on the support of two or more prior or subordinate claims. Lastly, the experts also examined and
questioned the relevance or irrelevance of a claim to the overall argument.
Experts Also Committed Reasoning Fallacies
To identify the types of reasoning fallacies made by participants, I observed all
participants’ final maps and compared them to the criterion map. I reported the frequencies of
each reasoning fallacy made by each participant in Table 4.23. All the novices in this study
(except for Novice 2) leapt to conclusions (did not correctly identify the mediating claims that
connect a low level claim to the main conclusion), made erroneous associations between nodes
(connected a node to the wrong parental/antecedent node), reversed causation (swapped the
cause and effect), and established insufficient cause (identified a single reason when multiple
reasons were needed to support a claim). Novice 2, on the other hand, connected all 14 claims
directly to the main conclusion to produce an argument diagram with only two levels, without
making errors in association or mistakenly reversing causation between nodes. In contrast, all the
experts in this study committed two reasoning fallacies - leaping to conclusions and making
erroneous associations, with Expert 2 committing reversed causation and establishing
insufficient cause.
133
Table 4.23
Frequencies of Reasoning Fallacies Observed in Final Argument Maps
Novices Experts
Reasoning fallacies 01 02 03 04 05 E01 E02 E03 E05
Leaping to
Conclusions 1 10 7 9 3 2 4 2 1
Irrelevance 4 N/A 1 3 7 2 1 1 4
Reversed Causation 1 N/A 1 1 1 1
Single Cause 1 5 3 3 1 1
Circular Reasoning 1
Identify the
distraction claim N N N N N Y Y Y Y
Identify the negative
association Y Y N N N N N Y N
Identify the main
conclusion N Y Y N/A Y Y Y Y Y
Expert 3 failed to identify the correct link between N5 (Increase selective attention) and
N2 (Help encode into long-term memory). Instead Expert 3 added N6 (Reduce overall cognitive
load) between N5 and N2 as a mediating factor. As a result, Expert 3 failed to correctly identify
N5’s connection to two of its subordinate nodes, “N7 (Personalize communication) and N11
(Add asynchronous audio). He concluded, for example, that N7 was not related to N6 (reduce
overall cognitive load) and instead, directly linked N7 and N11 (as subordinate claims of N5) to
the sub-conclusion N2, which was a leap to a conclusion. Table 2.24 and Figure 4.6 present the
mapping actions, verbal actions, and argument map of Expert 3, all of which illustrate the
moment when Expert 3 committed this reasoning fallacy.
134
Table 4.24
An Example of the Leap to the Conclusion Fallacy (Expert 3’s mapping and verbal action script)
14:04 Add a link on N12
to N5
MC
Click N7 IACE Personalize communication, I know that they said that
that helped but I can't remember the exact reason. I
suppose probably selective attention [silence]
Move N7 next to
N5
PN
Add a link on N7 to
N5
MC
RERRO I don't think that it really reduces cognitive load, so I am
going to-- I'm not sure if this is correct but
Delete a link on N7
to N5
DL
Move N7 under N2 PN
REASO I'm going to move it into to help encoding long term
memory.
For reasons to be independent to on cognitive load.
Table 4.25
An Example of the Leap to the Conclusion Fallacy (Expert 5’s mapping and verbal action script)
14:00 Click N5 IACE Note #1. Increasing selective attention should also
help to-- let's see, it should probably help to reduce
the overall cognitive load, I would imagine.
IACE So we'll go ahead and tie that up with that. It should
help to lower the cognitive load and help encode into
memory.
14:11 Add a link on N5
to N6
MC
reposition N7 to
south
IACE So personalizing communication, that seems to be not
related to the cognitive load at all, but it might be
something that would help to encode into long term
memory. If the student remembers the agents, it's like
likely to give the negative sort of feedback, but that's
not here.
Add a link on N7
to N2
MC So I'll put that there to connect that up.
136
The same reasoning fallacy (leaping to conclusions) was observed when Expert 5 initially
connected N5 to N6 using causal reasoning (see note #1 in Table 2.25). After inserting an arrow
from N5 to N6, Expert 5 analyzed N7 (use of personalized communication using a pedagogical
agent) and concluded that N7 did not relate to N6 (reduce overall cognitive load). For that
reason, Expert 5 directly linked N7 to N2 (help encode into long-term memory, the sub-
conclusion).
In addition to these observed reasoning fallacies, the task of constructing the argument
map also included three important tasks: identifying an irrelevant claim as distraction claim,
identifying a negative claim with an inverse relationship to its parent claim, and identifying the
main conclusion of the argument. All experts pointed out the irrelevance of claim N14
(Matching media to student learning style will increase learning) to the overall argument
whereas all novices failed to identify this claim as irrelevant to the main conclusion. The
reasoning of the experts seemed to be based on a review of the available evidence presented in
the article the participants read and reviewed prior to constructing the argument maps, whereas
the reasoning of the novices was based more on personal belief regarding the effects of media on
student learning style.
[13:00] I'm ignoring up here ‘match media to student's learning style’ because that's just
one, conceptually doesn't seem to fit.
[19:23] I would almost toss 14 out, though. I don't feel strongly about that.
(Expert 1)
[10:21] I don't know what the heck this does. ‘Match media to student learning style.’ I
don't understand why that is. I'm going to just leave this one out for now. That seems to
be,‘match media,’ that seems, you could see that as a conclusion. It seems more of the
motivation for this whole thing.
[15:00] So this one, ‘match media style’—I don't know where this one goes. I might
have to leave this one for now. I think ultimately he may not find a place.
[23:40] Match-- I don't know what to do with 14.
[28:21] Then, ‘match media style to learning style’—I don't know what to do with that.
That just seems to me not especially helpful to the overall argument.
(Expert 2)
137
[9:20] I'm not sure how to understand ‘match media’ this time. So I'm just going to leave
it. That could just be made broader than I first understood it. I've read in the study and
there's no such thing as learning style, because they haven't been able to—some people
learn better in auditory and some are better with visual. But there's not much evidence for
that.
[35:20] Only one I'm not sure about is 'match media to student learning style' because
understood them one way. I have read empirical studies that indicate that there is no such
thing as learning style. But I guess you could understand it more broadly to say using
media helps encode long term memory.
As long as you don't try to only give visual stuff to visual learners, or something like that.
I'm not sure how narrowly you can understand that... need to understand that, so I'll just
leave it there.
(Expert 3)
Finally, all experts correctly identified the main conclusion, N11 The use of multimedia
increases learning, while two of the five novices failed to correctly identify the main conclusion.
Identifying the negative association seemed to be the most difficult task for both novice and
expert groups since only three participants correctly identified the negative association between
Claim N1 and N4.
Summary of Main Qualitative Findings
The following is a summary of the main findings drawn from the qualitative analysis of
data derived from the video recordings of the argument diagram construction process:
1. The participants exhibited five global actions to analyze and construct the argument map.
Their five global actions of argument diagramming were: 1) Scan claims; 2) Identify the
main conclusion; 3) Structure the argument; 4) Review; and 5) Correct reasoning
fallacies. All experts performed the five global actions at higher levels than the novices
whereas the novices performed all or some parts of the actions at a lower level.
2. The experts tended to use top-down reasoning processes whereas the novices tend to use
bottom-up reasoning to structure the claims in their argument maps.
3. With regard to the searching approach, all participants used breadth-first searching
approach as opposed to using a depth-first approach.
138
4. The experts used elaborative argument analysis approach prior to inserting links between
the nodes – an approach that start first by categorizing the claims into groups, then
identifying an overall level of a claim, the dependencies between claims, causality, and
irrelevance between claims.
5. Although the experts did commit some reasoning fallacies as did the novices, the experts
all together tended to make fewer leaps to conclusion, associations with irrelevant claims,
reversed causation, and single cause fallacies.
139
Outliers and Limitations
Originally, five experts in the fields of argumentation and philosophy participated in this
study. Experts 3 and 5 experienced difficulty in understanding the argument mapping task due to
the imperative tone used some of the claims. For example, Expert 3 stated “Especially in
philosophy - in our field - if you put it into an imperative tone, you're not stating something that's
true or false. If I say, ‘Close the window.’ True or false? It's almost like asking a question. (…)
But if you say, ‘This helps with encoding long-term memory.’ That could be true, or that could
be false. (…) When I first saw this, some of these were put in the imperatival tone, the
grammatical category of imperatives, increased selective intention, reduce overall cognitive load,
Use of personalized communication skills, That to me, sounds like, ‘This is what you should do.’
But saying what some should do is not the same as describing a fact.”
Because Expert 4 showed a high level of frustration and discomfort with the jMAP
argument task, he asked frequent questions that led to my active involvement and frequent
interventions during the mapping process. As a result, I decided not to include his argument map
and verbal reports in the analysis. Even though Expert 3 stated that he had misunderstandings
about the task, he voluntarily proposed to re-construct his argument map. After reviewing his
mapping processes before/after interview, I decided to include his map due to my minimal
involvement during the construction of the map and due to his superior analytical skills.
140
Table 4.26
Summary of Quantitative and Qualitative Findings by Research Question
Quantitative Results/Findings Qualitative Results/Findings
Research
Question
#1
Experts reasoning Novices reasoning
1. RCRCIAPNRC
2. IACEMCIACEREASO
3. REVIEWREVIEW
4. DELETEDELETE (limited)
1. RCRC
2. IAPNIAMC
3. IACEMCIACEREASO
4. REVIEWREVIEW
5. MCMC (limited)
1. All experts performed the five
steps whereas some novices
performed parts of the five steps.
2. Experts tended to use top-down
reasoning whereas novices tended
to use bottom-up reasoning.
3. All participants used breadth-first
approach.
4. Experts performed more
elaborative reasoning skills than
novices.
5. All participants committed
reasoning errors.
Research
Question
#2
Experts’ unique reasoning Novices’ unique reasoning
1. RCRCIAPNRC
1. IAPNMCIA
Research
Question
#3
High performers’
unique reasoning
Low performers’
unique reasoning
1. RCRCIAPNRC
2. IACEMCIACEREASO
3. REVIEWREVIEW
4. DELETEDELETE (limited)
1. RCIAMC
141
CHAPTER 5
DISCUSSION
I started this study with two questions. How do people reason? What are the differences
between good reasoners and bad reasoners when they analyze a complex argument? To answer
these questions, I explored the argument mapping and reasoning processes used by expert and
novice reasoners, identified the differences in mapping and reasoning processes between the two
groups, and finally identified the mapping and reasoning processes associated with the high
quality argument maps. Using sequential analysis and qualitative analysis, I arrived at the
findings and results presented in Table 4.26.
In this chapter, I discuss the findings of this study according to the three research
questions and interpret the findings in light of the relevant literature. Then, I outline the
implications of the findings for educators who teach reasoning and critical thinking, as well as
the potential impact for students in higher education who lack reasoning skills. Lastly, I discuss
the limitations in the study and conclude with future recommendations for teaching reasoning in
higher education.
Research Question #1.
What Reasoning Processes do Experts and Novices Perform When Diagramming a
Complex Argument?
Overall, the state diagrams reveal that neither the experts nor the novices exhibited any
patterns that consisted of large and long sequences of five or more actions. Instead, the diagrams
comparing the top two experts versus the low performing novices revealed patterns in sequences
that consisted of two to four actions at most, with both experts and novices revealing roughly the
142
same number of patterns in the processes used to construct their argument diagrams. This latter
finding is consistent with the first reported qualitative finding – the finding that both the experts
and novices exhibited all or as many as five global actions to analyze and construct the argument
map (scan claims, identify main conclusion, structure the argument, review, and correct
reasoning fallacies).
The following is a description of each of the processes revealed by the experts and by the
novices, and a discussion of the meaning or significance of each observed process in relation to
its overall purpose/function and in relation to the findings from the qualitative analysis of the
argument mapping process.
Expert Processes. The expert process for positioning nodes was Read Claim Read
Claim Identify Assoc Position Node Read Claim. Experts’ positioning of nodes was
preceded by two cognitive actions: 1) comprehension of a claim; and 2) identification of a
claim’s overall association. This process indicates that the experts used this process to create an
initial portrayal or rough approximation of the argument structure by using this process to
iteratively position related nodes in close proximity based on the identified associations between
given nodes before inserting links between any of the nodes (as reported in the qualitative
finding #4). This iterative process of positioning and/or re-positioning nodes suggests that the
experts were are to: 1) recognize the overall complexity of the relationships between claims; and
2) take a flexible attitude toward the initial positions of nodes knowing that the initial positions
of the nodes are likely to change when examining and taking other premises into consideration.
For these reasons, experts tended to position associated premises in close proximity to one
another without inserting links to connect the premises.
143
The expert process for linking nodes was Identify Cause-Effect AssociationMake
ConnectionIdentify Cause-effect AssociationReason. Experts’ insertion of links between
two nodes was preceded by identifying the cause-effect association. Once the experts inserted a
link between two nodes based on the identified cause-effect relationship, the experts again
identified the cause-effect relationship and provided a verbal explanation of the noted casual-
effect association. This process shows that the experts were able to explain the reasons for
inserting specific links between nodes, and explain why it is that the experts were able to
produce more accurate argument diagrams (as reported in the qualitative finding #4). However,
the process of inserting connections and then identifying the cause-effect relationship between
the nodes may not be a process that experts inherently use to create their argument diagrams, but
instead, may be simply a behavior that was produced by the task demands of participating in the
think-aloud protocol where the participants were asked to explain their actions.
The expert process for reviewing nodes was performed in an iterative process as the
experts reviewed the chain of reasoning after adding a new node to an existing chain of linked
nodes. The qualitative observations (qualitative finding #1) revealed that experts reviewed the
links within a particular chain of nodes (starting from the lowest level node to and through the
chain of premises leading to the main conclusion) prior to adding a new node to an existing chain
of nodes. The experts then followed this action by conducting a final review of the entire chain
of premises leading up to the conclusion. Overall, this iterative process of reviewing the
connections between nodes suggests that expert reasoners take a deliberate and analytic approach
to making inferences between premises and conclusions.
The expert process for repeated deleting links was observed primarily in the actions of
one and only one expert. This expert constructed an initial argument map but then re-constructed
144
the entire argument map after realizing that he had misunderstood or misinterpreted the
requirements and purpose of the argument mapping task. Overall, the deletion of links rarely
occurred across the four experts because they were very careful and deliberate in the process of
inserting the links between nodes. The experts tried to clearly identify the causal relationships
between nodes before they inserted links between nodes. This deliberate and careful approach
led them to use the delete action on a very infrequent basis.
Novice Processes. The novice process of positioning nodes was preceded by only one
cognitive action - identifying an association. Novices identified the association between two
nodes and then positioned the node on the map to convey the association of one node to another
node. The qualitative observations revealed that some novices performed this process in a fast
and intuitive manner without pausing to carefully reflect on the precise nature and/or accuracy of
the association between two nodes. One plausible explanation for this observed behavior among
the novices is that the novices may not have been fully aware of and/or did not anticipate the true
complexity in the relationships between the premises and conclusion when analyzing the given
argument (or any argument in general). This process of positioning nodes by novices was
reported in the qualitative finding #4.
The novice processes for linking nodes were identified across three different processes.
Novices inserted links between two nodes after: 1) identifying an overall association and
positioning a node; 2) identifying cause-effect association; and 3) inserting a link between two
other nodes. Of these three processes used to insert links between nodes, the last process (the
iterative process of inserting links between nodes) was exhibited in only one of the five novices.
This particular novice positioned all the nodes in the diagram first and then inserted links
145
between the nodes to complete the argument map. For this reason, the repeated linking process
will be excluded from further discussion because this process was the product of one outlier.
The novice process for reviewing nodes appeared infrequently among the five novices.
Novices reviewed the previous chain when they added a new node but limited their review to one
chain and did not review the links up the chain to the final claim or conclusion. For example, if
they added a new node to the 5th level, then they reviewed the 4th 3rd nodes but not the entire
chain of links from 4th 3rd 2nd the main conclusion. One plausible explanations as to
why the novices did not thoroughly review the chain of reasoning is that the novices’ may not be
aware that the veracity of the conclusion relies on the veracity of each and every premise along
the chain of premises.
Research Question #2
What Differences Exist in the Reasoning Processes Used by Experts versus Novices?
Overall, the experts exhibited four reasoning processes used to position nodes, link
nodes, review linked nodes, and delete links between nodes. In contrast, the novices exhibited
five processes used primarily to read/identify claims, link nodes, review linked nodes. Omitting
the processes that were observed solely in one expert (iterative deletion of links) and solely in
one novice (iteratively making connections between nodes), the experts exhibited a total of three
processes whereas the novices exhibited a total of four processes. Of the three processes
exhibited by the experts, only one of the processes (iterative process used to positioning nodes)
was not exhibited and was different from those exhibited by the novices. Of the four processes
exhibited by the novices, only one of these four processes was substantively different from those
exhibited by the experts (the iterative process used to position nodes and inserting links between
nodes). As a result, the main difference between the processes used by experts versus the
146
processes used by novices was that the experts used an iterative process to position nodes,
whereas the novices used an iterative process to both position and link the nodes. The experts
positioned nodes by first reading the claims and identifying their inter-relationship before
positioning the nodes based on their identified relationship. Contrary to the experts, the novices
did not position nodes iteratively. Instead, the novices positioned a node and then immediately
inserted a link between the nodes. These noted differences in process and their meaning and
significance/implications are discussed in the next section under research question 3.
The phi coefficient tests conducted in this study provided some measure of association
between a particular two-action sequential pattern and group membership. For the expert
groups, the 1) Read Claim Identify Assoc., 2) ConnectionIdentify Cause-effect Assoc., and
3) Delete Delete patterns were found to be associated with experts’ action sequences (but in a
weak manner or to a small degree). However, none of these associations appear to lend
additional support for the two observed differences in the reasoning processes – processes that
consist of three or more sequences of actions. However, it must be noted that the phi coefficient
test only examines group association for actions pairs as opposed to testing and examining group
association for longer sequences of actions that more fully capture the processes of reasoning
used by the experts and the novices. As a result, the phi-coefficient tests and results were overall
considered to be not relevant or of central importance in this study.
Research Question #3
Which Processes Might Help Produce Diagrams of High versus Low Accuracy?
High Performers’ Unique Reasoning Processes.
The sequential patterns exhibited by the two highest performers provide some
suggestions as to what processes might help to produce diagrams of higher accuracy. The two
147
highest performers produced reasoning processes that were identical to three patterns observed
across all four experts: 1) the expert process for positioning nodes, 2) the process for linking
nodes, and 3) the process for reviewing nodes.
Why does the expert iterative process of positioning nodes help them to produce a more
accurate argument map? One possible reason is that experts are able to identify semantic
relationships by identifying similarity, comparison, inclusion and abstraction. As a result,
experts can easily organize complex claims into groups and better recognize and analyze their
hierarchical relationships. In other words, grouping claims by placing associated claims in close
proximity can help experts tease out their complex hierarchical relationships and produce more
accurate argument diagrams. Early in the mapping process, it was observed that the experts
verbally stated that they needed to position and reposition nodes until they believed that the
identified the final and correct position of each nodes in the process of working out the
hierarchical relationships between all the nodes. The use of iterative process of positioning nodes
enabled the experts to change the relationships between the nodes quickly and easily (without
having to delete and re-insert links between nodes) and hence enabled them to more efficiently
think through all the possible relationships between the nodes in the process of identifying the
correct links/connections between the nodes.
With regard to the process for linking nodes, the experts examined the cause-effect
relationship between two claims (e.g., B is the result of A, A helps B happen) to make decisions
on when to insert a link between two claims. This process helped experts to produce a more
accurate map because it required the experts to make more explicit their explanations and
justifications for inserting a link between two nodes (e.g., identifying a missing premise that
completes the connection between two premises).
148
After adding a node to the existing chain of nodes, the high performers reviewed the flow
of reasoning from the lowest node up to the main conclusion. This iterative and extended review
process helped them to detect reasoning fallacies (e.g., leaping to conclusions), refine the
hierarchical relationships between the nodes, and as a result, create more accurate maps.
Low Performers’ Unique Reasoning Process
One specific process for linking nodes (Identify Association Connect) appeared to be a
unique process used only by the two lowest performers based on the results of the phi coefficient
test. The two novices showed a tendency to immediately insert a link between claims once they
identified their association (e.g., A is related to B. or A is supporting B) without making more
explicit the precise cause-effect relationships between the two claims. How is it that this
particular process of linking nodes might hinder students’ ability to produce higher quality
argument maps? One possible reason is the novices (unlike the experts) may not have
recognized the true complexity of the given arguments (or other arguments in general). And as a
result, the novices were too quick and premature in inserting the links between claims instead of
taking the necessary time to position and re-position related claims in close proximity to
thoroughly explore all their possible relationships. Furthermore, adding links prematurely can
make it more difficult and discourage the novice from moving and repositioning nodes to correct
for logical fallacies because the process of changing the position of nodes (and their associations
with other nodes) requires students to delete the outdated links, re-insert new links, and/or re-
route existing links so that the links are not placed on top of other nodes and visually obstruct the
nodes.
149
Qualitative Findings and Discussion
Even though the sequential analysis provided indications of mapping actions that serve as
potential indicators of some of the reasoning processes identified in the qualitative analysis,
some important actions, such as identifying the main conclusion, recognizing reasoning errors,
identifying the independencies/dependencies between premises, and identifying irrelevant claims,
were not revealed in the results of the sequential analysis because these particular processes of
reasoning occurred at a very low frequency. For example, identifying the main conclusion likely
occurred one and only one time for each participant. Likewise, finding the irrelevant claim
(since there was only one irrelevant claim in the argument) also likely occurred one and only one
time. Furthermore, only one negative relationship was present in the criterion argument map. In
addition, five other reasoning processes that are perhaps more global in nature could not be
revealed from the sequential analysis of the mapping actions (at least not from the coding
scheme developed and used in this current student to code the mapping actions). From the
qualitative analysis, this study identified the following five reasoning processes.
1. Five-step to Argument Map: The experts performed five stages in the process of
constructing their argument maps: 1) read all claims, 2) identify the main conclusion; 3) structure
the map; 4) review the map; and 5) recognize reasoning errors and make revisions. Overall, this
five stage process is overall very similar to the seven-stage process identified by Scriven (1976).
2. Top-down versus Bottom-up: Second, experts tended to use a top-down reasoning
process whereas the novices tended to use a bottom-up reasoning process while analyzing and
constructing the argument maps. In particular, three experts out of four dominantly used top-
down reasoning and two novices used bottom-up reasoning (one novice used top-down, one
novice used both top-down and bottom-up, and the other novice’s processes did not fit into either
150
category). This result is consistent with the previous findings that show that the use of top-down
reasoning is one of main characteristics of experts (Cross, 2004).
3. Breadth-first Approach. Third, with respect to the types of searching processes, the
experts and novices used ‘breadth-first’ strategy over ‘depth-first’ strategy.
4. Sophisticated Reasoning Skills. Fourth, experts demonstrated more sophisticated
questions and strategies used to analyze relationships, posing questions like “What is the overall
level of this claim “, “What is the claim that supports the major premise?” “Are they supporting
the ordinate claim by working together (dependently as co-premise) or are they independent
reasons?”, “Is this claim relevant to the main argument?”, “Do they have a negative relationship
or a positive relationship?” “Is the relationship between claims strong or weak?” These types of
questions demonstrate how the experts used various analytical approaches to reason through and
to identify the associations between claims. Although some novices exhibited more
sophisticated reasoning skills than other novices, the novices overall demonstrated a lower level
in reasoning ability in terms of the types of questions that the experts posed to themselves. For
example, novices 2 and 4 used an intuitive and fast processing approach to produce a low quality
map. Novices 3 and 5 on the other hand exhibited some of the analytical questioning and
processes like the experts but were not able to carry through with the process to correct for errors
in their maps. Conversely, some novices were able to detect errors in their map, and yet were
unable to go through the proper questioning process to make corrections to the noted errors.
5. Reasoning Fallacies for Everyone. While the expert group performed better at
analyzing and constructing the argument map than the novice group, both experts and novices
committed fallacies in their reasoning. Previous research has shown that reasoning fallacies are
not associated with specific reasoning processes, neither deductive nor inductive reasoning
151
(Neuman, 2003; Ricco, 2003). Instead, reasoning fallacies are associated with the reasoner’s
level of content knowledge, especially argument from ignorance, circular reasoning, and slippery
slope (Hahns & Oaksford, 2007). Reasoning skills are also associated with factors such as
general cognitive ability (Weinstock, Neuman, & Glassner, 2006; Svedholm-Häkkinen, 2015) or
educational level. This study selected participants who were largely unfamiliar with the content.
So as expected, experts sometimes linked their claims without identifying the mediating claim to
the main conclusion (indication of leaping to conclusion and slippery slope fallacies) and they
linked claims (very technical and specific claims) to incorrect super-ordinate claims. However,
the experts in this study committed few errors in circular reasoning, and few errors in reversed
causation, and single cause fallacies.
On the other hand, the reasoning novices tended to also commit the fallacies of leaping to
conclusion and slippery slope as well as reversed causation due to the lack of content knowledge.
In addition, the novices committed the fallacies of irrelevance and single cause. The question
now is exactly how and why are these observed reasoning fallacies associated with “lack of
content knowledge” and/or “lack of reasoning skills”? Previous research has found that lack of
content knowledge is associated with specific reasoning fallacies, but does not identify nor
explain how lack of content knowledge produces deficiencies in the mapping and reasoning
processes that produce these specific fallacies in logic. It would be helpful to diagnose how lack
of content knowledge influences and changes the reasoning processes in ways that lead to logical
fallacies. By doing so, appropriate instruction and interventions can be presented to students to
ameliorate the negative effects of deficiencies in content knowledge.
Another interesting finding was that the experts tended to reason based on evidence
whereas the novices tended to reason based on personal belief. In this study, the participants
152
were presented a distraction claim that was not true (yet was highly believable and conceivable),
and not relevant to the main argument. In this study, the five novices included and linked the
distractor claim (“matching media to students’ learning style increases learning”) to node(s) in
the argument map without question. On the other hand, the experts raised questions about the
validity of the claim and correctly identified this claim as irrelevant to the main argument. These
findings suggest that the novices tended to accept the validity of a claim when the claim matched
their own belief system, whereas the experts demonstrated their ability to examine the distractor
claim based on logic as opposed to personal beliefs. Findings from prior studies has shown that
the bias from personal beliefs does not appear to interfere with the reasoning processes for
people with high reasoning skills, whereas the effects of personal beliefs is strong for people
with inadequate reasoning skills (Svedholm-Häkkinen, 2015). Given that nearly all the
participants in this study were not content experts in multimedia and instructional design, content
knowledge does not appear to be a factor in explaining the differences in logical fallacies
committed by the experts and novices. Instead, the observed differences appear to be associated
with differences in their skills with the processes of reasoning. From the dual-process
perspective, the experts in this study examined the claim’s association based on the use of logic
rather than personal belief due whereas the novices (with their weaker skills and experience with
reasoning) appeared to have relied on using more intuitive reasoning processes that rely on
personal beliefs – intuitive processes that were more prone to producing fallacies in logic.
153
Instructional and Software Design Implications
The findings in this study suggest that examining the reasoning processes of experts and
novices at both the micro level (mapping actions) and global level (reasoning strategies) can help
us identify the types of processes that can potentially help students’ produce higher quality
argument maps and achieve deeper understanding of complex arguments. The findings in this
study suggest that instructors should encourage students (when diagramming arguments) to
apply the following procedure: 1) use an iterative process of positioning nodes while carefully
examining the cause-effect relationships between nodes; 2) iteratively review links between
nodes across each chain of linked nodes; and 3) insert the links between nodes once their
hierarchical relationships have been examined. For example, students can be instructed at or
near the beginning of the mapping process to place associated claims in close proximity to one
another based on their careful review of the cause-effects relationships, but wait to insert links
between the nodes at a later time so that the nodes can be easily re-positioned as new
relationships are discovered. Once the nodes are placed in their desired location, students can
then be instructed to carefully and iteratively review the chain of links connecting the lowest
level premises up to the main conclusion while making explicit to students the common types of
fallacies they should look out for in their argument maps.
Although these three processes can also be used to facilitate the analysis of complex
arguments in other contexts that do not involve the use of diagrams (e.g., group discussions,
written essays), using these processes to construct argument diagrams is particularly important
for the following reasons. Diagrams are used heavily in industry to conduct root cause analysis
because it enables industries and organizations to chart out the multitude of variables and events
that contribute to breakdowns in process, and to identify the root causes for the breakdown so
154
that solutions that can be identified to directly address the root causes and not the symptoms of
the problem. Most of all, argument analysis is a highly complex process that can be learned
much more effectively when given sufficient practice and immediate feedback (Twardy, 2004;
van Gelder, 2001). One way to provide more practice and feedback is to embed these processes
directly into argument mapping software so that the software can scaffold these processes as
students are creating argument diagrams, diagnose the processes used by students in relation to
the desired/target processes, and provide immediate prompts and individualized feedback to
guide students through the map construction process.
To create argument mapping software that can scaffold these three processes, students
can be encouraged and/or required to use the iterative process of positioning nodes by disabling
the link insertion function in the software and making the linking function available only after a
student has positioned all or a subset of nodes. Second, the mapping software can be designed to
detect whether the student has positioned the outcome node near top of screen and monitor how
many times each node has been moved and/or re-positioned to indicate to what extent the
students is actively exploring and revising the node positions and their hierarchical structure.
Once the student feels that all the nodes have been placed in the desired location, the software
can provide a one-step function that automatically inserts links between all the nodes based on
each node’s proximity and relative position to other nodes. Before this auto linking function is
executed, however, the system can prompt the student to spend more time evaluating the
hierarchical relationships between the nodes if the system detects a deficiency in the number of
times the students executes the iterative node positioning process/action. These types of
software functions can be used and systematically manipulated to conduct controlled
155
experiments to test the effects of each specific process on students’ understanding of complex
arguments.
156
Limitations of the Study
Although this study provides insights into the mapping and reasoning processes used by
experts and novice graduate students, the findings are not conclusive because this study
employed an exploratory design rather than a controlled experimental design. As a result, the
cause effect relationship between the observed processes and map quality cannot be verified
using the data drawn from this study. Another limitation of this study is that its findings cannot
be generalized to the larger population. The targeted participants were experts who teach
argumentation courses and novice graduate students with limited argumentation experience. As a
result, the processes observed in the novices in this study may not be processes observed in
novices across other student populations (e.g., undergraduates, high school students).
Furthermore, the participants in this study were skewed by gender in both the expert (100%
male) and novice (80% female) groups. However, there is little evidence to indicate that gender
difference exist in the reasoning skills of college students (Kuhn, 1992). Prior content
knowledge, on the other hand, has been found to affect reasoning processes (Hahns & Oaksford,
2007). Due to the fact that expert 1 reported the highest level of content familiarity, there is the
possibility that expert 1’s reasoning processes may have skewed the reasoning processes found
in the expert group. However, expert 1 scored third highest in map accuracy even though he was
scored highest in content familiarity. Given these circumstances, any unintended effects from
uneven gender between groups and the levels of content familiarity were likely to be minimal in
my study, particularly given the overall consistency in the patterns of actions observed across the
four experts in this study.
In addition, the participants in this study used a computerized mapping tool, jMAP, to
analyze and construct their argument maps given 15 claims extracted in advance by the
157
researcher (not the participants). For this reason, the participants’ reasoning processes may have
been constrained by this particular aspect of the task design. As a result, this study’s findings
may not be applicable or generalizable to contexts where participants are required to perform the
entire argument analysis process (including the process of extracting and identifying the meaning
of premises) as described by Scriven (1976). For example, the premises in jMAP were not only
extracted in advance by the researcher, but they were also randomly arranged on the initial jMAP
screen. If the participants had been asked to extract the premises from the article on multimedia
principles, the participants would be able to identify each premise in immediate context to any
other related premises and conclusions stated and described within the article. As a result, having
the participants extract the premises directly from the article might simplify the process of
identifying the associations between premises so that less time is spent on positioning and re-
positioning nodes to explore their possible relationships.
Another limitation of this study was the presence of individual differences in the
participants’ ability to verbalize their thought processes. Even though all participants were
native English speakers, some participants verbalized their thought processes less frequently than
other participants. Some participants appeared to be unable to simultaneously talk out loud
while analyzing and constructing their argument maps even with frequent prompting from the
researcher. Sometimes, participants merely reported what they were doing (identifying their
actions) instead of presenting an explanation and justification for their actions. At this time, it
cannot be determined whether their difficulty in verbalizing their thought processes can be
attributed to their ability to apply reasoning skills or the unfamiliarity with the talk out loud
protocol.
158
Finally, the format of the sentences used to describe each premise within each node
should be stated in declarative form, not in imperative form. In this study, several sentences
started with a verb. From the interviews with two of the experts in argumentation, using
commands or imperative tones can hinder the argument analysis process and, can in fact, confuse
students. These imperative sentences confused two experts and due to this confusion, one of
these two experts (the one expert that was omitted from the data analysis) was not able to
complete the argument diagramming task without excessive prompting and guidance from the
researcher. Even though other participants did not report this problem, this could be a possible
threat to the validity of the findings reported in this study.
Limitations of Sequential Analysis
Sequential analysis requires a minimum number for the cell frequencies in each action
sequence. For this reason, if a marginal cell frequency is less than 5, sequential analysis cannot
detect it as an important pattern. Along the same lines, the critical p-value of .05 used in the
sequential analysis in this study to test for all possible action sequences was chosen because this
study was exploratory in nature. Perhaps a more conservative p value with increase in sample
size could be used to more accurately determine and identify differences between the specific
processes used by the experts versus the novices in future studies. Also, the phi coefficients
reported in this study only examined the magnitude of the difference between specific action
pairs, whereas the main focus of this study was to examine larger sequences of actions to identify
reasoning processes of 3 or more actions in a sequence. An alternative statistic is needed to
determine the magnitude of the differences in the frequency of two-, three-, and four-event
sequence exhibited by the experts and novices
159
Directions for Future Research
Most of all, no cause-effect relationships between the observed processes and
performance scores can be determined from the findings in this study because this study did not
use an experimental design. To test the effects of specific reasoning processes on performance,
different versions of an argument mapping software program can be developed to guide its users
in using different reasoning processes (e.g., top-down versus bottom-up) in order to test and
compare the effects of specific reasoning processes on map accuracy and student understanding
of complex arguments. Furthermore, the computerized argument tool could be programmed to
employ sequential analysis to compute and recognize what processes students are using and what
types of reasoning fallacies students are making to create the mechanism for delivering prompts,
feedback, and strategies that can help students produce accurate argument maps and achieve
more precise understanding of complex arguments.
Another recommendation is to conduct further research examining the relationships
between the mapping and reasoning processes identified in this study’s qualitative analysis of the
argument map construction process. Table 5.1 shows that in this study, mapping processes were
found to be potential indicators of three of the five main steps in the map construction process
(scan claims, construct argument, review) identified in this study’s qualitative analysis. As a
result, further research can be done to identify mapping actions associated with the other two
remaining steps in the map construction process (identify conclusion, correct fallacies). Future
studies can also refine the coding scheme developed in this study so that the sequential analysis
of mapping actions can: 1) detect, measure, and monitor students’ use of top-down, bottom-up,
depth first, and breadth first approaches used to construct arguments in step 3; and 2) determine
which reasoning processes tend to produce or not produce specific reasoning fallacies (e.g.,
160
leaping to conclusions). For example, the inclusion of a code into the coding scheme that
identifies the action of positioning one node below another node and the action of positioning a
node above an existing node will make it possible to examine to what extent participants are
using a top-down or bottom-up approach, respectively. Using this more elaborate coding
scheme, it may be possible to use sequential analysis to identify a broader range of approaches
that students are using and determine how these particular approaches affect the quality of
students’ argument maps and the frequency of particular fallacies in reasoning.
Table 5.1
Mapping Actions as Indicators of the Map Construction and Reasoning Processes
Expert Novice
1. Scan claims RC↔RC
2. Identify conclusion
3. Construct argument RC→RC→IA→PN→RC IA↔PN
IACE↔MC→REASON IA↔MC, IACE↔MC, MC↔MC
4. Review Review ↔ Review Review ↔ Review
5. Correct fallacies
161
Conclusion
As an educator and researcher in higher education, understanding student reasoning
processes is an important yet challenging task. The goal of higher education is to train students
to think critically and to use accurate reasoning skills to deal with complex arguments. The
findings in this study demonstrate that graduate students’ reasoning skills can vary in terms of
the strategies they employ (compared to processes used by experts) and the level of
understanding that can be achieved by employing specific strategies. As a result, educators need
to focus efforts on identifying students’ current reasoning skills and teaching more advanced
reasoning skills, in addition to teaching knowledge and content specific to their disciplines. One
way to improve students’ reasoning skills is to first identify, test, and model the reasoning
processes used by experts. Once these processes have been adequately tested and validated,
students reasoning skills can be improved by providing students with guidance on how to
perform these very processes. This guiding and modeling of processes can in the future be
implemented with computerized adaptive argument mapping software – software that can be
used for both diagnostic purposes (e.g., recognize students’ behavioral patterns and reasoning
fallacies, and then provide adaptive feedback to improve their performances) and instructional
purposes (i.e., to support and teach the use of specific processes). Although more evidence and
study are needed to fully identify and understand the reasoning processes of experts, this study
provides initial insights into how people reason when they are asked to create a complex
argument structure using a computerized mapping tool and also provides insights into the
potential use of sequential analysis as a means to identifying key processes of effective
argumentation.
162
APPENDIX A
PARTICIPANT’S PROFILE SURVEY
Your identification number is ________.
Please answer the following questions.
Age: ( )
Gender: (female, male)
Profession
I am currently (graduate student________ professor________).
If you are a student, how many semesters have you completed so far? ( )
If you are a professor, how many years have you been so? ( )
If you are a professor, what is your field of expertise?
Writing and reviewing journal experience
How many years have you been actively involved in writing journal articles?
How many articles have you published?
How many years have you served as a peer-reviewer for journal articles?
How many articles have you reviewed?
Are you familiar with the article ‘Six principles of effective e-Learning: What works and
why? (Ruth Clark) Please circle your familiarity of each principle using three scales (never
heard of it, know it some, very well know)
o Multimedia Principle (never heard of it, know it some, Know it very well)
o Congruity Principle (never heard of it, know it some, Know it very well)
o Modality Principle (never heard of it, know it some, Know it very well)
163
o Redundancy Principle (never heard of it, know it some, Know it very well)
o Coherence Principle (never heard of it, know it some, Know it very well)
o Personalization Principle (never heard of it, know it some, Know it very well)
Have you ever used the jMAP software to create a map? (Yes, No)
Have you used other mapping software to create a map such as concept maps or/and network
maps? (Yes, No)
164
APPENDIX B
SUMMARY OF SIX E-LEARNING PRINCIPLES
Original Article: Clark, R. (2002). Six principles of effective e-Learning: What works and why.
The e-Learning Developer’s Journal, 1-20. Retrieved from
http://faculty.washington.edu/farkas/HCDE510-Fall2012/ClarkMultimediaPrinciples(Mayer).pdf
The Multimedia Principle: adding graphics to words can improve learning.
By graphics we refer to a variety of illustrations including still graphics such as line
drawings, charts, and photographs and motion graphics such as animation and video.
Research has shown that graphics can improve learning. The trick is to use illustrations that
are congruent with the instructional message. Images added for entertainment or dramatic
value not only don’t improve learning but they can actually depress learning.
Mayer compared learning about various mechanical and scientific processes including how a
bicycle pump works and how lightning forms, from lessons that used words alone or used
words and pictures (including still graphics and animations). In most cases he found much
improved understanding when pictures were included.
Figure 1. e-Learning illustrating a biological process.
165
The Contiguity Principle: placing text near graphics improves learning.
Contiguity refers to the alignment of graphics and text on the screen. Often in e-Learning
when a scrolling screen is used, the words are placed at the top and the illustration is placed
under the words so that when you see the text you can’t see the graphic and vice versa. This
is a common violation of the contiguity principle that states that graphics and text related to
the graphics should be placed closed to each other on the screen.
Mayer compared learning about the science topics described above in versions where text
was placed separate from the visuals with versions where text was integrated on the screen
near the visuals. The visuals and text were identical in both versions. He found that the
integrated versions were more effective.
Figure 2. An example of application of the contiguity principle.
166
The Modality Principle: explaining graphics with audio improves learning.
If you have the technical capabilities to use other modalities like audio, it can substantially
improve learning outcomes. This is especially true of audio narration of an animation of a
complex visual in a topic that is relatively complex and unfamiliar to the learner.
Mayer compared learning from two e-Learning versions that explained graphics with exactly
the same words — only the modality was changed. Thus he compared learning from versions
that explained animations with words in text with versions that explained animations with
words in audio. In all comparisons, the narrated versions yielded better learning with an
average improvement of 80%.
Figure 3. Visual and supporting auditory information maximize working memory resources.
167
The Redundancy Principle: explaining graphics with audio and redundant text can hurt
learning.
Some e-Lessons provide words in text and in audio that reads the text. This might seem like a
good way to present information in several formats and thus improve learning. Controlled
research however, indicates that learning is actually depressed when a graphic is explained
by a combination of text and narration that reads the text.
In studies conducted by Mayer and by others, researchers have found that better transfer
learning is realized when graphics are explained by audio alone rather than by audio and text.
Mayer found similar results in two studies for an average gain of 79%. There are exceptions
to the redundancy principle as recently reported by Roxana Moreno and Mayer. In a
comparison of a scientific explanation presented with narration alone and with narration and
text, learning was significantly better in conditions that included both narration and text.
Figure 4. Presenting words in text and audio can overload working memory in presence of
graphics
168
The Coherence Principle: using gratuitous visuals, text, and sounds can hurt learning.
It’s common knowledge that e-Learning attrition can be a problem. In well-intended efforts
to spice up e-Learning, some designers use what is called a Las Vegas approach. In other
words, they add glitz and games to make the experience more engaging. The glitz can take a
variety of forms such as dramatic vignettes (in video or text) inserted to add interest,
background music to add appeal, or popular movie characters or themes to add entertainment
value.
In the 1980’s research on details presented in text that were related to a lesson explanation
but were extraneous in nature found them to depress learning. Such additions were called
“seductive details.” In more recent research, Mayer has found similar negative effects from
seductive details presented either via text or video. For example, in the lesson on lightning
formation, short descriptions of the vulnerability of golfers to lightning strikes and the effect
of lightning strikes on airplanes were added to the lesson. In six of six experiments, learners
who studied from the base lesson showed much greater learning than those who studied from
the enhanced versions.
Figure 5. A seductive detail from a quality lesson. From Clark and Mayer, 2002.
169
The Personalization Principle: Use conversational tone and pedagogical agents to increase
learning.
A series of interesting experiments summarized by Byron Reeves and Clifford Nass in their
book, The Media Equation, showed that people responded to computers following social
conventions that apply when responding to other people. For example, Reeves and Nass
found that when evaluating a computer program on the same computer that presented the
program, the ratings were higher than if the evaluation was made on a different computer.
People were unconsciously avoiding giving negative evaluations directly to the source.
Deeply ingrained conventions of social interaction tend to exert themselves unconsciously in
human-computer interactions. These findings prompted a series of experiments that show
that learning is better when the learner is socially engaged in a lesson either via
conversational language or by an informal learning agent.
Based on the work of Reeves and Nass, Mayer and others have established that learning
programs that engage the leaner directly by using first and second person language yield
better learning than the same programs that use more formal language. Likewise a number of
studies have shown that adding a learning agent — a character who offers instructional
advice — can also improve learning.
Figure 6. Jim serves as a pedagogical agent. With permission from Plato Learning Systems.
170
APPENDIX C
RETROSPECTIVE INTERVIEW QUESTIONS
1. Please share any thought you have in regards to your experiences with the argument
diagramming task.
2. Did you use any particular strategies to help you construct your argument map?
3. Can you describe the process you used to create your argument diagram? For example, what
was the first action you performed for the task? Why? What was the next action and why?
Describe how you went about identifying the relationships between the nodes.
4. What difficulties did you have while constructing your argument map? How did you solve
the difficulties you have had?
5. Have you read the article ‘Six e-learning principle before? YES NO
6. Please indicate on your argument map which principles you already knew prior to the start of
this argument mapping session. Please circle the nodes using blue pen.
7. Can you tell me which relationships in your diagram were difficult to identify and can you
explain why?
a. How did you identify the relationship between the premises?
b. Did you insert any links between premises by merely guessing their possible
relationships?
8. Were you able to think-aloud while constructing the map? Did you filter out certain thoughts
that you would or would not say? Why?
173
APPENDIX F
INVITATION EMAIL FOR THE SECOND CODER
Dear. Dr. ****
Thank you for accepting my invitation letter that requested you participating in my
research as a second coder. I will provide you 1) coding schemes, 2) Direction for coding and 3)
data files. Please read the direction and let me know if you have any questions. Thank you again.
174
APPENDIX G
CODING PROCEDURES FOR THE SECOND CODER
Objectives: Coding the verbal data using the given coding scheme.
This coding scheme was developed to identify the reasoning processes used in analyzing
argument and constructing an argument map using jMAP software. You have two files, excel
and video files. In the excel file, you can see the timeframe, map behavior, and verbal transcript.
In the video files, you can watch their argument mapping processes and hear their think-aloud
report. Since their verbal data were transcribed into the excel sheet, you just need to focus on
identifying the meaning of their utterances using the given category system. You will find that
some codes would not appear based on the data that you are dealing with.
The purpose of this coding task is to identify any reasoning processes involved in an
individual data. For this reason, you will not include some contents if they are not related to the
reasoning processes (E.g., the participant talked out loud “I will move this node because I want
to make it pretty, then this utterance will not be coded). You will focus on the verbal data that
represent individuals’ reasoning processes.
Read a direction Read the coding scheme Practice the coding examples Open the excel
file and video file at the same time and code the verbal data by thinking the meaning of verbal
report .
Sometimes you will notice that a participant reads claims with a particular purpose (e.g.,
searching for a claim that support the main conclusion, or major premise. In this case, you will
think the purpose of their behavior and code it as “searching for a claim” instead “read a claim”.
Sometimes a participant reads a claim in order to understand its content (reread it and they would
say I don’t understand what this meant). In this case, you can simply code it “RC, read a claim”.
Please watch video when you code think-aloud data so that you can understand what is going on
the time when the participant talked out loud.
Novice 03
https://www.youtube.com/watch?v=AuKafLZeEro&list=PLAZrSGvtmDs1nK5PmXnX
d7kSHRL22y-tt&index=4
Expert 05:
https://www.youtube.com/watch?v=t62Q1bjkZJw&list=PLAZrSGvtmDs1nK5PmXnXd
7kSHRL22y-tt&index=12
175
Code Meaning Examples in context
RC read a claim Read a claim - Encode; understanding of meaning
IMC Identify the main conclusion Identified the main conclusion
IL Identify a level of a claim Identify a claim's level and position it to top/bottom or right/left
PN* Position a node Move/position a node
IA Identify an association Noticed some associations (without saying Cause-effect). E.g., N3 and
N6 are related.
IACE Identify a cause-effect association Verbally state that N3 is feeded into N6. E.g., I'm connecting from N3 to
N6 since N3 will help N6
ITC Interpret a claim by his or her own words
INEG Identify a negative relationship Verbally state a negative relationship between two claims
ID Identify dependencies between claims Specify dependencies/commons between claims
IID Identify independencies between claims Specify that there are reasons to support the same claim but in different
reasons.
IIR Identify irrelevant claims to the main
argument
E.g., I think this is a different issue.
REASO Provide a reason for an association Explicitly state a reason why there is a relationship
MC* Make a connection (connect a link between
two nodes)
Add a link and connect two nodes
REVIE Review the flow of reasoning
RERRO Recognize a reasoning error
DL* Delete a link, detach a link Delete a link --> disconnect the relationship, or reserved the direction
176
As you can see from the figure above, map behaviors (2nd column) are already coded. What
you need to do is to read the sentence in Think-aloud (5th column) and code it to the 4th column,
titled Verbal Coding based on the coding scheme. You will type the code ONLY for the yellow
cells. Please watch the video while you analyze the verbal report in order to help your
understanding of each participant’s diagramming context.
Thank you so much for your participation as a second coder. Please don’t hesitate to contact me
if you have any questions/suggestions in regard to coding schemes.
Sincerely,
Time frame Map BehaviorBehavior
Coding
Verbal
CodingThink aloud
00:00-2:38So, I'm just going to read through all of these reasonings and see where I'll begin, how I'll
form the map. So I'm just going to read them out loud. Number one, enables a scrolling web browser for multiple graphics and text description.
I'm going to begin making a map. 2:39 Pointed N11 I'm going to pick number 11, use of multimedia increases learning.
First move N11 to
map MN I'm going to bring this over to the section and I'm going to--
3:07Click N11 and
move a little bit
because I think at least if it's off the main one-- there's definitely many things I can better
will derive from this argument.
Use of multimedia increases learning.
3:15 pointed N2 Okay, let's see. Helps encode into long term memory.
I think that definitely relates to--
First move N2 to
map MN
Pointed N11 and
N2 That's a good reason to have to use multimedia in e-learning. So
177
APPENDIX H
EXAMPLES OF CODING RESULTS FOR MAPPING BEHAVIOR AND
VERBAL REPORT
NOVICE 03
Time frame
Map Behavior
Behavior Coding
Verbal Coding
Think aloud
00:00-2:38
So, I'm just going to read through all of these reasonings and see where I'll begin, how I'll form the map. So I'm just going to read them out loud.
RC
Number one, enables a scrolling web browser for multiple graphics and text description.
RC
Helps encode into long term memory. Remove word for word audio narration of on-screen text.
RC Decrease load on visual working memory.
RC
Number five is increase selective attention. Oh okay, trying to figure out what that would connect to.
RC Increase selective attention.
RC Number six is reduce overall cognitive load.
RC
Number seven is personalize communication with use of pedagogical,
RC
yeah, I know what that means, pedagogical agents which is like-- yeah all right
RC number seven-- number eight, exclude gratuitous visuals.
RC Number nine exclude gratuitous sounds.
RC
Number ten add async audio narration and animated demonstration with reading text.
RC Number 11 use of multimedia increases learning.
IMC
All right, okay I'm just thinking-- okay yeah that's-- that'll be main one, main point.
RC
Number 12 is open new pop up window for an animated demonstration of complex content.
RC Number 13, exclude gratuitous text.
RC 14 is match mediate to students' learning style.
RC
15 is decrease load on auditory working memory impact student learning.
RC
I'm just going to repeat that one because-- lost count sorry. Number 15 is decrease load on auditory working memory impact student learning.
I'm going to begin making a map.
2:39 Pointed N11 IMC I'm going to pick number 11, use of multimedia increases learning.
178
First move N11 to map MN I'm going to bring this over to the section and I'm going to--
3:07 Click N11 and move a little bit
because I think at least if it's off the main one-- there's definitely many things I can better will derive from this argument.
RC Use of multimedia increases learning.
3:15 pointed N2 RC Okay, let's see. Helps encode into long term memory.
IA I think that definitely relates to--
First move N2 to map MN
Pointed N11 and N2 REASO That's a good reason to have to use multimedia in e-learning. So
Clicked N2 This one..
IACE
I'm going to-- for number two helps encode into long term memory. I'm going to add the arrow to connect to use of multimedia increases learning.
3:32 Add a link N2 MC
3:44 Okay. Go back to the selection.
3:52 pointed N5 RC Number five increase selective attention,
pointed N6 RC reduce overall cognitive load,
pointed N7 RC personalize communication with the use of pedagogial agents
pointed N14/clicked RC Match media to students learning style.
pointed N13/clicked RC Exclude gratuitous text.
pointed N12 RC
Okay. Open a new pop up window for animated demonstration of complex content.
pointed N5, N15, N11 RC
I'm trying to figure out... what other examples I can attach to main point, use of multimedia increases learning.
Let's see. I know what to do next, all right.
4:41 Clicked N7 RC Personalized communication with use of the pedagogical agent.
Clicked N14 RC Match media to student's learning style.
pointed N6, N12, N3 RC
Okay. Let's see. Remove word for word audio narration on-screen text.
Clicked N3 Okay. Trying to figure out
5:08 First Move N3 to map
MN
I chose remove word for word audio narration of on-screen text just because
Clicked N13 IA
I'm going to take this one, exclude gratuitous text. I think this one relates to remove word for word audio narration of on-screen text. So I'm going to add
First Move N13 to map MN
179
5:40
Add a link on N13 and connect to N3 MC the arrow to this one and connect.
5:47 pointed N15 - N1-N4 RC I'm trying to figure out what I will select next.
pointed N5 RC Increase selective attention.
IA
I'm going to add this as something increased selective attention as something that
First move N5 to map MN Derived from…
clicked N11 IACE Now number eleven use of multimedia increases learning.
6:23 Clicked N5 It increases selective attention.
Add a link on N5
Connect N5 to N11 MC
Reposition N5 I'm going to move this over [inaudible]
sort of, not so well.
6:39 pointed N9 -N14
Clicked N12 [Researcher] Talk aloud
Clicked N1
Yes. I'm trying to figure out if I'm doing this right . or if I'm doing it all wrong.
RC
Okay. number one is enable scrolling web browser for multiple graphics and text description.
IA
So it's not like-- that doesn't derive. That's not something that I'll input directly under use of multimedia increases learning. But I'm trying to figure out what that would go under.
7:21 Clicked N15 RC
Decrease load on auditory working memory impacts student learning. Okay, so decrease audio load on auditory working memory impacts student learning.
What does that go under?
7:50 Move N15 a little bit. Sorry, insane, okay.
7:55 pointed N12 and clicked RC
Open a new popup window for animated demonstration of complex content.
pointed N14 RC
Match media style. Match media to student's learning style. Let's see. Um..
IA
This one…... also will go under use of multimedia increases learning
8:13 First move N14 to map MN
180
8:22 Add an arrow on N14 MC so I'm going to add an arrow to it, connected to it.
Okay so I'm going to keep going.
Correcting the arrow point and dot.
Okay. Officially connected together.
8:34 Clicked N4 RC Number four, decrease load on visual working memory.
Clicked N6 RC
Number six, I'm just going through these and I'll figure out which one I'll work with next.
pointed N3, N2, N5,
Clicked N5 RC increase selective attention… increase selective.. Attention
9:10 Pointed N9, N8, N9
pointed N10 RC
Add async audio narration and animated demonstration with reading text.
RC
Add async audio narration and animated demonstration. Okay. Add async audio narration and animated demonstration..
That's too much.
INEG I feel like that's-- obviously that's a negative one.
First move N10 to map MN
9:47
Clicked N15 and back to the selection list Yeah. It's like,
Clicked N10 I'm going to add a red arrow to that because negative--
9:56
Add a link on N10 and connect N10 to N11 MC
this [?] add asynch audio narration and animated demonstration with reading text.
REASO
That's like overload sort of with sensory. With trying to read it and listening to it at the same time and then the same--
Changed the link to RED CA
10:18 Clicked on a empty space REASO
I don't know….. sometimes I get confused or less interested in what is going on if they're just going to read it to me.
10:31 Clicked on N15 I can read it myself.
Move N15 to different select position I think that's what I'll do.
181
pointed N10-N2
pointed N11 Review Use of multimedia increases learning.
Pointed N5 Review Increase selective attention.
10:55 pointed N10 Review
Add async audio narration and animated demonstration with reading text.
Clicked N5 RERRO
Okay actually I think this one, increase selective attention. How do I-- oh that's right just delete the arrow.
Deleted a link N5&N11 DL
No just kidding. Okay. Then I'll select the arrow. Having some trouble with this.
Move N5 under N10 REPN
Move 12 to Left for space I'm gonna move this.
Clicked N5 Okay. I'm going to add
Clicked N10
async audio narration and animated demonstration with reading text.
11:40 Clicked N5 IA
I'm going to add increase selective attention. Like I don't think increase selective attention is a bad thing. But
Clicked N10
it I guess with add async audio narration and animated demonstration with reading text-- okay. So
Clicked N5 ITC
if something is-- if someone is reading to you, like via a PowerPoint of everything that's written on the screen.
12:05 Clicked N10 IACE I've said before that maybe you wouldn't even pay attention but
Clicked N5 REASO
you would either-- you would probably actually either be reading it yourself. I always will just read the thing myself and and tone out the person who's talking or just listen to the person that's talking since I now that they're reading exactly what's on the screen.
12:29 Add a link on N5 IACE
So, I'm gonna have Increase selective attention... add it under async audio narration and animated demonstration with reading text.
12:43 Connect a link N5 to N10 MC
But I also don't think it's a bad thing so I'm just having it.
12:47 Changed a link to RED CA
So I guess it's another red one under the red one. Since it also has to be red. I just something to try to do.
13:01 Pointed N15 I'm going to keep going and see if what am doing is making sense.
RC Decrease load on visual working memory.
Pointed N4 and clicked N4 RC
Remove word for word audio narration of on-screen text.
182
Pointed N3 and Clicked N3 RC Match media….
13:32 move N3 a little bit. I think I'm having a harder time than it needs to be.
pointed N12 and clicked N12 RC
Open a new popup window for animated demonstration of complex content.
IA This helps encode into long term memory.
REASO
It's more intriguing and if it's something that's explained to you through animation,
First move N12 to map MN
14:01
Add a link on N12 and connect N12 to N2 MC
..you have a better way of having that stick with your long term memory.
pointed N11 REVIE Use of multimedia increases learning.
pointed N14-N11-N2 REVIE
Match media to student's learning styles. I'm just reading out loud to make sure that these are all, so far, making sense, or coherent points.
14:25 pointed N2 - N6 -N4 REVIE Match media to student's learning style.
move N4 a little bit IA
I'm trying to think if I can have multiple-- if I can have this arrow connect
Add a link on N14 and connect to N2 MC to…
REASO because this is true, I think.
14:49 pointed N14 IACE Matching media to student's learning style will
pointed N2 help them remember for much greater period of time.
15:03 Pointed N9 RC Exclude gratuitous sounds.
Clicked N8 RC Exclude gratuitous visuals.
pointed N9 I'm just thinking of what I'm going to do next, which if--
Clicked N3 and selected text and pointed nodes and links
how I'm going to-- I'm just thinking keep reading one of them. I'm like, "Okay is this right? Does this connect to this? Does it make sense for anything to connect to that or maybe there isn't another one that goes underneath it." I know this is not the way I'm supposed to be doing this but you know.
15:48 Pointed N4 RC Decrease load on visual working memory.
I'm going to-- I don't know where that one's going to go--
pointed N6 RC Reduce overall cognitive load.
183
Pointed and clicked N7 RC Personalize communication with use of pedagogical agents.
IA That will-- yeah, this is something that helps with …
16:14 First move N7 to map MN
pointed N14 Um..matching media to student's learning style.
pointed N7 Right! .. Well?
pointed N14, N2, N7 Yeah.
Pointed N2, Clicked N2, N7 IA
But it also helps encode into long term memory, I think, Because
Add a link on N7 REASO when you connect with them, connect with people,
Connect a link N7 to N2 MC
REASO
it's like a personalized way of doing it. Helps encode into long term memory.
16:48 Okay.
Clicked N9 REVIE Exclude gratuitous sounds.
pointed N3 - N9 REVIE Remove word for word audio narration on screen text.
Clicked N13 Yeah, that doesn't?--
move N13 little bit.
Click a link of N13 to N3 I'm taking this off because
pointed N9 IA
I think I meant to put -- and then to put exclude gratuitous sounds here I guess?
Clicked N13 and the link RERRO
I don't know what I was thinking when I did that. I'm going to get rid of the arrow that's connecting exclude gratuitous text to number three, which is remove word for word audio narration of on-screen text.
17:24 Deleted the link N13-N3 DL
Move N13 back to the start position REPN I'm going to put it back.
IA I'm just going to-- exclude gratuitous sounds….
Clicked N9 and move to map MN
pointed N3-N11 REASO
because removing the word for word audio narration of on-screen text; getting rid of the sounds will help someone focus better.
184
pointed N9 - N3
Move N9
clicked on N3 - N9
move N9 Sorry, I'm going to connect
17:59 Add a link on N9 IACE
exclude gratuitous sounds with an arrow. connecting to remove word for word audio narration of on-screen text.
Connect a link N9 to N3 MC
RC Reduce overall cognitive load.
pointed N13 RC Exclude gratuitous text.
Clicked N13 um.. Okay
pointed N8 RC Excluding gratuitous visuals.
pointed N12 RC
Open a new pop up window for an animated demonstration of complex content… um
18:30 pointed N1 RC
Enable scrolling web browser for multiple graphics and text description.
clicked N1 I don't know where that one goes at all but--
Pointed N4 - N14
Clicked N15 Um.. Okay.I'm just thinking about what I'm doing wrong.
RC
Open a new pop up window for an animated demonstration of complex content.
IA Right, yeah, that helps encode it to long term memory.
19:04 pointed N6 RC Reduce overall cognitive load.
pointed N15 RC
Decrease load on long term working memory…impact student learning.
Clicked N15 IA
Okay, So that one will go underneath remove word for word audio narration of on-screen text.
move to map MN
clicked N3 REASO
That way if you remove auditory element, it relates to removing the audio so that they can focus better. And then [inaudible].
19:51 pointed 15
Clicked N12 Okay.. I'm just trying to figure out if I'm doing this right.
Clicked N5 REVIE Increase selective attention. Yeah. See that will…
Pointed N10 Add async audio narration….
pointed N3
IA
I'm going to put reduce overall cognitive load over here because they all relate to-- these all relate to
pointed N6 MN
Clicked the link N10 -
185
N11
reposition N3, IA
not overloading students with numerous images, numerous sounds and text. And so like, focusing on one thing.
reposition N9
rearrange arrow.
So right now, I'm moving the arrows around, see how I'm going to map this out, because I can't--
20:41 Clicked N6 IL
so the first thing on top's going to be reduce overall cognitive load.
reposition N6 REPN
Clicked N3 and move N3 a little bit IA
I'm going to have remove word for word audio narration of on-screen text so that we'd only be focusing on the text.
Clicked N6 - N3 Okay… um….
Clicked N6-N3-N6 IACE
Now I'm thinking of reducing the overall cognitive load should go connect from helps encode
Moved N6 a little bit. into long term memory
Move N3 to West for space REASO because it does.
move N6 to a correct position I think.
Clicked N2 So I'm going to connect
Clicked N6
Add a link on N6 IACE
reduce overall cognitive load to helps encode into long term memory.
21:35
Connect a link N6 to N2
MC
Oh, sorry [inaudible]. Is that-- Red dot using the white dot and make the connection to here. And this one too, yeah. Right, right right. We need the red dot. Yeah. There we go. Struggling a little with the arrows. Okay so here we go.
21:57
Pointed N15-N3-N15 RC
So decrease load on auditory working memory Impact student learning. Decrease load on-- okay, for some reason I'm just like-- that sentence doesn't make sense to me, just because-- I don't know if I'm reading it wrong probably. Decrease load on auditory working memory. Impact student learning. Right. Okay where do I put that?
clicked N3 RC
Remove word for word audio narration of on-screen text. Remove word for word audio
Clicked N6 IA right so that's to-- that works with
Add a link MC reducing the overall cognitive load.
186
on N6
Delete the link on N6 That's not what I wanted to happen.
22:46 Clicked on N3
Add a link on N3 IACE So we're going to add an arrow to
Connect a link N3 to N6 reduce the overall cognitive load
pointed N15-N6-N15 RC
And then decrease load on auditory working memory….Impact student learning.
Clicked N15 IA So that is also relating to reducing overall cognitive load.
23:13 Add a link on N15 MC
Connect N15 to N6 You got to connect it to that one.
Clicked N9 and move down N9 RC Exclude gratuitous sounds.
pointed N3 RC Remove word for word--
Clicked N9 and pointed N3 IA related to both of these. Right, yeah. It relates to all of them.
23:38 pointed N6 RC Reduce overall cognitive load
pointed N9 RC Excluding gratuitous sounds.
clicked N4 and move N4 and move it back to the start position RC Decrease load on visual working memory.
clicked N5 and move down a little bit
clicked the arrown N5 to N10 RERRO I don't know if I'm doing this right.
I'm trying to figure out if…
Deleted the link N5 to N10 DL
reposition N10 for a space for N5 REPN
if I'm connecting the reasonings and the examples in the right way because increasing selective attention--
24:21 pointed N11 RC Use of multimedia increases learning.
187
pointed N5 RC Right, It increases selective attention,
clicked N3 RC so remove word for word audio narration.
I'm trying to figure out if I should just connect this(N5) to here (N11) or if I should just connect it to one thing under here(N6).
Add a link on N5 Because I think--
Connect a link N5 to N11 MC I think I can still add this to here.
move N3 for a space RC Remove word for word audio.
Add a link on N5 MC
. I feel like increase selective attention, if I'm adding too many arrows or
Connect a link N5 to N3 IA
if this works with remove word for word audio narration of on-screen text.
Clicked and move N9 to down for space REASO Just allowing only one thing being focus on at a time.
pointed N3-N5 RERRO
Wait no, because selective attention is having multiple things going on at once anyway. Okay, never mind.
25:28 Deleted the link N5 to N3 DL
Pointed N5 REASO I'm taking this arrow off because it doesn't--
Pointed N9 RC So exclude gratuitous sounds
move N9 around RC
Decrease load on auditory working memory impact student learning
I'm just trying to think of where to put exclude gratuitous sounds.
placed N9 under N15 and N12 MN
reposition N15 a little bit
Also, I know that it-- I think I'm focusing too much on just these three things.
Add a link on N9 I forgot about all these ones over here.
Connect a link N9 to N6 MC
reposition nodes to avoid overlap Not good. I'm trying to just make it physical.
26:30:00 clicked N12 REVIE
. Open a new pop up window for an animated demonstration of complex content. Right?
pointed N7 REVIE Personalize communication with the use of pedagogical agents.
188
pointed N2 REVIE Yes, it helps encode into long term memory.
pointed N14 REVIE Match media to students learning style.
clicked N4 RC Okay, decrease load on visual working memory.
I'm trying to figure out where this one would go.
clicked N8 RC Exclude gratuitous visuals.
clicked N12 RC
Open a new popup window for an animated demonstration of complex content.
27:03:00 RC Personalize--
27:05:00 clicked N1 RC
enable a scrolling web browser for multiple graphics and text description.
Where is that going to go? Remove-- I don't know.
RC
Enable a scrolling web browser for multiple graphics and text description.
RC
Async audio narration and animated demonstration with reading text. I know this is not as difficult as I'm making it but I'm just trying to figure out what this means.
27:58:00 clicked N8 RC Exclude gratuitous visuals.
RC Open a new pop-up window.
IA
So this, I believe this will go under open a new pop up window for animated demonstration of complex content.
move N8 to a map MN
Clicked N1 I don't think I'm doing this right.
add a link on N8
connect N8 to N6 MC
I'm going to add an arrow from exclude gratuitous visuals to reduce overall cognitive load.
clicked N12 pointed N1 RC
Trying to figure out where to put all these-- these two web browser ones. Open a new pop up window for animated demonstration of complex content. Right.
RC
Enable a scrolling web browser for multiple graphics and text description.
ITC Right, that's to have the text and images combined but it's not
That's good.
move N1 to a map MN I'll put this over here for the moment.
clicked N1 RC
Enable a scrolling web browser for multiple graphics and text description.
I'm just trying to think if this was-- if I'm just going to connect it to use of multimedia increases learning, because that doesn't explain why it does.
29:32:00 clicked N4
pointed N10 and clicked RC
Add async audio narration, animated demonstration with reading text. Right
189
pointed N13-pointed N4 RC Decrease load on visual working memory.
clicked N4 Just trying to think…
repositioned a map for a spaces IA
All right, so I'm going to add-- I think [chuckles] I don't think if I did that-- oh, it does do that. I'm going to move these upward so I can make space because I'm going add-- this looks ghastly, but I'm going to -- exclude gratuitous visuals under I'm going to have decreased visual load on visual working memory. . I'm going to have that under there because I think it's giving enough space for the layout I guess. I can just put it over here. Decrease load on visual working memory. I'm going to add
30:54:00 Add a link on N4 MC to exclude gratuitous visuals.
connect N4 to N8 So those connect.
31:19:00 clicked N13 and moved to a map IACE
Exclude gratuitous text. I'm going to add that to-- have this connect to decrease load on visual working memory.
add a link on 13 MC
connect N13 to N4
clicked N1 RC
Have this connect to enable a scrolling web browser for multiple graphics and text description.
Where is that going to go? [inaudible]
RC
Add async audio narration and animated demonstration with reading text.
RC
Enable a scrolling web browser for multiple graphics and text description.
RC Use personalized communication with use a pedagogical agent.
pointed N4 RC Decrease load on working memory
clicked N12 RC open new pop…
clicked N1
Add a link on N1 IA
Add this to personal communication with use of pedagogical agent. You know--
connect N1 to N7 MC
why would I do that? I don't know. I'm trying to think if I have to describe why I'm doing it but I have no good reason for why. That doesn't make any sense.
33:00:00
Clicked N14 REVIE
Then there's also I need to figure out this one match media to student's learning style. I feel like a lot of these-- all these relate to ways of doing that. Ways of matching media to a student's learning style.
Clicked N6 REVIE Reduce overall cognitive load-- yeah.
Clicked N12 REVIE
Open a pop up window for animated demonstration of complex content.
Clicked N2 REVIE Help encode to long term memory. Right? Okay.
Reposition
190
N9
Clicked N14 and pointed the overall map REVIE
Match media to student's learning style. That I feel like-- just not sure. I feel like others-- the way I want to make the map is crazy, if I connect these guys to match media to student's learning style.
Clicked N3
clicked N15 REIVE Decrease load on auditory working memory
clicked N9 (sigh)… hm…
read nodes
pointed N11
Clicked N10 REVIE
I trying to think as I'm doing this-- does this make sense? Add async audio narration and animated demonstrations with reading text.
N10 - N11 INEG
That's the only one that I thought would be not increasing learning
35:03:00
Yeah, pretty much. I'm a little unsure of it but I think I'm done. Yeah.
EXPERT 5
Time frame
Map Behavior MAP
CODING VERBAL CODING
Think aloud
0:00
Pointed N1 and moved N1 to map So first I'll just read all my claims on the side.
RC Enable scrolling web browser for multiple graphics and text description.
So I assume this is multimedia. This is one of the cases where its not a complete sentence.
RC Enable scrolling browser for multimedia graphics and text description.
MN I'll just pull it out to make it clear.
0:30
Pointed N2, N3, N4, N5, N6, N7, N8, n9, RC Helps encode into long term memory.
RC Remove word-for-word audio narration of onscreen text.
RC Decrease load on visual working memory and increase selective attention.
RC Reduce overall cognitive load.
RC Personalize communication with use of pedagogical agents.
RC Exclude gratuitous visuals,
RC exclude gratuitous sounds.
ID So you want to put these redundancies over here so there's a match up.
191
Move N9 and N8 MN
1:13 pointed N10, N11 RC
Add sync audio narration and animated demonstration with reading text.
IMC Use of multimedia increases learning.
This seems like an important sort of claim, a more general claim. So I set that out.
Pointed N12, N13 RC
Open new popup window for animated demonstration of complex content.
Exclude gratuitous text.
Moved N13 to map (closed to N8 & N9) ID So we have some more gratuitous stuff, I'll put over there.
1:50 Pointed N14 & N15 RC Match media to students' learning style
and decreased load of auditory working memory, impact student learning.
Moved N15 to map RC
Okay, So the decreasing load on auditory working memory impact student learning,
MN
1:58
reposition N11 to the bottom and reposition N15 above N11 REPN IACE
that seems like... we want to exclude-- excluding the gratuitous sounds was going to decrease the load on auditory working memory to impact student learning.
Add a link on N9 to N15 MC
So for now I'll put an arrow there just to see if that's really what I want and to remind myself that I connected those.
repositioned N11 to the center of the bottom. This conclusion is out of the way.
2:31 pointed N14, N6 RC So match media to students' learning style…..
reduce overall cognitive load.
Moved N6 to the above of N11 MN
RC Reducing the cognitive load..
2:50 Pointed N4 Decreasing the load on working visual memory. Right!
Moved N4 to map reposition N8 to above of N4
MN RepN
Excluding the gratuitous visuals would decrease the load on the working visual memory. So we'll put an arrow there.
2:58 Add a link on N8 to N4 MC
192
3:04 pointed N13 IA And… Also likely reduced the…. Also likely to work with that as well. But.. .
repositioned N9-N15 and N8-N4 to the above of N6 RepN RC
Reduce overall cognitive load. Let me get this out of the way. Gratuitous visuals, just making some space for myself.
Move N13 next to N8-N4 MN IACE
So reducing the overall cognitive load seems to be what we're doing by excluding these gratuitous sounds and gratuitous visuals.
3:50 pointed N10 RC Add async audio narration and animated demonstration with reading text.
Moved N10 to the map MN Right! So this is the useful auditory stuff.
Moved N1 to the map MN
Clicked N10 It should… hm.. Let's see.
Moved N5 to a map MN RC So we've got our increasing selective attention,
Moved N2 to a map MN RC we've got our encoding into long-term memory.
4:13 Clicked N2 and N5 ISC
Those are both important sub-conclusions that I'm going to need here.
Reposition N1 to the left-top RC
Enable scrolling web browser for multiple graphics and text description.
RepN
pointed N1 & N10 IA
Let's see; these are tasks that you want. Underneath those (N1 & N10)
pointed N5 & N2 we have the things that the tasks give you
Pointed N12 and moved it next to N1 RC
So open a new popup window for animated demonstration of complex content.
IA That's another thing we want up here.
4:42 RC Personalize communication with the use of pedagogical agents.
Moved N7 between N1 and N10 (at the lowest level) MN IA
So, Computer "Jim", I'll put that up here again.
Moved N14 to the left-bottom of the map. MN RC
Match media to students' learning styles. So………., it looks like we want something - let's see -
pointed N11
5:20 Pointed N6 Pointed N3 IA
For reducing the overall cognitive load and want you remove word-for-word audio narration of onscreen text.
Clicked N13, N6 IA
So the exclusion of gratuitous text seems to also reduce the overall cognitive load and it's probably related to this selective attention as well, so there are several connections here. ..The
193
selective attention…here.
reposition N2 RepN
Repoistion N5 next to N6 RepN IACE
Opening up a new popup window... will help with the selective attention
Rearrange the nodes.
I'm just arranging my boxes, so that I can see roughly where I think these things will go and then make the relevant connections. So, I'll move that there.
6:08 reposition N10 next to N8 RC
Add sync audio narration and animated demonstration with reading text.
IACE This will actually help with the decreasing the load on the visual memory.
6:36 Add a link on N10 to N4 MC IACE
So we'll put an arrow there. This should help with this as well. So let's see,
adding async audio narration and animated demonstration with reading text should help decrease the load of visual working memory
since the other audio memory's going to be at work as well.
7:00 pointed N15 -N9 chain RC
Decrease load on auditory working memory impacts student learning. So excluding those gratuitous sounds, so leave that alone for now.
7:10 reposiiton N5 under N12 IACE
So, Increase selective attention should be something that happens when you open up a new popup window….
since they have nothing else to compete with, the popup window should take over.
Add a link on N12to N5 MC
7:22 pointed N3 RC remove the word-for-word audio narration of onscreen text.
Clicked N3 So this is something that--
Moved N3 to map under N15 MN RC
let's see, decrease the load on the auditory working memory impact student learning.
IA So this is something that seems related to these as well. So excluding gratuitous sounds.
Reposition N15 and N9 to south Make some more space here.
Reposition N9 to north RepN IA
So excluding gratuitous sounds seems to be related to the removing the word-for-word audio narration of onscreen text.
pointed N3 IACE It decreases the load on the auditory working memory, which will impact student learning.
194
Reposition N3 between N14 and N9 RepN IA
So it seems like both of these, since removing word-for-word audio narration of onscreen text and excluding gratuitous sounds.
Clicked N9 CA
Excluding gratuitous sounds includes the word-for-word audio narration but that could be different, since excluding gratuitous sounds might just refer to music and things like that.
8:29 Add a link on N3 to N14 MC IID
For that reason I'll count them as separate reasons, bearing on decreasing load onto our working memory.
8:40 Pointed N1 RC Then we have enable scrolling web-based browser for multiple graphic and text description.
Moved N1 to map Skip
That should work with, let's see…This is going to be the... What's that first one that I...? Putting the text next to it. I'll leave that there for now.
Pointed N7 and moved it to south a little bit RC Personalized communication with use of pedagogical agents.
This is the agent Jim on the right of the computer.
pointed N2 and N5
Let's see. It seems like there's not a whole lot that matches up with that.
9:24
Reposition N7 a little bit to south IA
Matching the media to the students' learning style seems to be something that would connect up with potentially... several of these.
REPN
Moved N14 to the top-left of the screen MN We'll put this stuff to the side for now.
9:50
Pointed N2 and reposition N2 a little bit to east RC Helps encode into long term memory.
Reposition N4 and N3 to north RC
So the decreased load on visual working memory and the decreased load on... Let's see…
Reposition N15 to north RC the decreased load on auditory working memory,
resposition N2 below N4 and N15 IA
that was supposed to help encode things into the long term memory
Add a link on N4 to N2 MC IACE
So by decreasing the load we are helping to encode things into the memory there.
Add a link on N15 to N2 MC So I'll put that there for now. Here we go and let's see.
10:35 Pointed N6 RC Reducing the overall cognitive load,
reposition N13 a little bit to south IACE
that's going to be something that happens when you exclude the gratuitous--
Reposition N6 below N13
195
Reposition N11 to west … Ah! Let's see
Clicked the link on N4 to N2 So probably what I want to do actually is-- let's see…
Delete the link on N4 to N2 DL And then…
Delete the link on N15 to N2 DL okay… Great!
11:01
reposition N2 to south and reposition N6 below N4 and N15
Actually those are probably going to help reduce the overall cognitive load.
Reposition N13 to south IACE
So decreasing the load on working visual memory, decreasing the load on auditory working memory... those are both going to help reduce the overall cognitive load.
11:18 Add a link on N4 to N6 MC
Add a link on N15 to N6 MC So.. Put that here.
Reposition N2 below N6 add a link on N6 to N2 MC IACE
And that should help, then, to encode that into long term memory.
11:38 reposition N13 to south IA
So.. increasing the selective attention also seems to have some impact on that.
Pointed N7 RC Personalize communication with pedagogical agents,
Pointed N13 RC exclude gratuitous text.
Clicked N10 IA That seems to be something as well that maybe we want. Let's see,
reposition N10 and N8 to west IACE
We'll move this over here. There we go. We're adding async audio narration and animated demonstration with reading text. That should decrease load on the working visual memory.
reposition N13 next to N8 RepN IA Excluding gratuitous visuals will do that as well.
reposition N4 to south. IA And excluding gratuitous texts also will do that.
12:21 IACE So we'll put that here to connect that up. So all that's going to decrease the load on the visual working memory.
Add a link on N13 to N4 MC
reposition N3 to north, reposition N9, N15, N6 to organize the map Review
Removing word-for-word narration and excluding gratuitous sounds helps decrease auditory the working memory. All those things are going to help reduce the overall cognitive load the student has to deal with, which should help them to encode that into long term memory.
196
12:50 pointed N14 and N12 IA
Now we have open a new popup window, which increased selective attention. That's right.
reposition N7 to north RC Personalize communication with the use of pedagogical agents
clicked N1 RC And enable scrolling browser. Right
reposition N14 to south
reposition N1 to north RC
So the scrolling web browser and the text... multiple graphics and text description.
reposition N14 to West-south IIR
This seems to be something that is not really connected with any of those. It seems a separate issue.
pointed N1, N14, N7 reposition N11 a little bit to west INEG
So some of these are a little bit in tension with the others. So I want to make sure I have enough room for negative connections that I want to build in.
13:29
Move N11 (back and forth) IMC
So, the use of multimedia increases learning. These all seem to be going towards this sort of conclusion.
IACE So all of this stuff helps encode into long term memory, which will increase your learning.
13:48 Add a link on N2 to N11 MC So, I'll go ahead and put an arrow to connect to this one up here.
Clicked N5 IACE
Increasing selective attention should also help to-- let's see, it should probably help to reduce the overall cognitive load, I would imagine.
14:11 Add a linlk on N5 to N6 MC IACE
So we'll go ahead and tie that up with that. It should help to lower the cognitive load and help encode into memory.
reposition N7 to south IACE
So personalizing communication, that seems to be not related to the cognitive load at all, but it might be something that would help to encode into long term memory. If the student remembers the agents, it's like likely to give the negative sort of feedback, but that's not here.
Add a link on N7 to N2 MC So I'll put that there to connect that up.
14:50 Clicked N1 RC Then enabling scrolling web browser for multiple graphics and text description,
IA that's something that is related to the use of multimedia.
INEG
It's going to-- let's see, it's in tention withsome of these.... not necessarily in tension with some of gratuitous visuals, but certainly I want to make sure that whatever we include is necessary.
197
15:22
Clicked N14 Reposition N11 to west-south reposition N2 to west RepN RC So, matching the media to the students' learning style, let's see,
IA that should also help to encode into our long term memory.
Deleted the link on N2-N11
So I'll just give myself an extra bit of room there for another arrow.
Add a link on N2 to N11 MC So, we'll hook that up there.
Add a link on N14 to N2 MC IA
Matching the media to the students' learning style should create something to help encode into long term memory.
reposition N2 to south a little bit reposition N14 to south a little bit repoistion N7 to south a little bit
16:05 clicked N1 RC So then we're left with enabling scrolling web browser for multiple graphics and text description.
IA
This is something that likely I think would help the focus with the selective attention. It's not really in tension with anything like I first thought
Add a link on N1 to N5 MC ITC
Because it doesn't say to include a lot of text. It just says, "Include the scrolling web browser" which should help with selective attention."
REASO
When the students don't get all of the information all at once, they can scroll down and see just what they need to see as long as you have the graphics and text next to each other as it said.
pointed the map
I think that should be fine. I don't think that it have any negative arrow.
16:50
pointed the chain N1-N5, N12-N5, REVIE
Let me check and make sure that's right. So enabling a scrolling web browser, that should increase selective attention as should opening a new popup window
N5-N6, REVIE Both of those should be techniques to help increase selective attention,
N6-N2 REVIE which will reduce the overall cognitive load and help encode into long term memory.
pointed the chain N5-N6 RERRO
I'm not sure that increase in selective attention is something that reduces the overall cognitive load
198
IA It might be just the increasing the selective attention should help to encode into long term memory,
CONTA But probably would help encode in long term memory because you're reducing the cognitive load. So I think I'll leave that there.
N10, N8, N13 to N4-N6-N2
chain
REVIE Add sync audio narration and animated demonstration with reading text
REVIE , excluding gratuitous visuals
REVIE and excluding gratuitous texts will decrease the load on the working visual memory,
REVIE reduce the cognitive load and encode into long term memory.
N3 & N9 to N15-N6-N2-N11 chain
REVIE Then removing word-for-word audio narration,
REVIE excluding gratuitous sounds both decrease the auditory working memory,
REVIE reducing the cognitive load,
REVIE encoding into long term memory. That should work there.
N7-N2, N14-2-N11
REVIE Personalizing should help encode into long term and
REVIE matching media should students' long term...-- . So all of that helps to encode into long term memory.
REVIE That means that the use of multimedia increases learning. So I think that [inaudible]. We're good.
199
APPENDIX I
INTERVIEW TRANSCRIPT RESULTS
Transcription details:
Date: 24-Sep-2014
File: Exp05_interview
Transcription results:
Okay. First, please share any thoughts you have in regard to your experience with argument diagramming task.
Well let's see. I don't know if I have anything particular. I think generally, it seems like a-- I think I sort of treat it as a puzzle. Normally, I would try to find some sort of conclusion and see why I believe that and see why I believe these other things. So I think it's the way that I would typically think about certain things, especially as I write in philosophy, but it's different to see them all laid out visually. And it's also different when it's not the sort of topic that I'm used to.
So some of these I'm not sure. I think, "Well, it could be that there could be some negative connections, maybe up to the top, maybe these would be in conflict." And I also think, "Well, I don't know that they're exactly indication or anything like that." Yeah, it seems basically somewhat familiar and yet somewhat alien at the same time [chuckles].
Did you use any particular strategy to help you construct your argument map?
Let's see. Yes, I think generally, what I always try to do is find whatever I take to be the ultimate conclusion first. I find that and try to set that aside and then look for the reasons that would support that. As I was reading all the reasons, I noticed various similarities. So we're excluding text, we're excluding gratuitous visuals, we're excluding gratuitous sounds. I knew that all of those would typically go up towards the top and then go to support something about decreasing load on some sort of memory, so that helped me at least get that structure set. I started working with the conclusion and then ended up going back to the topics we use further out reasons, which helped me to see what they were supporting and then draw the connection in there together.
Normally, I think I would tend to work from the conclusion all the way back and then find those ultimate reasons, but I ended up finding the conclusion and then hopping back and then working my way back down to the conclusion. Then there were some that I wasn't quite sure where they would fit [chuckles], so I waited until I saw what sort of sub-conclusions I had to see where they tended to fit. That seems to be the general strategy.
200
Can you describe the process you used to create your argument diagram? For example-- well, you're already playing that [chuckles]. What was the first action you performed for the task and why? What was the next [exponent?] why they [put a?] thing? I think different [?]. Okay. Let's go.
Okay. So just--
That's first?
Yeah. So first, I find the conclusion. I try to pull that out and set that aside. At first, I was just reading through all the statements and then when I found the one that I thought was the conclusion, I put that on the side. Then I would start to group some of these ones that seem similar. It's about including various gratuitous things. Then I noticed the statements about decreasing the load on the working memory, and auditory memory, the visual memory and things like that. Those seem to connect together to reduce the overall cognitive load. So that helped to me draw those together. Then I could see that there was various other boxes that would also fit in as ways to decrease the load on the working visual memory or whatever which would fit in nicely with the overall cognitive load.
And then from that, I just had the extra ones that I saw. I guess the selective attention seem to fit in fairly well. I could know that the pop-up window would help to increase the selective attention. Enabling the browser also, then I had to fit that one in. I realized that probably increases selective attention, as well. And then the last two, they didn't seem to fit in with anything else I had. So they seem to just ways to encode in the long-term memory. That's why I put those last. These were the first ones I kind of figured out because I was focused on the other sort of connections and schemes.
I have a question. When I observed your performing, you were not sure about this link. And then you just left the link here. So can you elaborate?
Right. So the link between the increasing the selective attention and reducing the overall cognitive load. I wasn't sure why they're increasing the selective attention would be a way to encode it into the long-term memory, or if the way that it encodes it into the long-term memory is by reducing the overall cognitive load. When you increase their selective attention, there's less that is going on cognitively. There's less reduction in the cognitive load. That's the reason why it helps to encode it into the long-term memory.
If the reason that increasing the selective attention helps to encode it into long-term memory is because it reduces the overall cognitive load, then that's what I have diagrammed. That's what I seemed to think was going on. If that's not the case, then I would not want this arrow here. I'd want to take that out and put it straight down to encode in the long-term memory. Maybe it does that in a different way, but it seemed like that was a way that increasing the selective attention would help to encode it into long-term memory. So I left that arrow there.
You thought this one is mediate between selective attention and--
Right. Right. This is the way in which-- so it increases selective attention by reducing
201
the cognitive load. I'm sorry, you increase selective attention which means that you reduce the cognitive load and that helps to encode into long-term memory. If you don't have this middle way or if you don't reduce the cognitive load when you increase the selective attention in that way. And there are some other explanation that helps to explain why it encodes in the long-term memory, then I would want to take out that arrow.
Have you ever taken a psychology course [before then?]?
I took some psychology classes, undergrad. But it's been many, many years ago [chuckles].
So this terminology is not really new thing for you.
It's not entirely new, no. Especially the [common?] that I am using in my reason and critical thinking class now. It's sort of related to that, right? Especially the selective attention right now. So some of these terms are familiar, but the theories about how they work and connect up, I'm not as familiar with.
Okay. Next question. I think this is [going to do?]. What difficulties did you have while constructing your argument map? How did you solve that difficulties?
Right. Essentially, these few [crosstalk]-- yeah, I'm sorry. Because you got the screen. These three here - 1, 7, and 14 - I didn't see them as relating to what I had started to do over in this area. I had left them to the side and then I wasn't quite sure exactly where they were going to fit. I suppose that was a difficulty just in the sense that I wasn't quite sure where these would go in the long run. But once I had organized these in a way that I wanted to organize all the other, the working memory and the cognitive load stuff, then I realized that I already had 12, [as?] connect that to increase the selective attention.
And then when I thought about what enabling the scrolling web browser would do, that seems to limit what you can see, right, in the same way that the pop-up window does. So that seemed to fit up nicely with the increase in the selective attention, which just left these last two - 7 and 14 - and neither of those seem to fit into anything that I've done over here. But, it didn't seem that those, straightforwardly, lead to 11, that use multimedia increases learning. So it seems that I would need to connect through that with two, that they would encode them in long-term memory.
In the article - The Personalized Communication with Jim - right there, it was sort of connected but not really explicitly something that they were saying that would help to encode into long-term memory. But it seemed that when you have something that you can name and you become familiar with, it's a lot of easier to remember that sort of thing. It seemed like that was a clear path to go through. And then the same with matching the media to the students' learning style. If it's something that the way they learn, that's going to help to remember it better. So both of those seem like loose threads, but it seemed that they've best fit.
Do you believe media? That student have different learning style and you need to match auditory learning or visual learner or something like that. Do you think that's true?
202
Yeah. I think that certain students, yeah. They'd be ones that would match better with auditory or visual, right? Yeah. In that sense then, I could see that working, I suppose, some way possibly over in this area, if you can connect this up somehow. But I'm not sure how matching the media to their learning styles would fit in with say, excluding any of these, or adding. So it could be a reason, say, to add the sync audio narration and demonstration with the reading text, right? So that you get two sorts of ways that they're getting the information so that you can kind of match it to the learning style. But that's not so much matching into the learning style, it's just giving them several different learning styles so whichever one fits with them is the one they can appeal to or use.
Have you read the article?
No.
No. Okay. Please indicate on your argument map which principle you're [leading?] you prior to that start of this argument mapping session. For example, did you know about multimedia principle?
No. I don't think I knew--
You knew of that.
I don't think I knew any of them. I think, at least, certainly not by those names. I think I've heard of--
Like the concept?
Right. The concepts of--
Which one do you think?
Let's see.
This one is [?]-
Right. I think the redundancy principle for sure. And--
This one?
Right. No, I guess not. Let's see. It was the-- right, so I think that the redundancy and the coherence principle both, those basic concepts I was familiar with. To a lesser extent, maybe the personalization principle. But the other one is not really a [principle?].
So you'd already say this. Then can you tell me which relationship in your diagram were difficult to identify? Can you explain why? You already say something. Lets start with green and the relationship was kind of difficult or-- to make that one as a--?
The relationship between-- lets see. It's like this one and this one was a bit difficult to identify for me (N14-N2, N7-N2). I think probably, this one and this one, as well. Second [?] the little green check. I think this one was slightly at first up with this, but after thinking about it and realizing that amounted the same as 12, then this one (N1-N5, N5-N6)didn't really seem like it was that bad. I think all the rest of these, let's see, I felt fairly confident that those were ways to match up except for it. So, say, this one,
203
this sort of connection here would be the only other one that I wasn't quite so sure how it was going to work.
Look at this relationship, okay? So [add those?] think [on those?] audio narration, and animated demonstration with written tasks will decrease the load in visual-working memory or something?
Yeah. I think that when you add the audio narration with the reading text, so the thought there was that the student doesn't have to-- they might not even read the information, right, if you have the audio narration of the text. Or if they do read it, say, as an automatic type process, right? So it's like dual-process existing one something or other where they just happen to kind of read it but they don't have to spend their cognitive energy trying to read it and understand it as much. I was assuming that would decrease the load on the visual-working memory and it would allow them to just kind of learn in different ways?
They listen?
Right. They can hear it instead of reading it, right.
They can hear it instead of looking at it? Okay.
That's the thought, yeah.
This one is kind of important. When you insert a link between the [premise?] and you were not sure, did you use guessing? For example, like this one here or here, did you guess or just...?
Right. I think that I tried to not put any links in until I felt fairly confident that I wanted them there. If I did put a link in, the only reason why I wanted to delete it was to put something else in between as a separate step. I think that when it came to the end - and I just had these two left, 7 and 14 - I wouldn't say that I guessed at the relationship. I think I--
You had some reason.
Right. I could rationalize why I made the connections. I think I was the least confident, so I wasn't sure if I was creating a story to make this fit. That made sense because that was what was left, and it didn't seem to fit into what I'd already created on the other side. But I don't think there was anything that I just guessed, I suppose, in that sort of sense. Usually, I wasn't quite sure how this would work, so I would think about what that connection was, and I could see how there would be a connection there. So it was some sort of rationalizing process for [that?].
Okay. So you had your own rationale to connect.
Yeah. And I'm not sure that the rationale that I used is--
Correct of not, but anyway, you--
Right. But I could certainly defend it if I needed to [laughter].
This one is the last one, I think. Were you able to think aloud while constructing the map, or did you filter out certain thought that you would...?
204
Yeah. I don't think that I filtered out thoughts, at least not intentionally [chuckles] if I did. I think the thinking aloud, often-- there's a lot of reading what I'm seeing so that there's not just a silence. So I don't think that there was anything that I filtered out. There might have been fleeting thoughts that just didn't quite come to the surface, that I didn't quite say. But if that happened, it wasn't intentional sort of way.
How percent is your thought were verbalized, do you think?
Let's see. I don't know. I would assume probably in the 90s, 90%.
90? Do you remember some thought that you didn't verbalize?
I don't remember any thought that I didn't verbalized. So I would assume that it's probably pretty high, but I'd like to leave some room for error. Usually, I remember thinking, "Well, you know, I'll put this off to the side because I'm not sure exactly where this goes," but I would usually say that it's happened, and then I also say, "Well, there's this arrow here. I'm not sure if that should go there." Maybe one thing is I think that when I use the-- right. So this arrow here that you asked about, and so you heard me say something about it, right?
I heard. Yeah. You say, "It's direct link here" or like that.
Right. So at least I thought that I said this but maybe I didn't. I suppose you have the recording, you can check that I said, "No, I think this is okay" and I thought that I'd given some reason why I think that it's okay. But I don't know that I fully expressed the thought. So of course, now that we've talked about it, right, I have. That might be something that just I think, "No. I think this is going to be okay because I think that this is the way that it's going. It's reducing overall cognitive load." Other than that, I can't think of anything that-- I suppose if this one didn't come out either - since you asked about this one as well - I think this is something that I thought that adding the audio narration and the animated demonstration through reading text would decrease the visual working memory because the students wouldn't have to say--
Read the text.
Read it. Right. They don't have to read the text so that helps out with the visual working memory. They can just rely on listening to the text if they need to. I think that was the other thought that I can think of that might not have been verbalized in the way that would [be?]. I think that should be it.
That's great. Because you gave this also, this relationship based on your reason, right? When you are scrolling the browser and you can just press the information that you want to read so that increase selective attention, right?
Right. Yeah.
That was very nice.
205
REFERENCES
Austhink. (2006, December 7). Argument Mapping Tutorials. Retrieved from
http://rationale.austhink.com/rationale2.0/tutorials/
Baddeley, A.D., & Hitch, G.J. (1974). Working memory. In G.H Bower (Eds.), The psychology
of learning and motivation (pp.48-90). New York: Academic Press.
Bakeman, R., & Gottman, J.M. (1997). Observing Interaction: An Introduction to Sequential
Analysis. Cambridge: Cambridge University Press.
Bensley, A. D. (2010). A brief guide for teaching and assessing critical thinking in psychology.
Observer, 23(10), 49-53.
Bensley, D. A., Crowe, D. S., Bernhardt, P., Buckner, C., & Allman, A. L. (2010). Teaching and
Assessing Critical Thinking Skills for Argument Analysis in Psychology. Teaching of
Psychology, 37(2), 91-96. doi: 10.1080/00986281003626656
Binkley, M., Erstad, O, Herman, J., Raizen, S., Ripley, M., Rubmle, M. (2012). Defining 21st
Century skills. In P. Griffin, B. McGaw & E. Care (Eds.), Assessment and teaching of
21st century skills (pp. 17-66). New York: Springer.
Braak, S. W. van den., Oostendorp, H. van, Prakken, H., & Vreeswijk, G. A. (2006). A critical
review of argument visualization tools: Do users become better reasoners? In F. Grasso,
R. Kibble, & C. A. Reed (Eds.), Workshop Notes of the ECAI-2006 Workshop on
Computational Models of Natural Argument (pp. 67-75). Riva del Garda:Italy.
Case, K., Harrison, K., & Roskell, C. (2000). Differences in the clinical reasoning process of
expert and novice cardio respiratory physiotherapists. Physiotherapy, 86(1), 14-21.
Cellier, F., Eyrolle, C., & Marine, C. (1997). Expertise in dynamic environments. Results of
comparison between novice and expert operators in supervision of dynamic environment.
Ergonomics, 40 (1), 28–50.
Chi, M. T. H (2006). Two approaches to the study of experts’ characteristics. In Ericsson K.A.,
Charness, N., Feltovich, P., & Hoffman, R. (Eds.), Cambridge handbook of expertise and
expert performance. (pp. 121-30), Cambridge University Press.
Chi, M. T., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.),
Advances in the psychology of human intelligence (pp. 7-77). Hillsdale, NJ: Erlbaum.
Clark, R. (2002). Six principles of effective e-Learning: What works and why. The e-Learning
Developer’s Journal, 1-20. Retrieved from
http://faculty.washington.edu/farkas/HCDE510-
Fall2012/ClarkMultimediaPrinciples(Mayer).pdf
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational Psychological
Measurement, 20(1), 37-46.
Correia, V. (2011). Biased and fallacies: The role of motivated irrationality in fallacious
reasoning. Journal of Reasoning and Argumentation, 3(1), 107-127.
206
Cosgrove, R. (2011). Critical thinking in the Oxford tutorial: a call for an explicit and systematic
approach. Higher Education Research & Development, 30(3), 343-356. doi:
10.1080/07294360.2010.487259
Crespo, K.E., Torres, J. E., & Recio, M. E. (2004). Reasoning process characteristics in the
diagnostic skills of beginner, competent, and expert dentists. Journal of Dental
Education, 68(12), 1235-1244.
Cross, N. (2004). Expertise in design: an overview. Design Studies, 25(5), 427–441
Davies, M. (2011). Introduction to the special issue on critical thinking in higher education.
Higher Education Research & Development, 30(3), 255-260.
Davies, M. (2013). Critical thinking and the disciplines reconsidered. Higher Education
Research & Development, 32(4), 529-544.
De Neys, W. (2006). Dual Processing in Reasoning Two Systems but One Reasoner.
Psychological Science, 17(5), 428-433.
Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation
in classrooms. Science Education, 84(3), 287-312
Dwyer, C.P., Hogan, M.J., & Stewart, I. (2010). The evaluation of argument mapping as a
learning tool: Comparing the effects of map reading versus text reading on
comprehension and recall of arguments. Thinking Skills and Creativity, 5(1), 16-22.
Easterday, M. W., Aleven, V., & Scheines, R. (2007). Tis Better to Construct than to Receive?
The Effects of Diagram Tools on Causal Reasoning. In R. Luckin, K. R. Koedinger, & J.
Greer (Eds.), Frontiers in artificial intelligence and applications: Vol. 158. Artificial
intelligence in education: Building technology rich learning contexts that work. (pp. 93-
100). Amsterdam: IOS Press.
Elqayam, S. & Evans, St.B.T. (2011). Subtracting “ought” from “is”: Descriptivism versus
normativism in the study of human thinking. Behavioral and Brain Sciences, 34(5), 233-
290.
Ennis, R. H. (1982). Identifying implicit assumptions. Synthese, 51(1), 61-86. doi:
10.1007/bf00413849
Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research.
Educational Researcher, 18(3), 4-10.
Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of
thinking during experts' performance on representative task. In K. A. Ericsson, N.
Charness, P. Feltovich, and R. R. Hoffman, R. R. (Eds.). Cambridge handbook of
expertise and expert performance (pp. 223-242). Cambridge, UK: Cambridge University
Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis; Verbal reports as data (revised
edition). Cambridge, MA: Bradfordbooks/MIT Press.
207
Ericsson, K. A., & Simon, H.A. (1998). How to study thinking in everyday life: Contrasting
think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and
Activity, 5(3), 178-186.
Ericsson, K.A., & Lehmann, A.C. (1996). Expert and exceptional performance: Evidence of
maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
Ericsson, K.A., & Simon, H.A. (1983). Protocol analysis: Verbal reports as data. Cambridge,
Mass: MIT Press.
Evans, J. S. B. T. (2002). Logic and human reasoning: An assessment of the deduction paradigm.
Psychological Bulletin, 128(8), 978-996.
Evans, J. S. B. T. (2003). In two minds: Dual-process accounts of reasoning. Trends in Cognitive
Sciences, 7(10), 454-459. doi: 10.1016/j.tics.2003.08.012
Evans, J. S. B. T. (2006). The heuristic-analytic theory of reasoning: Extension and evaluation.
Psychonomic Bulletin & Review, 13, 378-395.
Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition.
Annual Review of Psychology, 59, 255-278. doi:
10.1146/annurev.psych.59.103006.093629
Evans, J. S. B. T., & Over, D. E. (1996). Rationality and reasoning. Hove, UK:Psychology
Farrington-Darby, T., & Wilson, J.R. (2006). The nature of expertise: Fundamental review.
Applied Ergonomics, 37(1), 17-32.
Feldon, D. (2007). The implications of research on expertise for curriculum and pedagogy.
Educational Psychology Review, 19(2), 91-110
Glaser, R., & Chi, M. T. H. (1988). Overview. In M.T. H. Chi, R. Glaser, & M. J. Farr (Eds.),
The nature of expertise (pp. 15-27). Hillsdale, NJ: Lawrence Erlbaum.
Goel, V., Buchel, C., Frith, C., & Dolan, R.J. (2000). Dissociation of mechanisms underlying
syllogistic reasoning. NeuroImage, 12, 504-514. Doi:10.1006/nimg.2000.0636
Gold, J., Holman, D., & Thorpe, R. (2002). The role of argument analysis and storytelling in
facilitating critical thinking. Management Learning, 33(3), 371-388.
Govier, T. (1987). Problems in Argument analysis and evaluation. Dordrecht: The Netherlands:
Foris Publications Holland.
Guba, E. G., & Lincoln, Y.S.(1994). Competing paradigms in qualitative research. In N. K.
Denzin and Y. Lincoln (Eds.), Handbook of qualitative research, (pp.105–117).
Thousand Oaks, CA: Sage
Hahn, U. & Oaksford, M. (2007). The rationality of informal argumentation: A Bayesian
approach to reasoning fallacies. Psychological Review, 114(3), 704–732
Harrell, M. (2007). Using argument diagramming software to teach critical thinking skills.
Proceedings of the 5th International Conference on Education and Information Systems,
Technologies and Applications. July, Orlando. 2007
208
Harrell, M. (2008). No Computer Program Required: Even Pencil-and-Paper Argument Mapping
Improves Critical-Thinking Skills. Teaching Philosophy, 31(4).
Harrell, M. (2011). Argument diagramming and critical thinking in introductory philosophy.
Higher Education Research & Development, 30(3), 371-385. doi:
10.1080/07294360.2010.502559
Heit, E. (2007). What is induction and why we study it? In Feeney, A., & Heit, E. (Eds.),
Inductive reasoning: Experimental, developmental, and computational approaches (pp.1-
24). Cambridge: Cambridge University Press.
Hitchcock, D., & Verheij, B. (2006). Introduction. In D.Hitchcok & B.Verheij (Eds.), Arguing on
the Toulmin Model: New essays in argument analysis and evaluation (pp. 1-23).
Dordrecht: The Netherlands: Springer.
Hoffman, R.R. (1996). Defining expertise. In Williams, R., Faulkner, W., & Fleck, J. (Eds.),
Exploring expertise (pp.81-100). Edinburgh, Scotland: University of Edinburgh Press.
Jeong, A. (2010). Assessing change in learners’ causal understanding using sequential analysis
and causal maps. In V.J. Shute, & B.J. Becker (Eds.), Innovative assessment for 21st
century: Supporting educational needs (pp. 187-206). New York: Springer-Verlag.
Jeong, A. (2012). DAT software. Retrieved January 31, 2012, http://myweb.fsu.edu/ajeong/dat
Jeong, A. (2012). jMAP. Retrieved from https://sites.google.com/site/causalmaps/
Jeong, A. C. (2014). Sequentially analyzing and modeling causal mapping processes that
produce high versus low Causal understanding. In Ifenthaler, D., & Hanewald, R. (Eds.),
Digital Knowledge Maps in Education: Technology Enhanced Support for Teachers and
Learners (pp. 239-252). Association of Educational Communication & Technology.
Johnson-Laird, P. N. (1983). Mental models. Cambridge, MA: Harvard University Press.
Johnson-Laird, P. N. (1999). Deductive reasoning. Annual Review of Psychology, 50(1), 109-
135.
Johnson-Laird, P. N. (2008). Mental models and deductive reasoning. In J. E. Adler & L. J. Rips
(Eds.), Reasoning: Studies of Human Inference and Its Foundation (pp. 206-222).
Cambridge, UK: Cambridge University Press.
Jonassen, D.H. (1997). Instructional design for well-structured and ill-structure problems.
Educational Technology Research and Development, 45(1), 656-694.
King, P. M., Wood, P. K., & Mines, R. A. (1990). Critical thinking among college and graduate
students. The Review of Higher Education 13(2), 167-186.
Klaczynski, P. A. (2000). Motivated Scientific Reasoning Biases, Epistemological Beliefs, and
Theory Polarization: A Two-Process Approach to Adolescent Cognition. Child
development, 71(5), 1347-1366. doi: 10.1111/1467-8624.00232
Klauer, K. C., Musch, J., & Naumer. B. (2000). On belief bias in syllogistic reasoning.
Psychological Review, 107(4), 852-884. doi:10.1037/0033-295X.107.4.852
209
Koro-Ljungberg, M., Doublas, E.P., Therriault, D., & Malcolm, Z. (2012). Reconceptulizing and
decentering think-aloud methodology in qualitative research. Qualitative Research, 0(0),
1-19.
Kuhn, D. (1991). The Skills of Argument. Cambridge: Cambridge University Press.
Kuhn, D. (1992). Thinking as argument. Harvard Educational Review, 62(2), 155-179
Kuhn, D. (1999). A developmental model of critical thinking. Educational Researcher, 28(2),
16-26.
Kuhn, D. (2007). Jumping to conclusions: Can people be counted on to make sound judgments?
Scientific American Mind, 18 (1, Feb/Mar), 44-51.
Kuhn, D., & Udell, W. (2003). The development of argument skills. Child Development, 74(5),
1245-1260.
Landis, R.R., & Joch G.G. (1977). The measurement of observer agreement for categorical data.
Biometrics, 33(1), 159-174.
Larkin, J.H. (1983). The role of problem representation in physics. In A.L. Stevens & D. Gentner
(Eds.), Mental Models, (pp.75-99). Hillsdale, NY: Erlbaum.
Lim, L. (2011). Beyond logic and argument analysis: Critical thinking, everyday problems and
democratic deliberation in Cambridge International Examinations' Thinking Skills
curriculum. Journal of Curriculum Studies, 43(6), 783-807. doi:
10.1080/00220272.2011.590231
Livingston, C., & Borko, H. (1989). Expert-novice differences in teaching: A cognitive analysis
and implications for teacher education. Journal of Teacher Education, 40(4), 36-42.
Madill, A., Jordan, A. & Shirley, C. (2000). Objectivity and reliability in qualitative analysis:
Realist, contextualist and radical constructionist epistemologies. British Journal of
Psychology, 91(1), 1–20. doi: 10.1348/000712600161646. Mahwah, NJ: Lawrence
Erlbaum Associates.
Mayer, R. E. (2010). Problem solving and reasoning. In V. Aukrust (Eds.), Learning and
Cognition (pp.112-117). Oxford, UK: Academic Press
McMillan, J. H. (1987). Enhancing college-students critical thinking: A review of studies.
Research in Higher Education, 26(1), 3-29. doi: 10.1007/bf00991931
Neuman, Y., & Weizman, E. (2003). The role of text representation in students’ ability to
identify fallacious arguments. Quarterly Journal of Experimental Psychology, 56, 849-
865
Nielsen, J. (1994). Estimating the number of subjects needed for a thinking aloud test.
International Journal of Human-Computer Studies, 41(3), 385-397.
Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical
Education, 39, 418-427
210
Oaksford, M. & Hahn, U. (2007). Induction, deduction, and argument strength in human
reasoning and argumentation. In Feeney, A., & Heit, E. (Eds.), Inductive reasoning:
Experimental, developmental, and computational approaches (pp.260-301). Cambridge:
Cambridge University Press.
Partnership for 21st Century Skills. (2009, December). P21 framework definitions. Retrieved
from http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf.
Patton, M. Q. (2001). Qualitative Research & Evaluation Methods, 3rd edition. Thousand oaks,
CA: Sage Publications.
Paul, R., & Elder, L. (2001). The miniature guide to critical thinking: Concepts & tools. Dillon
Beach, CA: The Foundation for Critical Thinking.
Pett, M.A. (1997). Nonparametric statistics in health care research: Statistics for small sample
and unusual distribution. Thousand Oaks, CA: Sage Publications .
Reimold, M., Slifstein, M., Heinz, A., Mueller-Schauenburg, W., & Bares, R. (2006). Effect of
spatial smoothing on t-maps: arguments for going back from t-maps to masked contrast
images. Journal of Cerebral Blood Flow and Metabolism, 26(6). doi:
10.1038/sj.jcbfm.9600231
Rider, Y., Thomason, N. (2008). Cognitive and pedagogical benefits of argument mapping:
L.A.M.P. guides the way to better thinking. In A., Okada. S.J.B., Shum, & T., Sherborne
(Eds.), Knowledge Cartography: Software tools and mapping techniques (Advanced
Information and Knowledge Processing) (pp.113-130). London: UK: Springer-Verlag.
Ruiz-Primo, M. A. & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in
science assessment. Journal of Research in Science Teaching, 33(6), 569-600.
Ruiz-Primo, M. A. (2004, September). Examining concept maps as an assessment tool.
In Concept maps: Theory, methodology, technology. Proceedings of the first
international conference on concept mapping (Vol. 1, pp. 555-562).
Schaeken, W. (2000). Deductive reasoning and strategies. Mahwah, N.J: L. Erlbaum Associates.
Schechter, J. (2013). Deductive reasoning. In H.E. Pashler (Eds.), Encyclopedia of the mind (pp.
227-231). Thousand Oaks, CA: Sage Publications.
Schunn, C. D., & Anderson, J. R. (1999). The generality/specificity of expertise in scientific
reasoning. Cognitive Science, 23(3), 337-370.
Scriven, M. (1976). Reasoning. New York: McGraw-Hill.
Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational
Behavior and Human Decision Processes, 53(2), 252-266.
Shaw, V.F. (1996). The cognitive processes in informal reasoning. Thinking & Reasoning, 2(1),
51-80.
Siegel, H. (1989). The Rationality of Science, Critical Thinking, and Science Education.
Synthese, 80(1), 9-41.
211
Simon, H. A. (1980). Problem solving and education. In Tuma, T., & Reif, R. (Eds.), Problem
Solving and Education: Issues in teaching and research (pp.81-96). Erlbaum, NJ:
Hillside.
Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning.
Mahwah, NJ: Elrbaum.
Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the
rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. doi:
10.1017/s0140525x00003435
Strube, P. (1989). A content-analysis of arguments and explanations presented to students in
physical science textbooks: A model and an example. International Journal of Science
Education, 11(2), 195-202.
Svedholm-Häkkinen, A.M. (2015). Highly reflective reasoners show no signs of belief
inhibition, Acta Psycholoica, 154, 69-76
Taylor, C. (1996). Defining science. Madison, WI: University of Wisconsin Press.
Taylor, V.L. (1971). The art of argument. Metuchen, N.J.: The Scarecrow Press.
The Chronicle of Higher Education & America Public Media’s Marketplace, (2012, December).
The role of higher education in career development: Employer perceptions. Retrieved
from http://chronicle.com/items/biz/pdf/Employers%20Survey.pdf.
Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation
data. American journal of evaluation, 27(2), 237-246.
Thompson, V. A. (2010). Towards a metacognitive dual process theory of conditional reasoning.
In Oaksford, M., & Chater, N. (Eds.), Cognition and Conditionals - Probability and
Logic in Human Thinking (335-354). Oxford, UK: Oxford University Press.
Toulmin, S. E. (1958). The uses of argument. Cambridge: Cambridge University Press.
Twardy, C. (2004). Argument maps improve critical thinking. Teaching Philosophy, 27, 95–116.
van Bruggen, J.M., Boshuizen, H.P.A., & Kirschner, P.A. (2003). A cognitive framework for
cooperative problem solving with argument visualization. In P.A. Kirschner, S.J.
Buckingham Shum, & C.S. Carr (Eds.) Visualizing argumentation: Software tools for
collaborative and educational sense-making (pp.25-47). London: Springer-Verlag.
van Bruggen, J.M., Kirschner, P.A., & Jochems, W. (2002). External representation of
argumentation in CSCL and the management of cognitive load. Learning and Instruction,
12(1), 121-138.
van Gelder, T. (2013). Argument mapping. In Pashler, H. (Eds.), Encyclopedia of the Mind (pp.
13-14). Thousand Oaks, CA: Sage.
Van Gelder, T. J. (2001). How to improve critical thinking using educational technology. In G.
Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.), Meeting at the crossroads:
Proceedings of the 18th annual conference of the Australasian Society for computers in
learning in tertiary education (pp. 539-548). Melbourne: Biomedical Multimedia Uni,
The University of Melboure.
212
Van Gelder, T. J. (2002a). Enhancing deliberation through computer supported argument
mapping. In Kirschner, P.A., Buckingham Shum S.J., Carr, C.S. (Eds.) Visualizing
argumentation: Software tools for collaborative and educational sense-making (pp. 97-
115). London: Springer-Verlag.
van Gelder, T. J. (2002b). Argument Mapping with Reason!Able. The American Philosophical
Association Newsletter on Philosophy and Computers, 85-90.
van Gelder, T. J. (2007). The Rationale for Rationale™. Law, Probability and Risk, 6, 23-42.
doi:10.1093/lpr/mgm032
Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. (1994). The think aloud method: A
practical guide to modeling cognitive processes. London: Academic Press.
Verschueren, N., Schaeken, W., & d’Ydewalle, G. (2005). A dual-process specification of causal
conditional reasoning. Thinking & Reasoning, 11, 278–293.
doi:10.1080/13546780442000178
Walton, D, & Gordon, T.F. (2009). Jumping to a conclusion: Fallacies and standards of proof.
Informal Logic, 29(2), 215-243.
Walton, D. (1999). Rethinking the fallacy of hasty generalization. Argumentation, 13(2), 161-
182.
Walton, D.N. (1989). Reasoned use of expertise in argumentation. Argumentations, 3, 59-73,
Weinstock, M.P., Neuman, Y., & Glassner, A. (2006). Identification of informal reasoning
fallacies as a function of epistemological level, grade level, and cognitive ability. Journal
of Educational Psychology, 89(2), 327–341
213
BIOGRAPHIC SKETCH
Areas of Interest
Critical thinking and reasoning processes in argument analysis
Implement of visual mapping tools in higher education
Student Motivation and online learning design
Education
Florida State University, Tallahassee, FL Sep.2007 – May 2015
Doctor of Philosophy in Instructional Systems & Learning Technology
Dissertation title: Modeling the reasoning processes in experts and novices’
argument diagramming task: Sequential Analysis of diagramming behavior and
Thinking-Aloud Data.
Florida State University, Tallahassee, FL Aug.2009 – Aug.2012
Master of Science in Measurement and Statistics
University of Utah, Salt Lake City, UT Sep. 2004 – Jul. 2005
Graduate Exchange Program in Teaching & Learning Department
Gyeongin National University of Education, Incheon, Korea Mar.2002 – Feb.2006
Master of Science in Elementary Computer Education,
Thesis title: A Study on How to Organize Concept Knowledge of Computer
Disciplines for the Elementary Level
Gyeongin National University of Education, Incheon, Korea Mar.1995 – Feb.1999
Bachelor of Education (Focus area: Korean Education)