THE CATHOLIC UNIVERSITY OF AMERICA
Deconstructing Constructivism: Modeling Causal Relationships Among Constructivist Learning Environment Factors and Student Outcomes in Introductory Chemistry
A DISSERTATION
Submitted to the Faculty of the
Department of Chemistry
School of Arts & Sciences
Of The Catholic University of America
In Partial Fulfillment of the Requirements
For the Degree
Doctor of Philosophy
By
Regis Komperda
Washington, D.C.
2016
Deconstructing Constructivism: Modeling Causal Relationships Among Constructivist Learning
Environment Factors and Student Outcomes in Introductory Chemistry
Regis Komperda, Ph.D.
Director: Diane M. Bunce, Ph.D.
The purpose of this dissertation is to test a model of relationships among factors
characterizing aspects of a student-centered constructivist learning environment and student
outcomes of satisfaction and academic achievement in introductory undergraduate chemistry
courses. Constructivism was chosen as the theoretical foundation for this research because of its
widespread use in chemical education research and practice. In a constructivist learning
environment the role of the teacher shifts from delivering content towards facilitating active
student engagement in activities that encourage individual knowledge construction through
discussion and application of content.
Constructivist approaches to teaching introductory chemistry courses have been adopted
by some instructors as a way to improve student outcomes, but little research has been done on
the causal relationships among particular aspects of the learning environment and student
outcomes. This makes it difficult for classroom teachers to know which aspects of a
constructivist teaching approach are critical to adopt and which may be modified to better suit a
particular learning environment while still improving student outcomes.
To investigate a model of these relationships, a survey designed to measure student
perceptions of three factors characterizing a constructivist learning environment in online
courses was adapted for use in face-to-face chemistry courses. These three factors, teaching
presence, social presence, and cognitive presence, were measured using a slightly modified
version of the Community of Inquiry (CoI) instrument. The student outcomes investigated in this
research were satisfaction and academic achievement, as measured by standardized American
Chemical Society (ACS) exam scores and course grades.
Structural equation modeling (SEM) was used to statistically model relationships among
the three presence factors and student outcome variables for 391 students enrolled in six sections
of a general chemistry course taught by four instructors at a single university using a common
textbook. The quantitative analysis of student data was supported by investigating the instructor's
approach to teaching using instructor responses to a modified version of the Approaches to
Teaching Inventory (ATI), semi-structured interview questions, and information available in the
course syllabus.
The results of the SEM analysis indicate that incoming math ability, as measured by ACT
math scores, has the largest effect on student academic achievement in introductory chemistry
courses. Of the three presence factors, cognitive presence has the largest direct effect on
academic achievement and student satisfaction. Teaching presence has a direct effect on
satisfaction similar in size to the effect of cognitive presence. The relationship between social
presence and student outcomes is found to be relatively small. Given the role that both teaching
and social presence play in influencing cognitive presence, these results suggest that classroom
teachers should emphasize the development of a learning environment with a large degree of
cognitive presence where students take ownership of their own learning process. This type of
learning environment can be supported by specific instructor behaviors such as facilitating
discussions and implementing group work focused on collaboration and developing shared
understandings.
ii
This dissertation by Regis Komperda fulfills the dissertation requirement for the doctoral degree in Chemical Education approved by Diane M. Bunce, Ph.D., as Director, and by Gregory Miller, Ph.D., Marc M. Sebrechts, Ph.D., and Gregory R. Hancock, Ph.D., as Readers.
__________________________________________
Diane M. Bunce, Ph.D., Director
__________________________________________
Gregory Miller, Ph.D., Reader
__________________________________________
Marc M. Sebrechts, Ph.D., Reader
__________________________________________
Gregory R. Hancock, Ph.D., Reader
iii
Table of Contents
List of Tables ............................................................................................................................... viii
List of Figures ................................................................................................................................. x
Chapter 1 ......................................................................................................................................... 1
Foundations of Constructivism ................................................................................................... 2
Constructivism in Chemical Education ...................................................................................... 3
Constructivism in Online Education ........................................................................................... 6
Research Model ........................................................................................................................ 10
Instructor Approaches to Teaching ........................................................................................... 14
Research Questions ................................................................................................................... 16
Modification and Pilot Study of Survey Instruments ............................................................... 16
Methodology and Sample Size ................................................................................................. 19
Data Analysis ............................................................................................................................ 21
Summary of Results and Implications for Teaching ................................................................ 22
Limitations and Future Research .............................................................................................. 25
Chapter 2 ....................................................................................................................................... 27
Philosophical and Psychological Foundations of Constructivism ............................................ 28
Constructivism as a Model of Learning .................................................................................... 30
Constructivism in Education ..................................................................................................... 35
Chemical Education .............................................................................................................. 41
Online Education .................................................................................................................. 44
Measuring a Constructivist Learning Environment and Student Outcomes ............................. 48
Defining Constructivism ....................................................................................................... 48
iv
Development of the Community of Inquiry Student Survey Instrument .............................. 50
Measuring Student Outcomes in Constructivist Learning Environments ............................. 62
Modeling the Influence of a Constructivist Learning Environment on Student Outcomes .. 67
Measuring Instructor Approaches to Teaching ......................................................................... 72
Research Questions ................................................................................................................... 78
Chapter 3 ....................................................................................................................................... 80
Modifications to ATI and CoI Wording ................................................................................... 80
Instructor and Student Survey Pilot Studies ............................................................................. 86
Recruitment of Participants for Pilot Studies ........................................................................ 86
Instructor Survey Pilot Study Methodology ......................................................................... 87
Student Survey Pilot Study Methodology ............................................................................. 88
Pilot Study Results .................................................................................................................... 89
Instructor Survey ................................................................................................................... 89
Student Survey ...................................................................................................................... 98
Survey Instrument Validity Evidence ..................................................................................... 101
Test Content ........................................................................................................................ 104
Response Process ................................................................................................................ 105
Internal Structure ................................................................................................................ 107
Relationships with Other Variables .................................................................................... 108
Consequences of Use .......................................................................................................... 109
Power Analysis for Sample Size Determination ..................................................................... 111
Overall Data-Model Fit ....................................................................................................... 112
v
Testing Parameters within the Model ................................................................................. 116
Selection of model parameters ........................................................................................ 117
Simplification of model-implied correlation matrix ....................................................... 119
Construction of the model-implied correlation matrix ................................................... 122
Calculation of model fit function values ......................................................................... 124
Methodology ........................................................................................................................... 127
Data Analysis .......................................................................................................................... 130
Qualitative Data Analysis ................................................................................................... 130
Quantitative Data Analysis ................................................................................................. 132
Data cleaning .................................................................................................................. 132
Assumptions for CFA and SEM analysis ....................................................................... 135
Internal structure of the CoI instrument .......................................................................... 137
Average scale scores and scale reliability ....................................................................... 141
Two-phase SEM Analysis ................................................................................................... 142
Chapter 4 ..................................................................................................................................... 146
Instructor Survey and Interview Results ................................................................................. 147
Student Data Analysis Results ................................................................................................ 150
Descriptive Statistics and Assumptions for SEM Analysis ................................................ 150
Confirmatory Factor Analysis of CoI Data ......................................................................... 154
CoI Scale Scores and Reliability ......................................................................................... 158
Structural Equation Model Results ..................................................................................... 162
Addressing the Research Questions ........................................................................................ 176
vi
Research Question 1 ........................................................................................................... 176
Research Question 2 ........................................................................................................... 180
Research Question 3 ........................................................................................................... 184
Summary of Results ................................................................................................................ 186
Chapter 5 ..................................................................................................................................... 188
Contribution of Results to Existing Literature ........................................................................ 189
Instructor and Student Ratings of Learning Environment .................................................. 189
Influence of a Constructivist Learning Environment on Student Outcomes ...................... 191
Implication of Results for Teaching Introductory Undergraduate Chemistry ........................ 197
Limitations and Future Research ............................................................................................ 200
Appendix A – Original CoI Items and Loadings ........................................................................ 205
Appendix B – ATI Revised for Pilot Study ................................................................................ 209
Appendix C – Student Survey Items Used in Pilot Study .......................................................... 211
Appendix D – ATI Revisions After Pilot Study ......................................................................... 214
Appendix E – Student Survey Revisions After Pilot Study ........................................................ 217
Appendix F – R Program for Calculating Coefficient H ............................................................ 220
Appendix G – Path Tracing and Matrix Determination for Hypothesized Research Model ...... 221
Appendix H – LISREL Syntax and Output for Power Analysis ................................................ 226
Appendix I – CoI and Satisfaction Instrument Used for Student Data Collection ..................... 230
Appendix J – Interview Transcript and Course Syllabus Coding Rubric ................................... 232
Appendix K – Mplus Model Syntax ........................................................................................... 233
Appendix L – Student Variable Correlations from Mplus .......................................................... 239
vii
References ................................................................................................................................... 242
viii
List of Tables Table 1: Original and Revised CoI Items ..................................................................................... 82 Table 2: Notation Used in Full Structural Equation Model in Figure 13 ................................... 115 Table 3: Sample Sizes for Testing Overall Data-Model Fit ....................................................... 116 Table 4: Sample Size Necessary to Test Each Focal Parameter Arranged from Largest to
Smallest ................................................................................................................................. 126 Table 5: Semi-Structured Instructor Interview Questions .......................................................... 130 Table 6: Evidence of Teaching Approaches Aligned with Constructivism from Interviews and
Syllabi ................................................................................................................................... 149 Table 7: Descriptive Statistics for Student Academic Variables .................................................151 Table 8: Descriptive Statistics for Student Survey Variables ......................................................151 Table 9: Independent t-tests for Differences in Academic Variables Based on Missing Survey
Responses .............................................................................................................................. 153 Table 10: Nested Model Comparison for Teaching Presence Factor ......................................... 156 Table 11: Social Presence Items with Added Error Covariance Terms ...................................... 157 Table 12: Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument .... 160 Table 13: Reliability Values for the Presence and Satisfaction Scales ....................................... 162 Table 14: Satisfaction Items from Student Survey ..................................................................... 165 Table 15: Model Parameter Values and Standard Errors (SE) from Final Research Model ...... 166 Table 16: Decomposed Standardized and Unstandardized Effects Among Variables in Structural
Model .................................................................................................................................... 169 Table 17: Standardized Loadings for Satisfaction Items in the Current Research and Existing
Literature ............................................................................................................................... 174 Table 18: Item on Each Presence Scale with Highest Mean Rating ............................................179
ix
Table 19: Cronbach’s Alpha for CoI Presence Scales in the Current Research and Existing
Literature ............................................................................................................................... 183 Table 20: Original CoI Item Wordings and Published Loadings, Paths, or Correlations ........... 205 Table 21: ATI Items Used in Pilot Study, Revised ATI Items, and Rationale for Revision ...... 214 Table 22: Algebraic Statements from Path Tracing .....................................................................221 Table 23: Correlation Matrix from Mplus ...................................................................................239
x
List of Figures
Figure 1. The Community of Inquiry model ................................................................................... 9 Figure 2. The hypothesized structural model of relationships among the three CoI presence
factors, math ability scores, and student outcomes ................................................................. 11 Figure 3. The Community of Inquiry model ................................................................................. 47 Figure 4. A factor model with three measured variables and one latent factor ............................ 54 Figure 5. A component model with three measured variables and one latent factor (component)
................................................................................................................................................. 54 Figure 6. A factor model with three measured variables and one latent factor with the error terms shown for each measured variable ................................................................................ 55 Figure 7. A model of hypothesized relationships among the 34 items on the CoI student
survey and the three presence factors ..................................................................................... 57 Figure 8. A model of hypothesized relationships among the three CoI presence factors
and student satisfaction ........................................................................................................... 60 Figure 9. A model of hypothesized relationships among the three CoI presence factors and
student satisfaction indicating the nonsignificant path between social presence and satisfaction .............................................................................................................................. 60
Figure 10. The hypothesized structural model of relationships among the three CoI presence
factors, math ability scores, and student outcomes ................................................................. 68 Figure 11. Teaching presence as a single factor with 13 indicator variables ............................... 71 Figure 12. Two correlated factors taking the place of a single teaching presence factor ............. 71 Figure 13. The primary structural equation model with all parameters labeled ......................... 114 Figure 14. The primary research model conceptualized with all focal parameters existing among
latent variables ...................................................................................................................... 120 Figure 15. The three-factor model of the CoI survey instrument used to inform the two-phase
SEM analysis ........................................................................................................................ 140 Figure 16. The CFA model used in the measurement phase of the SEM analysis ..................... 143
xi
Figure 17. The structural model used in the second phase of the SEM analysis ........................ 145 Figure 18. The three-factor model of the CoI survey with standardized parameter values ........ 159 Figure 19. Standardized values for focal parameters in the structural model and R2 values for
endogenous variables ............................................................................................................ 164
xii
Acknowledgements
I am deeply indebted to many people for their support and guidance. The dissertation
process includes both moments of exhilaration and moments of self-doubt and I have been
fortunate to have someone looking out for me every step of the way. The following people
provided the pricks of brightness I needed in order to see the light at the end of the tunnel and
continue moving forward.
First, I would like to thank my advisor, Dr. Diane Bunce, for her patience with the
process of extracting a dissertation topic from the tangled web of my mind and for allowing me
the independence to pursue a project that, in the early stages, seemed impossible. Her feedback
and assistance at every stage of this process were invaluable, and I truly appreciate the time and
energy she invested in reading and providing edits on multiple drafts of this dissertation.
Similarly, I want to thank Dr. Gregory Miller, Dr. Marc Sebrechts, and Dr. Gregory Hancock for
generously giving their time to serve on my committee and for their valuable insights and helpful
suggestions to improve this research. Additionally, I gratefully acknowledge all the instructors
and students who provided the data that ultimately made this research possible.
To my statistics professors, Dr. Michaela Zajicek-Farber and Dr. Hancock, thank you for
being exceptional teachers and helping me to understand, appreciate, and truly enjoy statistics in
a way I never thought I could. I especially want to thank Dr. Hancock for answering my
seemingly endless stream of questions with his characteristically cheerful attitude no matter how
frantic I became. I must also recognize Brian Johnston for giving me the opportunity to use
statistics “in the wild” and for providing me with access to Qualtrics to use for data collection.
Given the unpredictable nature of both my schedule and mood at various stages in this process I
xiii
also want to thank my coworkers for their moral support and for allowing me to vent every time
something did not go according to plan. This dissertation also benefitted greatly from numerous
conversations with colleagues who pointed me towards relevant literature, provided excellent
critiques of my logical arguments, and above all kept me company when I had spent too much
time alone writing or analyzing data.
None of this would have been possible without the constant encouragement of my family
and friends, particularly their unwavering optimism that I would eventually finish. Finally, I
want to thank the best old man dog in the world, Tolle. He provided the perfect mix of snuggles
and walk breaks that kept me active enough to avoid becoming permanently attached to my
laptop, and I am forever grateful for his companionship.
1
Chapter 1 Meta-analysis of 225 studies focusing on student academic performance in undergraduate
science, technology, engineering, and mathematics (STEM) courses demonstrates that students
in courses utilizing active learning techniques performed almost half a standard deviation (0.47)
better on assessments as compared to students exposed to traditional lectures (Freeman et al.,
2014). A similar result (0.39) was obtained when isolating the 22 chemistry course studies
included in the meta-analysis (Freeman et al., 2014). As defined by the researchers, active
learning “engages students in the process of learning through activities and/or discussion in class,
as opposed to passively listening to an expert. It emphasizes higher-order thinking and often
involves group work” (Freeman et al., 2014, p. 814). Another way to define these student-
centered teaching practices emphasizing knowledge construction through active engagement is
as constructivist.
This research utilizes constructivism as a theoretical framework to integrate research
from chemical education and online education. A model is presented which demonstrates how
specific aspects of a constructivist learning environment are hypothesized to influence student
outcomes of academic achievement and satisfaction. The primary focus of this research is the
testing of a structural equation model depicting relationships among student perceptions of
aspects of the learning environment characterizing a constructivist approach to teaching and
student outcomes. The analysis of student data is supported by data collected from the course
instructor regarding his or her approach to teaching. Acceptable fit between the hypothesized
model and the student data provides introductory undergraduate chemistry instructors with more
detailed information regarding which aspects of a constructivist approach to teaching have the
greatest influence on student academic and affective outcomes.
2
Foundations of Constructivism While the term constructivist has been used as an adjective to describe a collection of
student-centered teaching practices, constructivism can also be considered a philosophy as well a
model of learning. As a philosophy, constructivism includes various epistemological and
ontological viewpoints. Generally, constructivists believe that knowledge is constructed by each
individual as a result of his or her experiences (Bodner, 1986; von Glasersfeld, 1981/1984). The
philosophical aspects of constructivism are interwoven in the psychological research of Piaget
(1964/1997), who considered himself a genetic epistemologist and focused his research on
identifying and describing the internal processes carried out by each individual while engaged in
the process of learning.
Psychological research by Piaget, Vygotsky, and Ausubel focused on understanding the
cognitive processes at work when an individual constructs knowledge. The result of these efforts
is a model of learning that describes knowledge construction in terms of situating new
knowledge within existing cognitive structures. Piaget called this process assimilation
(1964/1997). Building on Piaget’s work, Ausubel (1960) determined that new information can
only become part of an existing cognitive structure if the new information is relevant to the
existing structure. Vygotsky’s (1978) research focused on the role of social interactions in
knowledge construction and demonstrated that under the guidance of an adult or more capable
peer, children could solve problems above their current individual developmental level.
Constructivism as a model of learning suggests that teaching practices aligned with the cognitive
development work of Piaget, Ausubel, and Vygotsky should enhance the knowledge construction
process in individuals.
3
Though constructivism is a model of learning not a theory of teaching, it provides a
framework through which teaching practices can be analyzed to determine if they are consistent
with the constructivist model of learning. Teaching practices aligned with constructivism are not
new and many have been successfully implemented in the past before constructivism became
prevalent in education (von Glasersfeld, 1989). In this way, constructivism does not necessarily
represent a new way to teach, but rather a new way to understand which teaching practices
should be effective due their alignment with a constructivist model of learning.
Some teaching practices consistent with constructivism include the instructor (1) acting
as a facilitator of learning, (2) identifying existing student cognitive structures in order to make
the connections between existing knowledge and new knowledge more explicit, (3) creating
authentic problem solving tasks, (4) fostering active student involvement in problem solving, (5)
supporting students working and communicating in groups to socially construct understanding,
(6) encouraging discussion of and reflection on the learning process, and (7) assessing more than
arriving at a correct answer (Duffy & Cunningham, 1996; Hyslop-Margison & Strobel, 2008;
Piaget, 1973; von Glasersfeld, 1989; Windschitl, 2002). In general, teaching practices aligned
with constructivism are student-centered and shift the classroom focus away from an instructor
who is dispensing information. By examining which teaching practices are effective for various
types of students and the alignment of these teaching practices with a constructivist model of
learning, the constructivist model of learning can be tested and refined.
Constructivism in Chemical Education The influence of constructivist teaching practices on student outcomes in chemistry
courses can be seen most directly in research on the use of process-oriented guided-inquiry
4
learning (POGIL; Hanson, 2006, 2008) and peer-led team learning (PLTL; Varma-Nelson &
Banks, 2013; Varma-Nelson & Coppola, 2005). In both POGIL and PLTL, one or more weekly
lectures or recitation sections are replaced with workshop sessions in which groups of students
work together to construct an understanding of chemical concepts. Critically, both PLTL and
POGIL approaches warn against using “drill” or “plug and chug” type problems that encourage
memorization and application of algorithms and instead encourage problems emphasizing
application of concepts or synthesis of new ideas to encourage more active learning. In this way,
both POGIL and PLTL incorporate teaching practices aligned with a constructivist model of
learning.
Investigations of the effect of constructivist teaching practices on student learning are
complicated by the difficulty of directly measuring student learning. The chemical education
research community relies on exam grades, course grades, and standardized American Chemical
Society (ACS) exam scores as ways to measure student learning (Conway, 2014; Gosser,
Kampmeier, & Varma-Nelson, 2010; Gupta, Burke, Mehta, & Greenbowe, 2015; Hall, Curtin-
Soydan, & Canelas, 2014; Lewis & Lewis, 2005; Mitchell, Ippolito, & Lewis, 2012; Ruder &
Hunnicutt, 2008; Tien, Roth, & Kampmeier, 2002). The beneficial effect of POGIL and POGIL-
style instruction on both final exam and final course grades has been demonstrated in a one-
semester organic and biochemistry course for pre-health professionals and in large enrollment
general and organic chemistry courses (Conway, 2014; Ruder & Hunnicutt, 2008). Similar
results have been reported for implementations of PLTL in general and organic chemistry
courses from a variety of institutions including community colleges and research universities
(Gosser et al., 2010; Lewis & Lewis, 2005; Mitchell et al., 2012).
5
Implementing more general constructivist teaching practices, not specifically POGIL and
PLTL, has been shown to improve student academic achievement in chemistry. Hall et al. (2014)
describe a supplemental discussion-type section that has “roots in social constructivism and
borrows elements from a number of learner-centered pedagogies” (p. 37). This program recruited
students with lower SAT scores who, after enrollment in the program, earned exam scores in
both their general and organic chemistry courses that were not statistically different from peers
entering with higher SAT scores. Considering SAT scores is necessary to show the effectiveness
of teaching practices across different groups of students, since a demonstrated relationship exists
between math ability scores, as measured by SAT and ACT math scores, and grades in
introductory college science courses (Lewis & Lewis, 2005; Nordstrom, 1990; H. E. Spencer,
1996; Tai, Sadler, & Mintzes, 2006; Xu & Lewis, 2011).
In addition to improvements in academic achievement in chemistry, student satisfaction
with the constructivist learning environment has also been reported in the chemical education
literature (Conway, 2014; Hall et al., 2014; Ruder & Hunnicutt, 2008; Tien et al., 2002). Though
satisfaction and attitude are frequently used interchangeably, Xu & Lewis (2011) identify an
emotional satisfaction aspect of student attitudes towards chemistry. The courses in which the
students in the Xu & Lewis study were enrolled were not specifically described as constructivist,
but a correlation of 0.35 was found between emotional satisfaction and ACS exam scores
indicating that a relationship exists between student satisfaction and academic outcomes.
However, larger correlations (0.45 and 0.46) existed between ACS exam scores and math ability
scores as measured by SAT math and ACT math scores, respectively (Xu & Lewis, 2011),
confirming the relationship between math ability and academic achievement in chemistry.
6
Taken together, these studies in the chemical education literature demonstrate that student
academic achievement in introductory chemistry courses is related to both the teaching approach
employed by the instructor and the preexisting math ability of the student. In addition, a
relationship has been demonstrated between academic achievement and student satisfaction with
a course. Some of the teaching approaches utilized in these studies have explicit ties to
constructivism while other studies adopt more general student-centered approaches to teaching
that are also aligned with constructivism. However, none of the studies explores the specific
aspects of the teaching approach that may influence student outcomes. Without this level of
detail, it is difficult for classroom instructors to know which aspects of a constructivist teaching
approach are critically important to adopt to influence student outcomes and which aspects can
be modified to better suit the style of a particular instructor.
Constructivism in Online Education Research in online education has taken a complementary approach to investigating the
role of constructivism. Whereas chemical education has emphasized the student outcomes that
occur as a result of adopting student-centered constructivist teaching practices, online education
research has focused on identifying aspects of the learning environment that influence student
outcomes of satisfaction and persistence. For online educators, constructivism is a useful
framework for engaging students and fostering the development of knowledge (Vrasidas, 2000)
without the constant presence of a teacher.
The first cohesive model of online learning came from the development of the
Community of Inquiry (CoI) model. The CoI model was developed from analysis of computer-
conferencing transcripts (D. R. Garrison, Anderson, & Archer, 2000) using a grounded theory
7
approach of working from data to develop a theory (Creswell, 2013). From this analysis, three
types of presence emerged as the foundation of the CoI model: cognitive presence, social
presence, and teaching presence. The influence of constructivism can be seen in each type of
presence.
Cognitive presence is defined as “the extent to which the participants in any particular
configuration of a community of inquiry are able to construct meaning through sustained
communication” (D. R. Garrison et al., 2000, p. 89). Here the idea of knowledge construction by
constructing meaning indicates that cognitive presence is not simply a matter of providing
content to students but rather is related to the degree to which the content fosters mental activity
on the part of the students. In the CoI model, cognitive presence is described in terms of a four
stage inquiry learning cycle. This definition also highlights the social nature of the CoI model
since the role of both the other students in the course and the instructor is to provide someone to
communicate with to construct meaning.
The three types of presence in the CoI model work together to encourage deep learning,
as illustrated by the fact that “cognitive presence…is more easily sustained when a significant
degree of social presence has been established” (D. R. Garrison et al., 2000, p. 95). The CoI
definition of social presence as “the ability of participants in a community of inquiry to project
themselves socially and emotionally, as ‘real’ people (i.e., their full personality)” (D. R. Garrison
et al., 2000, p. 94) indicates that social presence is only linked to a successful educational
experience when it supports affective as well as cognitive outcomes. These affective outcomes
such as finding “the interaction in the group enjoyable and personally fulfilling” (D. R. Garrison
8
et al., 2000, p. 89) are described as important for keeping students engaged and enrolled in the
online course.
The final aspect of the CoI model, teaching presence, describes the responsibility of the
instructor to establish a learning environment that supports the development of both social and
cognitive presence. The role of the instructor as a facilitator of learning invokes the student-
centered emphasis of constructivism. The overlap of teaching presence and cognitive presence is
necessary in order to select content that will encourage students to follow the stepwise inquiry
cycle that comprises cognitive presence, and the overlap of teaching presence and social
presence is necessary to set a course climate that will foster the development of a learning
community. A visual description of the CoI model can be seen in Figure 1. In this model the
three presence factors are represented by overlapping circles and the indicators are represented
by arrows pointing to each presence factor. The steps in the inquiry cycle are joined by dashed
arrows. These indicators highlight the types of activities hypothesized to improve learning when
undertaken by the instructor and students.
The CoI model indicators were operationalized by online education researchers to create
a student survey instrument that could be used to determine the degree to which a learning
environment was perceived by students as fostering these three types of presence. Since these
types of presence describe various aspects of a constructivist learning environment, this survey
can also be used as a tool to measure the degree to which an instructor has created a
constructivist learning environment. Utilizing student perceptions of the learning environment is
an advantageous technique because students are the intended target of the teaching approach
employed by the instructor. Additionally, students directly experience the learning environment
9
Figure 1. The Community of Inquiry model. Indicators of each type of presence are given in the arrows. Adapted from D. R. Garrison et al. (2000) and Swan (2003). over an extended period of time and are therefore able to provide a more complete description
of the learning environment than an observer who may not attend all class sessions or fully
engage in activities and assignments.
The development and use of the CoI student survey has been documented in the online
education literature with both undergraduate and graduate students enrolled in fully online
courses from various disciplines from engineering to education to business (Arbaugh, Bangert, &
Cleveland-Innes, 2010; Arbaugh, 2008; Bangert, 2008; D. R. Garrison, Cleveland-Innes, &
Fung, 2010; Joo, Lim, & Kim, 2011; Shea & Bidjerano, 2009). After the initial instrument
10
development studies demonstrated the utility of the 34-item CoI instrument, researchers began to
look at relationships among the three CoI presence factors (cognitive, social, and teaching) and
student outcome variables such as satisfaction and persistence (Joo et al., 2011) utilizing the
statistical technique of structural equation modeling (SEM). The CoI instrument items and
literature values for relationships among factors are provided in Appendix A.
The structural equations in SEM describe hypothesized causal relations among variables.
Causal relations provide more specific information about the hypothesized influences of
variables than correlations. Correlations among the three CoI presences were expected to exist
due to their overlapping nature in the CoI model, but the correlations could not provide
information about causal paths among the variables including the influence of cognitive, social,
and teaching presence on student outcomes. The purpose of the current research using SEM will
be to investigate the causal relationships among the three CoI presence factors and student
outcomes in face-to-face introductory chemistry courses.
Research Model The previously discussed studies in online and chemical education can be synthesized into
a single model hypothesizing the influence of a constructivist learning environment on student
outcomes of academic achievement in chemistry and satisfaction. In this model, the
constructivist learning environment is measured by the three CoI factors of cognitive presence,
social presence, and teaching presence. Academic achievement in chemistry is measured by the
outcomes typically used in chemical education research such as ACS exam scores and final
course grades. Student satisfaction is measured using a survey instrument as is typical in both
online and chemical education research.
11
The model in Figure 2 provides a diagrammatic representation of the hypothesized
structural relationships among these variables. Relationships, or paths, between two variables are
indicated with directional arrows. Latent variables, also called factors, are shown as ovals and
represent variables that are not measured directly but will be identified by statistical analysis of
student responses to CoI and student satisfaction survey items. Measured variables of math
ability and academic achievement in chemistry are shown as rectangles and are determined based
on student scores. The portion of the model showing the individual CoI and satisfaction survey
items has been omitted for simplicity in presentation.
Figure 2. The hypothesized structural model of relationships among the three CoI presence factors, math ability scores, and student outcomes.
12
Based on prior research in online education, teaching presence is hypothesized to directly
influence both cognitive and social presence while also indirectly influencing cognitive presence
through social presence. The multiple influences of teaching presence are due to the role of the
instructor in both selecting course content and setting the tone of interactions between the
instructor and students (D. R. Garrison et al., 2010; Shea & Bidjerano, 2009; Swan, 2003). From
the chemical education literature, the cognitive presence and teaching presence aspects of a
constructivist learning environment are expected to directly influence academic achievement in
chemistry as measured by ACS exam scores and final course grades (Conway, 2014; Gosser et
al., 2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder & Hunnicutt,
2008). The direct influence of teaching presence on ACS exam scores and final course grades is
hypothesized because “the instructor serves as an expert who plans instruction to stimulate
students’ interest, motivates their participation in the learning process, and facilitates their
learning” (Swan, 2003, p. 8). Though no studies with the CoI instrument have specifically
examined the influence of teaching presence on student academic outcomes, the role of the
instructor as expert and facilitator of learning suggests a causal relationship between teaching
presence and academic achievement in chemistry.
Additionally, in the CoI model selecting appropriate content is at the overlap of teaching
and cognitive presence. The selection of content and learning activities that facilitate student
knowledge construction is expected to influence the demonstration of that knowledge
construction through ACS exam scores and final course grades. For this reason, cognitive
presence is also hypothesized to have a causal influence on academic achievement in chemistry.
13
In addition to the influence of teaching and cognitive presence, math ability is also
expected to have a direct influence on academic achievement in chemistry (Lewis & Lewis,
2005; Mitchell et al., 2012; Nordstrom, 1990; Tien et al., 2002; Xu & Lewis, 2011). A path is
included to account for the influence of ACS exam scores on final course grades. The paths
between social presence and both ACS exam scores and final course grades are omitted because
the influence of social presence on these two outcomes is hypothesized to be indirect and
mediated primarily by cognitive presence. This indirect influence is hypothesized because of the
small correlation seen between social presence and perceived learning in earlier studies
(Arbaugh, 2008) and the assumption that social presence only affects academic outcomes when
cognitive presence provides an academic context for the social interactions among students.
Finally, from both the online and chemical education literature, cognitive presence, social
presence, and teaching presence are all expected to directly influence satisfaction (Conway,
2014; Hall et al., 2014; Joo et al., 2011; Ruder & Hunnicutt, 2008). A two-headed arrow is
included to account for a relationship between final course grades and student satisfaction
beyond the relationships already present in the model, but no causal direction is proposed for the
relationship. The lack of causality reflects the ongoing debate as to whether students who are
more satisfied with a course perform better academically or if performing better academically
causes students to be more satisfied with a course (Greenwald & Gillmore, 1997; Howard &
Maxwell, 1982).
This hypothesized model represents the integration of research in both chemical education
and online education through their shared use of constructivism as a foundation for the
development of teaching practices aligned with how students learn. In this model, student
14
perceptions and student performance are the variables used to model relationships among
constructivist learning environment factors and outcomes of student satisfaction and academic
achievement in introductory chemistry. However, student measurements should not provide the
only source of information about the degree to which a learning environment incorporates
constructivist principles. The course instructor can also provide data to support or refute the
picture of the learning environment portrayed by student responses to the CoI survey instrument.
Instructor Approaches to Teaching Previous studies in online and science education have utilized interviews with instructors in
order to determine their approach to teaching (Arbaugh & Benbunan-Fich, 2007; Prosser,
Trigwell, & Taylor, 1994; Trigwell, Prosser, & Taylor, 1994). In the Arbaugh and Benbunan-
Fich (2007) study, instructors’ responses were compared with information in the course syllabi
or course websites in order to support the researchers’ classification of the course as either
objectivist or constructivist and group-centered or individual-centered. All instructor reports in
the Arbaugh and Benbunan-Fich (2007) study were found to be consistent with information
provided in the course syllabi or websites.
Interviews were also used to inform phenomenographic qualitative research investigating
the approaches to teaching adopted by 24 instructors of first-year undergraduate chemistry and
physics (Prosser et al., 1994; Trigwell et al., 1994). In later research, Trigwell & Prosser (2004)
used the interview transcripts to develop an inventory that could be administered to instructors in
order to determine their approach to teaching in a particular context. Statistical analysis of this
Approaches to Teaching Inventory (ATI) indicates that the ATI is an acceptable instrument for
identifying two distinct instructor approaches to teaching in specific contexts (Prosser &
15
Trigwell, 2006; Trigwell, Prosser, & Ginns, 2005). One approach identified by the ATI can be
described as an information transmission teacher-focused approach (ITTF) while the other
represents a conceptual change student-focused approach (CCSF).
The ATI has been used in chemical education research to examine how the teaching
approaches of new university chemistry professors change after attending a short teaching
workshop emphasizing the use of student-centered teaching approaches (Stains, Pilarz, &
Chakraverty, 2015). The instructors who attended this workshop had statistically significantly
higher CCSF scores one week after the workshop compared to their CCSF scores before the
workshop. They also had significantly lower ITTF scores than a control group who did not attend
the workshop. These results suggest that the CCSF scale measures approaches to teaching
aligned with constructivism.
Combining information from the perspectives of two classroom stakeholders, the
instructor and students, provides a more complete picture of the learning environment created by
the instructor and experienced by the students. Measuring the learning environment from both
perspectives also allows for an examination of how well the ATI and CoI instruments measure
the existence of different aspects of a constructivist learning environment. Comparing responses
to the ATI and CoI items also provides support for the acceptability of the CoI instrument for
measuring indicators of a constructivist learning environment from the students’ perspective.
Additionally, the psychometric properties of the CoI survey must be examined quantitatively as a
result of the small modifications in wording required for the instrument to be used with students
in a face-to-face class. Combining CoI responses with measurements of student outcomes will
provide the data necessary to test the model proposed in Figure 2. Testing this model provides
16
information on how a constructivist learning environment, as perceived by students, affects
student satisfaction and academic achievement in chemistry. These research goals are
summarized by the following research questions.
Research Questions
1. Are self-reported instructor approaches to teaching consistent with student
perceptions of the learning environment?
2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for
measuring student perceptions of the indicators of a constructivist learning
environment in a face-to-face introductory undergraduate chemistry course?
3. To what degree does a constructivist learning environment, as measured by
student CoI survey responses, affect outcomes of student satisfaction and
academic achievement in chemistry, as measured by ACS exam scores and final
course grades, when the effect of math ability on academic achievement is
considered?
Modification and Pilot Study of Survey Instruments
Both the ATI and CoI survey instruments required modifications to their wording before
they could be used to address the research questions. Initially, items on both were kept as similar
as possible to their original wording. In the ATI, changes were made to the original
European/Australian wording to more closely align the language with US usage by changing the
term “subject” to “course.” As an example, the item worded “In this subject, I provide the
students with the information they will need to pass the formal assessments” was revised to “In
17
this course, I provide the students the information they will need to pass the formal assessments.”
Similar revisions occurred for all items on the ATI. Four items on the CoI were reworded to
reflect the intended face-to-face focus of the current research. For these items the word “online”
was replaced with “face-to-face”. Two additional items on the CoI were modified to align the
instrument with best practices in survey design (Krosnick & Presser, 2010). One change was
made so that all items related to teaching presence started with the same question stem, “The
instructor…” The second change split the item reading “Reflection on course content and
discussions helped me understand fundamental concepts in this class” into two separate items,
one addressing reflections on course content and the other addressing reflections on discussions.
Changes were also made to the scales of both instruments to provide labels for each of
the five scale points above the corresponding number. In addition, a “Not Applicable” option
was added to the CoI instrument with a numerical value of zero for situations in which an item
did not apply to a specific student or course. It was anticipated that this situation might occur
when formal, structured in-class discussions were not part of the course but the item asked about
discussions.
The survey instruments underwent a pilot study to ensure the items were being
interpreted as intended due to the wording changes and the planned use of the instruments with a
research population that differed from the population reported in the literature. Two types of
satisfaction items were included on the student survey during the pilot study to provide a
comparison of responses to traditional satisfaction items from the online education literature
(Bolliger & Wasilik, 2012) with responses to satisfaction items using a semantic differential
18
scale from the chemical education literature (Xu & Lewis, 2011). The instruments used for the
pilot study with instructors and students are provided in Appendix B and C, respectively.
Permission was obtained from the university institutional review board (IRB) to recruit
instructor and student participants for the pilot studies. Five chemistry instructors and five
undergraduate chemistry students participated in the pilot studies of the instructor and student
survey instruments. The protocol for both pilot studies was a think-aloud interview in which each
participant read the items on the survey instrument aloud and verbalized his or her rationale for
selecting a particular response (Krosnick & Presser, 2010). Analysis of these think-aloud
responses was used to determine if the interpretation of the items aligned with their intended
interpretation or if particular items needed to be reworded to ensure more accurate and consistent
interpretation. Both the instructors and students used a specific introductory chemistry course,
either recently taught or completed, as a reference when responding to the survey items. The
syllabus of the relevant course was collected and analyzed to look for consistency between
responses to the surveys and the classroom practices listed in the syllabus. The instructor pilot
study also included semi-structured interview questions to provide a description of the
instructor’s approach to teaching in his or her own words to compare against responses to the
ATI.
As a result of the instructor pilot study, extensive revisions were made to the ATI to
make the items less vague and ensure that each item focused more closely on course design and
actual classroom practices instead of instructor beliefs about teaching. The extensive revisions to
the ATI led to the conclusion that ATI responses should not be used in isolation in future
research. Instead, the main research study combined data from ATI responses with a semi-
19
structured instructor interview and analysis of the course syllabus to provide a more complete
picture of the approach to teaching utilized by the course instructor.
The CoI items did not require extensive revisions, though a few small changes were made
based on student responses to particular items. Student responses demonstrated that the semantic
differential items were best for capturing overall satisfaction with the course, so these were
chosen for use in the main research project instead of the traditional satisfaction items. Appendix
D and E contain the instructor and student survey instruments after revisions based on the results
of the pilot studies.
Methodology and Sample Size
The design of the main research study utilized a mixed methods approach in which both
quantitative and qualitative data were collected and analyzed to answer the research questions
(Creswell, 2014). A mixed methods approach was chosen in order to minimize some of the
limitations of a purely quantitative approach by integrating qualitative data to provide a more
comprehensive understanding of the learning environment. For this research, the primary focus
of the data collection and analysis was quantitative data obtained from instructor responses to the
ATI, student responses to the CoI and satisfaction instrument, and student achievement data in
the form of math ability scores on the math portion of the ACT, ACS exam scores, and final
course grades. The qualitative portion of the research consisted of short semi-structured
instructor interviews and analysis of the course syllabi.
The first research question was addressed by comparing quantitative data from student
responses to the CoI and instructor responses to the ATI with qualitative data obtained from the
instructor interview and course syllabus analysis. The second and third research questions were
20
addressed through statistical analysis of quantitative data. Structural equation modeling (SEM)
was chosen as the primary statistical data analysis methodology over other techniques such as
hierarchical linear modeling (HLM) because of the complex causal relationships investigated in
which the three CoI presence factors were hypothesized to influence each other as well as
multiple student outcomes (Bauer, 2003; Huta, 2014; Kline, 2011). The second research question
can be considered a subset of the third research question in which only relationships among the
three CoI presence factors were examined. Since the third research question can only be
answered by testing the hypothesized model of relationships among all student variables, the
necessary sample size for the research was determined from the model illustrated in Figure 2.
Two separate power analyses were conducted to determine the sample size necessary to
test overall data-model fit and to test the specific model parameters of interest in this research.
The results of both a priori power analyses indicated that a sample size of approximately 80
students would be sufficient to test both overall data-model fit and specific model parameters of
interest with power = .80 and alpha = .05. The number of instructors participating in the research
was determined by how many instructors taught the students whose data was analyzed.
The student data analyzed in this research were obtained from an existing data set for an
administration of the modified CoI instrument collected for another project investigating
predictors of student success in general chemistry. This data set contained almost 400 usable
anonymized student responses to the CoI and satisfaction survey instrument in addition to scores
on the first-semester ACS general chemistry exam, final course grades excluding laboratory
scores, and ACT math scores. These students were enrolled in six sections of a first semester
general chemistry course taught by four different instructors at a large, public, primarily
21
undergraduate institution (Indiana University Center for Postsecondary Research, n.d.). Though
the student data can be considered as grouped by classroom, the selection of classrooms was
based on availability not random sampling, as would have been necessary for the use of HLM
(Huta, 2014). All four instructors completed the ATI, provided their course syllabus, and
participated in the semi-structured interview.
Data Analysis
The instructor data obtained for this research had three components. The first component
was quantitative responses to the ATI. The second was course syllabi provided by the instructors.
The final component of the instructor data was transcripts of the instructors’ responses to semi-
structured interview questions asking for a description of their approach to teaching the
introductory undergraduate chemistry course from which the student data were collected.
Qualitative analysis and coding of the transcripts of these instructor interviews and course syllabi
demonstrated a major theme of student-centered teaching practices consistent with
constructivism. Ultimately, this qualitative data was analyzed along with the results of instructor
responses to the ATI and student responses to the CoI survey in order to address the first
research question.
The student data analyzed for this research only consisted of the quantitative data
available in the anonymized data set. The quantitative analysis of the student data began by
cleaning the data set to remove unusable student responses. As a result of the data cleaning steps
the number of usable student participants was 391. The 391 responses represent approximately
89% of the total 439 responses collected. Exclusion of 11% of the initial sample still provided
22
roughly five times the minimum sample size necessary to examine the overall data-model fit and
model focal parameters with sufficient power.
Descriptive statistics were computed in order to check assumptions regarding missing
data and normality necessary for selection of the correct estimation technique for use when
testing the models of the hypothesized structure of the CoI instrument and the overall research
model presented in Figure 2. This descriptive statistical data is presented in Chapter 4. Mplus
software (version 7.0) was used to test the hypothesized models.
Initially the internal structure of the CoI instrument was tested with confirmatory factor
analysis (CFA). The CFA results provided information regarding the best overall model for the
individual items and three factors that comprise the CoI instrument. After the CoI CFA,
structural equation modeling (SEM) analysis was used to test the model presented in Figure 2.
This analysis followed the two-phase SEM analysis recommended by Mueller & Hancock
(2008).
Summary of Results and Implications for Teaching
The first research question was addressed by integrating the qualitative data collected
from the instructor interview and syllabus analysis with instructor responses to the ATI and then
comparing this analysis to student responses to the CoI items. Analysis of instructor and student
data indicated that student and instructor perceptions of the learning environment were generally
aligned. This alignment was demonstrated by the students perceiving indicators of a
constructivist learning environment in their CoI responses while the instructors described
approaches to teaching that were consist with a student-centered constructivist approach to
23
teaching in their ATI responses, course syllabi, and interviews. These results are presented in
Chapter 4.
The answer to the second research question was determined from evidence for the
validity and reliability of the CoI survey scores. Some validity evidence resulted from how well
the data fit a hypothesized three-factor model of the CoI instrument based on models of the CoI
survey found in the online education literature (Arbaugh et al., 2010; Arbaugh, 2008; Bangert,
2008; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). The model !"
provided an indication of data-model fit, where smaller values relative to the degrees of freedom
of the model (df) indicate better data-model fit. Data-model fit was also examined using fit
indices such as the comparative fit index (CFI), the root mean square error of approximation
(RMSEA) along with its 90% confidence interval (CI90), and the standardized root mean square
residual (SRMR). Scaled fit indices were computed as a result of corrections to the model !" due
to the nonnormal distribution of the data. A slightly modified three-factor model of the CoI
instrument was found to have the following data-model fit: !scaled,+,-./0
" = 1028.717; CFIscaled =
0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR = 0.061. Though the CFI is below the
target value of 0.95, possibly indicating a relatively small amount of variance and covariance in
the data, the model is a good fit for the data as indicated by the RMSEA and SRMR values based
on joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). This result indicates
that the tested model of the internal structure of the CoI is a viable representation of the true
underlying relationships present in the data. In addition to the overall data-model fit information,
reliability information calculated for the three CoI scales also provides evidence to support the
conclusion that the CoI is an acceptable instrument. Therefore, these results provide a positive
24
answer to the second research question indicating that the modified CoI survey is an acceptable
instrument for measuring student perceptions of the indicators of a constructivist learning
environment in face-to-face introductory undergraduate chemistry courses.
The third research question was addressed broadly by the overall data-model fit for the
full hypothesized research model in Figure 2. This model again had acceptable data-model fit
(!scaled,+,-120
" = 1429.111; CFIscaled = 0.892; RMSEAscaled = 0.053, CI90 = [0.049, 0.056]; SRMR
= 0.065) though not all hypothesized relationships among variables were found to be statistically
significant. Decomposing the relationships between two variables into direct and indirect effects
allowed for an examination of how the variables in the model influenced each other. These
results indicate that while constructivist learning environment factors of teaching presence and
cognitive presence do appear to have an influence on student satisfaction and academic
outcomes, the influence of math ability on academic outcomes is larger. The moderately large
and significant effects of cognitive presence on academic outcomes and affective outcomes
shows that a more general adoption of constructivist teaching approaches, as measured by
cognitive presence, positively influences student outcomes, even after the effects of math ability
are considered.
Additionally, social presence appears to have very minimal influence on either academic
outcomes or student satisfaction. The small effect of social presence on student outcomes
indicates that only group work that encourages the development of cognitive presence through
the use of group activities that foster active student engagement with material in support of
constructing explanations and understanding will ultimately influence student satisfaction and
academic achievement. For classroom instructors, these results suggest that group work should
25
be implemented with careful thought as to how it supports the development of a constructivist
learning environment.
As a result of this research, information is now available about the nature of relationships
among the variables in the research model. Understanding the nature and magnitude of the
relationships among specific factors indicating a constructivist learning environment and student
outcomes supports instructors in making informed decisions about how to most effectively
approach teaching in their own classrooms to maximize student outcomes given the ever-present
limitations of time and energy. By examining the effects of specific aspects of a constructivist
learning environment, rather than the effects of adopting an entire approach to teaching,
instructors should have a better sense of how to implement constructivist teaching practices in a
way that meets the needs and preferences of the individual instructor and the particular
classroom environment.
Limitations and Future Research
The small number of instructors involved in the research limited the ability to provide
additional evidence for the validity of the ATI scores given the extensive revisions that occurred
as a result of the pilot study. Some evidence for the validity of the ATI scores was available
based on instructors’ descriptions of their approaches to teaching provided during the semi-
structured interviews, but it would be beneficial to collect ATI responses from a larger set of
instructors so that the internal structure of the revised ATI instrument could be tested with CFA.
Similarly, the use of preexisting student data made it impossible to conduct think-aloud
interviews with students who had completed the CoI survey while they were still enrolled in the
course in order to provide evidence for the validity of the CoI survey scores based on the
26
response process of the students. Future research should consider conducting think-aloud
interviews with students who complete the CoI survey to ensure their interpretation of the items
is aligned with the intended item interpretations.
Using data from instructors teaching the same course at the same university in the same
semester using the same textbook provided some benefits in minimizing differences across
classrooms so that the student data could be combined into a single data set, but it also limited
the generalizability of this research. Since the current research established a relationship among
the CoI presence factors and student outcomes in a single chemistry course taught by multiple
instructors, future research should gather data from additional chemistry courses in which a
wider variety of approaches to teaching have been adopted. This would allow the research model
to be tested more broadly and provide evidence either supporting or modifying the relationships
examined in this study.
27
Chapter 2
For almost 30 years, chemical educators have developed and evaluated new teaching
practices encouraging a shift from teacher-centered to student-centered classrooms. An example
of student-centered teaching practices would be students working in groups during class time to
actively solve difficult problems and construct their own knowledge of chemical concepts.
Teaching practices emphasizing individual knowledge construction can be classified as
constructivist. One way constructivist teaching practices have entered the chemistry classroom is
through specific pedagogies such as process-oriented guided-inquiry learning (POGIL; Hanson,
2006, 2008) and peer-led team learning (PLTL; Varma-Nelson & Banks, 2013; Varma-Nelson &
Coppola, 2005). Improvement in final exam and final course grades as well as high levels of
student satisfaction with the course have been demonstrated when POGIL and PLTL were
employed in undergraduate general and organic chemistry courses (Conway, 2014; Gosser et al.,
2010; Gupta et al., 2015; Hall et al., 2014; Lewis & Lewis, 2005; Ruder & Hunnicutt, 2008;
Smith, Wilson, Banks, Zhu, & Varma-Nelson, 2014; Tien et al., 2002).
Student-centered teaching practices aligned with constructivism have also been more
broadly adopted by educators in both face-to-face and online classrooms (Duffy & Cunningham,
1996; Hyslop-Margison & Strobel, 2008; Partlow & Gibbs, 2003; Phillips, 1995; Vrasidas,
2000). An instrument that can be used to measure student perceptions of the presence of
constructivist teaching practices through the Community of Inquiry (CoI) model has been
developed by researchers in online education (Arbaugh et al., 2008; D. R. Garrison et al., 2010;
Swan, Garrison, & Richardson, 2009). Additionally, an instrument has been developed to
determine the degree to which an instructor has adopted a student-centered approach to teaching
based on self-reported frequencies of utilizing various classroom practices (Prosser & Trigwell,
28
2006; Trigwell et al., 2005; Trigwell & Prosser, 2004). The combination of these two
instruments allows for the identification of a constructivist learning environment utilizing both
instructor and student perspectives. The prevalence of constructivism in educational research and
practice across disciplines and delivery methods speaks to its acceptance among instructors as a
useful model of how students learn.
Philosophical and Psychological Foundations of Constructivism Contemporary writer Ernst von Glasersfeld traced constructivist beliefs back to eighteenth
century philosophers Giambattista Vico and Immanuel Kant. As understood by von Glasersfeld,
some of the earliest evidence for a belief that individuals construct knowledge as a result of
experiences comes from Vico’s writing that “human truth is what man comes to know as he
builds it” (as cited in von Glasersfeld, 1981/1984, p. 7). While von Glasersfeld adopted Vico’s
epistemology, he also adopted Kant’s ontological perspective on the nature of reality. This
perspective rejects metaphysical realism and the belief that knowledge represents objective
reality experienced in the same way by all people. To von Glasersfeld, knowledge is not a
reflection of objective reality but instead knowledge represents the truth as constructed to fit
experiences. Specifically, von Glasersfeld’s (1981/1984) interpretation is that Kant believed “our
mind does not derive laws from nature, but imposes them [laws] on it [nature]” (p. 3).
Von Glasersfeld was one of the first to create an accessible interpretation of constructivist
philosophies. He considered his interpretation of constructivism as radical because it “breaks
with convention” (1981/1984, p. 5) of knowledge as reflecting objective reality. This radical
interpretation of constructivism led to many concerns regarding the adoption of constructivism in
science education. Of particular concern to scientists were von Glasersfeld’s beliefs that “science
29
(1) cannot reveal ‘objective truth,’ (2) is forever fallible, and (3) is not the most important thing
in the field of human experience” (1993, p. 37). This interpretation of constructivism was
strongly criticized by Suchting (1992), and others who have not adopted the more radical aspects
of von Glasersfeld’s philosophy of constructivism. However, the version of constructivism
brought to the chemical education community by Bodner (1986) and Tobin (1999) was heavily
influenced by von Glasersfeld’s radical definition of constructivism (1989, 1993).
Despite the radical version of constructivism introduced to the chemical education
community, Staver (1998), a researcher and former high school chemistry teacher, believed it
was possible to adopt constructivist beliefs about how knowledge is constructed without
rejecting the ability of constructed knowledge to match the objective reality. He expressed this
belief saying “many constructivists, including myself, choose to remain silent on the issue of
knowledge as a correspondence with the facts of reality” (p. 505). This position is described by
Wink (2014) who believes that most chemical educators either implicitly or passively reject the
radical constructivism of von Glasersfeld and Bodner. The silence of chemical educators on the
philosophical aspects of constructivism may also be a result of debates in the Journal of
Chemical Education about the incompatibility of radical constructivism and commonly held
beliefs about science as a search for objective knowledge (Bernal, 2006; Scerri, 2003).
Consequently, it appears that at least within the chemical education community the acceptance of
constructivism does not require a belief about what knowledge represents but only an acceptance
that knowledge is the result of an individual interpreting and making sense of experiences in
order to construct understanding.
30
Though von Glasersfeld recognized influences from Vico and Kant, he primarily credited
Swiss developmental psychologist and philosopher Jean Piaget’s research on the cognitive
development of children as the foundation of constructivism. Piaget’s study of how knowledge
develops overlapped with the fields of psychology and philosophy (Bodner, 1986; von
Glasersfeld, 1974). Piaget considered himself a genetic epistemologist, and described his
constructivist philosophy by explaining “for the genetic epistemologist, knowledge results from
continuous construction” (as cited in von Glasersfeld, 1974, p. 8).
Piaget primarily focused on the role of the individual, while others investigated the role of
social interactions in knowledge construction. John Dewey, an American educational reformer,
wrote in the early twentieth century that “meanings do not come into being without language,
and language implies two selves [e.g., teacher and student] involved in a conjoint or shared
understanding” (as cited in Garrison, 1995, p. 722). In this way, Dewey’s educational philosophy
emphasized the role of language and social interactions in constructing knowledge. Similar
beliefs influenced the research of Lev Vygotsky, a Soviet psychologist, on the role of language
and social interactions in how children develop (1978). The emphasis of Dewey and Vygotsky
on the role of social interactions in advancing cognitive development has become the basis of
social constructivism in education. Thus, the combination of philosophical beliefs and
psychological research provided support for the transition of constructivism from a philosophical
belief to a model of learning.
Constructivism as a Model of Learning
The constructivist emphasis on learning as individual knowledge construction contrasts
with earlier behaviorist models of learning based on stimulus-response experiments that placed
31
the emphasis for learning on conditions external to the learner (Good, Wandersee, & St. Julien,
1993; Jonassen, 1991). Teaching approaches aligned with a behaviorist model of learning were
useful for memorization of facts and training of specific behaviors, but were not successful in
developing more advanced conceptual understandings desired by many educators (Abraham,
2008; Fosnot & Perry, 2005; Matthews, 1993). Constructivist educational models instead
focused on describing the internal cognitive processes that lead to knowledge construction.
Piaget described his model of learning as simply adding two steps to the stimulus-response
model “it is indeed a stimulus-response theory, if you will, but first you add operations and then
you add equilibration” (1964/1997, p. 27). Both operations and equilibration are internal
processes carried out by each individual while learning is occurring. An operation is “an
interiorised [sic] action which modifies the object of knowledge” (p. 20) and equilibration is “a
process of self-regulation”. Piaget described one type of operation as the incorporation of new
knowledge into existing cognitive structures, which he called assimilation. For Piaget, “learning
is possible only when there is active assimilation. It is this activity on the part of the subject
which seems to me underplayed in the stimulus-response schema” (1964/1997, p. 26). According
to Piaget, in situations where new knowledge cannot be fit into existing cognitive structures, a
change in cognitive structures is required. He called the adjustment of cognitive structures to fit
the new knowledge accommodation. Assimilation and accommodation are expressions of
constructivist principles because they describe knowledge in terms of cognitive building
processes. While current educational theory has moved beyond Piaget’s theory that children
progress in a defined way through four stages of cognitive development (Bunce, 2001; Staver,
1998), his theory of the cognitive processes an individual uses to construct knowledge has
32
become a key element of constructivism applied as a model of learning. Piaget’s work provided a
foundation for research to identify factors that influence the ability of an individual to construct
knowledge.
Educational psychologist David Ausubel (1960) studied the role existing cognitive
structures play in constructing new cognitive structures. According to Ausubel’s subsumption
theory, new information can only become part of an existing cognitive structure if the new
information is relevant to the existing structure. This theory is closely linked to Piaget’s concept
of assimilation, but specifies that the existing structures must be tied to the incoming information
in some way so that the new information can be classified under the existing structures. This led
to Ausubel’s belief that “the most important single factor influencing learning is what the learner
already knows. Ascertain this and teach him accordingly” (as cited in Abraham, 2008, p. 56).
Ausubel’s emphasis on supporting the individual cognitive development of students closely
aligns his research with constructivism.
Cognitive scientist Donald Norman (1980) echoes Ausubel’s belief that learning is an
individual process influenced by the preexisting cognitive structures of the learner. Norman had
initially advocated a “web learning” model (1973) in which the role of the teacher was simply to
make as many connections as possible between pieces of knowledge in order to help the student
develop a robust and redundant network of knowledge that could be accessed in numerous ways.
However, Norman later acknowledged that this theory was too simplistic because it treated the
student as a “passive receptacle” (1980, p. 42). Norman’s revised understanding of knowledge
acquisition falls more in line with Piaget’s idea of active assimilation.
Although I do not yet understand the specific way by which new knowledge is acquired, it does involve active interpretation on the part of the student. The student comes to the
33
learning situation with a large set of preexisting ideas, and the material that is presented is interpreted according to those ideas. You cannot prevent it: I have tried…It is the student who decides what aspects of the material are important, what aspects are not. (1980, pp. 42–43)
Though the work of Piaget, Ausubel, and Norman emphasized the construction of
knowledge within the mind of each individual, this construction is often the result of interactions
with other individuals such as teachers or peers. The role of these interactions is emphasized in
the form of constructivism known as social constructivism. Vygotsky’s research on the
development of knowledge focused on social factors that influence learning and development but
still emphasized the internal nature of the process. Vygotsky employs the analogy of using an x-
ray to “reveal to the teacher how developmental processes stimulated by the course of school
learning are carried through inside the head of each individual child” (1978, p. 91). Social
interactions are important to Vygotsky’s zone of proximal development (ZPD), which he
described as “the distance between the actual developmental level as determined by individual
problem solving and the level of potential development as determined through problem solving
under adult guidance or in collaboration with more capable peers” (1978, p. 86). A critical
component of the ZPD is the presence of someone with more advanced knowledge, either a
teacher or peer. This implies that collaborative groups cannot be successful at improving
problem-solving abilities if all the group members are at the same developmental level or if an
expert is not available to provide guidance. The presence of the expert teacher is especially
important for the development of the most advanced group member since this individual is not in
the company of more capable peers.
The constructivist model of learning helps explain which teaching practices, such as
identifying preexisting cognitive structures of students and well-structured group work, should
34
improve knowledge construction in individuals. This model also unites key aspects of research
on the development of cognitive structures. The constructivist model of learning provides simple
analogies for cognitive processes that are easy to understand without delving into specifics of
neurons and other biological functions related to memory and learning. The analogy of
constructing or building up knowledge is especially meaningful in disciplines such as chemistry
that are typically taught as a progression of cumulative facts and concepts starting from atoms
and gradually increasing in complexity to discuss properties of materials at the macroscopic level
and eventually complex systems of reactions as in biochemistry or chemical engineering.
Scientists in general, and chemists in particular, should be familiar with the use of models
due to their importance in teaching phenomena such as atomic structure. An analogy can be
made between the use of the Bohr model of the atom to understand electron energy levels and
the use of constructivism as a model to understand how learning occurs. The Bohr model of the
atom provides a way to understand the position and color of the lines in the hydrogen spectrum
even though the electrons causing the spectral lines cannot be directly observed. Similarly,
constructivism provides a way to understand the success of various student-centered pedagogies
even though the knowledge structures of students cannot be directly observed. However, the
nature of models is that they are “simplified representations of phenomena or ideas” (Coll,
France, & Taylor, 2005, p. 185). In this way, models are imperfect and cannot accurately
represent all aspects of a phenomena or idea simultaneously. In the same way that the Bohr
model of the atom is not appropriate for more complex multi-electron atoms, the constructivist
model of learning becomes strained when asked to function as a philosophy, model of learning,
and pedagogy simultaneously.
35
Constructivism in Education
Von Glasersfeld (1989) realized that “by supplying a theoretical foundation that seems
compatible with what has worked in the past, constructivism may provide the thousands of less
intuitive educators an accessible way to improve their methods of instruction” (p. 138). While
constructivism can be used as a foundation from which to develop teaching practices aligned
with the constructivist model of learning, constructivism is not a theory of teaching (Committee
on Developments in the Science of Learning, 2000; Windschitl, 2002). The constructivist model
of learning developed from philosophical debate and psychological research speaks only to the
internal processes at work when an individual constructs knowledge. In some sense the phrase
“constructivist teaching practice” is an oxymoron because it is impossible for any teacher, no
matter how skilled, to implant knowledge directly in the mind of the student (Norman, 1980).
Though the constructivist model of learning does not prescribe specific teaching practices,
there has been a general acceptance of constructivism as an appropriate foundation for
developing new pedagogies. However, the development of teaching practices aligned with
constructivism is complicated by the many types of constructivism that can be found in the
literature. These are often referred to as “faces” (Good et al., 1993; Phillips, 1995) or “forms”
(Bodner, Klobuchar, & Geelan, 2001) of constructivism and typically share a core belief that
knowledge is constructed by individuals, but differ in their underlying ontologies or
epistemologies. This has led to vigorous debate and some misconceptions surrounding the
implications of constructivism for educational practice. Phillips (1995) believes that with the
acceptance of constructivism, “a weak or at least a controversial epistemology has become the
basis for a strong pedagogical policy” (p. 11). Fox (2001) takes a more cynical stance on the
36
acceptance of constructivism warning “it is in danger of becoming a general term of approbation
with but little content and an incoherent underlying epistemology” (p. 23).
The epistemological issues identified by Fox (2001) stem from simplistic interpretations of
some of the basic principles upon which all constructivists can agree. These misinterpretations
may be the result of interpretations and reinterpretations of constructivism by different authors in
order to make constructivism more accessible for classroom teachers. As a result, constructivism
may have become “little more than an educational slogan in the absence of conceptual
understanding and clarification” (Hyslop-Margison & Strobel, 2008, p. 73). For example, the
constructivist model of learning holds that all knowledge construction is an individual process,
yet the role of social interactions is frequently cited as critical for knowledge development. At
the extremes, these beliefs would appear to be incompatible and in opposition to one another:
either learning happens in complete isolation or it happens in the presence of others.
To borrow an example from chemistry, a parallel can be drawn to the simplistic way in
which students are sometimes introduced to the idea of ionic and covalent bonding. Ionic
bonding is often introduced as a bond that forms when electrons are lost by one atom and gained
by another. Covalent bonding is then taught as a completely separate type of bonding in which
electrons are shared between two atoms. This presentation sets up a false dichotomy for students
and leads to confusion later when the more complex topic of electronegativity is introduced. In
discussing electronegativity, atoms are described as having differing levels of attraction to the
electrons comprising the bond. When different combinations of atoms form a bond, the possible
types of bonds that can form range from completely ionic to completely covalent. Additionally, a
middle ground is introduced in which the bond cannot be described as purely ionic or purely
37
covalent. This middle ground of unequally shared electrons allows for a richer understanding of
bonding and is critically important to understanding the intermolecular forces that exist between
various atoms, ions, and molecules.
In the same way, a false dichotomy is presented in simplifying constructivism into
opposing camps of individual and social knowledge construction. A combination of these two
ends of the spectrum can be described as an individual constructing his or her own knowledge
while interacting with others. This conceptual middle ground acknowledges the role of both the
individual and his or her social environment. Recognizing that both the individual and the social
environment play a role in knowledge construction opens the door to a richer understanding of
how an effective learning environment can be designed. Many misconceptions about
constructivism stem from incomplete or simplified understandings of how constructivism should
influence educational practice.
Considering learning to be an entirely individual process leads to misconceptions that
constructivist teaching must allow students to discover all knowledge for themselves, thus
prohibiting the teacher from ever directly instructing students. A more correct interpretation is
that all knowledge construction is an individual process since it occurs within the mind of each
individual but this process can occur in any number of situations. Often these situations are
assumed to require the students to be physically active, but this is a misinterpretation of Piaget’s
“active assimilation” (1964/1997, p. 26). The activity described by Piaget is the active process of
knowledge organization that occurs anytime a student is mentally engaging with the material. As
described by Bächtold (2013), “knowledge construction implies activity of the mind but not
necessarily activity of the body” (p. 2478). The long historical tradition of education clearly
38
demonstrates that learning can occur while listening to a lecture or reading a textbook
(Committee on Developments in the Science of Learning, 2000), as long as the student is
mentally engaged in the process of knowledge construction.
Another misconception surrounding constructivism is that by focusing on how each
individual student constructs knowledge, the role of the teacher is deemphasized or even ignored
(Bodner et al., 2001). This concern is predicated on an incomplete understanding of
constructivism. In courses designed using the constructivist model of learning, the role of the
teacher becomes less about disseminating information and more about facilitating the
construction of knowledge in individual students. This new, but critically important role of the
teacher is clearly articulated by Piaget:
It is obvious that the teacher as organizer remains indispensable in order to create the situations and construct the initial devices which present useful problems to the child. Secondly, he is needed to provide counter-examples that compel reflection and reconsideration of over-hasty solutions. What is desired is that the teacher cease being a lecturer, satisfied with transmitting ready-made solutions; his role should rather be that of a mentor stimulating initiative and research. (1973, p. 16)
Although Coll & Taylor (2001) noted a concern that constructivism undermines the expert status
of the teacher, this is likely to be true only in the case where the teacher is regarded as an expert
due solely to his or her ability disseminate facts with no consideration for how to best help
students learn. However, Piaget’s belief that “the teacher-organizer should know not only his
own science, but also be well versed in the details of the development of the child’s or
adolescent’s mind” (1973, pp. 16-17) signifies that both content and pedagogical knowledge are
equally important. In this way, the expert status of the teacher is heightened because content
knowledge expertise must be paired with expertise in the ability to design and manage learning
environments that will encourage knowledge construction.
39
The ontological confusion regarding the role of constructivism in education seems to stem
primarily from von Glasersfeld’s radical interpretation. Here the misinterpretation is that if
knowledge is constructed by each student to make sense of his or her experiences then all student
ideas be must considered as equally valid since they represent reality for that student (Bodner et
al., 2001; Committee on Developments in the Science of Learning, 2000; Windschitl, 2002). For
science educators in particular this misinterpretation led to an erroneous belief that students
should be allowed to hold misconceptions that appear supported by everyday experience, such as
objects requiring a constant force to stay in motion or a belief in pseudoscience such as
intelligent design (Matthews, 1993; Mugaloglu, 2014). These interpretations by Matthews and
Mugaloglu do reflect von Glasersfeld’s radical constructivist beliefs in rejecting the idea of
knowledge as reflecting an objective truth and emphasizing knowledge as construction of reality
for that individual. However, these interpretations miss a key point made by von Glasersfeld
(1984) and summarized by Bodner et al. that “knowledge is no longer true or false; it either
works or it does not” (2001, p. 5). Von Glasersfeld and Bodner’s interpretation of radical
constructivism focuses on the viability of knowledge rather than its correspondence with an
objective reality.
Regardless of whether knowledge reflects an objective reality, constructivism requires
knowledge to be validated by relevant experiences. In this way, it is similar to how scientific
knowledge must be validated by relevant experimentation and evidence. In the context of social
constructivism, knowledge can be validated through group consensus. In considering
misconceptions or pseudoscientific beliefs held by students, there is no foundation for the
knowledge of the student to supersede the knowledge of the teacher. This is especially true
40
because the teacher is responsible for developing curriculum to test knowledge and facilitate
discussions that lead to understandings. A teacher can use the existing knowledge of students,
including misconceptions and pseudoscientific beliefs, to design classroom activities that provide
evidence contradicting the existing knowledge. As students have their knowledge tested and
found to be unsatisfactory, it is hypothesized that they will adjust their cognitive structures to
align with the new evidence and dismiss misconceptions and pseudoscientific beliefs. To this
end, one of the first uses of constructivism in chemical education was as a theoretical framework
for identifying and addressing student misconceptions (Bodner et al., 2001).
Another issue that can arise from a simplified view of constructivism is interpreting social
constructivism to imply that any type of group work is beneficial to learning. This can occur
when the role of Vytogsky’s ZPD is not fully understood. If the teacher understands that students
are placed in groups not because social interaction itself improves learning but because social
interactions allows a transition from one developmental level to another in the presence of a
more capable peer or adult, the teacher is likely to recognize the importance of monitoring and
providing guidance to the groups. If the current group of peers cannot provide an opportunity for
all students to advance developmentally, then groups may need to be periodically rearranged or
the teacher may need to provide additional support for more advanced students.
Though constructivism itself is a model of learning not a theory of teaching, it provides a
framework through which teaching practices can be analyzed to determine if they are aligned
with the constructivist model of learning. As von Glasersfeld (1989) made clear, teaching
practices aligned with constructivism are not new and many have been successfully implemented
in the past before constructivism became important in education. In this way, constructivism
41
does not necessarily represent a new way to teach, but rather a new way to understand which
teaching practices should be effective based on their alignment with a constructivist model of
learning. These teaching practices are typically student-centered and shift the classroom focus
away from a teacher who is dispensing information. Examples of teaching practices aligned with
constructivism include the teacher (1) acting as a facilitator of learning, (2) identifying existing
student cognitive structures in order to make the connections between existing knowledge and
new knowledge more explicit, (3) creating authentic problem solving tasks, (4) fostering active
student involvement in problem solving, (5) supporting students working and communicating in
groups to socially construct understanding, (6) encouraging discussion of and reflection on the
learning process, and (7) assessing more than arriving at a correct answer (Duffy & Cunningham,
1996; Hyslop-Margison & Strobel, 2008; Piaget, 1973; von Glasersfeld, 1989; Windschitl,
2002). By examining which teaching practices are more or less effective for various types of
students and the alignment of these teaching practices with a constructivist model of learning, the
constructivist model of learning can continue to be tested and refined.
Chemical Education
The chemical education community has a long history of utilizing current learning theories
to inform research and practice. Starting with the learning theories of Piaget and Ausubel
(Abraham, 2008; Herron, 1975; Novak, 1984; Nurrenbern, 2001), chemical educators have
gradually adopted constructivism as a foundation for both teaching and research. As previously
discussed, the version of constructivism brought to the chemical education community by
Bodner (2001; 1986) and Tobin (1999) was heavily influenced by von Glasersfeld’s radical
definition of constructivism (1989, 1993) and was met with some criticism by scientists
42
uncomfortable with the rejection of science as a process of discovering objective truths about the
natural world.
Perhaps unsurprisingly there is little evidence of radical constructivism in the current way
constructivism in used as a framework for teaching and research by chemical educators. Yet, a
Google Scholar search conducted in April 2016 listed over 1000 combined citations of Bodner’s
1986 and Bodner et al.’s 2001 constructivism articles, which interpreted von Glasersfeld for
chemical educators. Wink (2014) provides one possible explanation for the omission of radical
constructivism from chemical education by proposing a separation of the pedagogical and
philosophical components of constructivism. Instead of continuing to debate the philosophical
components of constructivism, chemical educators appear to have instead embraced
constructivism as a model of learning and focused on the pedagogical implications of
constructivism by providing opportunities for students to construct their own knowledge in a
student-centered learning environment. Regardless of the debate about the nature of reality that
surrounds constructivism as a philosophy, constructivism has still provided a sound theoretical
framework around which to design effective new approaches to chemical education teaching and
research.
Two specific pedagogies in chemical education that emphasize the instructor as a facilitator
of learning in more student-centered environments are process-oriented guided-inquiry learning
(POGIL; Hanson, 2006, 2008) and peer-lead team learning (PLTL; Varma-Nelson & Banks,
2013; Varma-Nelson & Coppola, 2005). In both POGIL and PLTL, one or more weekly lectures
or recitation sections are replaced with workshop sessions in which groups of students work
together to construct an understanding of chemical concepts. In this way, both POGIL and PLTL
43
incorporate teaching practices aligned with a constructivist model of learning. Evidence for
improved student learning outcomes as a result of utilizing POGIL and PLTL will be discussed
later in the chapter more detail.
Group work is a key component in POGIL and PLTL classrooms and aligns both
approaches with social constructivism (Abraham, 2008; Smith et al., 2014). The foundations of
PLTL include working cooperatively in teams to construct knowledge through discussion and
debate (Varma-Nelson & Coppola, 2005). This aligns with Tobin’s (1999) description of
constructivism as encouraging “learning from diverse sources and providing social contexts in
which individuals can make sense and test their emerging understandings” (p. 238). For POGIL,
the role of social constructivism is not explicitly stated, but can be inferred from
recommendations highlighting the importance of a “diversity of perspective and skills that
produces a rich exchange of ideas” (Hanson, 2006, p. 22). These diverse skills allow students to
help each other progress through Vygotsky’s ZPD under the guidance of someone functioning at
a more advanced level.
During group work, POGIL activities utilize a guided inquiry learning cycle. Student
groups explore a topic, then formulate a conceptual understanding, and eventually apply this
understanding in a new context. This cycle invokes constructivist principles of first constructing
knowledge to make sense of experiences then testing the viability of that knowledge. In PLTL,
less emphasis is placed on a specific learning cycle. However, it is critical that PLTL workshop
materials are “challenging, intended to encourage active learning and to work with groups”
(Varma-Nelson & Coppola, 2005, p. 8). Vygotsky’s ZPD is used to explain how the challenging
materials should be just beyond the immediate ability level of the students. Without
44
appropriately designed materials, there is no reason for students to work together in groups to
solve problems because the problems will be easy enough to solve individually (Varma-Nelson
& Coppola, 2005, p. 5). To avoid this, both PLTL and POGIL approaches warn against using
“drill” or “plug and chug” type problems that encourage memorization and application of
algorithms and instead encourage problems emphasizing application of concepts or synthesis of
new ideas.
Online Education
The philosophical debate surrounding constructivism is much less evident in online
education. It may be that online educators from disciplines with less empirical foundations are
more comfortable with understanding each person to have a conception of reality shaped by his
or her own experiences. Or, like chemical educators, online educators may also have separated
the philosophical and pedagogical components of constructivism. Then, instead of having a
philosophical debate, online educators instead focus on using a constructivist model of learning
to inform pedagogy.
For online educators, constructivism is a useful framework for engaging students and
fostering the development of knowledge (Vrasidas, 2000). Since much of the research in online
education is concerned with keeping students engaged and learning without the constant
presence of a teacher, it is not surprising that “constructivist models of learning are almost
exclusively recommended as a guide for the design and delivery of Internet-based courses”
(Bangert, 2008, p. 28). The first cohesive model of online learning came from the Community of
Inquiry (CoI) model. The CoI model was developed from analysis of computer-conferencing
transcripts (D. R. Garrison et al., 2000) using a grounded theory approach of working from data
45
to develop a theory (Creswell, 2013). From this analysis, three types of presence emerged as the
foundation of the CoI model: cognitive presence, social presence, and teaching presence. The
influence of constructivism can be seen in each type of presence.
Cognitive presence is defined as “the extent to which the participants in any particular
configuration of a community of inquiry are able to construct meaning through sustained
communication” (D. R. Garrison et al., 2000, p. 89). Here the idea of knowledge construction by
constructing meaning is incorporated directly into the definition of cognitive presence. This
indicates that cognitive presence is not simply a matter of providing content to students but
rather is related to the degree to which the content fosters mental activity on the part of the
students. This definition also highlights the social nature of the CoI model since the construction
process is thought to result from sustained communication. In this way, the CoI model does not
isolate cognitive presence from social presence and teaching presence. The role of both the other
students in the course and the instructor is to provide someone to communicate with in order to
engage in the process of socially constructing meaning.
The link between cognitive presence and social presence is further solidified by the idea
that “cognitive presence…is more easily sustained when a significant degree of social presence
has been established” (D. R. Garrison et al., 2000, p. 95). This can be interpreted as a restatement
of social constructivism in the sense that knowledge construction benefits from social
interactions due to the role of social interactions in “indirectly facilitating the process of critical
thinking” (D. R. Garrison et al., 2000, p. 89). The CoI definition of social presence as “the ability
of participants in a community of inquiry to project themselves socially and emotionally, as
‘real’ people (i.e., their full personality)” (D. R. Garrison et al., 2000, p. 94) indicates that social
46
presence is only linked to a successful educational experience when it supports affective
outcomes as well as cognitive outcomes. These affective outcomes such as finding “the
interaction in the group enjoyable and personally fulfilling” (D. R. Garrison et al., 2000, p. 89)
are described as important for keeping students engaged and enrolled in the online course.
The final aspect of the CoI model, teaching presence, describes the responsibility of the
instructor to establish a learning environment that supports the development of both social and
cognitive presence. The role of the instructor as a facilitator of learning invokes the student-
centered emphasis of constructivism. The overlap of teaching presence and cognitive presence is
necessary in order to select content that will encourage students to follow the inquiry cycle, and
the overlap of teaching presence and social presence is necessary to set a course climate that will
foster the development of a learning community. According to the CoI model, deep learning
results from the overlap of all three types of presence. Additionally, each type of presence
manifests in specific activities or behaviors that serve as indicators of a student-centered
classroom. In this way, the CoI model provides information about the types of activities that are
hypothesized to improve learning when undertaken by the instructor and students. A visual
description of the CoI model can be seen in Figure 3 in which the three presence factors are
represented by overlapping circles and the indicators are represented by arrows pointing to each
presence factor. These indicators will be discussed in more detail later as they relate to the
development of the student survey instrument designed to measure the extent to which a learning
environment is aligned with the CoI model.
47
Figure 3. The Community of Inquiry model. Indicators of each type of presence are given in the arrows. Adapted from D. R. Garrison et al. (2000) and Swan (2003). As with POGIL in chemical education, the social component of the CoI model is used to
support inquiry-based learning activities in online courses. In the CoI model, the inquiry cycle
has four steps: a triggering event, exploration, integration, and resolution or application. The
steps in the inquiry cycle are joined by dashed arrows in Figure 3. While POGIL and CoI share
foundations in social constructivism and inquiry, an important distinction between the two is that
POGIL represents a specific pedagogy developed from educational theories while the CoI is a
model developed to explain data. Although the CoI model was developed using data from online
48
courses, it encourages the adoption of teaching practices aligned with a constructivist model of
learning and should be more broadly applicable to other disciplines and delivery methods
including chemistry courses delivered in a face-to-face environment.
Measuring a Constructivist Learning Environment and Student Outcomes
Defining Constructivism
In spite of Tobin’s (1999) declaration of “moving on” from constructivism over fifteen
years ago, constructivism has continued to find acceptance as a framework for chemical
education research and teaching. Ferguson (2007) identified multiple studies utilizing
constructivism as a framework to research teaching strategies and student conceptions of
chemical concepts. Articles published in the Journal of Chemical Education as recently as 2015
either reference writings on constructivism, or have constructivism as a keyword (DeFever,
Bruce, & Bhattacharyya, 2015; Flynn & Ogilvie, 2015; Gupta et al., 2015; Stoyanovich, Gandhi,
& Flynn, 2015; Talanquer, 2015). However, the level of understanding and application of
constructivism is inconsistent across these five articles.
Only two articles explicitly discuss social constructivism in teaching (Gupta et al., 2015)
and radical constructivism as a theoretical framework for research (DeFever et al., 2015).
Talanquer (2015) describes the need to consider existing student cognitive structures before
teaching new concepts that would cause those structures to change, aligning his view of
knowledge development with constructivist principles. Yet, Talanquer does not make the
connection between his beliefs and constructivism, even though he references a book chapter
titled Constructivism and Troublesome Knowledge. The other two articles with constructivism as
a keyword abstain from discussing constructivism in the article’s text (Flynn & Ogilvie, 2015;
49
Stoyanovich et al., 2015), though Stoyanovich et al. (2015) use Vygotsky’s ZPD to rationalize
the order in which to teach acid-base concepts, which provides a tangential link to social
constructivism. These recent articles highlight the inconsistent application or reporting of
constructivism in chemical education, even in situations where authors appear to hold views
consistent with constructivist principles.
One possible interpretation of these publications is that constructivism does not have a
common meaning for all chemical educators. In considering the role of constructivism in
educational practice, there is a need to define what is meant by “constructivism” in order to
provide a clear foundation for measurements of constructivist learning environments. In
subsequent discussions of constructivism as applied to pedagogy, constructivism will be used to
describe an approach to teaching in which teaching practices are aligned with a constructivist
model of learning. That is not to say that constructivist classrooms never utilize lectures,
memorization, or independent work. Instead, a constructivist learning environment will be
defined here as one in which teaching practices have been adopted that are more student-centered
and have shifted the role of the instructor from a lecturer to a facilitator for at least part of the
instructional time. This definition of constructivism is narrow in that it explicitly avoids much of
the philosophical debate surrounding radical interpretations of constructivism and instead
emphasizes commonalities among the various types of constructivism that can be directly
applied to educational practice.
With a clear definition of a constructivist learning environment established, it is necessary
to determine how to measure the degree to which a learning environment can be considered
constructivist. One way to determine the degree to which student-centered teaching practices are
50
being used is to employ observational protocols such as the Reformed Teaching Observation
Protocol (RTOP) or the Classroom Observation Protocol for Undergraduate STEM (COPUS;
Lund et al., 2015; Stains et al., 2015). However, these protocols are labor intensive and typically
the external observer only watches a few class sessions. Instead, the perceptions of the two
classroom stakeholders, the students and the instructor, may provide a richer description of the
learning environment since the stakeholders experience the learning environment over the whole
semester.
Development of the Community of Inquiry Student Survey Instrument
Student perceptions are critical to understanding the degree to which an instructor has
created a constructivist learning environment because students are the intended target of the
instructional techniques employed by the instructor. Additionally, students directly experience
the learning environment over an extended period of time and are therefore able to provide a
more complete description than an observer who may not attend all class sessions or fully engage
in activities and assignments. However, judging by the inconsistent way in which researchers
and instructors understand constructivism, it is highly unlikely that undergraduate students in
their first or second year would be able to speak directly about constructivism in their learning
environments. Therefore, it makes more sense to ask students about indicators of a constructivist
learning environment, not constructivism itself.
A survey was developed by Bangert (2008) in which students were asked to rate the
presence of indicators of an online constructivist learning environment. These indicators
included the development of a learning community fostering interaction and thoughtful
discussion among students and the level of comfort a student had interacting with other students
51
and the course instructor. Bangert’s (2008) survey was an initial step towards developing a
student survey to measure a constructivist learning environment using the Community of Inquiry
(CoI) framework. Bangert developed and validated his survey instrument using both exploratory
factory analysis (EFA) and confirmatory factor analysis (CFA).
Factor analysis techniques differ from other statistical techniques such as the t-test or
ANOVA. The goal of most factor analysis techniques is to identify underlying unmeasured
variables called latent variables or factors which may be responsible for observed patterns of
correlations in the data. This contrasts with t-tests and ANOVAs that look for differences
between group means. However, there are factor analysis techniques such as structured means
modeling that can be used to look for differences in group means on latent variables (Hancock,
1997). The goal of EFA is to identify underlying factors while the goal of CFA is to confirm the
presence of factors hypothesized by the researcher prior to the analysis. Using EFA (n=404),
Bangert was able to show that 23 of the original 26 items on his survey had mathematical
relationships to four latent factors related to student evaluations of online teaching effectiveness
including: (1) student-faculty interaction, (2) cooperation among students, (3) active learning,
and (4) time on task.
After EFA, Bangert (2008) used a new sample (n=403) for a CFA with the 23 items that
had previously been shown to be related to the four identified factors. The purpose of the CFA is
to provide a statistical description of how well the hypothesized model fits the provided data. A
variety of fit indices are used to evaluate data-model fit. The indices used by Bangert include the
root mean square error of approximation (RMSEA) and the comparative fit index (CFI), but the
model !" was not provided. Both the RMSEA and CFI values can range from 0 to 1. The
52
RMSEA is a parsimonious fit index that describes how well a model fits the data while taking
the simplicity of the model into account. Acceptable RMSEA values are below 0.06 (Hu &
Bentler, 1999) and indicate that the model explains a relatively large portion of the variance and
covariance among variables while also minimizing the number of relationships among variables.
The CFI is an incremental fit index that describes how much better the hypothesized model is
compared to a null model with no relationships among variables. Larger CFI values indicate that
the model explains a large amount of variance and covariance beyond what the null model
explains while smaller CFI values indicate that either the model is poor or that only weak
relationships are present in the data. Bangert’s CFA model had a RMSEA of 0.042 with the 90%
confidence interval (0.038 to 0.047) remaining below the cutoff of 0.06 and CFI equal to 0.99,
suggesting that the model with four latent factors was a good fit for the data.
Items similar to those used in Bangert’s (2008) study were utilized in developing a survey
specifically to measure the latent factors of cognitive presence, social presence, and teaching
presence in the CoI model (Arbaugh, 2008; Arbaugh et al., 2008). The items used in these
studies are provided in Appendix A. The indicators for each type of presence, previously seen in
Figure 3 (p. 47), were operationalized to create items designed to address the various aspects of
each of the three underlying presence factors. The four indicators for cognitive presence are
related to the use of the inquiry cycle in learning activities. The first stage of the inquiry cycle is
a triggering event. This was operationalized as items asking about the degree to which the
students experienced curiosity or interest related to course activities and problems. The second
stage, exploration, was measured by statements related to the students’ motivation to explore
questions posed in class by utilizing a variety of information sources. In the third stage of the
53
inquiry cycle, integration, students are asked about the degree to which they were able to
combine and reflect on the information obtained in the exploration stage. Items addressing the
final stage, resolution, described the ability of students to construct explanations and apply them
to solve problems in the course or in their daily lives.
The three indicators of social presence identified by the CoI model are the ability of the
students to express their emotions, engage in productive discourse with other students, and
collaborate. Productive discourse allows students to freely exchange ideas while collaboration
results in a cohesive group working together to solve a problem. The emotional aspect of social
presence is most prominent in the CoI survey items and is related to the affective domain in
which students feel they belong to the community of learners, are able to form impressions of
other students, and feel comfortable interacting and participating in discussions, including
disagreeing and acknowledging other viewpoints. Collaboration and discourse were
operationalized as items asking about the degree to which social interaction occurred during
communication and the development of a sense of collaboration through discussions. This
operationalization of social presence does not directly address the idea of group work or
collaborative assignments but rather treats social presence as a subtler sense of community
developed among students working towards a common goal.
Teaching presence items are most similar to those found on traditional student evaluation
forms. In the CoI model teaching presence has three indicators including instructional design,
facilitation, and feedback. Instructional design items cover traditional student expectations of an
instructor to communicate course topics, goals, due dates and instructions. Facilitation items are
more closely related to constructivism in asking about how well the instructor guided students
54
towards understanding, focused discussions, encouraged exploration, developed a sense of
community, fostered engagement, and kept students on task. Lastly, items related to instructor
feedback focused not on grades but rather on the ability of the instructor to help students identify
areas of strength and weakness and areas of agreement and disagreement related to course topics.
Early studies with the CoI survey instrument used principal component analysis (PCA) to
investigate the underlying structure of the instrument (Arbaugh, 2008; Arbaugh et al., 2008,
2010). In PCA, a large set of variables is reduced to a smaller set of new variables that explain as
much of the total variance in the original variables as possible. In PCA, these reduced variables
are more correctly called components, not factors, though the term factor is frequently used to
describe the results of PCA (Tabachnick & Fidell, 2007). PCA and factor analysis techniques can
produce mathematically similar results even though their corresponding algebraic operations on
the covariances of the set of variables are different (Velicer & Jackson, 1990). Conceptually, the
difference between PCA and factor analysis techniques such as CFA can be illustrated by
Figures 4 and 5, which show three measured variables, represented by boxes (V1, V2, & V3),
and their relationship to an underlying latent factor (or component), represented by an oval (F1).
Figure 4. A factor model with three measured variables and one latent factor.
Figure 5. A component model with three measured variables and one latent factor (component).
55
The primary difference between Figure 4 and Figure 5 is the directionality of the arrows, or
paths, between the measured variables and the factor or component; this directionally is the
critical conceptual difference between the two techniques. In the CFA model (Figure 4), the
arrows point from the factor to the variables because the factor is the underlying mechanism
thought to be causing the observed correlations, or covariance, among the measured variables. In
CFA the variables correlate, or covary, because they share variance in common with the factor.
In the PCA model (Figure 5), the arrows point from the variables to the factor (component)
because the variables are being combined in an optimal way to create the factor (component).
Another distinction between the two techniques is that in PCA the components are built to
explain all of the variance in the measured variables including variance due to measurement
error. In CFA the error is not included as part of the shared variance and is separated from the
measured variables and the factor, as seen in Figure 6. The benefit of CFA is that a factor is a
more pure representation of the underlying theoretical construct because it exists separately from
Figure 6. A factor model with three measured variables and one latent factor with the error terms shown for each measured variable.
the error. In PCA the component represents an optimally weighted combination of variables, not
any underlying theory, and is only as reliable as the variables that have been combined to form it.
56
The theoretical foundation of the CoI model suggests that factor analysis techniques are a
better match for instrument development than PCA since the three types of presence in the CoI
model are the underlying factors the survey items are attempting to measure. However, PCA can
be an appropriate technique in some situations because, unlike factor analysis, PCA allows for
the easy computation of summary or factor scores for each individual. These factor scores can be
calculated because the factor is a weighted composite of item scores. When PCA is used on the
CoI survey instrument, scores can be calculated for each survey respondent on cognitive, social,
and teaching presence. These scores can then be used in regression and group comparison
techniques as seen in the Arbaugh (2008) and Arbaugh et al. (2010) studies, respectively.
After the initial instrument development studies demonstrated the utility of the 34-item CoI
instrument in a variety of online course settings (Arbaugh, 2008; Arbaugh et al., 2008, 2010),
researchers began to look at relationships among the three CoI presence factors (cognitive,
social, and teaching), demographic variables such as gender, age, and academic level (Shea &
Bidjerano, 2009), and student outcome variables such as satisfaction and persistence (Joo et al.,
2011). Both studies (Joo et al., 2011; Shea & Bidjerano, 2009) used structural equation modeling
(SEM) to examine hypothesized causal relationships (paths) among the latent variables.
The structural equations in SEM describe the causal relationships among variables. Causal
relationships provide more specific information about the hypothesized influences of factors than
the correlations among factors seen in previous studies with the CoI survey instrument (Arbaugh
et al., 2008). Correlations between the three CoI presences were expected to exist due to their
overlapping nature in the CoI model, but the correlations could not provide information about
causal paths between factors. When using SEM, the relationships among measured variables and
57
factors must be specified a priori. Causal relationships should have a theoretical rationale for
either existing or being excluded from the model. As an example, the model typically proposed
for the 34-item CoI instrument and the three CoI presence factors is shown in Figure 7 (D. R.
Garrison et al., 2010; Shea & Bidjerano, 2009).
Figure 7. A model of hypothesized relationships among the 34 items on the CoI student survey and the three presence factors. Error terms have been omitted to minimize clutter in the model. Based on models in D. R. Garrison et al. (2010) and Shea & Bidjerano (2009).
This model can be described in terms of a measurement portion and a structural portion
(Mueller & Hancock, 2008). The measurement portion of the model shows the relationships
between the measured variables and the latent variables. In this case, the measurement portion of
the model shows how the 34 items on the CoI instrument are related to the three presence factors.
58
The structural portion of the model describes the causal relationships among the latent variables
of teaching presence, cognitive presence, and social presence. As seen in Figure 7, teaching
presence is hypothesized to have a causal influence on both social and cognitive presence and
social presence is hypothesized to have a causal influence on cognitive presence. Shea and
Bidjerano (2009) justify these relationships because the role of the instructor is to develop the
course environment which would cause teaching presence to influence both cognitive and social
presence. Additionally, Shea and Bidjerano believe that social presence acts as a mediator
between teaching presence and cognitive presence. The influence of social presence on cognitive
presence is also reinforced by the original description of cognitive presence as being supported
by social presence (D. R. Garrison et al., 2000). This hypothesized structure creates both a direct
effect of teaching presence on cognitive presence and an indirect effect of teaching presence on
cognitive presence through social presence.
In addition to finding evidence supporting causal relationships among the three CoI
presence factors, Shea and Bidjerano (2009) examined the causal influence of gender, age, and
academic level on student ratings of teaching presence. Their overall model contained the 34 CoI
survey items linked to the three presence factors, as shown in Figure 7, along with the three
additional measured variables of gender, age, and academic level. The fit statistics for the model
were acceptable, but not particularly good (!+,-1"3
" =11155.16; CFI = 0.95 and RMSEA = 0.08)
considering the customary threshold is at or above 0.95 for the CFI and at or below 0.06 for the
RMSEA. The large !" value is somewhat expected due to the large number of degrees of
freedom and the large sample size (n=2159). An absolute fit index called the standardized root
mean square residual (SRMR) was determined to be 0.05. This value is in line with
59
recommendations for SRMR to be at or below 0.08 (Hu & Bentler, 1999) and provides some
support for the authors’ claim of having acceptable data-model fit. Having acceptable data-model
fit allows for interpretation of the values determined for the paths between two variables. Only
the paths from gender to teaching presence and age to teaching presence were found to be
statistically significant at p < .05. Although statistically significant, the standardized paths, which
can be interpreted like regression coefficients, were very small for gender (0.04) and for age
(0.08). It is likely that these paths are statistically significant due to the large sample size utilized
in the research, but their practical significance is minimal within the overall CoI model. This
implies that gender and age have only a small influence on student perceptions of teaching
presence.
Also utilizing the CoI survey instrument, Joo et al. (2011) developed a model in which CoI
factors were hypothesized to have a causal bearing on student satisfaction and persistence in
courses at an online university in Korea (Shin, 2003). In this model, all three CoI presence
factors were hypothesized to have a direct influence on student satisfaction with the online
university as a whole. The structural portion of their model describing hypothesized relationships
among CoI presence factors and satisfaction is shown in Figure 8. A similar causal relationship
was hypothesized among teaching, social, and cognitive presence as in the Shea and Bidjerano
(2009) study. After initially testing their model with 709 students, Joo et al. (2011) found the
path between social presence and satisfaction to not be statistically significant. This
nonsignificant path is shown in Figure 9 with a lighter dashed arrow.
60
One reason the path from social presence to student satisfaction may have been found to be
nonsignificant is that the model may not have had enough power to detect a statistically
significant path. The power may have been low if the sample size was not appropriate for the
number of degrees of freedom in the model (Hancock, 2006). In SEM, degrees of freedom are
calculated by subtracting the number of parameters in the model from the number of unique
pieces of information provided by the measured variable variance/covariance matrix. The
degrees of freedom for this model (46) were relatively small because the researchers did not
allow individual CoI survey items to load on their respective factors. Instead, the 34 individual
CoI items and eight satisfaction items were reduced to two measured variables per factor through
item parceling, a technique which combines multiple measured variables into a composite by
taking either the average or sum.
Item parceling can be used to minimize the possibility of overweighting a latent factor that
has more paths to measured variables than other latent factors in the model (Little, Cunningham,
Shahar, & Widaman, 2002). Additionally, item parceling can reduce measurement error by
Figure 8. A model of hypothesized relationships among the three CoI presence factors and student satisfaction adapted from Joo et al. (2011).
Figure 9. A model of hypothesized relationships among the three CoI presence factors and student satisfaction indicating the nonsignificant path between social presence and satisfaction. Adapted from Joo et al. (2011).
61
reducing the number of variables in the analysis and increasing the likelihood that the
assumption of multivariate normality is met before running the statistical analysis. A drawback
of item parceling is that it decreases the degrees of freedom of the model by reducing the number
of unique pieces of information relative to the number of parameters. Reducing the number of
degrees of freedom can then reduce the power to detect a statistically significant path.
After finding the path between social presence and satisfaction to not be statistically
significant, the authors removed it from further analysis in order to improve their model fit.
However, no theoretical argument was presented to support the statistical argument for removal
of the path. Since each path in the model represents a belief structure based on a theoretical
understanding of the relationships among variables, the path should only have been removed for
a theoretical reason, not simply to improve model fit. Without a sound theoretical reason for
social presence not to influence student satisfaction, it is best to simply report the path as
nonsignificant and consider possible reasons for this outcome, such as having low power in the
analysis or the fact that the presence items were asked about a specific course and the satisfaction
items were related to satisfaction with the online university as a whole.
Through multiple studies, online education researchers have provided evidence for the
validity and reliability of the results obtained when using the CoI survey to measure student
perceptions of the three types of presence aligned with a constructivist learning environment
using diverse populations of online learners in a variety of disciplines (Arbaugh, 2008; Arbaugh
et al., 2008, 2010; Joo et al., 2011; Shea & Bidjerano, 2009). While the CoI survey instrument
has been used to measure three aspects of a constructivist learning environment for online
courses, there is little published research investigating the relationship among these three factors
62
and student outcomes beyond perceived learning or satisfaction (Arbaugh & Benbunan-Fich,
2007; Arbaugh, 2008; Joo et al., 2011). Currently, no online education research has investigated
relationships among the three presence factors thought to indicate a constructivist learning
environment and measured student outcomes of academic achievement as measured by grades or
standardized exam scores. This may be because comparing grades or locating appropriate
standardized exams is more difficult in online education research since the students surveyed are
typically from numerous disciplines and include both graduate and undergraduate students often
from different institutions (Arbaugh, 2008; Arbaugh et al., 2008, 2010). Additionally, the online
education literature is typically interested in student satisfaction with the online delivery method,
not student satisfaction with the learning environment or educational outcomes (Arbaugh &
Benbunan-Fich, 2007; Arbaugh, 2000, 2008; Joo et al., 2011).
Measuring Student Outcomes in Constructivist Learning Environments
In contrast, chemical education research has investigated student outcomes of both
academic achievement and satisfaction after implementation of specific teaching practices
aligned with social constructivism, such as POGIL and PLTL, or implementation of more
general teaching practices aligned with constructivism (Conway, 2014; Gosser et al., 2010;
Gupta et al., 2015; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder &
Hunnicutt, 2008; Tien et al., 2002). Utilizing academic achievement in chemistry and satisfaction
as dependent variables indicates an underlying assumption that aligning teaching practices with
constructivism should improve student learning and thus result in students who perform better
academically and are more satisfied with their learning experience. These studies have been
narrowly focused on introductory chemistry courses in general or organic chemistry, which
63
creates smaller differences between courses and makes comparisons across courses more
meaningful than comparing across multiple disciplines and academic levels.
While student learning is difficult to measure directly, exam grades, course grades, and
standardized American Chemical Society (ACS) exam scores are frequently employed in the
chemical education literature as ways to measure student learning (Conway, 2014; Gosser et al.,
2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder & Hunnicutt, 2008;
Tien et al., 2002). The beneficial effect of POGIL and POGIL-style instruction on both final
exam and final course grades has been demonstrated in a one-semester organic and biochemistry
course for pre-health professionals and in large enrollment general and organic chemistry courses
(Conway, 2014; Ruder & Hunnicutt, 2008). Similar results have been reported for
implementations of PLTL in general and organic chemistry courses from a variety of institutions
including community colleges and research universities (Gosser et al., 2010; Lewis & Lewis,
2005; Mitchell et al., 2012). However, instructors rarely discuss the psychometric properties of
their final exams or final course grades, so without more detailed information about these
achievement measures, the conclusions drawn from these studies must be interpreted cautiously.
Standardized ACS exams represent a more psychometrically sound tool available to
chemical educators for measuring student achievement. These exams have been in use since
1934 and are continuously revised and updated to reflect changes in teaching practices and
curricula (Brandriet, Reed, & Holme, 2015). The ACS exams are developed by national
committees independent of a specific classroom instructor, the validity and reliability of the
scores are evaluated, and the items on the version of the exam currently in use are not made
publicly available. Therefore, scores on these exams can be considered an acceptable measure of
64
students’ content knowledge (Lewis, 2014). In studies where ACS exams were used as final
exams, students experiencing PLTL showed similar or improved ACS exam scores relative to
students experiencing traditional lecture-based instruction (Lewis & Lewis, 2005; Mitchell et al.,
2012). These results indicate that PLTL instruction holds students to the same standard of
content knowledge as traditional instruction.
The benefits of implementing PLTL instruction were supported by taking initial
differences between students into account by controlling for SAT scores in the statistical analysis
(Lewis & Lewis, 2005; Mitchell et al., 2012; Tien et al., 2002). Considering SAT scores is
necessary to show the effectiveness of teaching practices across different groups of students,
since a demonstrated relationship exists between SAT math scores and grades in introductory
college science courses (H. E. Spencer, 1996; Tai et al., 2006). Since not all students take the
SAT in preparation for college, Nordstrom (1990) and Xu & Lewis (2011) looked at the
correlation between SAT math and ACT math scores for freshmen chemistry students and found
it to be approximately 0.70. This indicates that the ACT and SAT math scores are measuring
similar abilities in students. This is further supported by the similar standardized regression
coefficients of around 0.40 obtained by Nordstrom (1990) when predicting chemistry course
performance and by Lewis & Lewis (2005) and Xu & Lewis (2011) when predicting ACS exam
scores from the SAT and ACT math ability scores along with other variables.
Even implementing more general constructivist teaching practices, not specifically POGIL
and PLTL, has been shown to improve student academic achievement in chemistry. Hall et al.
(2014) describe a supplemental discussion-type section that has “roots in social constructivism
and borrows elements from a number of learner-centered pedagogies” (p. 37). This program
65
recruited students with lower SAT scores who, after enrollment in the program, earned exam
scores in both their general and organic chemistry courses that were not statistically different
from peers entering with higher SAT scores. The adoption of guided inquiry techniques and
collaborative group work was also found to increase critical thinking in a first semester general
chemistry laboratory course (Gupta et al., 2015). Here, social constructivism is invoked to
explain how “critical thinking develops in students through interactions with the teacher and
among students” (Gupta et al., 2015, p. 37). These two studies provide support for the idea that
the adoption of more general teaching practices aligned with constructivism, not just the
implementation of a specific constructivist pedagogy such as POGIL or PLTL, may improve
student academic achievement in introductory undergraduate chemistry courses.
In addition to improvements in academic achievement in chemistry, student satisfaction
with the constructivist learning environment has also been reported in the chemical education
literature (Conway, 2014; Hall et al., 2014; Ruder & Hunnicutt, 2008; Tien et al., 2002). In
contrast to the measurement of satisfaction with the online delivery aspect of the course seen in
the online education literature (Arbaugh & Benbunan-Fich, 2007; Arbaugh, 2000, 2008; Joo et
al., 2011), the chemical education literature typically reports student satisfaction and attitudes
towards a specific course learning environment. Students are generally positive about the
teaching practices employed in the learning environment, even though they often note an
increase in the amount of work done in class. The satisfaction instrument used by Hall et al.
(2014) highlighted the importance of student-student interactions by asking students to rate how
supportive they found their study group and how comfortable they felt contributing to
conversations about course materials. Students in the Tien et al. (2002) study reported that
66
working with peers helped their learning even more than the course lectures. Conway (2014)
reported students enjoyed working in groups and felt that they learned the material better than in
more traditionally taught courses. Similarly, Ruder & Hunnicutt (2008) reported students had
positive attitudes towards group work and a belief that they were learning from other students.
These studies reflect the affective outcomes of social presence as described by Garrison et al.
(2000).
Though satisfaction and attitude are frequently used interchangeably, Gardner (1975)
distinguishes satisfaction as one component of an attitude towards something. Similarly, Xu &
Lewis (2011) identify an emotional satisfaction component of student attitudes towards
chemistry. Xu & Lewis (2011) revised the Attitude towards the Subject of Chemistry Inventory
(ASCI) into a shorter version with the original semantic differential scale. In this semantic
differential scale, students indicate their position on a scale between two opposite words, such as
“satisfying” and “frustrating” instead of the more traditional response scale in which students
indicate their degree of agreement or disagreement with a particular statement. Though the
courses in which the students in the Xu & Lewis study were enrolled were not specifically
described as constructivist, a correlation of 0.35 was demonstrated between emotional
satisfaction and ACS exam scores indicating that a relationship exists between student
satisfaction and academic outcomes. However, a larger correlations (0.45 and 0.46) existed
between ACS exam scores and math ability scores as measured by SAT math and ACT math
scores, respectively (Xu & Lewis, 2011). This result confirmed the relationship between math
ability and academic achievement in introductory chemistry courses.
67
Modeling the Influence of a Constructivist Learning Environment on Student Outcomes
The previously discussed studies in online and chemical education can be synthesized into
a single model showing the influence of a constructivist learning environment on student
outcomes of academic achievement in chemistry and satisfaction. In this model, the
constructivist learning environment is measured by the three CoI factors of cognitive presence,
social presence, and teaching presence. Academic achievement in chemistry is measured by the
outcomes typically used in chemical education research such as ACS exam scores and final
course grades. Student satisfaction is measured using a survey instrument as is typical in both
online and chemical education research.
The model in Figure 10 provides a diagrammatic representation of the hypothesized
structural relationships among these latent and measured variables. The latent variables are
shown as ovals and represent variables that are not measured directly, but will be identified by
analysis of student responses to the CoI survey instrument and student satisfaction survey items.
The measured variables of math ability and academic achievement in chemistry are shown as
rectangles and are determined based on student scores. The portion of the model showing the
individual CoI and satisfaction survey items has been omitted for clarity.
Based on prior research in online education, the same relationship among cognitive
presence, social presence, and teaching presence is expected as seen previously in Figure 8 (p.
60). Teaching presence is hypothesized to directly influence both cognitive and social presence
while also indirectly influencing cognitive presence through social presence. The multiple
influences of teaching presence are due to the role of the instructor in both selecting course
68
Figure 10. The hypothesized structural model of relationships among the three CoI presence factors, math ability scores, and student outcomes. content and setting the tone of interactions between the instructor and students (D. R. Garrison et
al., 2010; Shea & Bidjerano, 2009; Swan, 2003). Building on existing research in online
education, the proposed model in Figure 10 adds causal relationships among the CoI presence
factors and student outcomes of academic achievement and satisfaction.
From the chemical education literature, the cognitive presence and teaching presence
aspects of a constructivist learning environment are expected to directly influence academic
achievement in chemistry as measured by ACS exam scores and final course grades (Conway,
2014; Gosser et al., 2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder &
Hunnicutt, 2008; Tien et al., 2002). The direct influence of teaching presence on ACS exam
scores and final course grades is hypothesized because “the instructor serves as an expert who
69
plans instruction to stimulate students’ interest, motivates their participation in the learning
process, and facilitates their learning” (Swan, 2003, p. 8). Though no studies with the CoI
instrument have specifically examined the influence of teaching presence on student academic
outcomes, the role of the instructor as expert and facilitator of learning suggests a causal
relationship between teaching presence and academic achievement in chemistry. Additionally, in
the CoI model selecting appropriate content is at the overlap of teaching and cognitive presence.
The selection of content and learning activities that facilitate student knowledge construction is
expected to influence the demonstration of that knowledge construction through ACS exam
scores and final course grades. For this reason, cognitive presence is also hypothesized to have a
causal influence on academic achievement in chemistry.
In addition to the influence of teaching and cognitive presence, math ability is expected to
have a direct influence on academic achievement in chemistry (Lewis & Lewis, 2005; Mitchell
et al., 2012; Nordstrom, 1990; Tien et al., 2002; Xu & Lewis, 2011). A path is also included to
account for the influence of ACS exam scores on final course grades. The paths between social
presence and ACS exam scores and final course grades are omitted because the influence of
social presence on these two outcomes is hypothesized to be indirect. This indirect influence is
hypothesized because of the small correlation seen between social presence and perceived
learning in earlier studies (0.19; Arbaugh, 2008) and the assumption that social presence only
affects academic outcomes when cognitive presence provides an academic context for the social
interactions among students. Finally, from both the online and chemical education literature,
cognitive presence, social presence, and teaching presence are all expected to directly influence
satisfaction (Conway, 2014; Hall et al., 2014; Joo et al., 2011; Ruder & Hunnicutt, 2008). A two-
70
headed arrow is included to account for a relationship between final course grades and student
satisfaction beyond the relationships present in the hypothesized model, but no causal direction is
proposed for the relationship. The lack of causality reflects the ongoing debate as to whether
students who are more satisfied with a course perform better academically or if performing better
academically causes students to be more satisfied with a course (Greenwald & Gillmore, 1997;
Howard & Maxwell, 1982).
The model proposed in Figure 10 represents only one possible relationship among this set
of variables. Even if this model is shown to have good fit with collected data, it does not
necessarily represent the only viable model. Early research using the CoI has shown that
teaching presence may not be a single factor, but may be best modeled as two factors (Arbaugh,
2007; Shea, Sau Li, & Pickett, 2006). In this alternate model, teaching presence is instead
conceptualized as a pre-course instructor activity factor and an in-course instructor activity
factor. Pre-course instructor activities are typically done outside of the class period and include
the design and organization of the course. Items 1–4 on the CoI instrument address the results of
pre-course activities such as communicating course goals, topic, due dates, and instructions. In-
course instructor activities are typically done during the class period and include facilitation of
student learning and direct instruction (Arbaugh et al., 2008). Items 5-13 on the CoI instrument
address the in-course instructor activity factor. A visualization of the difference between
teaching presence as a single factor and a two correlated factors can be seen in Figures 11 and
12. During the data analysis process, these two competing models can be tested and statistically
compared to see which provides a better fit for the data.
71
The primary model presented in Figure 10 relies on measurements of student perception
and student performance to model relationships among constructivist learning environment
factors and outcomes of student satisfaction and academic achievement in chemistry. This model
represents the integration of recent research in both chemical education and online education
through their shared use of constructivism as a foundation for the development of teaching
practices aligned with how students learn. Structural equation modeling (SEM) was chosen as
the primary statistical data analysis methodology because of the complex causal relationships
investigated in which the three CoI presence factors were hypothesized to influence each other as
well as multiple student outcomes (Bauer, 2003; Huta, 2014; Kline, 2011). While SEM is a
Figure 11. Teaching presence as a single factor with 13 indicator variables.
Figure 12. Two correlated factors taking the place of a single teaching presence factor.
72
sound statistical technique for analyzing relationships among measured and latent variables,
student measurements should not provide the only source of information about the degree to
which a learning environment incorporates constructivist principles. Information about the
learning environment provided by the course instructor can be used to support or refute the
picture of the learning environment portrayed by student responses to the CoI survey instrument.
Measuring Instructor Approaches to Teaching Though student perceptions of the learning environment can provide valuable information,
it is necessary to remember that “undergrads may not be sophisticated enough to distinguish
between facilitation and direct instruction” (D. R. Garrison & Arbaugh, 2007, p. 165), which is
an important distinction between constructivist and objectivist learning environments. For this
reason, the most complete description possible of a learning environment necessitates the
combination of student perceptions with information obtained from the course instructor.
However, given the various interpretations and implementations of constructivism previously
discussed in recent publications in the Journal of Chemical Education, it may not be possible to
directly ask instructors about constructivism and obtain information aligned with the more
general definition of constructivism proposed for this research. Additionally, it may be possible
that some chemical educators who hold constructivist beliefs about learning or teach in ways that
are aligned with constructivism may be unaware that they are in agreement with constructivist
principles. Other chemical educators may be aware that constructivism is currently popular, but
they may not understand the specific principles associated with constructivism. Therefore, it may
be more effective to ask instructors to describe their approach to teaching and then have the
researcher interpret the instructor responses through a constructivist lens.
73
Previous studies in online and science education have utilized interviews with instructors in
order to determine their approach to teaching (Arbaugh & Benbunan-Fich, 2007; Prosser et al.,
1994; Trigwell et al., 1994). As a portion of their research on the importance of interactions in an
online environment Arbaugh and Benbunan-Fich (2007) used semi-structured interviews to ask
MBA course instructors “whether their courses were based primarily in fact/concept
dissemination via online lectures, or based on knowledge construction by students” (p. 857).
Instructors were also asked about the relative amount of individual and group work in their
courses. Instructors’ responses were compared with information in the course syllabi or course
websites in order to support the researchers’ classification of the course as objectivist or
constructivist and group-centered or individual-centered. All instructor reports in the Arbaugh
and Benbunan-Fich (2007) study were found to be consistent with information provided in the
course syllabi or websites.
Interviews were also used to inform phenomenographic qualitative research investigating
ideas held about teaching and learning by 24 instructors of first-year undergraduate chemistry
and physics courses in order to see how these ideas influenced the approaches to teaching
adopted by the instructors (Prosser et al., 1994; Trigwell et al., 1994). The categories of
approaches to teaching developed in these studies were based on the interviews and were not
hypothesized in advance of collecting data as in the Arbaugh and Benbunan-Fich (2007)
research. The five categories that emerged from an analysis of the interviews were (1) a teacher-
centered information transmission approach, (2) a teacher-centered approach emphasizing
student acquisition of concepts, (3) an approach utilizing interactions between the teacher and
student to facilitate student acquisition of concepts, (4) a student-centered approach emphasizing
74
student development of concepts, and (5) a student-centered approach emphasizing students’
conceptual change. These five categories can be conceptualized as a continuum ranging from a
more objectivist approach focusing on the teacher acting as an external source of information to
a more constructivist approach focusing on changing the knowledge structures of individual
students.
In later research, Trigwell & Prosser (1996) used the interview transcripts to develop an
inventory that could be administered to instructors in order to determine their approach to
teaching in a particular context. Trigwell & Prosser clearly state that the inventory is context
dependent and should not be used to classify an instructor but rather to classify an instructor’s
approach to teaching for a particular course. Even within the same subject, instructors often
adopt different approaches to teaching for different classes depending on the situation. As an
example, the same instructor may be more likely to utilize a teacher-centered information
transmission approach with a large-enrollment introductory level course but utilize a student-
centered approach emphasizing conceptual change when working with upper level undergraduate
or graduate students on independent research projects which by their nature have smaller
enrollments.
From the interview responses, Trigwell & Prosser (2004) selected statements that served as
indicators of various teaching approaches. A principle components analysis (PCA) was
performed on a 39-item version of the inventory with responses from a total of 58 university
chemistry and physics instructors, 11 of whom had participated in the original interviews. As a
result of this analysis, the student-teacher interaction subscale was removed due to a high degree
of overlap with the student-centered subscale. Eventually, the inventory was shortened and a
75
confirmatory factor analysis (CFA) was undertaken with over 650 participants from universities
in 15 countries in disciplines representative of the variety typically taught at the university level.
Most recently, a CFA was undertaken for the 16-item version of the Approaches to
Teaching Inventory (ATI) with responses from over 1000 instructors at universities in the UK,
US, Scandinavia, and Hong Kong in a range of disciplines (Prosser & Trigwell, 2006). However,
this sample was heavily weighted towards engineering instructors from Sweden who comprised
over half the sample. Multiple CFA models were tested and the two models with the best fit were
found to be a four factor model with four covariance terms added between individual item errors
(CFI = 0.934, RMSEA = 0.041, and SRMR = 0.043) and a two factor model with three
covariance terms added between individual item errors and one item loading on both factors (CFI
= 0.931, RMSEA = 0.040, and SRMR = 0.043). No model !" values were provided. Though the
fit statistics for the two models are similar, Prosser & Trigwell state a preference for the two-
factor model due to the high degree of correlation between two of the subscales in the four-factor
model (over 0.90). In the two-factor model, one factor can be described as the information
transmission teacher-focused approach (ITTF) while the other represents a conceptual change
student-focused approach (CCSF).
It is not entirely clear from the Prosser & Trigwell (2006) article whether or not the error
covariance terms were hypothesized prior to beginning the analysis or whether they resulted
from modification indices provided by the software used in the analysis. Given the relatively
poor fit of the two factor model with no error covariances (CFI = 0.865, RMSEA = 0.055, and
SRMR = 0.056), it seems likely that the error covariances and additional item loading were
added based on modification indices provided by the software to show opportunities to improve
76
model fit. While the RMSEA and SRMR values are within an acceptable fit range, the low CFI
value indicates a possibility that even though the model did an acceptable job explaining
relationships among the data, the relationships were relatively weak to begin with. Though
nothing prohibits a researcher from modifying models after seeing the results, modifications
should be supported by a theoretical reason in addition to a mathematical reason (Mueller &
Hancock, 2008).
Prosser & Trigwell (2006) explain that the items linked by the error covariance terms were
somewhat redundant and therefore their linking is supported by their similar item wordings. The
item that was allowed to load both the ITTF and CCSF factors was related to the use of
textbooks to provide information. The rationale for this item being related to both approaches is
due to the inclusion of instructors from multiple disciplines in the research sample and the
differing ways in which instructors in the humanities and sciences regard the use of textbooks.
However, no evidence was provided for this in terms of conducting a separate analysis with
samples separated by discipline. Further work with the ATI removed some problematic items
and introduced new items, resulting in a 22-item inventory that had an improved CFI value
without the use of error covariance terms (CFI = 0.95; RMSEA = 0.06 CI90=[0.057, 0.072];
SRMR = 0.08). This model is based on data from only 318 instructors in a range of disciplines at
Australian and UK institutions (Trigwell et al., 2005).
Ultimately, the development of the ATI from qualitative interviews and the quantitative
results from these three studies (Prosser & Trigwell, 2006; Trigwell et al., 2005; Trigwell &
Prosser, 2004) indicate that the ATI is an acceptable instrument for identifying two distinct
instructor approaches to teaching in specific contexts. Cronbach’s alpha values for each
77
approach’s subscale range from 0.66 to 0.86 (Prosser & Trigwell, 2006; Trigwell et al., 2005;
Trigwell & Prosser, 2004) indicating good internal consistency of the subscales based on the
generally accepted, but somewhat arbitrary cutoff of 0.70 (Arjoon, Xu, & Lewis, 2013). The
results of the two factor CFA consistently demonstrate a negative correlation (–0.26; Prosser &
Trigwell, 2006; –0.35 Trigwell et al., 2005) between the factors representing the two approaches.
The negative values support the idea that these factors describe distinct instructor approaches to
teaching in a specific context. The ATI has also been used in chemical education research to
examine how the teaching approaches of new university chemistry professors change after
attending a short teaching workshop emphasizing the use student-centered teaching approaches
(Stains et al., 2015). The instructors who attended this workshop had statistically significantly
higher CCSF scores one week after the workshop compared to their CCSF scores before the
workshop, suggesting that the CCSF scale measures approaches to teaching aligned with
constructivism. The instructors also had significantly lower ITTF scores than a control group
who did not attend the workshop.
Combining information from the perspectives of two classroom stakeholders, the instructor
and students, provides a more complete picture of the learning environment created by the
instructor and experienced by the students. Measuring the learning environment from both
perspectives also provides information regarding how well the ATI and CoI measure the
presence of indicators of a constructivist learning environment. For example, if the instructor’s
approach to teaching in a particular course is closer to the conceptual change student-centered
approach, it would be expected that the student CoI responses would show a high degree of
teaching and cognitive presence. Additionally, if the instructor indicates on the ATI that he or
78
she sets aside class time for students to discuss course topics amongst themselves, this is likely to
be reflected in students’ agreement with CoI items related to course discussions.
Comparing responses to the ATI and CoI determines whether or not the CoI instrument,
when modified for face-to-face introductory undergraduate chemistry courses, is an acceptable
instrument for measuring indicators of a constructivist learning environment from the students’
perspective by providing evidence for the validity of the item responses. Additionally, the
psychometric properties of the modified CoI survey are examined quantitatively to provide
additional evidence for the validity and reliability of the scores. If the modified CoI survey
proves to be an acceptable instrument for this population of students, then combining CoI
responses with measurements of student outcomes will provide the data necessary to test the
model proposed in Figure 10 (p. 68). Testing the fit of this model allows for an examination of
how a constructivist learning environment, as perceived by students, affects student satisfaction
and academic achievement in chemistry. These research goals are summarized by the following
research questions.
Research Questions
1. Are self-reported instructor approaches to teaching consistent with student
perceptions of the learning environment?
2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for
measuring student perceptions of the indicators of a constructivist learning
environment in a face-to-face introductory undergraduate chemistry course?
3. To what degree does a constructivist learning environment, as measured by
student CoI survey responses, affect outcomes of student satisfaction and
79
academic achievement in chemistry, as measured by ACS exam scores and final
course grades when the effect of math ability on academic achievement is
considered
80
Chapter 3 This research relies on both student and instructor survey instruments to measure the
extent to which a constructivist learning environment exists in a given face-to-face introductory
undergraduate chemistry course. These instruments required slight modifications to their
wording to better align them with the intended research situation and population. Though the
literature contains documentation of prior use of the CoI and the ATI, modifications to the
wording of items necessitated a small study to pilot the reworded instruments prior to their use in
the main research project.
In addition to this preparation for the main research study, two separate power analyses
were conducted to determine the sample size necessary to test overall data-model fit and to test
the specific model parameters of interest for this research. The methodology for this research
primarily focuses on obtaining quantitative data in the form of survey responses and student
achievement data. A small qualitative strand is embedded in the collection of instructor data. For
this reason, the data analysis procedures utilized to answer the research questions are
predominately related to SEM analysis.
Modifications to ATI and CoI Wording
Though both the ATI and the CoI have been previously used in published research and
have information available on their development and use with various populations of instructors
and students (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al., 2010; Joo et al.,
2011; Prosser & Trigwell, 2006; Shea & Bidjerano, 2009; Trigwell et al., 2005; Trigwell &
Prosser, 2004), evidence for the validity of the instrument scores must be demonstrated in each
use (American Educational Research Association, American Psychological Association, &
81
National Council on Measurement in Education, 2014). For this particular study, the researcher
initially made small modifications to the wording of specific items on both the ATI and CoI in
order to better align the items with best practices in survey instrument design (Krosnick &
Presser, 2010) and ensure the wording was appropriate for the research population of instructors
and students enrolled in face-to-face introductory chemistry courses. These wording changes in
combination with the use of the instruments with this specific population necessitated an initial
pilot study of the survey instruments to check if the items were being interpreted as intended.
The pilot study procedures and results are described in greater detail after a discussion of the
modifications to particular survey items.
One of the authors of the ATI, K. Trigwell (personal communication, August 13, 2015),
was contacted for permission to use the ATI. The researcher was provided the revised 22-item
ATI-R (Trigwell et al., 2005) along with scoring directions. Initial changes were made to the
original European/Australian wording of the ATI-R by the researcher to more closely align the
language with US usage. As an example, item 11 on the original ATI-R is “In this subject, I
provide the students with the information they will need to pass the formal assessments.” Here,
the term “subject” is used in a way that would be more similar to the US usage of “course.”
Therefore, this item was initially revised to “In this course, I provide the students the information
they will need to pass the formal assessments.” This revision process occurred for all 22 items on
the ATI-R.
In this initial stage of revisions, the items were kept as similar as possible to their original
wording by only changing the word “subject” to “course” where appropriate. The response scale
was not changed, but the layout of the instrument was modified to label each of the five scale
82
points above its corresponding number. This modification was done to minimize the cognitive
effort necessary for respondents to select the most appropriate scale point by removing the need
for respondents to remember the labels or refer back to descriptions given in the instrument
directions (Krosnick & Presser, 2010). The revised instrument, response scale, and directions
used in the instrument pilot study with chemistry course instructors can be found in Appendix B.
A similar process was undertaken to modify the CoI instrument. The CoI items have been
published numerous times with only slight variations across research groups (Arbaugh et al.,
2010; Arbaugh, 2008; D. R. Garrison et al., 2010; Shea & Bidjerano, 2009). The version of the
CoI used as a starting point for the current research was published by Arbaugh, Cleveland-Innes,
and Diaz (2008). The CoI was developed by Canadian researchers and did not have the same
wording issues due to language differences as the ATI. However, since the CoI was originally
designed for students in online courses four items were reworded to reflect the intended face-to-
face population of the current research. These four original items and the modified version can
be seen in Table 1.
Table 1 Original and Revised CoI Items
Original CoI items (Appendix A)
Revised CoI items used in pilot study (Appendix C)
16. Online or web-based communication is an excellent medium for social interaction
Q16. Face-to-face communication is an excellent medium for social interaction
17. I felt comfortable conversing through the online medium
Q17. I felt comfortable conversing face-to-face in class
22. Online discussions help me to develop a sense of collaboration
Q22. In-class discussions helped me to develop a sense of collaboration
28. Online discussions were valuable in helping me appreciate different perspectives
Q29. In-class discussions were valuable in helping me appreciate different perspectives
83
In addition to modifications of the four items in Table 1 to remove references to an online
course environment, two other small wording changes were made to the instrument before it was
piloted with students who had completed face-to-face chemistry courses. These changes were
made so that the instrument reflected best practices in survey design (Krosnick & Presser, 2010).
Twelve of the thirteen items designed to measure teaching presence all started with the same
question stem, “The instructor…” but one item was originally worded “Instructor actions
reinforced the development of a sense of community among course participants.” The item was
changed to “The instructor reinforced the development of a sense of community among course
participants”. This change was made to maintain the consistency of the stem across all items
designed to measure teaching presence.
The second change was made to an item originally worded as “Reflection on course
content and discussions helped me understand fundamental concepts in this class.” As written,
this item is considered double-barreled because it asks two questions simultaneously (Krosnick
& Presser, 2010). This type of item poses a problem for respondents because if the respondent
agrees with only one part of the question but not the other it is difficult for the respondent to
choose a response that accurately reflects his or her opinion. For example, if reflections on
course content helped the student understand concepts but the student never reflected on course
discussions, the student may not know whether to select agree or disagree. Therefore, this item
was split into two separate items, one addressing reflections on course content and the other
addressing reflections on discussions.
The CoI instrument is most typically given using a five-point scale ranging from strongly
agree to strongly disagree (Arbaugh et al., 2008; D. R. Garrison et al., 2010; Shea & Bidjerano,
84
2009). The five-point scale has been found to be an acceptable number of scale points in both the
survey methodology literature and the structural equation modeling literature (Finney &
DiStefano, 2013; Krosnick & Presser, 2010). In structural equation modeling research,
approximately normal ordinal data with at least five categories can be treated as continuous data
without distorting the model fit indices (Finney & DiStefano, 2013). From a survey methodology
perspective, a five-point scale has performed better than shorter scales and similarly to seven-
point scales. Methodological studies have shown scale performance to worsen when the
instrument contains more than seven points (Krosnick & Presser, 2010). When given an
unlabeled scale, some research has shown that respondents naturally divide it into five scale
points (Krosnick & Presser, 2010).
Another benefit to the five-point scale is that it is easy to provide distinctive scale labels
for each point in a five-point scale. For the pilot study, these labels were provided as strongly
agree, agree, unsure, disagree, and strongly disagree. Survey research has demonstrated that
labeling all scale points improves reliability of the responses and respondent satisfaction with the
survey instrument (Krosnick & Presser, 2010). In addition, an option was added with a numerical
value of zero and the label “Not Applicable” for situations in which an item did not apply to a
specific student or course. It was anticipated that this situation might occur for courses that did
not use formal, structured in-class discussions since many CoI items reference course
discussions.
In addition to small differences in wording among the versions of the CoI found in the
literature, the items are sometimes grouped by type of presence (Arbaugh et al., 2008, 2010; D.
R. Garrison, Cleveland-Innes, & Fung, 2004; Shea & Bidjerano, 2009), sometimes randomized
85
(D. R. Garrison et al., 2010), and sometimes no information is provided on the order of item
presentation (Joo et al., 2011). The recommendations of Krosnick & Presser (2010) were
followed in grouping the items by type of presence, but these groupings were not explicitly
labeled to avoid biasing student responses to the items. With these groupings, the version of the
CoI instrument used in the pilot study with students had items 1-13 addressing teaching
presence, items 14-22 addressing social presence and items 23-36 addressing cognitive presence,
excluding item 28. Item 28 was designed as a check to catch respondents who may not be
reading carefully due to disinterest or fatigue by simply asking the respondent to select the
option corresponding to “Disagree”. Including this item allows for the exclusion of a set of
responses where “Disagree” was not selected since it is likely that the other options selected by
the respondent do not accurately reflect a thoughtful evaluation of the learning environment.
In addition to the CoI items, students in the pilot study were also provided with five
traditional satisfaction items on a five-point scale from Bollinger and Wasilik (2012) and four
satisfaction items on a five-point semantic differential scale from Xu & Lewis (2011). Prior
research with the traditional satisfaction items had used a five-point scale (Bolliger & Wasilik,
2012) but the semantic differential items were changed from the original seven-point scale used
by Xu & Lewis (2011) to a five-point scale in order to maintain consistency in response scale
length across all items on the student survey instrument used in the pilot study. The complete
instrument used for the pilot study with students is found in Appendix C.
86
Instructor and Student Survey Pilot Studies
Recruitment of Participants for Pilot Studies
The pilot study methodology was similar for both the ATI and the combined
CoI/satisfaction items on the student survey. The sample for the ATI pilot study was drawn from
a population of instructors of the classroom portion, not laboratory sections, of 100-level face-to-
face chemistry courses and the sample for the CoI/satisfaction item pilot study was drawn from a
population of students who had completed 100-level face-to-face chemistry courses. Both the
instructor and student samples were recruited from the population at a midsize private research
university (Indiana University Center for Postsecondary Research, n.d.). After obtaining
permission for the pilot study from the institutional review board (IRB) at the university,
recruitment emails were sent to both instructors and students informing them of the purpose of
the study and soliciting their participation.
Only instructors currently listed on the university chemistry department’s website who
had taught a 100-level chemistry course in the previous two years received the instructor
recruitment email. The student recruitment email was sent to all student members of the
Chemistry Club at the university. Chemistry Club members were selected as the target
population since these students were likely to have completed 100-level chemistry courses at the
university and their email addresses were available on the Chemistry Club website. Two
recruitment emails were sent to both instructors and students; the second email was sent
approximately a week after the first. Five instructors and five students participated in the pilot
study. All ten participants (instructor and student) gave permission to have their responses audio
recorded during the pilot study.
87
Instructor Survey Pilot Study Methodology
In the pilot study for the ATI, instructors were first asked a series of open-ended
interview questions related to the length of their teaching career, their general approach to
teaching chemistry, any changes they have made to their teaching approach over time, their
specific approach to teaching 100-level courses, and their approach to teaching in the 100-level
course they have most recently taught. The syllabus for the most recently taught 100-level course
was either provided by the instructor or brought to the interview by the researcher in order to aid
the instructor in recalling specific course details and to provide an artifact for the researcher to
use for comparison with the instructor’s responses. The answers to the open-ended interview
questions about the instructor’s approach to teaching were used to provide information about the
background of each instructor and to inform interpretation of the instructor’s responses to the
ATI.
The instrument testing phase of the pilot study was conducted after the instructors
responded to the open-ended interview questions. During this phase, instructors were provided
with the 22-item ATI instrument revised by the researcher as discussed earlier in this chapter
(see Appendix B). The instrument testing protocol was a think-aloud interview in which the
instructor read each item aloud and verbalized his or her rationale for selecting a particular
response (Krosnick & Presser, 2010). If the instructor indicated confusion or uncertainty in
selecting a response, additional follow-up questions were asked by the researcher to probe
possible reasons for this difficulty, including unclear wording of the item or the item not being
applicable in a particular classroom environment. Following completion of the survey
instrument, the instructor was asked to provide any general or overall feedback on the survey
88
instrument and asked if the survey instrument satisfactorily captured his or her approach to
teaching for the particular 100-level course that was the focus of the ATI responses.
Student Survey Pilot Study Methodology
The pilot study for the CoI and satisfaction items followed a similar format to the second
phase of the instructor pilot study. Prior to meeting with each student participant, the student was
asked to provide the course number and semester of enrollment for the first 100-level chemistry
course completed at the university. The researcher used this information to obtain a copy of the
course syllabus from the university’s online syllabus manager. As with the instructor pilot study,
this syllabus was provided to each student to aid the student’s recall of specific course details and
to provide an artifact for the researcher to use for comparison with the student’s interview
responses. The CoI and satisfaction item pilot study protocol started with the think-aloud
interview.
During the think-aloud interview students were provided with the 36-item CoI instrument
revised by the researcher along with the two sets of satisfaction items (see Appendix C). If the
student indicated confusion or uncertainty in selecting a response, additional follow-up questions
were asked by the researcher to probe possible reasons for this difficulty, including unclear
wording of the item or the item not being applicable in a particular classroom environment.
Following completion of the survey instrument, the student was asked if he or she had any
difficulty with the semantic differential set of satisfaction items, if the student had a preference
for the traditional or semantic differential satisfaction items, for any other general comments or
feedback on the CoI or satisfaction items, and if the items seemed to cover all relevant aspects of
89
the course that would help someone understand the student’s experience within the course
environment.
Pilot Study Results
Instructor Survey
The five instructors participating in the pilot study reported between four and 40 years of
experience teaching chemistry. Four instructors were male and one was female. The most
recently taught 100-level courses used for reference in the instructor interviews were first
semester general chemistry, GOB (general, organic, and biochemistry) for nursing students, and
a chemistry course for non-science majors. Typical enrollment for these courses ranged from
approximately 40 students in the GOB and non-science majors courses to approximately 100
students in the first semester general chemistry courses.
All instructors identified at least a few issues with item wording during the think-aloud
portion of the interview. During the think-aloud each instructor was directed to provide a short
explanation of why he or she chose a particular response to each ATI item and also asked to
elaborate on particular words or phrases that were unclear or confusing. The most frequent
comment from instructors was that particular phrases were unclear or too open to interpretation.
For example, the first item on the ATI asks about students focusing on “what I provide them” but
instructors were unsure if this meant only material created by the instructor such as lecture notes
and handouts, or if this included all course material selected by the instructor including the
textbook and supplemental materials.
Similarly, instructors were unclear on what the phrases “formal assessment items”,
“teaching time”, “key texts and readings”, “teaching sessions”, “good set of notes”, “good
90
presentation of information”, and “information base” meant in questions 2, 5, 6, 8, 9, 10, 11, 15,
16, 17 and 22. For these items, modifications were made to the wording to more closely align
each item with the interpretation adopted by most instructors. In items 5, 8, and 15 the phrases
“teaching sessions” and “teaching time” were removed to accommodate the broader
interpretation of teaching in the course to include communicating with students beyond the
scheduled class meeting times. This interpretation included office hours, email exchanges, and
course websites as places where instructors could “make available opportunities for students in
this course to discuss their changing understanding of the subjects” as described in item 13. This
interpretation is consistent with student responses to the CoI items in which the students
considered office hours and email exchanges as places where the instructor could communicate
important course information and provide feedback to students.
Other modifications included item 12 which was identified as too vague in asking about
“any questions that students may put to me” and was changed to “any questions about course
content that students may ask”. Additionally, item 14 was expanded to cover the frequently
mentioned idea that students could be annotating notes provided by the instructor and not
necessarily writing their own from scratch. In addition to the unclear and vague items, some
items used language that instructors found to be too loaded, such as the phrase “I deliberately
provoke debate and discussion” in item 8. This wording was interpreted as antagonistic and was
changed to the more neutral phrase “I encourage debate and discussion”. For the same reason,
the word “question” in item 15 was changed to “discuss”.
Some of the items also seemed to have implied judgments, such as the phrase “a lot of
facts” in item 4. Instructors generally agreed that students should be presented with facts as part
91
of the course but did not feel comfortable agreeing with the idea of presenting a lot of facts since
this seemed to have a negative connotation. For the same reason the qualifiers “good” or “a lot”
were removed from items 10, 15 and 16 to make them more neutral.
Another issue that arose during the think-aloud was the relevance of the frequency
response scale for certain ATI items. The frequently scale ranged from “only rarely” to “almost
always” and did not seem appropriate for items asking for instructor judgments using phrases
like “it is important” or “it is better”. As an example, the second item on the version of the ATI
used in the pilot study read “It is important that this course should be completely described in
terms of specific objectives that relate to formal assessment items.” On a frequency scale, it is
unclear if a response of “almost always” indicates that the instructor almost always believes it is
important to describe the course in terms of specific objectives related to formal assessment
items or if the course itself is almost always described in terms of specific objectives related to
formal assessment items. Due to the difficulty of answering these items on the frequency
response scale, they were reworded to remove the belief component and be more clearly focused
on actual classroom practices. As a result, the second item was changed to read “This course is
completely described in terms of specific objectives that relate to course assessments.” With this
new wording the item could now be answered with respect to how frequently specific objectives
are related to course assessments. This change and similar changes to items 4, 10, 14, 17, 18, 20,
and 21 better align the ATI with its purpose in the context of the current research, which is to
provide a description of the learning environment from the instructor’s perspective. Appendix D
contains the full list of ATI items, revisions made to the items after the pilot study, and the
specific rationale for each change.
92
While all but four items on the ATI required some degree of revision to their wording,
each instructor’s interpretation of the items during the think-aloud protocol indicated that
generally each instructor had an interpretation of the item that agreed with the intended focus of
the item based on whether it was designed to address the information transmission teacher-
focused (ITTF) or the conceptual change student-focused (CCSF) scale (Trigwell & Prosser,
2004). For this reason, scale scores were calculated for each instructor and compared to their
responses to the open-ended interview questions which asked about their approaches to teaching.
Responses to the open-ended interview questions were examined through the lens of the two
teaching approaches the ATI was designed to measure, ITTF and CCSF. The scale scores and
interview responses were also compared to the information available in the course syllabus.
The self-reported approaches to teaching varied for each instructor but a majority of the
instructors described the importance of having students solve chemistry problems both as a way
to learn chemistry and also as a way to demonstrate their knowledge. A subset of these
instructors also commented on wanting to provide students with class time to practice and
discuss problem solving with each other but felt limited in their ability to provide this class time.
These comments are interpreted as indicating that a majority of the instructors perceived a
conflict between wanting to adopt a more student-focused classroom environment, interpreted as
a more CCSF approach, and feeling limited by the need to use class time to provide information
to students, interpreted as a more ITTF approach. One instructor emphasized the role of
discussions in providing an impetus for students to learn content and as a way for students to
demonstrate their knowledge outside of solving problems, which provides another illustration of
a more CCSF approach.
93
Following the directions provided by Trigwell for use of the ATI-R, scores were
calculated on each scale for each instructor’s approach to teaching the particular course
discussed during the interview. These calculations were done by summing the responses to items
on each scale and dividing by the total number of items on the scale. This produced an average
score for each instructor on each scale ranging from 1 (Only Rarely) to 5 (Almost Always).
Examination of the scores showed that all five ITTF scores were in a relatively narrow range
from 3.5 to 4.4 while CCSF scores ranged from 3.0 to 4.8.
Additionally, there was no clear relationship between ITTF and CCSF scores. That is,
higher CCSF scores did not correspond to lower ITTF scores and higher ITTF scores did not
correspond to lower CCSF scores. This result was unexpected and may indicate that the two
approaches to teaching are not mutually exclusive for the 100-level chemistry courses described
by the instructors. One possible explanation for this result could be if the sample of five
instructors was atypical of those surveyed in other research with the ATI. Another possibility
may be that ITTF and CCSF scores have a negative correlation in aggregate across multiple
instructors but when focusing on an individual instructor describing his or her approach to
teaching a specific course, there is no relationship between ITTF scores and CCSF scores.
Addressing the first possible explanation requires looking at the samples used to develop
and refine the ATI. The initial phenomenographic interviews that led to the development of the
ATI were conducted with 24 instructors of first-year undergraduate chemistry and physics
courses at two Australian universities (Trigwell et al., 1994). The courses taught by these
instructors included courses for “engineers, life scientists, nurses and dentists as well as courses
for chemists and physicists” (Trigwell et al., 1994, p. 219). The instructors “held positions from
94
lecturers to professors, they all conducted lectures as part of their teaching and most were
involved in tutorials and/or laboratory teaching” (Trigwell et al., 1994, p. 219). The five
instructors interviewed for this pilot study comprised the entire population of instructors teaching
100-level chemistry lecture courses the semester the research was conducted and in that way are
fully representative of instructors at this midsize private university in the United States. In
addition, all interviewed instructors conducted lectures as part of their teaching and taught
courses targeted at the same population of students as in the Prosser, Trigwell & Taylor (1994)
study. Differences in US and Australian language were the primary reason for the small
modifications to the wording of the ATI items in changing “subject” to “course” for the pilot
study but the other similarities between the two groups indicate that they are analogous groups of
instructors.
While the group of instructors participating in the pilot study was comparable to the
group of instructors participating in the initial phenomenographic interviews used to develop the
ATI items, the ATI has also been tested with a much larger and more diverse population of
instructors. A majority of the research in the development and use of the ATI was conducted
with instructors in Europe, Australia, and Hong Kong who taught both in a variety of disciplines
in addition to chemistry and at various levels other than introductory courses (Prosser &
Trigwell, 2006; Trigwell et al., 2005). The use of the ATI with this range of instructors suggests
that it should be broadly applicable to other disciplines and that the sample of instructors
participating in this pilot study should be well within the population for which the ATI is
applicable. To confirm that the approaches to teaching 100-level chemistry courses described by
the sample of instructors participating in the pilot study are typical of instructors at other US
95
universities, future work with the revised ATI should be extended to include 100-level chemistry
course instructors at other US universities.
The second possible reason for the observation that ITTF and CCSF scores were not
related in this sample of instructors may be that almost all introductory chemistry courses are
heavily focused on content delivery. This emphasis on content delivery explains the high and
relatively narrow range of observed ITTF scores among all approaches to teaching 100-level
chemistry courses. The large amount of content in 100-level chemistry courses is frequently
discussed in the chemical education literature (J. N. Spencer, 1992; Talanquer & Pollard, 2010).
Given this environment for 100-level chemistry courses, it may be that the ITTF scale is
providing little relevant information about an instructor’s approach to teaching 100-level
chemistry courses since all instructors are concerned with covering a large amount of content.
This interpretation is further supported in the literature (Stains et al., 2015) by the lack of change
in ITTF scores after new chemistry professors attend a workshop emphasizing student-centered
teaching techniques, even though CCSF scores significantly increased for this group.
The chemical education literature also discusses the efficiency of lecture for delivering
content to large numbers of students even while recognizing that this information transfer
method of teaching is not as effective as more student-centered teaching approaches (Chambers
& Blake, 2007; Toto & Booth, 2008). The difficult balance between lecture and more student-
centered teaching approaches was apparent in the open-ended interviews with the instructors
participating in this study. All five instructors commented on wanting to spend less class time
lecturing but feeling constrained by the number of students enrolled in the course, the type of
material presented, or the limited amount of contact time with students. In spite of these
96
constraints, all instructors had been able to integrate some student-centered approaches into a
portion of their class time. The degree to which these student-centered teaching approaches had
been integrated into a particular course could be seen in interview responses, CCSF scale scores,
and the course syllabus.
The CCSF scale scores show that, on average, all five instructors’ approaches to teaching
could be considered as emphasizing a conceptual change, student-focused approach
approximately half the time. The instructors’ descriptions of their approaches to teaching in these
specific 100-level courses were compared with their responses to the ATI and the course
information available in their syllabi. This comparison indicated that the CCSF scale on the ATI
was able to detect a student-centered approach to teaching aligned with the definition used in this
research in which the role of the instructor was shifted from a lecturer to a facilitator for at least
part of the instructional time. The two instructors who described an approach to teaching that
was “student-centered” either in their interview or in their course syllabus corresponded to the
two highest CCSF scale scores. These two approaches to teaching still utilized lectures as the
primary method of delivering content to students but set aside some class time for small group or
whole class discussions. In this way these instructors adopted a teaching approach in which they
act as a facilitator for a portion of the instructional time which is reflected in the CCSF scores for
these courses.
Though these two instructors spent some class time utilizing a student-centered approach
to teaching, this did not mean that the instructors emphasized teaching content, as measured by
the ITTF scale, to any lesser degree than the other instructors who participated in the ATI pilot
study. While this sample of five instructors is relatively small, the results suggest that in the
97
context of approaches to teaching 100-level chemistry courses the ITTF and CCSF scales should
not be considered as describing mutually exclusive approaches to teaching. Clearly it is possible
for instructors to adopt aspects of a student-centered approach to teaching as part of an approach
that delivers content to students in a lecture format. However, it remains unclear whether this
result is due to the small sample size or the relative homogeneity of instructors and courses that
are the focus of the study. To address these concerns future research with the revised ATI should
be extended to instructors of chemistry courses beyond the 100-level to see if a similar
relationship is seen between ITTF and CCSF scores on the individual instructor level.
The scope of this research on constructivist learning environments in 100-level chemistry
courses did not necessitate the inclusion of a sample of instructors that would be large enough to
perform any statistical analyses, such as CFA, that could examine the internal structure of the
revised ATI instrument in Appendix D. Without the ability to demonstrate that the items in the
revised ATI are still related to the expected factors representing the ITTF and CCSF approaches
to teaching, it did not seem acceptable to use the revised ATI as the sole method of collecting
data on the degree to which a course instructor has created a constructivist learning environment.
For this reason, when the revised ATI was used in the main research project, a short semi-
structured interview was also conducted with the instructors to provide an opportunity for each
instructor to describe her or her approach to teaching. The interview was then used to inform
ATI scale score interpretation in the same way the open-ended question responses informed the
results of the pilot study. The results of the pilot study also indicated that only the CCSF scale
would provide information relevant to the degree to which a course instructor has created a
constructivist learning environment utilizing student-centered approaches to teaching.
98
Student Survey
The five students participating in the pilot study had all completed first semester general
chemistry courses. Four students had taken the course within one year of participating in this
pilot study and the other had taken the course three years prior to the interview. The four students
who had taken first semester general chemistry within the last year had to write a paper as part of
their course requirements. This group of four students also contained two pairs of students who
had been enrolled in the same course during the same semester. As a result of this overlap, the
five students represented courses taught by three of the five instructors interviewed in the
instructor portion of the pilot study. Two of the three female students and none of the male
students interviewed had been enrolled in an honors section of general chemistry. The class sizes
reported by the students ranged from approximately 15 students in the honors general chemistry
course to approximately 100 students in the regular section of general chemistry.
Due to recruitment of students in the Chemistry Club and the voluntary nature of
participation in the pilot study, these five students may not be representative of the entire
population of students enrolled in 100-level chemistry courses. Specifically, these students are
likely to have had more positive experiences in their chemistry courses and higher course grades,
both of which are possible motivations for joining the Chemistry Club. The students were not
asked to report their course grades as part of the pilot study, but the three students who
mentioned their course grade during the interview reported receiving grades of either B+ or A.
The other two students reported being happy with the grade they received. As a result, these five
students may have been more predisposed to provide positive answers to the items on the survey
than the average student enrolled in the courses. However, since the focus of the pilot study was
99
not on student ratings of the course but instead on how students interpreted the items, this
potential bias was not anticipated to influence the results of the think-aloud interviews.
Unlike the instructor pilot study, the student think-aloud interviews did not reveal many
issues with wording of CoI or satisfaction items. The only item that caused a problem for all five
students was item 5, a teaching presence item, which read “The instructor was helpful in
identifying areas of agreement and disagreement on course topics that helped me to learn.” The
students all indicated confusion on what was meant by “areas of agreement and disagreement”.
The most common student interpretation was related to agreement or disagreement on what
topics to include in the course or the order in which to present the course topics. These
interpretations appear to speak to the course design and organization component of teaching
presence. However, this interpretation is not supported by the literature discussing the
development of the CoI. Arbaugh (2008) describes identifying areas of agreement and
disagreement as one role of the instructor related to the facilitating discourse component of
teaching presence. Based on the intended focus of this item, it was reworded as “The instructor
was helpful in facilitating discussions on course topics that helped me to learn.”
Another wording issue occurred for item 27, a cognitive presence item, which initially
read “Brainstorming and finding relevant information helped me resolve content related
questions.” Student responses to this item indicated that brainstorming and finding relevant
information were not perceived as a single process but rather as two distinct phases of problem
solving. Students described “brainstorming” as the mental organization and planning that
occurred before “finding relevant information” which was described as looking up information
necessary to solve a problem. For most students these two processes were related, but some
100
students indicated that one half of the process was more helpful than the other half. Since some
students interpreted this item as double-barreled, it was split into two separate items. In addition,
a few students noticed inconsistent use of the phrases “course activities” and “learning activities”
so these were all changed to the phrase “course learning activities”.
The think-aloud interviews did not indicate any issues with student understanding of the
semantic differential satisfaction items. The follow-up question asking about student preference
for either the traditional or semantic differential set of satisfaction items revealed that students
preferred the traditional set of items because they felt those items addressed more specific
aspects of the course while the semantic differential items provided a more overall assessment of
the course. Even though students preferred the traditional satisfaction items, their description of
the semantic differential items as a more holistic view of their satisfaction with the course aligns
these items more closely with their purpose in this research project. The satisfaction items are
intended to function as indicators of the latent variable of overall student satisfaction with the
course and therefore the semantic differential items were chosen for use in conjunction with the
CoI items in the main research project.
Small changes were also made to the response scale options based on how students were
using the “Unsure” and “Not Applicable” options for the CoI items. Probing student use of the
unsure option revealed that students were treating it more as a midpoint between agree and
disagree than to communicate uncertainty in their response. For this reason, it was renamed as
“Neutral” in the revised version of the CoI. This change also increases the correspondence
between the CoI response scale and the response scale for the semantic differential satisfaction
items where the middle category functions a neutral midpoint between the two opposite words.
101
Initially a “Not Applicable” option was included as a zero on the CoI response scale, as
in the Shea & Bidjerano (2009) study, to accommodate situations where the course did not utilize
formal discussions. However, the students in the pilot study did not use this option consistently
and often were able to provide a response to items about course discussions even when the
course did not utilize formal structured discussions. The students typically tried to find anything
in the class that could be considered a discussion, such as the instructor asking questions of the
class during lecture.
Looking at pairs of students who were in the same course environment revealed that
when one student selected the “Not Applicable” option, the other student did not and instead
used the other scale options to describe his or her perception of the learning environment. This
inconsistent use of the “Not Applicable” option could have caused issues during the analysis and
interpretation of results when the CoI was used with a larger sample of students. For this reason,
the “Not Applicable” option was removed in order to encourage all students to select a response
that best described the learning environment. Encouraging respondents to provide their opinions
by removing don’t know and not applicable options is in alignment with best practices in survey
design (Krosnick & Presser, 2010). With the “Not Applicable” option removed, the response
scale for the CoI items was changed to a more standard appearance with 1 (Strongly Disagree)
on the left and 5 (Strongly Agree) on the right. The response scale for the semantic differential
items was also changed in this way, matching the original arrangement in Xu & Lewis (2011).
The full revised CoI and satisfaction student survey instrument is available in Appendix E.
Survey Instrument Validity Evidence
Much recent discussion of validity related to instruments used for measurement in
102
chemical education research utilizes the description of validity as a single concept with multiple
aspects (Arjoon et al., 2013; Wren & Barbera, 2013; Xu & Lewis, 2011). This conceptualization
of validity is also present in the most recent edition of the Standards for Educational and
Psychological Testing, known as the Standards, a collaboration of the American Educational
Research Association (AERA), the American Psychological Association (APA), and the
National Council on Measurement in Education (NCME). In the Standards, validity is described
as “a unitary concept. It is the degree to which all of the accumulated evidence supports the
intended interpretation of test scores for the proposed use” (AERA et al., 2014, p. 14). The
unitary nature of validity is also discussed at length by Messick (1989, 1995) who further
emphasizes that providing evidence for validity is a continuous process. Due to the continuous
nature of this process, Messick explains that “validity is a matter of degree, not all or none”
(1989, p. 13). Finally, both Messick (1995) and the Standards (2014) are clear that validity is not
a property that can be ascribed to a particular test but rather it is the test scores themselves that
must have evidence provided for the validity of their meaning in a particular context.
The unitary concept of validity replaced older descriptions of validity as having distinct
types including content validity, criterion-related validity, predictive validity, concurrent
validity, and face validity (AERA et al., 2014; Messick, 1989). Instead of recognizing distinct
types of validity, Messick (1989) defines construct validity as the overarching form of validity
because it provides “an integration of any evidence that bears on the interpretation or meaning of
the test scores” (p. 17). Messick (1995) describes six aspects of validity that can be used as
criteria to evaluate construct validity. These six aspects are content, substantive, structural,
generalizability, external, and consequential. Messick’s six aspects are integrated into the five
103
types of evidence for validity described in the Standards (2014) where the generalizability and
external aspects are integrated into a larger category of providing evidence for relationships
among the scores and other variables. The five types of evidence in the Standards (2014) are: (1)
evidence based on test content, (2) evidence based on response processes, (3) evidence based on
internal structure, (4), evidence based on relations to other variables, and (5) evidence for
validity and consequences of testing. These types of evidence for validity will be used as a
framework through which to evaluate the evidence that exists for the validity of ATI, CoI, and
student satisfaction scores in the context of the current research. While the Standards (2014)
specify that not all types of evidence are necessary in all situations, each type of evidence will be
discussed here and the presence or lack of evidence described for the ATI, CoI, and student
satisfaction items.
According to the Standards (2014), it is necessary to clearly define the proposed
interpretation of scores for which evidence is being provided. In the case of both the ATI and the
CoI, the underlying construct that both instruments are designed to measure is the degree to
which a constructivist learning environment exists for a particular 100-level chemistry course.
The ATI uses the instructor’s point of view to measure the frequency with which various
classroom activities and instructor behaviors occur. The proposed interpretation of ATI scores is
that these frequencies can be used to determine the degree to which an instructor’s approach to
teaching is aligned with constructivist principles of student-centered teaching. The CoI uses the
student’s point of view to measure the degree to which the student perceived various indicators
of a constructivist learning environment related to teaching presence, social presence, and
cognitive presence. The proposed interpretation of CoI scores is that the student perceptions can
104
be used as indicators of the three factors that encompass salient aspects of a constructivist
learning environment. Lastly, the student satisfaction items will be interpreted as indicators of
the latent variable of overall satisfaction with the course. With these proposed interpretations in
mind, the evidence for validity can now be described and evaluated. Since the ATI, CoI, and
satisfaction items were drawn from existing research, some evidence for validity will come from
reports by the test developers and test users in the literature (AERA et al., 2014) in addition to
evidence generated from the pilot study described in the previous section. Validity evidence from
use of the ATI, CoI, and satisfaction items in the main study will be reported in Chapter 5 along
with other interpretations generated from analysis of the survey response data.
Test Content The first source of validity evidence described by Messick (1995) and the Standards
(2014) relates to the content of the test. The test content must be both representative of the
construct and appropriate for the context. The development of the CoI provides the clearest
evidence of its representativeness in measuring a constructivist learning environment because the
survey items were specifically designed to measure the three presence factors described by the
Community of Inquiry model, which is a constructivist view of learning in online environments
(Swan et al., 2009). The ATI does not have an explicit link to constructivist learning
environments, but the pilot study interviews revealed that the student-centered aspect of the
conceptual change student-focused (CCSF) scale addressed many aspects relevant to a
constructivist learning environment such as the instructor acting as a facilitator and encouraging
active student involvement in discussions and explanations of problem solving. However, the
pilot study interviews also revealed that instructors were concerned that the ATI alone may not
105
fully address all aspects of teaching for a particular course. Since there is evidence for the
possibility of construct underrepresentation when using the ATI, it was used in combination with
instructor interviews and analysis of the course syllabus to provide a more complete description
of an instructor’s approach to teaching.
The original use of the CoI in measuring a constructivist learning environment in online
courses meant that some items were not appropriate for use with students in face-to-face
chemistry courses and necessitated rewording. However, a majority of the items addressed
indicators of a constructivist learning environment that could also be found in face-to-face
courses and were therefore appropriate for the current research context. The ATI was developed
from interviews of university chemistry and physics instructors and was therefore expected to be
appropriate for use in the current research with minimal changes to reflect the US language
conventions. While significant rewording of the ATI items was found to be necessary, the pilot
study interviews also revealed that the items on the information transmission teacher-focused
(ITTF) scale may not be relevant or appropriate for analysis in the current research context since
they did not provide information relevant to the development of a constructivist learning
environment. In contrast, the satisfaction items were developed specifically for use with college-
level chemistry students and are therefore representative of the satisfaction construct and
appropriate for the current research context (Xu & Lewis, 2011).
Response Process The second source of evidence for validity, described as substantive by Messick (1995)
and as evidence based on response process by the Standards (2014) speaks most directly to the
purpose of the pilot study. The results from the pilot study indicate that both the CoI and
106
satisfaction items were generally interpreted as intended, though the sample of students may not
have been fully representative of the entire population of students enrolled in 100-level
chemistry courses. During the pilot study interviews students demonstrated the ability to focus
only on the lecture portion of their chemistry course and had very few problems understanding
what each item was asking, providing evidence that the CoI and satisfaction items were in fact
measuring perceptions of the classroom environment. The notable exceptions to this were item 5,
“The instructor was helpful in identifying areas of agreement and disagreement on course topics
that helped me to learn,” and item 16, “Face-to-face communication is an excellent medium for
social interaction.” As previously discussed, item 5 was reworded to more clearly align it with its
intended function as an indicator of teaching presence related to facilitation of discussions. The
issue with item 16 was that students in the pilot study did not understand the relevance of this
item to the overall purpose of the survey since it seemed to be asking more about their beliefs
than any particular aspect of the classroom. However, item 16 was not removed from the
instrument to maintain comparability with previous administrations of the CoI. The functioning
of this item will be discussed in greater detail in Chapter 4 as part of the interpretation of the
results of administering the survey to a larger group of students in the main study.
Student responses to the two sets of satisfaction items indicated that the semantic
differential satisfaction items appeared to be measuring a more holistic sense of satisfaction with
the course. This understanding is in alignment with their intended interpretation as indicators of
overall student satisfaction both in prior research (Xu & Lewis, 2011) and the current research.
The pilot study with the ATI indicated extensive problems with instructors’ response
processes due to unclear wording of the items and the irrelevance of the frequency scale to items
107
worded more as measures of instructors’ beliefs than actual classroom practices. These problems
resulted in the rewording of 18 of the 22 ATI items to both clarify the items and improve their
ability to be answered on the frequency response scale. For the items that did not require
rewording, the pilot study indicated that instructors were able to answer each item by considering
their approach to teaching a specific chemistry course and were providing answers based on
actual classroom practices. This provides evidence that when the items are clear and appropriate
for the response scale, the instructor response process is in alignment with the intended goals of
the instrument to provide a measurement of classroom practices related to the ITTF and CCSF
scales.
Internal Structure Evidence based on internal structure primarily comes from existing research with the
CoI, satisfaction items, and ATI in which their internal structures were examined with factor
analysis techniques such as EFA and CFA. Evidence for the internal structure of all three
instruments was discussed extensively in Chapter 2 and was the primary reason the instruments
were selected for use in the current research. This evidence remains relevant for the satisfaction
items since they were unchanged in the pilot study. Given the small scale of the pilot study, it
was not possible to examine the results using any factor analysis techniques prior to
implementation of the modified instrument in the main study. The results of using the CoI and
satisfaction items with students in face-to-face chemistry courses in the main study are provide in
Chapter 4 and provide validity evidence based on the internal structure of the instrument in this
new context.
The substantial revisions to the ATI items limits the applicability of previous factor
108
analysis results to the revised version of the instrument. Since the primary research focused on
collecting data from students enrolled in the same class, too few instructors participated to
provide enough ATI responses for factor analysis of the results. Therefore, evidence for the
internal structure of the revised ATI will remain an open area for future research.
Relationships with Other Variables
The fourth type of evidence for validity described in the Standards (2014) is evidence for
relationships with other variables. This encompasses convergent, discriminant, predictive, and
concurrent relationships with other variables in addition to the generalizability of the validity
evidence. Due to the limited use of the semantic differential satisfaction items in the chemical
education literature, the modification of the CoI for use with students in face-to-face chemistry
courses, and the numerous revisions to the original ATI items, there is little evidence for the
generalizability of these instruments beyond the current study. Their use in the main research
study provides some evidence for their generalizability, discussed in Chapter 5, but this source of
evidence will remain minimal until the instruments are used more widely in chemical education
research.
However, some evidence for relationships between these instrument scores and other
variables can be found in the literature. The latent construct of emotional satisfaction measured
by the semantic differential items has been shown to correlate moderately with ACS exam scores
but less well with ACT or SAT math scores providing predictive and discriminant evidence for
the validity of the satisfaction item responses (Xu & Lewis, 2011). Similarly, the teaching
presence and cognitive presence factors on the CoI have been shown to have an influence on
student satisfaction (Joo et al., 2011). The existence of these relationships in the data collected
109
for the main research study will be presented in Chapter 4 and discussed in the context of
validity evidence in Chapter 5.
Research utilizing the ATI with new and first-year chemistry faculty teaching primarily
upper-level undergraduate and graduate chemistry courses at research intensive institutions who
were exposed to evidence-based instructional practices at a two-day summer workshop showed
an increase in CCSF scores one week after attending the workshop (Stains et al., 2015). Since
these evidence-based practices primarily emphasized student-centered teaching techniques, this
provides evidence for a relationship between CCSF scores and a pedagogical training variable.
Additionally, the workshop group was shown to use more student-centered teaching techniques
as measured by classroom observation scores from the Reformed Teaching Observation Protocol
(RTOP) and the Classroom Observation Protocol for Undergraduate STEM (COPUS; Stains et
al., 2015). This provides convergent evidence for the validity of the CCSF scores obtained with
the original form of the ATI. Additional convergent evidence was found in the pilot study based
on the alignment of CCSF scores with instructors’ descriptions of their approach to teaching and
information found in their course syllabi. Further validity evidence from instructor interviews
and syllabus analysis conducted for the main study will be discussed in Chapter 5.
Consequences of Use
The final category of evidence for validity comes from how the test scores will be used.
These consequences for use are related to how the creators of the instrument intended the scores
to be used, any claims that might be made beyond these intended interpretations, and any
unintended consequences of use. For all three instruments, care has been taken to trace the
instrument development back to its theoretical foundations, as discussed in Chapter 2, and ensure
110
that its use in the current research is in alignment with the intention of the creators. This is
especially a concern in using the ATI scores only as a measure of an approach to teaching a
particular course, not an assessment of any particular instructor (Prosser & Trigwell, 2006). The
decision to include instructor interviews in the main research study is in alignment with the
warning that the ATI “is not intended for use in gathering a full, rich self-report of teaching”
(Prosser & Trigwell, 2006, p. 405). The satisfaction and CoI items will only be used in their
intended capacity as indictors of the latent variables of student satisfaction with a chemistry
course (Xu & Lewis, 2011) and latent variables representing three aspects of a constructivist
learning environment (Arbaugh, 2008; Arbaugh et al., 2008; D. R. Garrison et al., 2000). Steps
will be taken to limit unintended consequences of using these instruments by clearly describing
the limitations of score interpretation in the context of the current research and protecting the
identity of all respondents when results are reported.
The constant generation of new evidence for the validity of CoI, satisfaction, and ATI
instrument scores means that the discussion of validity should be ongoing and revisited at every
stage of the research. As a result of the pilot study, there is clear evidence for the validity of both
the CoI and satisfaction instrument scores in the context of the current research. These two
instruments did not require substantial revisions and student interviews provided evidence for
alignment of student responses with the intended use of the instruments to provide indicators of
student perceptions of three aspects of a constructivist learning environment and overall student
satisfaction with the course. The pilot study provided less clear evidence for the validity of ATI
scores for use as the only means with which to determine the degree to which an instructor’s
approach to teaching is aligned with constructivist principles of student-centered teaching.
111
However, the alignment of the CCSF scale with instructor interviews and course syllabi provides
convergent evidence for the validity of this scale as does its use in other chemical education
research. Due to the significant revision of the ATI items, additional validity evidence was
examined after its use in the main research study. Though the CoI and satisfaction items were not
significantly reworded, their use in the main research study also provided an opportunity to
examine additional evidence for the validity of the scores in the context of the current research,
discussed in Chapter 5.
Power Analysis for Sample Size Determination In previous work with the CoI, recommendations for sample size in structural equation
modeling were based on rules of thumb suggesting that a sample size of 200 participants was
adequate for small to medium sized structural equation models with fair to good reliability (D. R.
Garrison et al., 2010; Tabachnick & Fidell, 2007). However, these general recommendations
overlook the importance of statistical power in the determination of sample size (Hancock &
French, 2013). Conducting a power analysis prior to collecting data allowed for the
determination of the sample size necessary to test both overall data-model fit and individual
parameters within the model.
Power analysis calculations assume that the data collected to test the model meet the
assumption of multivariate normality, that the model shows the correct relationships among
variables, and that correct parameters are used in the calculations. Previous research with the
CoI, satisfaction items, SAT and ACT math scores, and ACS exams have indicated that the data
meet normality assumptions (Joo et al., 2011; Shea & Bidjerano, 2009; Xu & Lewis, 2011) so it
was anticipated that the data analyzed for the current research met this assumption as well. Every
112
effort was made to ensure the correctness of the model and the parameters used in the power
analysis calculations. However, since the primary goal of the research was to test the model, it
was impossible to definitively know its correctness prior to collecting and analyzing data.
Overall Data-Model Fit
Power analysis for overall data-model fit was done utilizing the root mean square error of
approximation (RMSEA) fit statistic. As discussed in Chapter 2, smaller RMSEA values indicate
better data-model fit. When conducting power analysis for overall data-model fit an RMSEA
value of 0.05 is typically chosen as the cut-off between acceptable and unacceptable fit (Hancock
& French, 2013). The goal of the overall data-model fit power analysis is to reject a null
hypothesis that the RMSEA = 0.05 in favor of an alternate hypothesis where the RMSEA < 0.05.
The specific value of the RMSEA to be used in the alternate hypothesis must be specified prior
to conducting this power analysis. The RMSEA value chosen for the alternate hypothesis
represents the RMSEA value expected to be obtained when testing the fit between the
hypothesized model and the collected data. An expected RMSEA value of 0.00 indicates a
perfectly specified model and requires the smallest sample size to obtain the desired power
because the model perfectly represents the observed relationships in the data, but it is highly
unlikely that the data collected will have a perfect fit with the hypothesized model. At the other
extreme, an alternate RMSEA value of 0.04 is a much more conservative estimate of data-model
fit, but requires a much larger sample size to obtain the desired power due to the mismatch
between the relationships in the data and the model. Hancock and French (2013) suggest setting
the RMSEA value equal to 0.02 for the alternate hypothesis to provide a balance between
“unrealistic optimism” and “impracticality” (p. 127).
113
In addition to selecting an alternate RMSEA value, desired values for statistical power and
alpha level must be selected and the degrees of freedom for the model being tested need to be
determined. For this analysis the power was set to 0.80 and alpha was set to 0.05, as is typical in
hypothesis testing (Hancock & French, 2013). Calculating the degrees of freedom (df) of the
hypothesized model required knowing the number of unique variances and covariances of the
measured variables (u) and the number of model parameters (t) so that the number of model
parameters can be subtracted from the number of unique variances and covariances. This results
in the formula for degrees of freedom shown in Equation 1.
45 = 7 − 9 (1)
The number of unique variances and covariances can be determined from the number of
measured variables (p) in the model using the formula in Equation 2 (Mueller & Hancock, 2008).
7 =
:(: + 1)
2 (2)
For this research, the measured variables are the 36 items on the revised version of the CoI, the
four student satisfaction items, and the outcome variables of math ability, ACS exam scores, and
final course grades. This is a total of 43 measured variables, p, and following from Equation 2,
there must be a total of 946 unique variances and covariances.
To determine the value of t, it is helpful to construct the primary model used in this
research with all model parameters included and labeled. This full model is shown in Figure 13.
The model in Figure 13 is based on the model in Figure 10, provided in Chapter 2 on p. 68, but
includes all the hypothesized relationships among measured variables, latent variables, and
error/residual terms. The residual terms are called errors for measured variables and disturbances
114
Figure 13. The primary hypothesized structural equation model with all parameters labeled. Focal parameters of interest to this research are marked with asterisks.
115
for latent variables. The variables and error terms are labeled using the Bentler-Weeks labeling
convention and the model parameters are labeled using the abc system developed by Hancock &
Mueller (2006). A summary of the notation used in the model is provided in Table 2.
Table 2 Notation Used in Full Structural Equation Model in Figure 13
Symbol Meaning Rectangle Measured variable Oval Latent variable V Measured variable label F Latent variable/factor label E Error term/measured variable residual D Disturbance/latent factor residual bto,from Path between two variables where subscripts indicate where the path ends and
where it originated c Variance for a single variable or covariance between two variables T Nonstandard notation, used in this model to indicate a teaching presence item S Nonstandard notation, used in this model to indicate a social presence item C Nonstandard notation, used in this model to indicate a cognitive presence item SS Nonstandard notation, used in this model to indicate a student satisfaction item
Focusing first on the paths between variables (b) there are 36 paths representing survey
items loading on their respective factors and 13 paths between the factors and the measured
student variables. This comprises a total of 49 paths. Four paths between a survey item and its
respective factor have been fixed to 1 as a way to provide scale for the factor and are therefore
not parameters that will be estimated. There are a total of 48 variance and covariance
terms (c) representing the variances and error/disturbance variances of the 43 measured variables
and the four latent variables in addition to the covariance term between student satisfaction and
final course grades. Summing the 49 paths and 48 variances/covariances gives a total of 97
model parameters that must be estimated. Substituting the calculated values of u and t into
Equation 1 results in 849 degrees of freedom for the model in Figure 13.
Using the value of 849 for degrees of freedom, three power analyses were conducted in
116
order to estimate the sample size necessary to reject a null hypothesis of unacceptable data-
model fit (RMSEA = 0.05) with a power of 0.80 and alpha = 0.05. These calculations were
performed using a web-based power calculator for RMSEA (Preacher & Coffman, 2006). The
results were rounded up to the next whole number. The first calculation was done with the
alternate RMSEA value set at 0.00 to provide the most optimistic determination of sample size,
which was 53 participants. The second calculation was done with the alternate RMSEA value set
at 0.04 to provide the most conservative determination of sample size, which was found to be
178 participants. Finally, the calculation was done with the alternate RMSEA value set at 0.02 to
provide the most realistic determination of sample size (Hancock & French, 2013), which was
found to be 64 participants. These results are summarized in Table 3.
Table 3 Sample Sizes for Testing Overall Data-Model Fit
Alternate RMSEA value Resulting sample size 0.00 53 0.02 64 0.04 178
Note. For each analysis power = 0.80, alpha = 0.05, df = 849, and the null RMSEA = 0.05
Testing Parameters within the Model
Additional power analyses were conducted to determine the sample sizes necessary for
testing individual parameters within the model. While the whole model contains 97 parameters
that must be estimated, not all of these parameters are important in the context of the current
research questions. The primary purpose of the research is to examine relationships among the
three CoI presence factors (teaching presence, social presence, and cognitive presence), the
student satisfaction factor, and the measured variables of math ability, ACS exam scores, and
final course grades. These relationships include the 13 directional paths between measured
117
variables and factors as well as the covariance term between student satisfaction and final course
grades. In total, there are 14 parameters that can be considered the focal parameters for this
research. These are the same parameters explicitly shown as arrows in Figure 10 (p. 68) and
indicated by asterisks in Figure 13. The other parameters are termed peripheral parameters
(Hancock & French, 2013). The peripheral parameters will be estimated as part of the model, but
their values are not as important for addressing the primary research question.
Selection of model parameters
The steps outlined by Hancock & French (2013) were followed to calculate the sample
size necessary to determine if each focal parameter is non-zero with acceptable power. As with
the overall data-model fit power analysis, for each of the 14 single parameter tests the decision
was made to set power = 0.80 and alpha = 0.05. Next, numerical values were selected for all
parameters in the model, both focal and peripheral. The numerical values selected were all
standardized paths or correlations between variables, since these are most frequently available in
the published literature. In obtaining numerical values from the literature, preference was given
to path values obtained as a result of SEM analysis rather than correlations. This preference was
given because the path values that result from SEM analysis represent the magnitude of the
direct relationship between two variables after controlling for the effects of other variables while
correlations include the effects of other variables and are therefore less directly applicable to
being used as estimates for path values in the current research model.
Path values obtained from SEM analyses with the CoI included the loadings of the original
34 individual CoI items onto their respective factors (Arbaugh, 2008; Arbaugh et al., 2008, 2010;
D. R. Garrison et al., 2010; Shea & Bidjerano, 2009) and the paths among the three CoI presence
118
factors (D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). A table of the path
values obtained from the CoI literature can be found in Appendix A. Standardized paths to
satisfaction from teaching presence and cognitive presence were available from Joo et al. (2011),
but due to the removal of the path between satisfaction and social presence in the Joo et al.
analysis, a literature value could not be obtained for this path and instead the correlation between
social presence and satisfaction from Arbaugh (2008) was used.
Path values for the satisfaction items loading on the satisfaction factor were obtained from
the CFA performed by Xu & Lewis (2011) showing the loading of the semantic differential
satisfaction items on the emotional satisfaction factor. The covariance term between the residual
terms for final course grades and the satisfaction factor was estimated from SEM analysis by
Greenwald & Gillmore (1997) showing the relationship between an expected course grade factor
and a course evaluation factor. In situations where multiple analyses had been conducted and
multiple literature values were available for a path, the smallest value was chosen in order to
provide the most conservative estimate for the path value.
The value for the path between math ability and ACS exam scores came from the
standardized regression coefficient for SAT math scores presented in Lewis & Lewis (2005),
which was smaller than, but similar to, the correlations between ACT math, SAT math, and ACS
exam scores found in Xu & Lewis (2011). The relationship between math ability and final course
grades in 100-level chemistry courses was chosen from the lowest correlation found in the
literature (Craney & Armstrong, 1985; Nordstrom, 1990). The values for the paths among
teaching presence, cognitive presence, ACS exam scores, and final course grades are unique to
this research and were not available from the literature. Therefore, these paths were estimated
119
based on correlations among teaching presence, cognitive presence, and students’ perceived
learning in online MBA courses (Arbaugh, 2008). Finally, the value for the path between ACS
exam scores and final course grades was estimated using the average weight of final exam scores
based on the syllabi provided by the instructors participating in the pilot study interviews. This
decision was made based on the fact that ACS exams are often used as final exams (Lewis &
Lewis, 2005). All of the standardized values that were selected are provided in Table 4 along
with the results of the power analysis for testing model parameters.
Simplification of model-implied correlation matrix
Instead of using all 43 measured variables shown in the model in Figure 13 to calculate
the model-implied correlation or variance/covariance matrix the model was conceptualized as
containing seven latent variables instead of only four. This change allowed the model-implied
matrix to consist of only seven “pseudo” variables each loading on a single factor instead of 43
measured variables. The three new latent variables are the three achievement variables of math
ability, ACS exam scores, and final course grades each forced to load on a single latent variable
with a fixed loading of 1 and a fixed error variance of zero. In this way the new latent variables
are mathematically equivalent to the measured variables, but can be treated as latent
variables in analyses. These “dummy” latent variables are labeled “F5” (math ability), “F6”
(ACS exam scores), and “F7” (final course grades) in the model shown in Figure 14.
The four preexisting latent variables of teaching presence, social presence, cognitive
presence, and satisfaction were conceptualized a having a single measured variable loading on
each latent variable. This single measured variable serves as a placeholder for the multiple
measured variables loading on each factor. These placeholder variables are labeled “V1”, “V2”,
120
Figure 14. The primary research model conceptualized with all focal parameters existing between latent variables.
121
“V3” and “V4” in the model shown in Figure 14 and represent a combination of the individual
survey items loading on the teaching presence (“V1”), social presence (“V2”), cognitive
presence (“V3”), and satisfaction (“V4”) factors. The values for the loadings of these placeholder
variables and their error covariance terms are calculated using coefficient H, a value representing
the construct reliability for each factor calculated from the standardized loadings for each of the
measured variables previously loading on that factor (Hancock, 2001).
The formula for coefficient H is shown in Equation 3, where ℓ represents the
standardized loading for each of the measured variables (k) loading on a particular factor.
A =
1
1 +1
ℓB
"
(1 − ℓB
")
C
B-D
(3)
The calculation for HF4 is shown in Equation 4 using the loadings for each of the four semantic
differential satisfaction items provided in Xu & Lewis (2011).
AF4 =
1
1 +1
0. 74"
(1 − 0.74")+
0. 77"
1 − 0.77"+
0.83"
(1 − 0.83")+
0.48"
(1 − 0.48")
= 0.838 (4)
The value of HF4 determined from Equation 4 was then used to determine the values for the path
from F4 to “V4” and the error covariance for “E4” according to the formulas shown in Figure 14.
Similar calculations were performed for the three CoI presence factors utilizing a small function
written by the researcher for use with the statistical software R version 3.1.1 (R Core Team,
2014). The function takes the loadings as inputs and outputs a value for coefficient H. The code
for this function is available in Appendix F.
122
Construction of the model-implied correlation matrix
With values obtained for all the necessary model parameters, the next step of the power
analysis involved constructing the matrix containing the model-implied numeric relations among
the seven latent variables depicted in Figure 14. Since standardized values were used for the
parameter values, this matrix functions as a correlation matrix with values of 1 on the diagonals.
The mathematical determination of the matrix was done using path tracing rules to construct the
algebraic formulas that show how the total correlation between two variables results from
relationships between these two variables and other variables in the model. In other words, the
correlation between two variables is the sum of all possible paths between the two variables that
obey the path tracing rules developed by Wright (1934). In tracing the possible paths between
two variables any number of forward arrows (à) can be passed through, moving from tail to tip.
As an example, in Figure 14 there is a direct relationship between “F5” and “F7” represented by
the path bF7F5. There is also an indirect relationship between “F5” and “F7” through “F6”, this
trace would be represented as the product of the two arrows traveled along when moving from
“F5” to “F6” and “F6” to “F7”, represented as bF6F5bF7F6.
The path tracing rules allow for backwards movement along arrows from tip to tail (ß),
but once an arrow has been passed through in the forward direction, backwards movement is no
longer allowed. Additionally, each variable can only be passed through once in a particular trace.
Finally, only one two-headed arrow can appear in any trace. Since there are no more allowed
ways to move between “F5” and “F7” the total correlation between the two factors must be equal
to the sum of the two traces, that is bF7F5 + bF6F5bF7F6. However, the matrix is constructed to show
the model-implied correlations between variables, not factors. Therefore, an additional step was
123
added to each trace so that it proceeded from V5 (math ability) to V7 (final course grades).
Examination of Figure 14 shows that both traces now start with backwards motion through the
path connecting V5 to “F5” and proceed with the forward motion through the arrows previously
described, ending with forward motion from “F7” to V7. Since the paths connecting V5 to “F5”
and “F7” to V7 were fixed to 1, multiplying each trace by 1 twice will have no impact on the
numerical result obtained once the previously determined parameter estimates are substituted in
for bF7F5, bF6F5, and bF7F6.
A similar procedure was followed when starting or ending a trace with the composite
variables “V1” through “V4”, but in these cases the path between the factor and the composite
variable is equal to the square root of coefficient H for that factor. This path is equal to the
square root of coefficient H because coefficient H represents the construct reliability for that
factor, and the square root of the reliability is equal to the standardized loading for a single
indicator variable loading on that factor (Hancock & French, 2013). This value must be included
in the algebraic determination of each trace since it is not equal to 1 and will therefore affect the
calculated value for each trace.
Application of the path tracing rules described above resulted in the development of a
series of algebraic statements into which the previously estimated model parameters could be
substituted in order to calculate a model-implied correlation matrix for the seven variables
depicted in Figure 14. The full list of algebraic statements is available in Appendix G along with
the R code used to generate the matrix. Since there were no allowed ways to get from V5 to
“V1”, “V2”, “V3”, or “V4”, these correlations were set to zero.
Executing this code with the parameter estimates from the literature resulted in a
124
correlation matrix with values greater than one. Even though correlations greater than one are
impossible, this result was obtained because many of the literature values chosen for paths were
correlations and therefore overestimated the direct relationships between two variables when
substituted into the algebraic statements. Each correlation value obtained from the literature was
revised downward by an interval no greater than 0.05 until the matrix no longer contained values
greater than one. The resulting matrix is shown at the end of Appendix G. After obtaining this
matrix, its determinant was calculated to ensure it was positive since a non-positive definite
matrix would cause the software to be unable to run in later steps of the analysis. If a negative
determinant had been obtained, the literature correlations would have again been revised until the
matrix had a positive determinant (G. R. Hancock, personal communication, August 7, 2015).
Calculation of model fit function values
The final stage of the power analysis for testing individual model parameters utilized
structural equation modeling software to calculate the model fit function value (FML) each time a
focal parameter was constrained to zero. Constraining a parameter to zero is the mathematical
equivalent of removing it from the model, causing the fit of the model to degrade. For all models
tested in this stage, the sample size was arbitrarily set to 1001 because the actual sample size was
unknown but the modeling software required a sample size to perform the model fit calculations
(Hancock & French, 2013). The software used for this stage of the analysis was LISREL 9.10
Student Edition (Jöreskog & Sörbom, 2015).
Prior to constraining each focal parameter to zero the model-implied correlation matrix
was provided to LISREL along with relationship statements describing the model in terms of
relationships among variables and factors and setting path values and variance terms equal to the
125
values obtained from the literature and coefficient H calculations. The purpose of this step was to
check that the algebraic path tracing had been done correctly by showing that the data provided
in the model-implied correlation matrix exactly matched the model described by the relationship
statements. The result of this step was perfect data-model fit (χ2 = 0.00, p = 1.00), showing that
the specified model fit the provided data. Additionally, a visual inspection of the path diagram
output from LISREL indicated that the model had been specified correctly and matched the
model in Figure 14. The LISREL syntax used for this step and the corresponding text output are
provided in Appendix H.
With the accuracy of the model-implied correlation matrix generated by R and the
relationships statements entered as LISREL syntax confirmed, each one of the 14 focal
parameters was constrained to zero in turn. As a result, data-model fit results were obtained for
LISREL models where each model had a single focal parameter constrained to zero. Though 14
total models were run, two resulted in fatal errors in LISREL so only 12 sample size values were
obtained. The two missing sample size values were not expected to indicate the need for sample
sizes larger than the 12 obtained because the obtained sample sizes were for testing focal
parameters representing the largest and smallest hypothesized paths. As expected, constraining
each focal parameter to zero degraded the fit of model. This poor fit can be seen in the FML
values obtained for each model in which a focal parameter was constrained to zero (FML(θR)).
To calculate the sample size necessary to test each focal parameter, the value of the
noncentrality parameter (λ) must also be obtained. The noncentrality parameter follows a χ2
distribution and is determined based on the power level, alpha level, and degrees of freedom for
each test (Hancock & French, 2013). In this case of testing individual model parameters, the
126
degrees of freedom are equal to 1, the power level was set at 0.80 and alpha was set equal to
0.05. Using the table in Hancock & French (2013), λ was determined to be 7.85. The minimum
sample size necessary to test each focal parameter with power = 0.80 was then calculated using
Equation 5. The results of the twelve sample size determinations are provided in Table 4.
L = 1 +
M
NML(θR)
(5)
Table 4 Sample Size Necessary to Test Each Focal Parameter Arranged from Largest to Smallest
Focal parameter(s)
Standardized value used in correlation matrix
Minimum fit function value (FML(θR))
Sample size
bF3F2 0.30 0.103 78
bF6F1 0.30 0.147 55
bF6F3 0.35 0.194 42
bF2F1 0.52 0.231 35
bF3F1 0.49 0.261 32
bF6F5 0.414 0.292 28
bF4F2 0.30 0.320 26
bF7F6 0.18 0.349 24
bF7F1 0.30 0.712 13
bF7F3 0.35 0.870 11
bF7F5 0.414 1.171 8
cF4F7 0.38 1.233 8
bF4F3 and bF4F1 0.26 and 0.24 Fatal error obtained
Note. For each model, power = 0.80 and alpha = 0.05, and λ = 7.85
Examining the results of applying Equation 5 to the 12 FML(θR) values shows that as the
values of FML(θR) decreased the sample size necessary to detect each focal parameter increased.
This was the anticipated result because a smaller FML(θR) value for each parameter indicates that
127
constraining the parameter to zero, which is equivalent to removing it from the model, does not
influence data-model fit to a large degree. When the focal parameter has a smaller influence on
the model fit, more participants would be necessary to achieve a power of 0.80. The path from
social presence to cognitive presence (bF3F2) had the smallest FML(θR) value (0.103) and therefore
requires in the largest sample size (n = 78) to ensure it is tested with a power of 0.80 when alpha
= 0.05. The end result of this power analyses for model parameters is that a sample size of at
least 78 participants was targeted in order for all focal parameter tests of significance be
conducted with power = 0.80. A minimum sample size of 78 participants also exceeds the
sample size necessary to test overall data-model with power = 0.80 when the alternate RMSEA
value was set at 0.02.
Methodology
The main research study utilized a mixed methods approach in which both quantitative
and qualitative data were used answer the research questions (Creswell, 2014). A mixed methods
approach was chosen in order to minimize some of the limitations of a purely quantitative
methodology by integrating qualitative data to provide a more complete understanding of the
learning environment. Specifically, an embedded design was chosen for this research.
In an embedded design, one type of data collection is embedded within the larger
collection of a different data type (Creswell & Clark, 2007). For this research, the primary data
were quantitative data obtained from anonymized student responses to the CoI and satisfaction
instrument in addition to instructor responses to the ATI. Additional quantitative data were
student achievement measures: math ability scores on the math portion of the ACT (SAT scores
were not available), ACS exam scores, and final course grades. The qualitative portion of the
128
research consisted of short, semi-structured instructor interviews and analysis of the course
syllabus.
In an embedded design, the timing of the two types of data collection can be either
sequential or concurrent (Creswell & Clark, 2007). The collection of instructor data in this study
was concurrent. The instructor first completed the ATI and then participated in the semi-
structured interview. The timing of the instructor data collection and student data collection was
sequential. The student data analyzed in this research were from an existing data set collected by
another chemical education researcher for a separate project investigating predictors of student
success in general chemistry. The student data were collected prior to conducting instructor
interviews.
The student data were collected from six sections of a first semester general chemistry
course taught by four different instructors at a large, public, primarily undergraduate institution
(Indiana University Center for Postsecondary Research, n.d.). Though the student data can be
considered as grouped by classroom, the selection of classrooms was based on availability not
random sampling as would have been necessary for the use of a data analysis technique such as
hierarchical linear modeling (Huta, 2014). A total of 439 students are represented in the data set,
but complete data were not available for all students. The data set contained numeric responses
to the modified CoI and satisfaction survey instrument in addition to the number of items
answered correctly on the first-semester ACS general chemistry exam (Form GC15FG), final
course grades as a percentage of points earned after excluding laboratory scores, and ACT math
scores. Students completed the survey instrument during a regular class period using Scantron®
forms to record their responses. The survey was administered by the course instructor within two
129
weeks of the end of the semester during the normal end of course evaluation period for the
university.
To accommodate the Scantron® forms used at the participating institution the numeric
responses 1–5 (Strongly Disagree to Strongly Agree) were replaced with letters A–E and the
directions were modified slightly by the course instructor. The version of the instrument used at
the participating institution is provided in Appendix I. Student grades and ACT math scores were
not matched to survey responses by the course instructor until after final course grades had been
submitted. The collection and anonymization of this data was undertaken with institutional
review board (IRB) approval as an exempt research study by the university where the data were
collected. IRB permission from this researcher’s institution and IRB permission from the
institution where the data were collected were granted approving the use of the anonymous
student data set in the current research study.
Instructor data collection began by obtaining IRB permission from the home institution of
the researcher to conduct instructor interviews and collect ATI responses. In addition, consent
was obtained from the chair of the chemistry department where the student data were collected to
contact the instructors to invite them to participate in the current research. The four instructors
who taught the six sections of the course were invited participate in the research the semester
after teaching the class from which student data were collected. Each instructor completed the
revised ATI (items listed in Appendix D) as an online survey administered through Qualtrics
(http://www.qualtrics.com/) and responded to semi-structured interview questions during a video
conference. All four instructors consented to participate in the research.
Unlike the pilot study, the instructors were not asked to follow a think-aloud protocol to
130
respond to the ATI. After completion of the ATI, the instructors participated in the semi-
structured interview. With consent of the instructors, the interviews were audio recorded. The
interview questions were designed to provide the instructor with an opportunity to address any
issues that may have occurred when responding to the ATI and to allow the instructor to describe
his or her approach to teaching while providing details about the course. The questions used
during the semi-structured interviews are provided in Table 5. The researcher also requested a
copy of the syllabus of the introductory undergraduate chemistry course taught by the instructor
in which the students who provided data were enrolled.
Table 5 Semi-Structured Instructor Interview Questions
Question1. Are there any responses on the ATI you would like to provide an explanation for or
discuss in more detail? 2. In your own words, please describe your approach to teaching the course you used as a
reference for completing the ATI. For example, how do you decide what topics to cover or how to structure a class period?
3. In your own words, please describe how you prepare for a typical meeting of the course you used as a reference for completing the ATI.
4. Please describe the course you used as a reference for completing the ATI. For example, how long have you been teaching this course, what kind of topics are covered, and what type of students are enrolled in the course (chemistry majors, non-majors, etc.)?
Data Analysis
Qualitative Data Analysis
Analysis of the qualitative data collected for this research began with transcription of the
recorded instructor interviews. A second chemical education graduate student researcher listened
to a subset of the recorded interviews and spot checked the transcription both for accuracy and to
ensure that the semi-structured interview questions did not lead the instructor (Creswell, 2013, p.
131
259). Analysis and coding of the transcript and course syllabus used a major theme of student-
centered teaching practices consistent with constructivism.
Coding of the interview transcripts and course syllabi was done by hand using a rubric
developed specifically for this research, provided in Appendix J. The codes used in the rubric
were based on the focus of the interview and indicators of a constructivist learning environment.
These codes included (1) incorporation of student learning activities to replace a portion of
lecture time, (2) the role of the instructor as a facilitator, (3) instructor conceptions of student
learning, (4) use of group work, (5) emphasis on student understanding of concepts, (6)
incorporation of authentic problem solving tasks, (7) students being asked to test or apply
knowledge, and (8) use of discussions to probe student understanding.
To ensure validity of the coding process a second chemical education researcher coded
all the interview transcripts and syllabi using the same rubric. This chemical education researcher
was a faculty member familiar with constructivism but not directly involved in the current
research. Before beginning the coding process, the operationalized definition of a constructivist
approach to teaching utilized in this research was discussed with the researcher. This definition
of a constructivist approach to teaching describes an approach that is more student-centered and
shifts the role of the instructor from a lecturer to a facilitator for at least part of the instructional
time. The coding scheme was discussed to ensure both coders interpreted the codes in the same
way. After both coders had applied the rubric to each set of instructor data, scores were
compared to ensure consistency in interpretation of the indicators. If necessary, interpretations
were clarified and each coder had an opportunity to revise the score that had been assigned. After
coding all four sets of instructor data, the percent agreement of the two coders was 94%.
132
As part of the coding process both coders identified quotes from the interview transcripts
or course syllabi that provided evidence for the specific codes representing student-centered
constructivist approaches to teaching. Similarities in selection of quotes by both coders were
used to provide additional evidence for the validity of the coding. This qualitative data was
analyzed along with the results of instructor responses to the ATI and student responses to the
CoI survey in order to address the first research question.
Quantitative Data Analysis
Data cleaning Analysis of the quantitative data collected for this research began by checking the
instructor survey, student survey, and student achievement data to look for obvious errors in data
entry or otherwise corrupted data. This checking process included determining the minimum and
maximum value for each of the measured variables to ensure it was within an acceptable range.
For example, both instructor and student survey item responses were expected to be whole
numbers between 1 and 5. All four instructor survey responses were within the correct range. In
the student data set survey response values of 9 were present, these indicated missing values as
reported by the scanning software used at the institution where the data were collected.
Additionally, a few students had survey response values of 6 (F) which were from students who
were not paying attention to the fact that the scale ended at 5 (E). Both values of 9 and 6 were
recoded as missing data.
In the student data set, it was anticipated that the final course grade data would be a
percentage score between 0 and 100 but one student earned a final course grade of 100.02% and
this value was left in the data set. The range for the math ability score was expected to be
133
between 1 and 36 since ACT math scores (ACT, 2015) were present in the student data set. All
ACT math values in the data set fell between 16 and 35. The raw ACS exam scores could range
from 0 to 70 and the student data set contained values from 14 to 63.
The student survey responses were checked to see if any student did not select “disagree”
for item 29 as instructed (Please select “Disagree” for this item). Any student not selecting
“disagree” for this item had his or her data excluded from further analysis; a total of 17 students
were removed from the data set based on this criterion. Additional data cleaning involved
removal of 15 students from the data set for whom no course grade was available since these
cases represented students who withdrew from the course before the end of the semester. Finally,
an additional 16 students were removed from the data set because they were missing both ACS
exam scores and CoI responses. Missing the ACS exam score meant these students did not take
the final exam for the course and missing CoI responses meant that these students were also not
present on the day that the CoI was administered. These were likely students who did not
officially withdraw from the course but were no longer actively attending class meetings. This
interpretation is supported by the low course grades for these 16 students, ranging from 0.23% to
50.93%.
As a result of these data cleaning steps, the number of usable student participants was
391. The 391 responses represent approximately 89% of the total 439 responses collected.
Exclusion of 11% of the initial sample still provided five times the minimum sample size
necessary to examine the overall data-model fit and model focal parameters with sufficient
power. No instructor survey responses were excluded from the analysis since all four instructors
responded to all 22 items on the modified ATI survey instrument.
134
Of these 391 cases, only 307 students had complete data on all variables. Course grade
was the only variable for which all students had complete data. The missing data rate for ACS
exam scores was 1.5% and 6.6% for ACT math scores. Between 12% and 14% of students were
missing data on one or more of the survey items. More detailed information about the missing
data rate for each survey variable is provided in Chapter 4.
The next step of the quantitative data analysis was to look at outliers in the student data
set. Outliers were not examined in the instructor data set since it only contained four responses.
Univariate outliers were examined for the three continuous student variables: ACS exam scores,
ACT math scores, and course grades. Univariate outliers were considered to be any student with
scores more than three standard deviations above or below the average on any of the three
continuous variables. No outliers were identified for the ACS exam scores or ACT math scores.
Eight students were identified as having course grades three standard deviations below the
average due to earning less than 40% of the course points with laboratory points excluded.
However, since these 8 students only represented about 2% of the student data and a robust
estimation technique not requiring the assumption of normality was used in the SEM analyses
these negative outliers were left in the student data set.
Multivariate outliers were examined by calculating the Mahalanobis distance based on
the 40 student survey variables and three academic variables. Using the cutoff value of 77.42 for
a chi-square test with p = 0.001 and df = 43 there were 18 multivariate outliers identified. One
issue with identifying multivariate outliers using the Mahalanobis distance is that only
individuals with complete data on all 43 variables could be considered in the analysis whereas
the SEM analyses were conducted with all cases using a technique robust to missing data. For
135
this reason, and because the 18 cases represented less than 5% of the data set, the multivariate
outliers were not removed from the student data set.
Assumptions for CFA and SEM analysis
The typical estimation technique employed when performing CFA and SEM analysis is
full information maximum likelihood (FIML), often abbreviated as simply ML. This estimation
technique is robust to missing data and does not require deletion of incomplete cases or
imputation of missing values, unlike older statistical techniques used to deal with missing data
(Enders, 2013). Instead, iterative algorithms are used to find the most likely values for the
missing data given the assumption of multivariate normality and the relationships among other
values in the data.
The use of ML assumes that the missing data are random. This could mean either missing
completely at random (MCAR) or missing at random (MAR). For MCAR, missing values for a
variable, such as a response to a survey item, must not be missing because of what the value
would have been, such as a low rating, or because of a relationship to other variables in the
model, such as a low course grade. For missing at random (MAR), values must not be missing
because of what the value would have been, but may be missing because they depend on the
value of another variable. It is difficult to demonstrate MCAR, but MAR may be demonstrated if
examination of the data reveals that the missing survey data may be more likely for students with
high or low scores on academic achievement variables. If another variable looks like it may be
responsible for the missing data, this variable must be included in the model as an auxiliary
variable. As an auxiliary variable it must be allowed to correlate with the error variance terms for
the other measured variables in the model (Enders, 2013).
136
To check if the missing survey responses met the assumption of MAR, three separate
independent t-tests were conducted with one of the three academic variables (ACT math, ACS
exam, and course grade) as the dependent variable. The two groups being compared were
students with complete responses to all items on the survey instrument and students missing one
or more responses to survey items. The results of these t-tests, presented in Chapter 4, indicated
that students with incomplete responses to the survey items had statistically significantly (p <
0.05) lower scores on the academic variables than students who responded to all the survey
items. As a result, ACT math scores, ACS exam scores, and final course grades were included as
auxiliary variables in all analyses involving only the CoI instrument. Since the three academic
variables were included in the full research model used for the SEM analysis, they did not need
to be modeled as auxiliary variables in that analysis.
The responses to the survey were given on a five-point scale, and therefore they are
considered ordinal data and are inherently not normally distributed. However, research has
shown that with at least five scale points, ordinal data following an approximately normal
distribution can be treated as normally distributed continuous data for ML estimation and no
major distortion is seen in model-fit indices (Finney & DiStefano, 2013). However, slight
underestimation may occur for loadings and standard errors leading to an increase in Type I error
(Finney & DiStefano, 2013). If the categorical data are not approximately normal, this may lead
to an increase in the !" value for the model and a decrease in the CFI, increasing the likelihood
of finding the model to have unacceptable fit (Finney & DiStefano, 2013).
Skew of the survey items was checked by visually examining histograms created for each
of the survey variables and by calculating a z-score for skewness equal to the absolute value of
137
the skewness value divided by the standard error of the skewness value. All survey items except
one were found to have nonnormal distributions and skewness z-scores greater than 1.98
indicating a skewness value significantly different from 0 (p < .05) due to a majority of students
selecting responses of Agree and Strongly Agree to the CoI items and positive responses to the
satisfaction items. Descriptive statistics for the student variables are provided in Chapter 4.
As a response to the categorical nature of the data and the nonnormal distribution of the
survey responses a correction factor was applied to all CFA and SEM analysis. The Satorra–
Bentler correction is a commonly used robust estimation technique that corrects the model !"
value and standard errors but does not adjust parameter estimates since these are generally robust
to nonnormality (Finney & DiStefano, 2013). However, use of the Satorra–Bentler correction
requires either a complete data set or the use of listwise deletion to create a complete data set.
The limitations of LISREL made it impossible to deal with both the missing data and the
nonnormal distribution of the data simultaneously. Instead of reducing the sample size by using
the Satorra-Bentler correction and listwise deletion, the MLR estimator was used in Mplus
(version 7.0). The MLR estimator also corrects the model !" value and standard errors while not
adjusting parameter estimates, but MLR can be used with missing data (Muthén & Muthén,
1998-2015). Therefore, MLR was chosen to provide scaled estimates of model fit and parameter
significance for all CFA and SEM analysis.
Internal structure of the CoI instrument
The second research question was addressed by examining the internal structure of the
CoI instrument with a series of confirmatory factor analyses. First, a CFA was conducted with
the 13 teaching presence items to determine whether a single factor or two-factor model was a
138
better fit for the latent variable of teaching presence (Figures 8 and 9, p. 60). As discussed in
Chapter 2, some research with the CoI had indicated that the teaching presence factor may be
better described as two correlated factors instead of a single factor. In the two factor model one
factor, which items 1–4 were expected to load on, relates to pre-course activities performed by
the instructor that are typically accomplished in the syllabus or communicated via
announcements (Arbaugh, 2007; Shea et al., 2006). The second factor, which items 5–13 were
expected to load on, relates to in-course instructor activities such as facilitating discussions and
providing feedback.
These two models can be described as nested because the parameters of one model are a
subset of the parameters in the other model. In the case of the two competing models for teaching
presence, the single factor model can be considered as nested within the two-factor model
because the single factor model is a special case of the two-factor model where the correlation
between the two factors is constrained to 1. Due to this relationship, the two models can be
statistically compared by looking at the difference between their !" values and their respective
degrees of freedom (Mueller & Hancock, 2008). However, due to the use the MLR estimator
which produced a scaled !" value, the difference in !" values must be examined in the context of
the scaling correction factor. Each of the two models has its own scaling correction factor that is
simply the ratio of the uncorrected !" value to the corrected !" value. The difference test scaling
correction (cdiff) can be calculated from the scaling correction for each model and the degrees of
freedom for each model, shown in Equation 6 where the subscript 1 indicates the nested one-
factor model and the subscript 2 indicates the full two-factor model.
Odiff =
OD 45D − O" 45"
(45D − 45")
(6)
139
A !" difference test can then be conducted by subtracting the uncorrected !" value of the full
model from the uncorrected !" value of the reduced model and dividing by the difference scaling
correction. This can be seen mathematically in Equation 7. The degrees of freedom for the
∆!
dif
"=
!D
"− !
"
"
Odiff
(7)
resulting !" value are the difference in the number of degrees of freedom of the two models. The
result of conducting the !" difference test indicated that the two factor model of teaching
presence was a better fit for the data (!+,-D
"= 15.695, p < 0.001), and that an error covariance
term should be added between item 12 and item 13. The results of this analysis and specific
details of the !" difference calculation are presented in Chapter 4 and the syntax used to run the
models is provided in Appendix K. Since the in-course factor is most directly related to the
intended construct of classroom teaching practices additional analysis with the CoI instrument
only used items 5-13 as indicators of the teaching presence factor. The fit for this smaller nine-
item teaching presence model with three academic auxiliary variables and the error covariance
between item 12 and item 13 was acceptable (!scaled,+,-"1
"= 79.902; CFIscaled = 0.960;
RMSEAscaled = 0.073, CI90=[0.055, 0.091]; SRMR = 0.046) using the joint criteria of CFI ≥ 0.96
and SRMR ≤ 0.09 when the RMSEA is not considered (Hu & Bentler, 1999).
Next, a CFA was conducted to test the data-model fit for the full three factor CoI model
by including the items loading on the social presence and cognitive presence factors. The error
covariance between item 12 and 13 was included along with the three academic auxiliary
variables. After the addition of the error covariance terms between three pairs of satisfaction
items, shown in Figure 15, acceptable data model fit was obtained (!scaled,+,-./0
"= 1028.717;
140
Figure 15. The three-factor model of the CoI survey used to inform the two-phase SEM analysis.
141
CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR = 0.061) based on joint
criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 when the CFI is not considered (Hu & Bentler,
1999). This model of the CoI instrument provided an answer to the second research question,
discussed in more detail in Chapter 5, and was the model of relationships among CoI variables
used in the analysis of the full research model.
Average scale scores and scale reliability
After determining the item loadings for the three CoI factors, the next stage of the
analysis was to determine average scale scores for both CoI and ATI instruments. Average scale
scores for each of the three types of presence were determined by first computing an average
teaching, social, and cognitive scale score for each individual. The average scale score for each
individual was computed by adding the numeric responses for each item on a particular scale and
dividing by the total number of items on that scale. Then, the overall average scale score was
computed by adding all the individual scale scores and dividing by the total number of
individuals. For this analysis, only individuals with complete data for all CoI items were
included in the calculations. A similar analysis was undertaken for the satisfaction items, after
reversing the coding of the last item so that all four items had the positive response associated
with the number 1 on the scale. On the ATI only items representing the CCSF scale were
analyzed since the pilot study had indicated that only the CCSF scale score would be relevant to
the current research on constructivist learning environments.
Scale reliability was calculated in two ways for each of the three types of presence on the
CoI. Cronbach’s alpha was calculated using the raw student response data and provides
information about the reliability of each of the presence scales when item response values are
142
combined as either averages or sums to create scale scores and can be interpreted as the internal
consistency of responses to the survey items. Coefficient H provides information about the
reliability of the underlying latent factor representing each type of presence and can be
interpreted as the stability of the factor. Coefficient H was calculated, using the formula
previously presented in Equation 3, from the standardized loadings produced as a result of
conducting the three-factor CFA depicted in Figure 15. These results are presented in Chapter 4.
Scale reliability was not determined for the ATI since there were only four instructor responses.
Two-phase SEM Analysis To address the third research question, a two-phase SEM analysis (Mueller & Hancock,
2008) was implemented for analysis of the full model in Figure 13 (p. 114), after modifying it
based on the results of the CoI analysis shown in Figure 15. In the first phase, the measurement
portion of the model was tested. The measurement portion of the model focuses on the
relationships between the measured variables and their respective factors. In the measurement
phase of the analysis, the causal paths among factors are replaced with correlational
relationships. Only if the measurement portion of the model has acceptable data-model fit can
the structural portion of the model be tested in the second phase by replacing the correlational
relationships with the originally hypothesized relationships.
For the measurement phase, the overall research model was conceptualized as having
seven latent variables, similar to the model used in the power analysis for testing model focal
parameters. The first four latent variables are the three presence factors from the CoI plus the
student satisfaction factor. The indicator variables from the student survey are shown loading on
their respective factors in Figure 16. Note that the measurement model shown in Figure 16
143
Figure 16. The CFA model used in the measurement phase of the SEM analysis.
144
reflects modifications to the number of indicator variables for teaching presence and the addition
of error covariance terms as a result of the earlier CFA steps. The final three latent variables are
“dummy” latent variables for math ability, ACS exam score, and final course grade. These
“dummy” latent variables are mathematically equivalent to their respective indicator variables
because the loading from the latent variable to the indicator variable is constrained to one and the
error variance of the indicator variable is constrained to zero.
Unlike the model used for power analysis, the measurement model in Figure 16 allows all
seven factors to correlate with each other. This correlational model is a seven-factor CFA model.
This CFA model includes relationships between variables, like social presence and ACS exam
scores, that were not hypothesized to have a direct connection in the full structural model.
Allowing all seven factors to correlate with each other means that any poor data-model fit must
have been due to problems with how individual items are loading on the factors. The fit of this
model was acceptable (!scaled,+,-12X
"= 1374.826; CFIscaled = 0.899 ; RMSEAscaled = 0.051,
CI90=[0.047, 0.055] ; SRMR = 0.058) based on joint criteria of RMSEA ≤ 0.06, and SRMR ≤
0.09 (Hu & Bentler, 1999), and as a result, analysis proceeded to the second phase.
The second phase of the SEM analysis reintroduced the hypothesized paths among
variables and was, therefore, a test of the model in Figure 13 (p. 114), with the modifications that
were introduced during the earlier CoI survey analyses. This modified model is shown in Figure
17. As expected, the data-model fit degraded slightly after the introduction of the causal paths
between variables, but the overall fit of the model was acceptable (!scaled,+,-120
"= 1429.111;
CFIscaled = 0.892; RMSEAscaled = 0.053, CI90=[0.049, 0.056] ; SRMR = 0.065) so no additional
analysis took place. Detailed results of this analysis are presented in Chapter 4.
145
Figure 17. The structural model used in the second phase of the SEM analysis.
146
Chapter 4
The results of the previously described analyses are presented in this chapter, beginning
with a summary of the quantitative instructor data from responses to the Approaches to Teaching
Inventory (ATI) and qualitative instructor data from coding the instructor interviews and course
syllabi. A majority of the results presented in this chapter come from quantitative analysis of the
anonymized student data set. The student data analysis results begin with presentation of
descriptive statistics for the student academic variables and survey responses followed by the
results of three independent t-tests to support the inclusion of auxiliary variables in the
confirmatory factory analysis (CFA) of the Community of Inquiry (CoI) survey instrument.
Next, the results of using the student survey data to test models of the CoI instrument are
presented including rationale for modifications made to the initially hypothesized models. Then,
the results of using the entire student data set in the two-phase structural equation modeling
(SEM) analysis are provided.
The last part of this chapter interprets the results of the instructor and student data
analysis in the context of addressing the three research questions. The first research question is
addressed by examining the alignment of instructor and student descriptions of the learning
environment. The results of the CFA of the CoI instrument are used to address the second
research question by providing evidence for the validity and reliability of the CoI instrument
scores. Lastly, the results of the SEM analysis are used to address the third research question
regarding how a constructivist learning environment affects student outcomes of academic
achievement and satisfaction.
147
Instructor Survey and Interview Results
Following the directions provided by Trigwell for use of the ATI-R (personal
communication, August 13, 2015), scores were calculated on the conceptual change student-
focused (CCSF) scale for each instructor’s approach to teaching the general chemistry course
discussed during the interview. In the main study, only the CCSF scale score information was
calculated because the pilot study results indicated that the information transmission teacher-
focused (ITTF) scale did not provide information relevant to instructors’ development of a
student-centered constructivist learning environment. The average scale score calculations were
done by summing the responses to the 11 items on the CCSF scale (3, 5, 7, 8, 13, 14, 15, 17, 18,
20, 21) and dividing by the total number of items on the scale. This produces an average CCSF
score for each instructor ranging from 1 (Only Rarely) to 5 (Almost Always) with a midpoint of
3 (About Half the Time). The individual instructor CCSF scores range from 3.5 to 4.6 with a
mean score of 4.1 indicating that, on average, instructors could be considered as emphasizing a
conceptual change, student-focused approach in their classroom more than half the time.
Individual instructor CCSF scores are not provided since the focus of this research is not to
highlight instructor differences but instead to look at the correspondence between instructor and
student perceptions of the learning environment.
The instructor interview transcripts and course syllabi were coded using the rubric in
Appendix J following the procedure described in Chapter 3 with both the researcher and a
second chemical education researcher coding the instructor data. The coding process results in
two sets of scores for each instructor on eight indicators of a student-centered constructivist
learning environment. These indicators are (1) incorporation of student learning activities to
148
replace a portion of lecture time, (2) the role of the instructor as a facilitator, (3) instructor
conceptions of student learning, (4) use of group work, (5) emphasis on student understanding of
concepts, (6) incorporation of authentic problem solving tasks, (7) students being asked to test or
apply knowledge, and (8) use of discussions to probe student understanding. Each indicator was
rated on a scale ranging from 0 (no evidence) to 3 (multiple pieces of evidence). The percent
agreement between the two coders is 94% and disagreements between the coders are no larger
than one scale point out of the four scale points indicating a high degree of consistency of the
assigned codes. During the coding process both coders identified quotes from the interview
transcripts and course syllabi that provide evidence of the indicators of a student-centered
constructivist learning environment. Table 6 provides examples of this evidence for the two
indicators that most directly relate to the development of a student-centered constructivist
learning environment, namely, how the instructor uses class time and opportunities the instructor
provides for learning to take place. The evidence provided in Table 6 was was generated by
instructors self-reporting classroom practices and approaches to teaching in both the semi-
structured interview and in their own individually written course syllabus.
149
Table 6 Evidence of Teaching Approaches Aligned with Constructivism from Interviews and Syllabi Instructor Use of class time Opportunities for learning
A
“During most lectures, you will have the opportunity to work with your classmates in small groups.”
“A classroom environment in which students actively engage with the content promotes in-depth learning.”
B
“We would spend some time with me [instructor] presenting some information and then I’ll ask them [students] to do something with that information...where I ask them to work together in groups.”
“The teacher’s fundamental task is to get students to engage in learning activities.”
C
“I try to arrange the class so that there are at least two problem-solving periods in a class. I prefer not to talk for more than 30 minutes in one [75 minute] class period.” “These activities will be done in groups in most cases.”
No specific description of learning provided.
D
“Every day my students are all placed in groups so they work with those groups the entire semester and they sit with them in their class and there’s always some opportunity for them to work on something, discuss things, that’s a key piece.”
“My focus for teaching is structuring learning opportunities so that the students can construct their own understanding of concepts.”
All four instructors describe an approach to teaching that targets spending no more than
half of the class time presenting information to students in a lecture format. In support of this
approach, all four instructors utilize group work to some degree in their course, either by creating
formal structured groups or by encouraging students to create informal groups based on other
students sitting nearby that day. Additionally, three out of the four instructors describe learning
as requiring active engagement with the material or mention the construction of individual
knowledge either on their course syllabus or during the interview. The analysis of instructor data
provides evidence that the four instructors have an approach to teaching which shifts their role
150
from a lecturer to a facilitator for at least part of instructional time. The results from coding the
instructor interviews and course syllabi are in alignment with the average CCSF scale scores
which indicate that the instructors emphasize a conceptual change, student-focused approach in
their classroom more than half the time. This aligns the teaching approaches of all four
instructors with the definition of a constructivist approach to teaching used in this research which
is an approach where teaching practices have been adopted that are more student-centered and
have shifted the role of the instructor from a lecturer to a facilitator for at least part of the
instructional time.
Student Data Analysis Results
Descriptive Statistics and Assumptions for SEM Analysis
After undertaking the data cleaning steps described in Chapter 3, descriptive statistics
were calculated for all student variables in the data set to check if the data meet the assumptions
for conducting the CFA and SEM analyses. For the three continuous student academic variables
of final course grades, ACS exam scores, and ACT math scores, the descriptive statistics include
the response rate, mean, standard error of the mean, standard deviation, minimum, maximum,
skewness, and standard error of the skewness. A summary of these values is presented in Table
7. A similar table was constructed for the categorical student survey variables and is presented in
Table 8. Table 8 also includes the frequencies for each categorical response selected for each
survey item. A correlation matrix for the student variables is provided in Appendix L.
151
Table 7 Descriptive Statistics for Student Academic Variables
Variable n % missing Mean (SE) SD Min Max Skew (SE) Final course grades 391 0.0 77.51 (0.65) 12.91 25.75 100.02 –1.11 (0.12) ACS exam scores 385 1.5 39.35 (0.51) 9.94 14 63 –0.00 (0.12) Math ability (ACT) 365 6.6 25.58 (0.20) 3.80 16 35 –0.01 (0.13)
Note. SE = standard error Table 8 Descriptive Statistics for Student Survey Variables
Item n %
missing Mean (SE) SD Skew (SE) Response Frequencies 1 2 3 4 5
T1 (Q1) 345 11.76 4.39 (0.04) 0.82 –1.76 (0.13) 5 10 15 132 183 T2 (Q2) 345 11.76 4.30 (0.04) 0.81 –1.35 (0.13) 4 7 32 140 162 T3 (Q3) 345 11.76 4.40 (0.04) 0.75 –1.47 (0.13) 3 4 25 132 181 T4 (Q4) 345 11.76 4.61 (0.04) 0.67 –1.95 (0.13) 1 5 15 87 237 T5 (Q5) 345 11.76 4.19 (0.05) 0.88 –1.16 (0.13) 5 11 44 140 145 T6 (Q6) 344 12.02 4.17 (0.05) 0.95 –1.09 (0.13) 5 17 50 114 158 T7 (Q7) 345 11.76 4.12 (0.05) 0.88 –1.00 (0.13) 5 10 55 142 133 T8 (Q8) 345 11.76 4.14 (0.05) 0.87 –1.17 (0.13) 6 10 44 153 132 T9 (Q9) 345 11.76 3.96 (0.05) 0.87 –0.57 (0.13) 3 13 82 145 102 T10 (Q10) 345 11.76 3.85 (0.05) 0.92 –0.52 (0.13) 4 20 90 141 90 T11 (Q11) 345 11.76 4.08 (0.05) 0.91 –0.98 (0.13) 5 16 53 144 127 T12 (Q12) 345 11.76 3.78 (0.06) 1.03 –0.51 (0.13) 7 32 91 116 99 T13 (Q13) 345 11.76 4.10 (0.05) 0.97 –1.07 (0.13) 7 18 51 126 143 S1 (Q14) 345 11.76 3.71 (0.06) 1.03 –0.40 (0.13) 7 34 101 112 91 S2 (Q15) 345 11.76 3.81 (0.05) 0.95 –0.47 (0.13) 5 22 98 130 90 S3 (Q16) 345 11.76 4.39 (0.04) 0.76 –1.39 (0.13) 3 3 30 128 181 S4 (Q17) 345 11.76 4.09 (0.05) 0.92 –1.04 (0.13) 5 18 46 147 129 S5 (Q18) 345 11.76 3.79 (0.06) 1.03 –0.64 (0.13) 8 34 75 132 96 S6 (Q19) 344 12.02 4.16 (0.04) 0.82 –0.88 (0.13) 1 14 43 157 129 S7 (Q20) 344 12.02 3.85 (0.05) 0.88 –0.73 (0.13) 5 20 73 171 75 S8 (Q21) 343 12.28 3.92 (0.05) 0.87 –0.57 (0.13) 4 11 86 149 93 S9 (Q22) 343 12.28 3.84 (0.05) 0.88 –0.56 (0.13) 4 18 86 156 79 Note. SE = standard error; for T, S and C items 1 = Strongly Disagree and 5 = Strongly Agree
152
Table 8, continued Descriptive Statistics for Student Survey Variables
% missing
Response Frequencies Item n Mean (SE) SD Skew (SE) 1 2 3 4 5
C1 (Q23) 344 12.02 3.58 (0.05) 0.99 –0.41 (0.13) 9 38 103 132 62 C2 (Q24) 344 12.02 3.53 (0.05) 0.94 –0.38 (0.13) 8 36 113 138 49 C3 (Q25) 344 12.02 3.33 (0.06) 1.06 –0.16 (0.13) 15 58 120 100 51 C4 (Q26) 344 12.02 3.71 (0.06) 1.04 –0.63 (0.13) 9 44 65 147 79 C5 (Q27) 344 12.02 3.74 (0.05) 0.96 –0.59 (0.13) 7 30 84 149 74 C6 (Q28) 343 12.28 3.91 (0.05) 0.85 –0.83 (0.13) 5 15 65 180 78 C7 (Q30) 344 12.02 3.82 (0.05) 0.90 –0.64 (0.13) 7 14 92 151 80 C8 (Q31) 344 12.02 3.99 (0.04) 0.77 –0.59 (0.13) 2 8 68 181 85 C9 (Q32) 344 12.02 4.02 (0.04) 0.83 –0.97 (0.13) 6 7 59 175 97 C10 (Q33) 344 12.02 3.99 (0.04) 0.83 –0.88 (0.13) 5 10 61 176 92 C11 (Q34) 344 12.02 3.84 (0.05) 0.92 –0.66 (0.13) 7 15 88 149 85 C12 (Q35) 344 12.02 3.80 (0.05) 0.90 –0.70 (0.13) 7 19 83 163 72 C13 (Q36) 344 12.02 3.83 (0.05) 0.87 –0.76 (0.13) 5 21 70 179 69 C14 (Q37) 344 12.02 3.68 (0.06) 1.05 –0.58 (0.13) 12 36 84 131 81 SS1 (Q38) 338 13.55 1.85 (0.06) 1.02 –1.09 (0.13) 164 94 54 19 7 SS2 (Q39) 336 14.07 2.38 (0.07) 1.19 –0.59 (0.13) 91 111 70 42 22 SS3 (Q40) 337 13.81 2.13 (0.06) 1.09 –0.73 (0.13) 118 108 69 32 10 SS4 (Q41) 335 14.32 4.16 (0.06) 1.07 –1.31 (0.13) 11 23 33 101 167
Note. SE = standard error; for T, S and C items 1 = Strongly Disagree and 5 = Strongly Agree; SS items were on a semantic differential scale where 1 was positive word for SS1–SS3 (comfortable, satisfying, pleasant) and 1 was negative for SS4 (chaotic)
The skew value of –1.11 in Table 7 reflects the negative skew of the course grades as a
result of the decision to leave negative outliers with low course grades in the data set since the
MLR estimator used in Mplus was robust to violations of normality. The robustness of MLR to
nonnormality is also why none of the survey items are transformed even though they exhibited
skew. Table 7 also shows a relatively small amount of missing data for the three student
academic variables. There are no missing course grade data, a 1.5% missing data rate for ACS
exams scores and a 6.6% missing data rate for ACT math scores. The missing data rate is higher
for the student survey variables in Table 8, ranging from 12-14% missing. The missing data rate
153
for the student survey variables is addressed by testing for a relationship between missing student
survey responses and student academic variables.
Three separate independent t-tests were conducted with one of the three academic
variables (final course grade, ACS exam scores, and ACT math scores) as the dependent variable
to evaluate the assumption that the missing student survey data were either missing completely at
random or missing at random. The two groups being compared are students with complete
responses to all items on the survey instrument and students missing one or more responses to
survey items. The results of these t-tests are presented in Table 9.
Table 9 Independent t-tests for Differences in Academic Variables Based on Missing Survey Responses
Outcome variable Group n Mean (SE) SD
Mean difference (SE) t df p
Final course grades Complete survey 330 79.80 (0.59) 10.66 14.66 (2.21) 6.63 69.36a < .001
Incomplete survey 61 65.14 (2.13) 16.64 ACS exam scores
Complete survey 327 40.36 (0.53) 9.51 6.76 (1.38) 4.92 383 < .001 Incomplete survey 58 33.60 (1.37) 10.44
Math ability (ACT) Complete survey 310 25.76 (0.21) 3.71 1.26 (0.55) 2.27 363 .02
Incomplete survey 55 24.51 (0.56) 4.16 aEqual variances could not be assumed across groups
For each test the group of students with incomplete survey responses has statistically
significantly (p < .05) lower scores on the academic variable than students who responded to all
the survey items. These results demonstrate that the academic achievement variables could be a
possible mechanism responsible for explaining the missing student survey response data. That is,
students with lower scores on academic achievement variables are more likely to be missing
responses to the survey items. Since the academic achievement variables have a relationship with
154
the missing data, they are included in the model of the CoI instrument in order to meet the
assumptions for missing data in the maximum likelihood estimation technique used in analysis of
the CoI instrument. As a result, final course grades, ACS exam scores, and ACT math scores are
included as auxiliary variables in all analyses involving only the CoI instrument. Since the three
academic variables are included in the full research model used for the SEM analysis, their
relationship with the three CoI presence factors is already part of the model and therefore they do
not need to be modeled as auxiliary variables in that analysis.
Confirmatory Factor Analysis of CoI Data
The first aspect of the internal structure of the CoI instrument tested are the two
competing models of teaching presence. For all CFA and SEM analyses scaled fit indices were
computed as a result of using the MLR estimator. The single factor model has all 13 teaching
presence items loading on a single teaching presence factor. This model does not show good
data-model fit (!scaled,+,-1/
"= 219.044; CFIscaled = 0.916; RMSEAscaled = 0.083, CI90=[0.071,
0.095]; SRMR = 0.048) with only the SRMR value within an acceptable range based on target
values for each fit index of CFI ≥ 0.95, RMSEA ≤ 0.06, and SRMR ≤ 0.08 (Hu & Bentler,
1999). Examination of the modification indices suggests that adding an error covariance term
between items 12 and 13 would improve the data-model fit. This modification is justifiable since
both item 12 and item 13 start with the same stem related to the instructor providing feedback,
i.e., “The instructor provided feedback that helped me understand my strengths and weaknesses
relative to the course’s goals and objectives” and “The instructor provided feedback in a timely
fashion.” None of the other suggested modifications were found to have any theoretical
155
justification. As a result of adding the error covariance term between items 12 and 13 the fit of
the single factor teaching presence model improves slightly (!scaled,+,-1.
"= 204.752; CFIscaled =
0.928; RMSEAscaled = 0.075, CI90=[0.064, 0.087]; SRMR = 0.050). However, this model still
only meets the SRMR target criterion indicating that the single factor model of teaching presence
may not be a good fit for the data. The low CFI and high RMSEA combined with the acceptable
SRMR indicates that there may not be much variance or covariance in the teaching presence
items for the model to explain.
As described in Chapter 2 and 3, the literature indicates that teaching presence may be
better modeled as having two correlated factors. In this two-factor model, one factor represents
pre-course activities completed by the instructor, such as reminding students of due dates, and
the second factor represents in-course activities conducted by the instructor, such as facilitating
discussions. In this model, survey items 1 through 4 are associated with the factor representing
pre-course activities and items 5 through 13 are associated with the factor representing in-course
activities. The error covariance term between items 12 and 13 is also included based on the
results of testing the single factor model reported previously.
The two-factor model of teaching presence shows improved fit relative to the one-factor
model (!scaled,+,-13
"= 186.799; CFIscaled = 0.937; RMSEAscaled = 0.071, CI90=[0.059, 0.083];
SRMR = 0.048), but the values of the fit indices still indicate that there may not be much
variance or covariance in the teaching presence items for the model to explain. The competing
one-factor and two-factor models for teaching presence are then compared with the nested !"
comparison described in Chapter 3. This comparison uses the scaling correction factor (c) for
each model, computed as the ratio of the scaled and unscaled !" values, and the difference test
156
scaling correction (cdiff) calculated using the degrees of freedom. The resulting scaled !"
difference is 15.695, which is a statistically significant !" value with one degree of freedom at p
< .001. The calculations and result are summarized in Table 10.
Table 10 Nested Model Comparison for Teaching Presence Factor
Model Y2 df c cdiff Y
2diff p(df = 1)
Two-factor model – unscaled 244.258 63 1.308 1.544 15.695 < .001 Two-factor model – scaled 186.799
One-factor model – unscaled 268.491 64 1.311
One-factor model – scaled 204.752
It is determined from this comparison of the two models that the two-factor model is a
better model of teaching presence, indicating that teaching presence is best described as having
separate pre-course and in-course aspects. Before moving on to testing a model with all three CoI
presence factors, a final teaching presence model is tested which includes just the in-course
survey items (5–13) loading on a single teaching presence factor and maintaining the error
covariance between items 12 and 13. This model cannot be statistically compared to the other
teaching presence models since it contains a different number of items. This 9-item single factor
teaching presence model shows acceptable data-model fit (!scaled,+,-"1
"= 79.902; CFIscaled =
0.960; RMSEAscaled = 0.073, CI90=[0.055, 0.091]; SRMR = 0.046), particularly with regards to
the CFI and SRMR using the joint criteria of CFI ≥ 0.96 and SRMR ≤ 0.09 (Hu & Bentler,
1999). The values of the CFI and SRMR indicate that when considering only items 5–13, there
may be stronger relations in the data and the model did a good job explaining the variances and
covariances. As a result, this is determined to be an acceptable model of teaching presence for
use in additional data analysis steps. This model of teaching presence using only items
157
describing in-course instructor activities and behaviors also better aligns with the research goals
of examining aspects of the classroom learning environment created by the instructor.
Next, a model of the full CoI instrument is tested which includes all three presence
factors. All three presence factors are allowed to correlate because of their overlapping nature in
the original CoI model. The initial three factor model for the CoI instrument includes the error
covariance term between item 12 and 13. Based on modification indices provided by Mplus,
three additional error covariance terms are added between pairs of social presence items that
asked students to rate conceptually similar aspects of social presence. These items are provided
in Table 11.
Table 11 Social Presence Items with Added Error Covariance Terms
Item Wording S1 (Q14) Getting to know other course participants gave me a sense of belonging in the
course S2 (Q15) I was able to form distinct impressions of some course participants S3 (Q16) Face-to-face communication is an excellent medium for social interaction S4 (Q17) I felt comfortable conversing face-to-face in class S9 Q(22) In-class discussions helped me to develop a sense of collaboration
The relationship between the error variances of items S1 and S2 is likely due to both
items asking students about other course participants. Similarly, both items S3 and S4 ask the
students about face-to-face communication in the course. The relationship between the error
variances of S4 and S9 is supported by both items asking the students about in-class discussions.
As a result of adding these error covariance terms, the overall data-model fit of the three factor
CoI model (!scaled,+,-./0
"= 1028.717; CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052,
0.061]; SRMR = 0.061) is determined to be acceptable based on joint criteria of RMSEA ≤ 0.06,
158
and SRMR ≤ 0.09 (Hu & Bentler, 1999). The low CFI value may be a result of relatively weak
relations in the data. Figure 18 shows the standardized values for all model parameters and Table
12 contains values for both standardized and unstandardized model parameters.
The model of the CoI instrument in Figure 18 demonstrates the same three-factor
structure seen in online education research using the original instrument. In addition, the model
results in Figure 18 and Table 12 show relationships among the three presence factors, which is
expected based on the overlapping nature of the three types of presence in the originally
conceptualized CoI model (D. R. Garrison et al., 2000). However, the model of the CoI
instrument in Figure 18 differs slightly from what was previously described in the literature due
to the removal of the four teaching presence items addressing instructor activities outside of class
and the addition of the four error covariance terms.
CoI Scale Scores and Reliability Once the internal structure of the CoI instrument has been determined to generally agree
with what had been seen in the literature, except for the exclusion of survey items 1 through 4,
item scale scores and scale reliabilities are calculated. In the same way that the average CCSF
scale score was calculated from the instructor data, the student data are used to calculate average
scale scores for teaching presence, social presence, and cognitive presence. The average scale
scores are 4.04 for teaching presence, 3.95 for social presence, and 3.77 for cognitive presence
on a five-point scale where 1 is Strongly Disagree and 5 is Strongly Agree. These average scale
scores indicate that students generally agreed with CoI item statements, providing evidence for
the students’ perception of indicators of a constructivist learning environment.
159
Figure 18. The three-factor model of the CoI survey with standardized parameter values. Asterisks indicate values significant at p < 0.001
160
Table 12 Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument
Variable Standardized Unstandardized
From To Parameter
(SE) Variance/ Residual Parameter (SE)
Variance/ Residual
Teaching presence 1 (0) 0.506 (0.081) T5 (Q5) 0.798 (0.031) 0.363 (0.050) 1 (0) 0.288 (0.035) T6 (Q6) 0.830 (0.024) 0.311 (0.041) 1.121 (0.066) 0.287 (0.029) T7 (Q7) 0.720 (0.036) 0.482 (0.052) 0.897 (0.071) 0.379 (0.033) T8 (Q8) 0.798 (0.029) 0.364 (0.046) 0.996 (0.074) 0.287 (0.033) T9 (Q9) 0.614 (0.046) 0.622 (0.056) 0.758 (0.075) 0.479 (0.046) T10 (Q10) 0.546 (0.055) 0.702 (0.060) 0.703 (0.077) 0.590 (0.050) T11 (Q11) 0.749 (0.033) 0.439 (0.049) 0.969 (0.070) 0.372 (0.040) T12 (Q12) 0.617 (0.043) 0.619 (0.053) 0.903 (0.074) 0.670 (0.054) T13 (Q13) 0.623 (0.044) 0.612 (0.055) 0.855 (0.080) 0.583 (0.051) Social presence 1 (0) 0.447 (0.073) S1 (Q14) 0.651 (0.041) 0.577 (0.054) 1 (0) 0.608 (0.056) S2 (Q15) 0.584 (0.042) 0.659 (0.049) 0.827 (0.076) 0.592 (0.044) S3 (Q16) 0.538 (0.053) 0.710 (0.057) 0.608 (0.080) 0.405 (0.043) S4 (Q17) 0.756 (0.035) 0.428 (0.052) 1.038 (0.092) 0.360 (0.048) S5 (Q18) 0.766 (0.029) 0.413 (0.044) 1.181 (0.104) 0.438 (0.048) S6 (Q19) 0.786 (0.035) 0.383 (0.055) 0.965 (0.081) 0.258 (0.035) S7 (Q20) 0.733 (0.033) 0.462 (0.048) 0.967 (0.085) 0.359 (0.041) S8 (Q21) 0.733 (0.033) 0.463 (0.048) 0.953 (0.090) 0.350 (0.033) S9 (Q22) 0.709 (0.034) 0.498 (0.049) 0.934 (0.095) 0.386 (0.039) Cognitive presence 1 (0) 0.537 (0.075) C1 (Q23) 0.729 (0.030) 0.469 (0.044) 1 (0) 0.474 (0.038) C2 (Q24) 0.700 (0.045) 0.510 (0.063) 0.912 (0.064) 0.465 (0.056) C3 (Q25) 0.763 (0.026) 0.418 (0.040) 1.111 (0.063) 0.476 (0.043) C4 (Q26) 0.498 (0.055) 0.752 (0.055) 0.717 (0.099) 0.837 (0.065) C5 (Q27) 0.627 (0.046) 0.607 (0.058) 0.827 (0.077) 0.568 (0.054) C6 (Q28) 0.653 (0.047) 0.574 (0.062) 0.765 (0.077) 0.424 (0.039) C7 (Q30) 0.642 (0.044) 0.588 (0.056) 0.801 (0.076) 0.491 (0.045) C8 (Q31) 0.722 (0.040) 0.478 (0.058) 0.767 (0.063) 0.289 (0.029) C9 (Q32) 0.805 (0.029) 0.351 (0.047) 0.924 (0.071) 0.249 (0.026) C10 (Q33) 0.784 (0.027) 0.385 (0.042) 0.904 (0.060) 0.275 (0.027) C11 (Q34) 0.810 (0.028) 0.344 (0.045) 1.029 (0.076) 0.298 (0.034) C12 (Q35) 0.796 (0.027) 0.366 (0.043) 0.998 (0.068) 0.309 (0.027) C13 (Q36) 0.792 (0.026) 0.373 (0.041) 0.954 (0.067) 0.291 (0.025) C14 (Q37) 0.719 (0.031) 0.483 (0.044) 1.042 (0.071) 0.545 (0.043) Note. All parameters significant at p < 0.001
161
Table 12, continued Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument
Variables Standardized Unstandardized Between Parameter (SE) Parameter (SE)
Teaching Social –0.667 (0.049) –0.317 (0.050) Teaching Cognitive –0.794 (0.039) –0.414 (0.060) Social Cognitive –0.674 (0.049) –0.330 (0.052) T12 (Q12) T13 (Q13) –0.272 (0.065) –0.170 (0.043) S1 (Q14) S2 (Q15) –0.400 (0.058) –0.240 (0.042) S3 (Q16) S4 (Q17) –0.374 (0.074) –0.143 (0.038) S4 (Q17) S9 (Q22) –0.359 (0.072) –0.134 (0.029)
Note. All parameters significant at p < 0.001
Reliability values for the three presence scales are calculated in two ways. First,
Cronbach’s alpha values are calculated for each scale in order to allow for comparison with
existing literature and to provide information about the scale reliability when item responses are
combined either as averages or sums to create scale scores (Arbaugh, 2008; Arbaugh et al., 2008,
2010; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). Second, construct
reliabilities are calculated from the standardized item loadings using coefficient H to provide
information about the reliability of the underlying latent presence factor. Both Cronbach’s alpha
and coefficient H values were greater than 0.89 for all three scales. These values are within the
generally accepted range of 0.70 or above (Arjoon et al., 2013; Mueller & Hancock, 2010).
These results are presented in Table 13. Both sets of calculations were performed using the
statistical software R (version 3.1.1; R Core Team, 2014). Cronbach’s alpha is calculated using
the alpha function in the psych package (version 1.4.8.11; Revelle, 2015), and coefficient H is
calculated using the function written in R by the researcher, provided in Appendix F.
162
Table 13 Reliability Values for the Presence and Satisfaction Scales
Scale Cronbach’s alpha Coefficient H Teaching presence 0.892 0.912 Social presence 0.894 0.904 Cognitive presence 0.932 0.944
The combination of the high reliability values for the CoI presence scales and the
acceptable overall-data model fit provides evidence that, with the sample of students in this
research, the CoI survey is able to measure student perceptions of indicators of a constructivist
learning environment. Other than the removal of items 1 through 4, the three-factor model of the
CoI instrument tested in this research corresponds to other research conducted with the CoI
instrument in online courses (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al.,
2010; Joo et al., 2011; Shea & Bidjerano, 2009). This relationship suggests that modifications to
the wording of the CoI instrument as a result of the pilot study did not impact its functioning.
Since these results demonstrate that the CoI instrument is an acceptable instrument for measuring
student perceptions of indicators of a constructivist learning environment, the CoI data are used
to test the full research model relating aspects of a constructivist learning environment to student
outcomes.
Structural Equation Model Results The full research model is tested in a two-phase SEM analysis process (Mueller &
Hancock, 2008). In the first phase of the SEM analysis, the measurement portion of the model is
tested in order to understand how the data fit the model under the least restrictive set of
conditions where all causal paths between variables are replaced with correlational relationships.
The measurement model includes modifications made to the CoI items such as removing survey
163
items 1–4 and adding the four error covariance terms. The fit of the measurement model is
acceptable (!scaled,+,-12X
"= 1374.826; CFIscaled = 0.899; RMSEAscaled = 0.051, CI90=[0.047,
0.055]; SRMR = 0.058) based on the joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu &
Bentler, 1999). None of the modifications suggested by the Mplus software are theoretically
justified so no changes are made before moving on to test the structural model.
The second phase of the SEM analysis reintroduces the hypothesized causal paths among
variables. As expected, the data-model fit degrades slightly after the introduction of the causal
paths between variables (!scaled,+,-120
"= 1429.111; CFIscaled = 0.892; RMSEAscaled = 0.053,
CI90=[0.049, 0.056]; SRMR = 0.065), but the overall fit of the model is acceptable based on joint
criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). The low CFI value
indicates a possibility that even though the model does an acceptable job explaining relationships
among the data, as demonstrated by the low RMSEA and SRMR values, the relationships were
relatively weak to begin with. Again, none of the modifications suggested by the Mplus software
are theoretically justified so this model is retained as the final research model. Having an
acceptable data-model fit indicates that the hypothesized model is a viable representation of the
true underlying relationships present in the data and provides support for the overall goal of the
research, which is to develop and test a model of relationships among aspects of a constructivist
learning environment and student outcomes of academic achievement and satisfaction. However,
not all of the originally hypothesized paths between variables, which are the focal parameters in
this research, are found to be statistically significant. Four previously hypothesized causal paths
and one nondirectional relationship are found to not be statistically significant at p < .05. These
four hypothesized causal paths are the direct effect of math ability on final course grades, the
164
direct effects of teaching presence on ACS exam scores and final course grades, and the direct
effect of social presence on student satisfaction. The hypothesized relationship between the
residual error terms for student satisfaction and final course grades is also found to not be
statistically significant. All other hypothesized paths are statistically significant. Figure 19 shows
the structural portion of the final research model with standardized values for all focal
parameters.
Figure 19. Standardized values for focal parameters in the structural model and R2 values for endogenous variables. Solid arrows and asterisks indicate paths significant at p < .05. The sign of the relationships with the satisfaction variable are reversed to reflect the scale of the satisfaction items. The standardized path values in Figure 19 can be interpreted similarly to standardized
regression coefficients. Dashed arrows indicate hypothesized paths that are found to not be
statistically significant at p < .05. The sign of relationships with the satisfaction latent variable
165
are reversed in Figure 19 to reflect the fact that the scale of three of the four satisfaction items,
provided in Table 14, is opposite from the scale for the other items on the survey instrument.
Table 14 Satisfaction Items from Student Survey THE CHEMISTRY
COURSE WAS… Middle
Q38(SS1). Comfortable 1 2 5 4 5 Uncomfortable Q39(SS2). Satisfying 1 2 5 4 5 Frustrating Q40(SS3). Pleasant 1 2 5 4 5 Unpleasant Q41(SS4). Chaotic 1 2 5 4 5 Organized
The scale of satisfaction items SS1, SS2, and SS3 presents a positive course description at the
low (1) end of the sale and a negative course description at the high end of the scale (5), but for
every other item on the survey instrument, a scale value of 5 corresponds to agreement with a
positive statement about the classroom environment. Due to these differences in the direction of
the scale, the original model output has a negative relationship between the three presence scales
and the satisfaction scale indicating that as the three presence scales increase the negative
responses to the satisfaction items decrease. To aid in interpretation of the model, the sign of
these relationships is reversed so that values can be more easily interpreted as when the three
presence scales increase the positive responses to the satisfaction items increase.
A full listing of all model parameters, both standardized and unstandardized, including
individual survey items loadings on their respective factors is presented in Table 15. In Table 15
the loadings of satisfaction items SS1, SS2, and SS3 onto the satisfaction factor are reversed
from the sign in the Mplus output so that the sign of their loading is consistent with the
relationships to the student satisfaction latent variable in the full model in Figure 19. Similarly,
the signs of the paths to satisfaction are reversed in Table 15 so that they are consistent with
Figure 19.
166
Table 15 Model Parameter Values and Standard Errors (SE) from Final Research Model
Variable Standardized Unstandardized
From To Parameter
(SE) Variance/ Residual
Parameter (SE)
Variance/ Residual
Teaching presence 1 (0) *0.502 (0.081) *T5 (Q5) 0.798 (0.032) 0.362 (0.051) 1 (0) 0.285 (0.035) *T6 (Q6) 0.836 (0.024) 0.302 (0.039) 1.133 (0.069) 0.279 (0.029) *T7 (Q7) 0.715 (0.038) 0.489 (0.055) 0.896 (0.074) 0.386 (0.034) *T8 (Q8) 0.797 (0.029) 0.365 (0.045) 0.991 (0.076) 0.284 (0.032) *T9 (Q9) 0.608 (0.047) 0.630 (0.057) 0.753 (0.076) 0.485 (0.045) *T10 (Q10) 0.540 (0.055) 0.708 (0.059) 0.700 (0.077) 0.597 (0.049) *T11 (Q11) 0.749 (0.033) 0.438 (0.049) 0.973 (0.073) 0.371 (0.039) *T12 (Q12) 0.612 (0.043) 0.625 (0.053) 0.893 (0.075) 0.667 (0.054) *T13 (Q13) 0.626 (0.043) 0.609 (0.054) 0.863 (0.083) 0.581 (0.051) Social presence *0.553 (0.063) *0.251 (0.044) *S1 (Q14) 0.654 (0.041) 0.573 (0.053) 1 (0) 0.608 (0.056) *S2 (Q15) 0.584 (0.042) 0.659 (0.049) 0.823 (0.076) 0.593 (0.044) *S3 (Q16) 0.541 (0.054) 0.707 (0.058) 0.608 (0.080) 0.405 (0.043) *S4 (Q17) 0.756 (0.035) 0.428 (0.053) 1.031 (0.090) 0.360 (0.048) *S5 (Q18) 0.770 (0.028) 0.407 (0.043) 1.183 (0.103) 0.436 (0.047) *S6 (Q19) 0.785 (0.035) 0.383 (0.055) 0.957 (0.079) 0.258 (0.035) *S7 (Q20) 0.735 (0.033) 0.460 (0.048) 0.964 (0.083) 0.359 (0.041) *S8 (Q21) 0.733 (0.033) 0.462 (0.048) 0.949 (0.088) 0.350 (0.033) *S9 (Q22) 0.712 (0.034) 0.493 (0.048) 0.936 (0.094) 0.385 (0.038) Cognitive presence *0.334 (0.056) *0.180 (0.033) *C1 (Q23) 0.731 (0.030) 0.466 (0.044) 1 (0) 0.469 (0.038) *C2 (Q24) 0.699 (0.046) 0.511 (0.064) 0.906 (0.065) 0.462 (0.054) *C3 (Q25) 0.769 (0.025) 0.409 (0.039) 1.123 (0.060) 0.470 (0.043) *C4 (Q26) 0.484 (0.057) 0.766 (0.055) 0.689 (0.099) 0.836 (0.064) *C5 (Q27) 0.626 (0.046) 0.608 (0.058) 0.827 (0.077) 0.571 (0.054) *C6 (Q28) 0.645 (0.049) 0.585 (0.063) 0.750 (0.078) 0.426 (0.039) *C7 (Q30) 0.636 (0.044) 0.596 (0.056) 0.789 (0.074) 0.495 (0.045) *C8 (Q31) 0.721 (0.040) 0.480 (0.057) 0.765 (0.062) 0.290 (0.029) *C9 (Q32) 0.804 (0.029) 0.353 (0.047) 0.926 (0.070) 0.252 (0.026) *C10 (Q33) 0.783 (0.027) 0.387 (0.043) 0.901 (0.061) 0.276 (0.027) *C11 (Q34) 0.806 (0.029) 0.350 (0.047) 1.020 (0.076) 0.302 (0.035) *C12 (Q35) 0.797 (0.027) 0.364 (0.044) 0.995 (0.068) 0.305 (0.028) *C13 (Q36) 0.792 (0.026) 0.373 (0.042) 0.949 (0.066) 0.289 (0.025) *C14 (Q37) 0.726 (0.030) 0.473 (0.043) 1.054 (0.070) 0.538 (0.042) Satisfaction *0.508 (0.077) *0.346 (0.062) *SS1(Q38) –0.804# (0.028) 0.353 (0.045) –1# (0) 0.372 (0.045) *SS2 (Q39) –0.868# (0.018) 0.246 (0.031) –1.271# (0.082) 0.359 (0.041) *SS3 (Q40) –0.892# (0.022) 0.204 (0.039) –1.189# (0.076) 0.247 (0.045) *SS4 (Q41) –0.573# (0.053) 0.672 (0.061) –0.745# (0.075) 0.774 (0.098)
167
Table 15, continued Model Parameter Values and Standard Errors (SE) from Final Research Model
Variables Standardized Unstandardized Variance / Residual (SE) Variance /Residual (SE)
Math ability 1 (0) *14.44 (1.037) *ACS exam scores 0.708 (0.039) 72.963 (5.351) *Final course grades 0.253 (0.022) 40.206 (4.255)
From To Parameter (SE) Parameter (SE)
Teaching presence *Social presence –0.668 (0.047) –0.635 (0.080)
*Cognitive presence –0.613 (0.077) –0.634 (0.100) *Satisfaction –0.388# (0.117) ––0.452# (0.142)
ACS exam scores –0.036 (0.107) –0.521 (1.534) Final course grades –0.023 (0.066) –0.408 (1.168)
Social presence *Cognitive presence 0.268 (0.072) –0.292 (0.075)
Satisfaction –0.053# (0.077) –0.065# (0.093) Cognitive presence
*Satisfaction 0.392# (0.114) 0.441# (0.130) *ACS exam scores 0.288 (0.097) 3.991 (1.389)
*Final course grades 0.140 (0.067) 2.416 (1.186) Math ability
*ACS exam scores 0.474 (0.035) 1.266 (0.104) Final course grades 0.042 (0.035) 0.141 (0.114)
ACS exam scores *Final course grades 0.790 (0.026) 0.981 (0.054)
Between Satisfaction Final course grades –0.080# (0.090) – –0.300# (0.334)#
T12 (Q12) T13 (Q13) –*0.273 (0.064) –*0.170 (0.043) S1 (Q14) S2 (Q15) –*0.401 (0.057) –*0.241 (0.042) S3 (Q16) S4 (Q17) –*0.374 (0.074) –*0.143 (0.038) S4 (Q17) S9 (Q22) *–0.362 (0.071) *–0.135 (0.029)
Note. Asterisks indicate significant at p < 0.05, #indicates sign change to reflect scale of satisfaction items
The values associated with the arrows in Figure 19 only provide information about the
direct effects between variables in the structural model, so indirect effects are requested from
Mplus in order to better understand how one variable may influence another variable by passing
through one or more additional variables. As an example, math ability has a significant direct
168
effect on ACS exam scores, but a nonsignificant direct effect on final course grades. As
described in Chapter 3, according to path tracing rules (Wright, 1934) the value of the indirect
effect of math ability on final course grades through ACS exam scores is mathematically equal to
the value of the direct effect from math ability to ACS exam scores multiplied by the value of the
direct effect from ACS exam scores to final course grades. The statistical significance of this
indirect effect, and all other indirect effects in the model, was determined by examining the
bootstrapped 95% confidence intervals calculated by Mplus. Bootstrapping procedures are used
to provide a more robust estimation of the significance of the indirect effects. This robust
estimation is necessary not only due to the nonnormality present in the data but also because
indirect effects are a product of multiple direct effects and are therefore unlikely to follow
normal distribution (Williams & MacKinnon, 2008). The bootstrapping procedures use the
student data as the pool from which to repeatedly draw random samples from the data to use in
calculating parameter estimates and confidence intervals (Field, 2013; Muthén & Muthén, 2010).
These results are presented in Table 16.
169
Table 16 Decomposed Standardized and Unstandardized Effects Among Variables in Structural Model Variables Standardized Unstandardized To From Direct Indirect Total Direct Indirect Total Final course grades (R2 = 0.747)
Teaching presence –0.023 0.263* 0.286* –0.408 4.681* 5.088* Social presence – 0.099* 0.099* – 1.849* 1.849*
Cognitive presence –0.140* 0.228* 0.368* –2.416* 3.917* 6.333* Math ability –0.042* 0.374* 0.416* –0.141* 1.242* 1.383* ACS exam scores –0.790* – 0.790* –0.981* – 0.981* ACS exam scores (R2 = 0.292)
Teaching presence –0.036* 0.228* 0.192* –0.521 3.272* 2.751* Social presence – 0.077* 0.077* – 1.165* 1.165*
Cognitive presence –0.288* – 0.288* –3.991* – 3.991* Math ability –0.474* – 0.474* –1.266* – 1.266* Student satisfaction (R2 = 0.492)
Teaching presence –0.388* 0.275* 0.663* –0.452* 0.321* 0.773* Social presence –0.053 0.105* 0.052 –0.065* 0.129* 0.064
Cognitive presence –0.392* – 0.392* –0.441* – 0.441* Cognitive presence (R2 = 0.666)
Teaching presence –0.613* 0.179* 0.792* –0.634* 0.185* 0.820* Social presence –0.268* – 0.268* –0.292* – 0.292*
Social presence (R2 = 0.447) Teaching presence –0.668* – 0.668* –0.635* – 0.635* *p < 0.05. Note. The sign of the paths to the satisfaction items are reversed.
In addition to the standardized values for the focal parameters of interest in this research,
Figure 19 and Table 16 also provide R2 values for all endogenous variables in the structural
model, which are the only variables for which R2 values could be calculated. The R2 values
indicate how much variance is explained by the causal paths pointing towards these variables.
Based on this data, the proposed research model is seen to explain almost 75% of the variance in
final course grades, 67% of the variance in cognitive presence ratings, approximately 50% of the
variance in student satisfaction, approximately to 45% of the variance is social presence ratings
and about 30% of the variance in ACS exam scores.
170
ACS exam scores represent a measurement of students’ chemistry content knowledge
based on a standardized national exam. About 30% of the variance in ACS exam scores (R2 =
0.292) is explained by the relationships included in this model. The data in Table 16 show that
only math ability and cognitive presence have statistically significant direct effects on ACS exam
scores. The direct effect of cognitive presence on ACS exam scores is about half the size of the
direct effect of math ability on ACS exam scores. This result indicates that the incoming math
ability of the students has more influence on their performance on the ACS exam than the degree
of cognitive presence in the classroom environment.
Teaching presence was hypothesized to have a direct effect on ACS exam scores in the
original research model, but the data do not support this hypothesis. However, the indirect effects
of teaching presence on ACS exam scores are statistically significant. There are multiple indirect
paths available in the model to move from teaching presence to ACS exam scores. The two
indirect effects with the largest values are the indirect effect of teaching presence through
cognitive presence to ACS exam scores (standardized = 0.177; unstandardized = 2.535) and the
longer indirect effect from teaching presence through social presence then through cognitive
presence to ACS exam scores (standardized = 0.052; unstandardized = 0.740). Both of these
indirect effects are statistically significant. Social presence was not hypothesized to have a direct
effect on ACS exam scores, but is found to have a significant, though small, indirect effect on
ACS exam scores due to the indirect path from social presence to cognitive presence then to ACS
exam scores.
Comparing the magnitude of the direct effect of math ability on ACS exam scores with
the magnitude of other direct and indirect effects on ACS exam scores demonstrates that math
171
ability has the largest influence on ACS exam scores. This relationship between math ability and
performance on the ACS exam is consistent with published research (Lewis & Lewis, 2005; Xu
& Lewis, 2011). The relationship between math ability and ACS exam performance for this
student data set is further supported by examination of the specific first semester general
chemistry ACS exam taken by the students (Form GC15FG) in which approximately one third of
the 70 items were classified by the researcher and an instructor involved in the research as
primarily emphasizing mathematical manipulation to arrive at the correct answer. Of the three
classroom environment factors, cognitive presence has both the largest influence on ACS exam
scores and the only significant direct effect on ACS scores of the three presence factors. The
influences of both teaching presence and social presence on ACS exam scores were only
statistically significant as indirect effects when passing through cognitive presence. These results
indicate that, in addition to math ability, creating a learning environment in which students are
engaged in constructing their own explanations and applying their knowledge has a beneficial
effect on students’ chemistry content knowledge.
The other measurement of students’ academic achievement used in this research is final
course grades. The final course grades in this research do not include the laboratory component
of the course, and in that respect are only an assessment of students’ performance in the
classroom portion of the course. Since the ACS exam is used as a final exam in the courses
studied in this research, students’ ACS exam scores are equal to between 15% and 25% of the
final course grade, depending on the instructor. Thus, the direct effect of ACS exam scores on
final course grades is by far the largest of all the variables. This explains why the model is able
to explain nearly 75% of the variance in final course grade (R2 = 0.747). While math ability does
172
not have a statistically significant direct effect on final course grades, the indirect effect of math
ability on final course grades through ACS exam scores is larger than any of the other total
effects on final course grades. These results indicate that for this sample of students, final course
grades are primarily influenced by the students’ scores on the final exam and their incoming
math ability and that the classroom environment factors have a smaller influence on final course
grades.
Similar to what was seen for ACS exam scores, cognitive presence is the only one of the
three presence factors that has a significant direct effect on final course grades. The large indirect
effect of cognitive presence on final course grades is a result of the path from cognitive presence
through ACS exam scores to final course grades. Teaching presence was originally hypothesized
to have a direct effect on final course grades, but the data do not support this hypothesis.
However, the sum of the multiple indirect effects of teaching presence on final course grades is
statistically significant. The largest indirect effect of teaching presence on final course grades is
the path from teaching presence to cognitive presence through ACS exam scores and ultimately
to final course grades (standardized = 0.140; unstandardized = 2.485). This indirect effect of
teaching presence is statistically significant. The next largest indirect effect of teaching presence
on final course grades is from teaching presence to cognitive presence to final course grades
(standardized = 0.086, unstandardized = 1.533) and is also statistically significant. These results
highlight the large influence ACS exam scores have on final course grades in this data set since
the size of the indirect effect of teaching presence falls almost in half when the path through ACS
exam scores is not included.
173
The third course environment factor, social presence, was not hypothesized to have a
direct effect on final course grades, so only the indirect effects of social presence on final course
grades are tested in the model. The significant indirect effect of social presence on final course
grades is a result of the indirect effect from social presence to cognitive presence through ACS
exam scores to final course grades (standardized = 0.061, unstandardized = 1.143). The indirect
effect from social presence through cognitive presence and then to final course grades, skipping
ACS exam scores, is not statistically significant. These results indicate that, for this sample,
social presence has a relatively small influence on final course grades and that this influence is
primarily as a result of the indirect effect through ACS exam scores to final course grades.
The large influence of ACS exams scores on final course grades, both directly and
indirectly, is mainly a result of the inclusion of ACS exams scores within the calculated final
course grades for this particular data set. The lack of a direct effect of math ability on final
course grades may indicate that the exams written by the instructors, which comprise between
30% and 45% of the final course grade, do not include as much mathematical content as the ACS
exam. Similarly to ACS exam scores, the only classroom environment factor that directly
influences final course grades is cognitive presence. Teaching presence and social presence only
significantly influence final course grades indirectly through cognitive presence and ACS exam
scores. These results highlight the importance of creating a learning environment where students
can engage in learning activities. These results also demonstrate that teaching presence and
social presence are influential only insofar as they support the types of learning activities that
develop cognitive presence.
174
Though the original model hypothesized a relationship between the residual error terms
for final course grades and student satisfaction, this relationship is not found to be statistically
significant. This result indicates that the correlations between final course grades and the four
individual satisfaction item responses, ranging from 0.24 and 0.52, were accounted for by the
causal relationships already present in the model. The hypothesized path from social presence to
student satisfaction is also found to not be statistically significant. While this result was
unexpected, the nonsignificant path from social presence to student satisfaction replicates results
of previous research (Joo et al., 2011). The implication of the nonsignificant path from social
presence to satisfaction in both this study and in the literature suggests that students’ comfort
level interacting with other students in the course does not directly influence students’
satisfaction with the course.
The magnitude of the individual survey item loadings for the student satisfaction factor
are larger than the values seen in previous research but trend in a similar way (Xu & Lewis,
2011), indicating that the satisfaction items in this study are functioning as intended. These
results are presented in Table 17.
Table 17 Standardized Loadings for Satisfaction Items in Current Research and Existing Literature
SS1 SS2 SS3 SS4 1 = comfortable
5 = uncomfortable 1 = satisfying 5 = frustrating
1 = pleasant 5 = unpleasant
1 = chaotic 5 = organized
Current research –0.80 –0.87 –0.89 0.57 Xu & Lewis (2011) –0.74 –0.77 –0.83 0.48
Note. The scale in Xu & Lewis (2011) ranged from 1 to 7; the sign of the loadings has been adjusted to be consistent with Figure 19 and Table 15. Additionally, the Cronbach’s alpha (0.857) and coefficient H value (0.903) of the satisfaction
scale are well above the generally accepted value of 0.70. This indicates an acceptable reliability
175
level of both the satisfaction scale and the latent variable of satisfaction (Arjoon et al., 2013;
Mueller & Hancock, 2010). These results suggest that the way satisfaction was measured in the
current research is in alignment with previous research and support the relationships student
satisfaction has with other variables in the research model.
Satisfaction is the only student outcome variable on which teaching presence had a
significant direct effect. The direct effects of teaching presence and cognitive presence on
student satisfaction were both similar in size and primarily responsible for the model being able
to explain nearly 50% of the variance in student satisfaction ratings (R2 = 0.492). The major
contributor to the indirect effect of teaching presence on student satisfaction is the path from
teaching presence to satisfaction through cognitive presence. The indirect effect of social
presence on satisfaction through cognitive presence is also significant, even though the total
effect of social presence on satisfaction is not significant. These results highlight the large
influence cognitive presence has on student satisfaction in this data set. Cognitive presence
influences satisfaction in two ways, directly and also indirectly as a mediator between other
variables and satisfaction. Combining these results with the nonsignificant relationship between
final course grades and satisfaction suggests that the students surveyed for this research are most
satisfied when the instructor facilitates the development of a learning environment where
students experience a high degree of cognitive presence. The role of the instructor as a facilitator
is important in creating the teaching presence that both directly and indirectly influences student
satisfaction. The results of this model also demonstrate that both teaching and cognitive presence
have a larger influence on student satisfaction than social presence. The smaller influence of
social presence on student satisfaction as compared to teaching presence and cognitive presence
176
suggests that individual engagement in the learning process has more influence on students’
satisfaction with the course than the students’ comfort level during group activities or
discussions.
Considering the results of the structural equation model as a whole indicates that while
constructivist learning environment factors of teaching presence and cognitive presence do
appear to have an influence on student satisfaction and academic outcomes, the influence of
math ability on academic outcomes is larger than either one. Cognitive presence has a greater
direct effect on the academic variables and satisfaction than teaching presence. However, the
direct effect of teaching presence on satisfaction is similar to the direct effect of cognitive
presence on satisfaction. Social presence appears to have minimal influence on both academic
outcomes and student satisfaction. In light of the nonsignificant connection between student
satisfaction and final course grades beyond the causal relationships already present in the model,
it appears that academic outcomes are strongly related to course learning activities while student
satisfaction is strongly related to both learning activities and specific instructor behaviors.
Addressing the Research Questions
Research Question 1 1. Are self-reported instructor approaches to teaching consistent with student perceptions of
the learning environment?
Measurement of the learning environment from the instructor’s perspective began with
modification of the Approaches to Teaching Inventory (ATI) for use in US classrooms. Pilot
testing of the ATI revealed that the conceptual change student-focused (CCSF) scale was most
177
closely aligned with the research goal of identifying student-centered teaching approaches where
the role of the instructor shifted to that of a facilitator for a portion of class time. During the
instructor facilitation portion of the course, students were given opportunities to actively
construct their own chemistry knowledge. This conclusion is a result of the pilot study
instructors who described an approach to teaching that was “student-centered”, either in their
interview or in their course syllabus, having the highest CCSF scores. Analysis of the ATI pilot
study results indicated the need to conduct a follow-up interview and analysis of the course
syllabus in order to provide a more complete description of an instructor’s approach to teaching.
Though a formal think-aloud interview was not conducted as part of the instructor data
collection process in the main study, the first question in the semi-structured interview provided
the instructors with an opportunity to explain or clarify their responses to the ATI if desired. All
instructors in the main study used this opportunity to clarify their interpretation or response to at
least one survey item. In all cases the instructor’s understanding of the item was in alignment
with its intended interpretation. In this way, the semi-structured interviews in the main study
provide evidence for the validity of the response process used to answer the extensively revised
ATI items.
When the four instructors in the main study provided responses to the ATI, the CCSF
scale scores calculated for each of the individual instructors range from 3.5 to 4.6 with a mean
score of 4.1. Since the response scale ranges from 1 (only rarely) to 5 (almost always) with the
midpoint (3) indicating something the instructor reports doing about half the time the mean score
of 4.1 indicates that, on average, the instructors self-report emphasizing a conceptual change,
student-focused approach in their classroom more than half the time. The items on the CCSF
178
scale include statements such as “I provide opportunities so that the students can discuss, among
themselves, key concepts and ideas in this course” and “My teaching in this course helps
students question their own understanding of the subject matter.” Agreement with these
statements and similar statements on the ATI is supported by the instructor interviews and
information available in the instructor-written course syllabus.
Student perceptions of the learning environments created by the instructors are examined
by computing the average scale scores for the three presence factors measured by the
Community of Inquiry (CoI) student survey instrument. The average scale scores are 4.04 for
teaching presence, 3.95 for social presence, and 3.77 for cognitive presence on a five-point scale
where 1 is strongly disagree and 5 is strongly agree. These averages indicate that the students in
the instructors’ classrooms generally have a positive perception of the three factors associated
with a student-centered constructivist learning environment. These results align the student
perceptions of the learning environment with instructor descriptions of their approach to teaching
the introductory chemistry course.
Closer examination of individual student survey item averages supports the instructors’
description of using an approach to teaching that encourages students to discuss course ideas
among themselves and construct their own understanding of the course material. The items on
each of the three presence scales with the highest average scores represent items the students rate
most favorability on the response scale from 1 (Strongly disagree) to 5 (Strongly agree). The
wording and average scores for these three items are provided in Table 18.
179
Table 18 Item on Each Presence Scale with Highest Mean Rating
Item Mean (SE) SD Wording
T5 (Q5) 4.19 (0.05) 0.88 The instructor was helpful in facilitating discussions on course topics that helped me to learn.
S3 (Q16) 4.39 (0.04) 0.76 Face-to-face communication is an excellent medium for social interaction.
C9 (Q32) 4.02 (0.04) 0.83 Course learning activities helped me construct explanations/solutions
Note. SE = standard error; SD = standard deviation; response scale from 1 (Strongly disagree) to 5 (Strongly agree)
The teaching presence item with the highest average score, T5, provides evidence from
the students’ perspective that the instructors utilize course discussions in a way that helped
students learn. The cognitive presence item with the highest average score, C9, relates directly to
the idea of having students construct their own understanding and aligns the students’
perceptions of the learning environment to their instructors’ own descriptions of how
opportunities for student learning are provided. The social presence item with the highest
average score, S3, is less directly applicable to understanding specific instructional strategies.
This is the same social presence item that was identified in the pilot study as the item students
described as asking more about their beliefs than any particular aspect of the classroom
environment. This item was left on the CoI instrument in order to provide an opportunity to
compare its functioning in this research with previous research using the CoI. Item S3 also shows
the smallest relationship to the social presence factor in the CFA of the CoI instrument,
indicating that while students generally agreed with the item, it is not highly representative of the
latent variable of social presence measured by the CoI instrument. Taken together, these results
demonstrate that self-reported instructor approaches to teaching, as measured by the CCSF scale
180
of the ATI, a short interview, and the course syllabus are consistent with student perceptions of
the learning environment as measured by the CoI at both the item and scale levels.
Research Question 2 2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for
measuring student perceptions of the indicators of a constructivist learning environment
in a face-to-face introductory undergraduate chemistry course?
The acceptability of the CoI survey as an instrument to measure student perceptions of
indicators of a constructivist learning environment is determined from evidence for the validity
and reliability of the CoI survey scores and underlying latent presence factors generated from
analysis of data in the main study. The analyses conducted in the main portion of this research
provide additional evidence for the validity and reliability of the student survey scores beyond
the evidence that was available after conducting the pilot study. Two types of additional validity
evidence generated in the main study are evidence for the internal structure of the CoI and
evidence based on relationships with CoI item responses and other variables. The reliability
evidence for the CoI survey scores and latent presence factors comes from determination of
Cronbach’s alpha and coefficient H values for the three presence scales of the CoI instrument.
Both the validity and reliability evidence are used to support the use of the CoI instrument as an
acceptable instrument for measuring student perceptions of indicators of a constructivist learning
environment in a face-to-face introductory chemistry course.
Evidence for the internal structure of the CoI items is primarily based on the results of the
confirmatory factor analyses. After implementing the modifications to the CoI instrument
181
including removing items 1–4 from the teaching presence factor and adding four error
covariance terms, the fit of the three factor model of the CoI instrument is found to be acceptable
(!scaled,+,-./0
"= 1028.717; CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR
= 0.061) based on joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). All
items load on their intended factors, and no modifications are necessary which would have
resulted in cross loading items on multiple factors. This model of the CoI instrument differs
slightly from what was previously described in the literature due to the removal of the four
teaching presence items addressing instructor activities outside of class time. However, this
separation of the teaching presence factor into two separate but related factors was hypothesized
as a possible result of the analysis based on theory and previous research. The values of the
loadings for the remaining items on the CoI instrument are consistent with what has been
reported in the literature (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al., 2010;
Joo et al., 2011; Shea & Bidjerano, 2009). Therefore, it is concluded that the model for the
internal structure of the CoI shown in Figure 18 (p. 158) is a viable representation of the true
underlying relationships present in the data. This evidence for the validity of the revised CoI
items is particularly important to addressing the second research question given the
modifications to some of the items and the use of the instrument with face-to-face chemistry
students rather than students in an online course. In addition to evidence for the validity of CoI
scores as a result of the internal structure of the CoI instrument, additional evidence is provided
by relationships between CoI scores and other research variables.
Evidence for the validity of the CoI survey scores based on their relationships with other
variables was partially examined in the context of the first research question. The consistency of
182
student and instructor descriptions of the learning environment provides support for the
acceptability of the CoI instrument as a tool to measure student perceptions of the learning
environment. Triangulating the instructor and student quantitative survey response data with the
qualitative instructor data provides convergent evidence that student perceptions of the learning
environment are consistent with self-reported instructor approaches to teaching. The relationship
between the three types of presence measured by the CoI instrument and student satisfaction was
tested during the SEM analysis of the full research model. The results of this analysis show that
both teaching and cognitive presence directly influence student satisfaction, but that social
presence does not directly influence satisfaction. These relationships among the three CoI
presence factors and student satisfaction are consistent with previous research (Joo et al., 2011)
and therefore support the conclusion that the relationships the CoI variables have with other
variables in this research provide evidence for the validity of the CoI survey scores.
Reliability information about the three CoI presence scales, as determined from
Cronbach’s alpha and coefficient H values, also provides evidence to support the conclusion that
the CoI is an acceptable instrument for measuring aspects of a constructivist learning
environment. The alpha and coefficient H values for the three presence scales were all 0.89 or
above, within the generally accepted range of 0.70 or above (Arjoon et al., 2013; Mueller &
Hancock, 2010). The alpha values for the three CoI scales are also consistent with alpha values
reported in other studies utilizing the CoI instrument (Arbaugh, 2008; Arbaugh et al., 2008,
2010; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009), as seen in Table 19.
183
Table 19 Cronbach’s Alpha for CoI Presence Scales in the Current Research and Existing Literature
Teaching presence Social presence Cognitive presence Current research 0.89 0.89 0.93 Arbaugh (2008) 0.95 0.87 0.90 Arbaugh et al. (2008) 0.94 0.91 0.95 Arbaugh et al. (2010) 0.96 0.91 0.95 D. R. Garrison et al. (2010) 0.93 0.87 0.91 Joo et al. (2011) 0.89 0.84 0.82 Shea & Bidjerano (2009) 0.96 0.92 0.95
The Cronbach’s alpha values in Table 19 from both this research using the modified CoI
instrument and from other research with the original CoI items designed for use in online courses
are all above 0.82 and indicate a high degree of internal consistency of the three presence scales.
Additionally, the range of Cronbach’s alpha values is similar across all seven studies listed in
Table 19 and provides evidence for the internal consistency of the three presence scales when
used with a wide variety of students enrolled in a range of different types of courses. The large
coefficient H values for the teaching, social, and cognitive presence factors calculated in this
research (0.912, 0.904, and 0.944, respectively) indicate a large degree of stability of the factors
and therefore greater anticipated reliability over repeated administrations (Mueller & Hancock,
2010). Taken together, both the Cronbach’s alpha and coefficient H values calculated in this
research demonstrate that modifying the CoI survey for use in a face-to-face environment does
not degrade its internal consistency relative to previously administered versions of the
instrument.
The previously described analysis of the student CoI responses provides evidence for
both the validity and reliability of the CoI scores from the internal structure of the CoI
instrument, from relationships among the three CoI presence factors and other research variables,
184
and from the calculated Cronbach’s alpha and coefficient H values. The three-factor structure of
the CoI instrument tested in the current research has acceptable fit with the student data and is
consistent with the structure of the CoI instrument seen in previous research with the CoI
instrument in online courses. The alignment of both individual CoI item averages and CoI scale
averages with data obtained from instructor ATI responses, interviews, and course syllabi
demonstrates that CoI scores can be used to provide an acceptable measurement of the learning
environment experienced by students. The calculated reliability information from Cronbach’s
alpha and coefficient H values suggests that the presence scales on the CoI instrument can be
considered reliable measurements of students’ perceptions of three aspects of a constructivist
learning environment. The combination of both validity and reliability evidence provides strong
support for the acceptability of the CoI as an instrument to measure student perceptions of
indicators of a constructivist learning environment in a face-to-face introductory undergraduate
chemistry course.
Research Question 3 3. To what degree does a constructivist learning environment, as measured by student CoI
survey responses, affect outcomes of student satisfaction and academic achievement in
chemistry, as measured by ACS exam scores and final course grades when the effect of
math ability on academic achievement is considered?
The primary goal of this research, as described by the third research question, is to use
the hypothesized model of relationships among the three CoI presence factors and student
outcomes to examine the influence of a constructivist learning environment on student
185
satisfaction and academic achievement. Since acceptable data-model fit was obtained, as
discussed on p. 163, the specific relationships among the three presence factors and student
outcomes of satisfaction and academic achievement are examined in order to provide detail
about the relative influence each variable had on student outcomes.
Four of the originally hypothesized direct effects between variables in the research model
are found to be not statistically significant at p < .05. These direct effects are the paths from math
ability to final course grades, from teaching presence to ACS exam scores, from teaching
presence to final course grades, and from social presence to student satisfaction. Not finding
these paths to be statistically significant does not mean that, for example, teaching presence does
not influence student academic outcomes in any way. Instead this result indicates that teaching
presence in isolation does not influence academic outcomes and instead teaching presence
influences academic outcomes through its relationship with other variables in the model, such as
cognitive presence. This multi-step influence describes the indirect effect of teaching presence on
academic outcomes through cognitive presence and supports the constructivist model of learning
that describes the role of the instructor as the person who influences the development of a
learning environment where students can experience cognitive presence by facilitating learning
activities where students can actively engage with the course material in order to construct their
own understanding.
Examination of the direct and indirect effects in the research model indicates that math
ability has the largest influence on academic outcomes. Of the three learning environment
factors, teaching presence and cognitive presence influence both student satisfaction and
academic outcomes directly and indirectly. In contrast, social presence does not have a large
186
influence on either satisfaction or academic outcomes. These results indicate that, of the learning
environment factors within the instructor’s control, the most essential aspect of the learning
environment for instructors to develop is cognitive presence. The large influence of cognitive
presence on student outcomes supports the constructivist model of learning in demonstrating the
importance of having students engage in learning activities that require them to explore content,
synthesize information, construct their own explanations, and apply their knowledge to solve
problems.
Summary of Results The results described in this chapter provide a description of the classroom environment
of six sections of an introductory undergraduate chemistry courses from both the instructors’ and
students’ perspectives. The four instructors participating in this research describe approaches to
teaching which emphasize the creation of a student-centered constructivist learning environment
more than half the time. Students in those sections positively rate three aspects of a constructivist
learning environment as measured by teaching presence, cognitive presence, and social presence
and those ratings are consistent with instructor descriptions of the learning environment. The
consistency between the instructors’ and students’ perceptions of the classroom environment
allows for the investigation of relationships among the three aspects of a constructivist learning
environment and student outcomes of academic achievement, as measured by ACS exam scores
and final course grades, and student satisfaction.
Using SEM analysis to test the hypothesized model of relationships among the three
presence factors describing aspects of constructivist learning environment, math ability, and
student outcomes of academic achievement and satisfaction demonstrates that math ability has
187
the largest influence on academic achievement and that only the cognitive presence aspect of the
learning environment directly influences academic achievement. While neither teaching presence
nor social presence directly influences academic achievement, both types of presence play a role
in supporting the development of cognitive presence and therefore both indirectly influence
academic achievement. Additionally, student satisfaction is directly influenced by both cognitive
presence and teaching presence and indirectly influenced by social presence to a small degree,
but satisfaction does not appear to have a relationship with final course grades beyond the causal
relationships present in the model. These results highlight the important role of the instructor in
creating the constructivist learning environment that fosters a high degree of cognitive presence
for the students which ultimately influences both academic outcomes and student satisfaction.
The influence of a constructivist learning environment on student outcomes will be examined in
Chapter 5 as it relates to implications for teaching introductory undergraduate chemistry courses.
188
Chapter 5
The primary goal of this research is to examine relationships among constructivist
learning environment factors and student outcomes in introductory undergraduate chemistry
courses. In order to examine these relationships, it was necessary to determine if the Community
of Inquiry (CoI) student survey instrument provides an acceptable measurement of student
perceptions of indicators of a constructivist learning environment. The acceptability of the CoI
student survey instrument is addressed by examining evidence for both the validity and the
reliability of the CoI survey scores. Validity evidence is generated from the internal structure of
the CoI instrument in addition to relationships between CoI responses and instructor data
collected in this research. Reliability evidence comes from determination of Cronbach’s alpha
and coefficient H values for the teaching presence, cognitive presence, and social presence
scales.
Once the acceptability of the CoI instrument is determined, the primary research model is
tested. This model contains hypothesized relationships among constructivist learning
environment factors and student outcomes of satisfaction and academic achievement. As shown
in Chapter 4, this model has an acceptable data-model fit though not all hypothesized
relationships among variables are found to be statistically significant. In this chapter, the
conclusions drawn from the results presented in Chapter 4 are discussed within the context of
existing literature related to constructivist learning environments. Implications for teaching,
limitations of the current research, and future research directions are also presented.
189
Contribution of Results to Existing Literature
Instructor and Student Ratings of Learning Environment
The two survey instruments used in this research, the Approaches to Teaching Inventory
(ATI) and the CoI, are drawn from existing instruments but required modifications in order to be
used in this particular research context with instructors and students in face-to-face introductory
undergraduate chemistry courses at US institutions. The ATI was originally developed for use in
Australia/Europe and the CoI was developed by Canadian researchers for use in online courses.
As a result of the instrument modifications, it was necessary to conduct pilot studies including
think-aloud interviews to ensure the items are being interpreted as intended by the target
population. These pilot studies resulted in additional modifications to the ATI and the CoI before
they could be used to collect data in the main research study.
At every stage of use, evidence for the validity of the ATI and CoI scores was examined
to test whether the scores were providing acceptable measurements of the learning environment
from both the instructors’ and students’ perspectives. Some evidence for the validity of the CoI
scores results from triangulating student responses with the course instructors’ descriptions of
their approach to teaching from their ATI survey responses. This investigation included
examining instructors’ conceptual change student-focused (CCSF) scale scores and interview
responses along with information available in instructors’ course syllabi. The results presented in
Chapter 4 indicate that student and instructor perceptions of the learning environment are
generally aligned. This alignment is demonstrated by the students perceiving indicators of a
constructivist learning environment while the instructors describe approaches to teaching that are
consistent with a student-centered constructivist approach to teaching.
190
Additional validity evidence is provided by testing the hypothesized internal structure of
the CoI instrument in the main study using confirmatory factor analysis (CFA). The
hypothesized model of the CoI instrument has acceptable data-model fit after only minor
modifications including the addition of four error covariance terms between CoI items and the
exclusion of four items found to be more closely related to activities performed by the instructor
outside of the classroom environment. In this model, the CoI items clearly load on the individual
factors they are hypothesized to be related to which were teaching presence, social presence, and
cognitive presence. Using this version of the CoI instrument, the reliability of the scales is
calculated in two ways. Cronbach’s alpha values assess the internal consistency of responses to
items on the presence scales while coefficient H values represent the reliability of each of the
underlying presence factors. Both measures of reliability are found to be well above the
generally accepted threshold values.
These results indicate that the modified CoI instrument and the CCSF scale of the
modified ATI are appropriate instruments for measuring aspects of a constructivist learning
environment created by the instructor and experienced by the students in face-to-face
introductory undergraduate chemistry courses. The ATI had previously been used in chemical
education research (Stains et al., 2015), but the current research indicates that perhaps only the
CCSF scale provides an acceptable measurement of student-centered approaches to teaching in
introductory chemistry courses since it measures the self-reported frequency of instructor
behaviors such as facilitating discussions and encouraging students to develop their own
understanding of course material. The modification of the CoI instrument for use with students in
a face-to-face learning environment represents the first measure of aspects of a constructivist
191
learning environment in chemical education research, and therefore provides a new tool for use
by researchers.
Influence of a Constructivist Learning Environment on Student Outcomes
The main goal of this research is to determine if it is possible to measure specific aspects
of a constructivist learning environment in order to provide more detailed information regarding
how a constructivist learning environment affects student outcomes. Prior to this research, the
existing literature had provided evidence that the adoption of constructivist teaching practices led
to improvements in student outcomes, but there was little evidence to support an explanation of
how the teaching practices were influencing student outcomes. As a result, constructivism had
become “little more than an educational slogan in the absence of conceptual understanding”
(Hyslop-Margison & Strobel, 2008, p. 73).
Constructivism is used as a theoretical framework in this research to build a model of
how aspects of a constructivist learning environment are hypothesized to affect student outcomes
of academic achievement and satisfaction. The model of a constructivist learning environment
described by the Community of Inquiry (D. R. Garrison et al., 2000) provides support for the
deconstruction of constructivism into three interrelated aspects: teaching presence, cognitive
presence, and social presence. These aspects of a constructivist learning environment are
hypothesized to influence each other as well as other student outcomes such as student
satisfaction with the course, content knowledge, and final course grades. While not all of the
hypothesized relationships are found to be present in the student data used in this research, the
relationships that are present support a constructivist model of learning in which students are
192
responsible for constructing their own understanding. The results also support the role of the
instructor in the constructivist model of learning which is to facilitate the knowledge
construction process by creating a learning environment in which students can actively engage
with the course material.
The results of testing the hypothesized research model with 391 students enrolled in six
sections of an introductory undergraduate chemistry course taught by four different instructors
demonstrate that math ability plays the largest role in influencing student academic outcomes as
measured by American Chemical Society (ACS) exam scores and final course grades. The strong
influence of math ability on student academic achievement in chemistry is well documented in
the chemical education literature (Lewis & Lewis, 2005; Mitchell et al., 2012; Nordstrom, 1990;
Tien et al., 2002; Xu & Lewis, 2011). This result is further supported by examination of the ACS
exam taken by the students (Form GC15FG) in which approximately one third of the 70 items
were classified by the researcher and an instructor involved in the research as primarily
emphasizing mathematical manipulation to arrive at a correct answer.
While the incoming math ability of the students is largely outside of the control of the
instructor, except in situations where a math requirement is placed on student enrollment in the
course, the instructor does have control over the type of learning environment experienced by the
students. Of the three presence factors measured by the CoI instrument used in the research
model, cognitive presence is shown to have the most influence on student academic outcomes
and satisfaction. Cognitive presence is the aspect of the learning environment most directly
aligned with the focus of the constructivist model of learning. The constructivist model
emphasizes individual knowledge construction through active engagement with information and
193
assimilation of that information within existing cognitive structures (Ausubel, 1960; Norman,
1980; Piaget, 1964/1997; Vygotsky, 1978). This large influence of cognitive presence on the
academic outcomes used in the research model provides evidence to support the constructivist
model of learning.
The research model also provides support for the role that teaching presence and social
presence play in influencing cognitive presence. In this way, both the instructor and students in
the learning environment indirectly influence student academic outcomes by supporting the
development of cognitive presence. These results highlight the importance of the instructor in
creating a learning environment that provides students with opportunities to construct their own
understanding of course material. As described in the constructivist model of learning, the role of
the instructor does shift to that of a facilitator of learning, but this is no way deemphasizes the
importance of the instructor, instead it speaks to the need for the instructor to have both content
and pedagogical knowledge of student learning (Bodner et al., 2001; Coll & Taylor, 2001;
Piaget, 1973).
The influence of both teaching presence and social presence on cognitive presence also
addresses some misconceptions surrounding how constructivist learning environments should be
structured to support the development of cognitive presence. The role of the instructor in
supporting cognitive presence demonstrates that developing a constructivist learning
environment does not imply that students should be left completely on their own to discover
knowledge (Bodner et al., 2001; Committee on Developments in the Science of Learning, 2000;
Matthews, 1993; Mugaloglu, 2014; Windschitl, 2002). Instead, in a constructivist learning
194
environment the instructor provides the structure and feedback that allows students to construct
their own knowledge, while still providing support and guidance when needed.
The role of the instructor in supporting the development of both social presence and
cognitive presence demonstrated by the research model is in alignment with Vygotsky’s (1978)
zone of proximal development (ZPD) in which the presence of a more capable adult or peer is
necessary to support a student solving a problem that is above his or her current development
level. The lack of a direct influence of social presence on student academic outcomes in the
research model addresses the common misconception that any type of group work will improve
student learning. In the research model, social presence only influences student academic
outcomes insofar as it supports the development of cognitive presence. This result is consistent
with Vygotsky’s theory of the ZPD in that group work can support students’ ability to solve
difficult problems. However, ultimately students must be able to solve the problems for
themselves in order to perform well on individual assessments. The research model shows that
instructors can influence the development of student’s problem solving ability directly through
their interactions with individual students but also indirectly by using social presence to create a
collaborative learning environment where students work together to create a community of
learners who help each other to construct knowledge. In addition to influencing student academic
outcomes, the research model also addresses how the three presence factors influence student
satisfaction.
The influence of the constructivist learning environment on student satisfaction in the
research model is primarily a result of the direct effects that both teaching presence and cognitive
presence have on student satisfaction. While social presence was hypothesized to directly
195
influence satisfaction no evidence to support this relationship is found from testing the model
with the data used in this research. One interpretation of these results is that because teaching
presence and cognitive presence have similar influences on student satisfaction, it may not be
necessary for instructors to be overly concerned if students are initially unhappy with taking
more control over their own learning in a constructivist learning environment. The results
suggest that the resulting increase in cognitive presence will ultimately have a positive influence
on student satisfaction. Examining the influence of various aspects of a constructivist learning
environment on student outcomes provides instructors with more specific information regarding
how changing specific aspects of their approach to teaching may ultimately influence student
satisfaction and academic achievement.
This research model demonstrates that cognitive presence is the primary aspect of a
constructivist learning environment that influences student academic achievement and
satisfaction. In this model, teaching presence is responsible for influencing the development of
cognitive presence, with social presence playing a smaller role in influencing cognitive presence.
These results suggest that the adoption of a constructivist approach to teaching primarily
influences student outcomes by encouraging students to actively engage with the material in
order to explore, discuss, construct, test, and apply their own understandings. This active
engagement is mainly facilitated by the instructor, but can also be supported by peers when
group work is designed to encourage supportive collaboration and discussion. The ability of this
research model to provide a more detailed understanding of how a constructivist learning
environment influences student outcomes can help explain improved student outcomes when
specific pedagogies such as process-oriented guided-inquiry learning (POGIL) and peer-led team
196
learning (PLTL) are implemented in chemistry classrooms (Gosser et al., 2010; Hanson, 2006,
2008; Lewis & Lewis, 2005; Mitchell et al., 2012; Tien et al., 2002; Varma-Nelson & Banks,
2013; Varma-Nelson & Coppola, 2005). The results of this study can also help explain the
effectiveness of the adoption of more general constructivist approaches to teaching in chemistry
courses (Freeman et al., 2014; Gupta et al., 2015; Hall et al., 2014) as well as recommendations
for the use of constructivist approaches to teaching online courses (Bangert, 2008; Vrasidas,
2000). Previously, it was unclear which aspects of pedagogies such as POGIL and PLTL are
responsible for improvements in student outcomes. This research indicates that cognitive
presence has the most influence on student outcomes. With this knowledge, instructors can
consider other teaching approaches, besides POGIL and PLTL, which can foster cognitive
presence which may provide for more gradual implementation of constructivist approaches to
teaching as opposed to the more encompassing constructivist approaches of POGIL and PLTL
that when implemented involve a major reorganization of an instructor’s approach to teaching.
The final contribution this research makes is to highlight the benefits that can be gained
from looking to other fields for theoretical foundations, instruments, and methodologies. The CoI
model and student survey instrument were originally developed by online education researchers
but can be applied to chemical education research due to the fact that the two fields share a
foundation in constructivism. The use of structural equation modeling (SEM) as methodology for
developing and testing a research model is much more prevalent in online education research (D.
R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009), but is becoming more widely
used in chemical education research, primarily as a way to develop and test survey instruments
(Xu & Lewis, 2011). SEM is a versatile analysis tool that can be used to examine complex
197
questions and causal relationships among variables with relatively simple data collection
procedures. Use of SEM in chemical education research could help examine many other
multifaceted relationships among variables which is becoming a more common analysis practice
in the field.
Implication of Results for Teaching Introductory Undergraduate Chemistry The results of testing the final structural equation model indicate that the adoption of
approaches to teaching corresponding to cognitive presence items on the CoI instrument could
contribute to improvements in student academic outcomes, even after the effects of math ability
are considered. These teaching approaches include having students construct their own
explanations and solutions rather than copy those provided by the instructor or textbook and
asking students to explore problems related to their life outside the classroom thus providing
opportunities for students to apply their chemistry knowledge to new situations. Though these
teaching approaches are often incorporated into specific pedagogies such as POGIL and PLTL,
the fact that none of the instructors in this study utilize these specific pedagogies demonstrates
that aspects of cognitive presence can be incorporated into the classroom without adopting a
completely new pedagogy. That is, this research suggests that a more general adoption of
constructivist teaching approaches positively influences student outcomes, as had been seen in
other studies (Freeman et al., 2014; Gupta et al., 2015; Hall et al., 2014), but this research is the
first to attribute the effect directly to the cognitive presence aspect of the learning environment.
While the adoption of a constructivist teaching approach must be supported and
facilitated by the course instructor, the nonsignificant direct effects of teaching presence on both
198
ACS exam scores and final course grades indicates that student academic success is not directly
influenced by specific instructor behaviors identified by students. Instead the significance of the
indirect effect of teaching presence on academic outcomes through cognitive presence indicates
that instructor behaviors influence the development of a constructivist learning environment by
creating the conditions for cognitive presence to occur. This interpretation is supported by the
large and significant direct effect of teaching presence on cognitive presence. For classroom
instructors, these results suggest that classroom behaviors such as facilitating discussions and
providing feedback are important only insofar as they contribute to creating an environment that
fosters student construction and application of knowledge as measured by cognitive presence. To
support construction and application of knowledge instructors could consider asking students to
explain the reasoning behind the processes used to solve problems in order to provide evidence
that students have internalized the knowledge necessary to solve the problem or instructors could
utilize problems that require student to apply multiple concepts simultaneously.
Satisfaction was the only student outcome on which teaching presence has a significant
direct effect. This direct effect is slightly smaller than the direct effect of cognitive presence, but
demonstrates that while specific instructor behaviors may make students feel more satisfied with
the course, students are also satisfied when they are engaged in constructing their own
knowledge, as measured by cognitive presence. The nonsignificant correlation between the
residual error terms for student satisfaction and final course grades illustrates that the causal
relationships between other variables in the model explains the correlation between satisfaction
and grades without requiring a direct link between satisfaction and grades. These results may
encourage instructors who initially receive negative student feedback while switching to
199
constructivist teaching approaches that place the instructor in the role of a facilitator. If students
are unfamiliar with being active participants in their own learning, they may at first be
uncomfortable with the instructor in this facilitator role but this initial dissatisfaction does not
necessarily preclude student academic success.
The minimal direct influence of social presence on satisfaction and student academic
outcomes indicates that, on its own, students’ comfort while interacting with their peers does not
play a large role in their affective or academic outcomes. This result was unexpected, but the lack
of direct influence of social presence on student satisfaction had been seen in previous research
with the CoI instrument (Joo et al., 2011). The largest effect of social presence is its direct effect
on cognitive presence. As a result, social presence indirectly influences satisfaction and
academic outcomes through cognitive presence. These results suggest that collaborative work or
discussions that encourage the development of cognitive presence through the use of activities
that foster active student engagement with material in support of constructing explanations and
understanding will ultimately influence student satisfaction and academic achievement. This
conclusion addresses a common misconception surrounding social constructivism, which is the
idea that simply having students work in groups will improve their achievement and satisfaction.
Based on the model tested in this research, classroom instructors should implement purposeful
group work or discussions with careful thought as to how the activity supports the development
of a constructivist learning environment.
This type of structured group work is used in both POGIL and PLTL teaching
approaches. The benefits to structured group work include the opportunity for students to
become comfortable working with various personalities and learn to work together towards a
200
common learning goal. A critical aspect of structured group work is holding students accountable
for the work that is done so that all members are encouraged to participate and all members have
an opportunity to engage in the learning process. Since it is ultimately the individual who must
construct his or her own knowledge, it is important that group work is structured to encourage
and support individual knowledge construction within the context of a larger group. This
encouragement and support speaks to the role of the instructor in facilitating structured group
work and connects back to the direct influence that teaching presence has on social presence in
the research model.
Limitations and Future Research The small number of instructors participating in this research limits the ability to provide
additional evidence for the validity of the ATI scores. Since the ATI has been extensively revised
from its original form it would be important to see if the internal structure of the revised
instrument matches its original two-factor structure. It is also important to test whether the
revised instrument scores still have a relationship to scores from observational protocols such as
the Reformed Teaching Observation Protocol (RTOP) and Classroom Observation Protocol for
Undergraduate STEM (COPUS). To test the internal structure of the ATI it would be best to
obtain a sample of chemistry instructors representing a variety of different chemistry courses
from the introductory undergraduate level through the graduate level. Comparing a larger set of
ATI scores to RTOP or COPUS scores would provide additional evidence regarding whether the
ATI is an accurate measurement of an instructor’s approach to teaching. This larger data set of
ATI responses would also provide more information on whether underreporting of adoption of
constructivist approaches to teaching is a common occurrence. The possibility of underreporting
201
a constructivist approach to teaching arose because the instructor in the main research study who
described the most student-centered constructivist approach to teaching did not have the highest
self-rating on the CCSF scale. Without a larger sample size, it is difficult to know if this result is
related to the particular instructor’s self-perception or if there is a more general issue with
instructors underreporting the adoption of student-centered approaches to teaching.
Limitations also exist related to providing evidence for the validity of the CoI scores. The
think-aloud interviews in the pilot study were conducted with the participation of Chemistry
Club students who may not have been representative of the more general population of students
enrolled in introductory undergraduate chemistry courses. Think-aloud interviews were not
conducted with the students in the data set whose responses to the CoI were analyzed in the main
portion of the research. It would be important for future research to conduct think-aloud
interviews with students enrolled in introductory chemistry courses at a variety of institutions in
order to provide validity evidence based on the response processes for a more diverse group of
students.
Another limitation of this research is the relative homogeneity of the approaches to
teaching utilized by the instructors whose teaching practices and student data were analyzed.
Though this homogeneity was helpful in minimizing differences in the university setting, course
materials, and student backgrounds so that the student data could be combined into a single data
set, it also limits the generalizability of this research. All four participating instructors actively
engage in chemical education research and are therefore familiar with the benefits of more
student-centered approaches to teaching. This is reflected in their CCSF scores, course syllabi,
and descriptions of teaching practices. Given this limitation, it is important for future research to
202
obtain an independent data set to cross-validate the results of this model. This cross-validation
would investigate whether or not the current research model is over fitted to the characteristics of
this particular sample.
While the size of the combined data set is large enough to test the model focal parameters
with sufficient power, there are not enough students represented in the data set to permit testing
the model separately for each instructor. Though the approaches to teaching described by the
instructors are relatively homogeneous, only one instructor uses formal structured student groups
which remained constant throughout the semester. With a larger sample of students and
instructors, it would be possible to test separate models for classrooms which utilize formal and
informal groups. This would provide information about whether the influence of social presence
changes based on the level of structure of student groups. Incorporating other measurement
techniques such as the RTOP or the COPUS could also help determine if the CoI scores are
sensitive to changes in approach to teaching across different learning environments, as was
demonstrated for the ATI in other literature (Stains et al., 2015).
Since the current research provides evidence for a relationship among the CoI presence
factors and student outcomes in a single chemistry course taught by multiple instructors, future
research should gather data from additional chemistry courses in which a wider variety of
approaches to teaching have been adopted. It would be especially useful to include classrooms
taught using known constructivist teaching approaches such as POGIL and PLTL and classrooms
taught using a more traditional lecture-based approach. This would allow the research model to
be tested more broadly and provide additional evidence either supporting or modifying the
relationships examined in this study.
203
If the relationships identified in this study are evident in a larger sample of classrooms,
then specific interventions could be developed to improve the aspects of the learning
environment that are identified as having the greatest influence on student outcomes, i.e.
cognitive presence and teaching presence. These interventions would not necessarily require the
development of new learning activities, but could simply help instructors identify specific
practices that would help them improve cognitive presence and teaching presence in their own
classrooms. As an example, many instructors recognize the importance of having students solve
problems during class, but may not realize that providing students an opportunity to solve
problems on their own, as opposed to watching the instructor solve the problem, encourages the
development of cognitive presence by providing students with an opportunity to engage with the
material by testing and applying their understanding. By helping the instructor understand why
problem solving is important, the instructor could be guided towards ways of implementing
problem solving that provide additional opportunities for the development of cognitive presence.
These opportunities could include asking more complex multi-stage problems that provide
students with opportunities to incorporate knowledge from a variety of topics or more open-
ended authentic problems based in real-world scenarios that may have more than one possible
solution. In this way the instructor would have a better understanding of how different types of
classroom activities influence student outcomes. This type of intervention would not require the
development of entirely new instructional materials, but simply provide the basis for a better
choice of teaching approaches. Ultimately, it is hoped that by providing more detailed
information about how aspects of a constructivist learning environment influence student
204
outcomes, instructors have a stronger research-based foundation available when selecting a
teaching approach that best fits their needs while also improving student outcomes.
App
endi
x A
– O
rigi
nal C
oI It
ems a
nd L
oadi
ngs
Tabl
e 20
O
rigi
nal C
oI It
em W
ordi
ngs a
nd P
ublis
hed
Load
ings
, Pat
hs, o
r Cor
rela
tions
O
rigi
nal C
oI In
stru
men
t Ite
m
Publ
ishe
d L
oadi
ngs*
Teac
hing
Pre
senc
e A
B
C
D
E
F G
1.
The
inst
ruct
or c
lear
ly c
omm
unic
ates
impo
rtant
cou
rse
topi
cs
0.71
0.
826
–0.8
79
–0.8
8 0.
80
0.80
3 0.
75
2.
The
inst
ruct
or c
lear
ly c
omm
unic
ates
impo
rtant
cou
rse
goal
s 0.
71
0.87
7 –0
.891
–0
.84
0.78
0.
829
0.71
3.
The
inst
ruct
or p
rovi
des c
lear
inst
ruct
ions
on
how
to p
artic
ipat
e in
co
urse
lear
ning
act
iviti
es
0.67
0.
592
–0.8
75
–0
.80
0.75
0.
722
0.70
4.
The
inst
ruct
or c
lear
ly c
omm
unic
ates
impo
rtant
due
dat
es/ti
me
fram
es fo
r lea
rnin
g ac
tiviti
es
0.
611
–0.8
71
–0.7
4 0.
69
0.52
5 0.
55
5.
The
inst
ruct
or is
hel
pful
in id
entif
ying
are
as o
f agr
eem
ent a
nd
disa
gree
men
t on
cour
se to
pics
that
hel
ped
me
to le
arn
0.
579
–0.8
76
–0.8
6 0.
87
0.69
7 0.
64
6.
The
inst
ruct
or is
hel
pful
in g
uidi
ng th
e cl
ass t
owar
ds u
nder
stan
ding
co
urse
topi
cs in
a w
ay th
at h
elps
me
clar
ify m
y th
inki
ng
0.83
# 0.
575
–0.8
36
–0.8
7 0.
90
0.74
0/0.
651+
0.86
/0.
72+
7.
The
inst
ruct
or h
elps
to k
eep
cour
se p
artic
ipan
ts e
ngag
ed a
nd
parti
cipa
ting
in p
rodu
ctiv
e di
alog
ue
0.86
0.
633
–0.7
00
–0.8
5 0.
88
0.68
5 0.
74
8.
The
inst
ruct
or h
elps
kee
p th
e co
urse
par
ticip
ants
on
task
in a
way
th
at h
elps
me
to le
arn
0.86
0.
579
–0.7
04
–0.8
7 0.
90
0.70
5/0.
758+
0.78
/0.
82+
9.
The
inst
ruct
or e
ncou
rage
s cou
rse
parti
cipa
nts t
o ex
plor
e ne
w
conc
epts
in th
is c
ours
e
0.52
3 –0
.555
–0
.77
0.85
0.
689
0.72
10.
Inst
ruct
or a
ctio
ns re
info
rced
the
deve
lopm
ent o
f a se
nse
of
com
mun
ity a
mon
g co
urse
par
ticip
ants
0.
67#
0.56
9 0.
591
–0.7
9 0.
87
0.64
5 0.
74
205
Ori
gina
l CoI
Inst
rum
ent I
tem
Pu
blis
hed
Loa
ding
s*
Teac
hing
Pre
senc
e A
B
C
D
E
F G
11.
The
inst
ruct
or h
elps
to fo
cus d
iscu
ssio
n on
rele
vant
issu
es in
a w
ay
that
hel
ps m
e to
lear
n 0.
81
0.42
5 –0
.682
–0
.74
0.85
0.
645
0.83
12.
The
inst
ruct
or p
rovi
des f
eedb
ack
that
hel
ps m
e un
ders
tand
my
stre
ngth
s and
wea
knes
ses (
rela
tive
to th
e co
urse
’s g
oals
and
ob
ject
ives
)#
0.80
# 0.
649#
–0.5
80
–0.7
5 0.
89
13.
The
inst
ruct
or p
rovi
des f
eedb
ack
in a
tim
ely
fash
ion
0.
513
–0.6
63
–0.7
5 0.
74
0.55
7 0.
44
Soci
al P
rese
nce
14.
Get
ting
to k
now
oth
er c
ours
e pa
rtici
pant
s giv
es m
e a
sens
e of
be
long
ing
in th
e co
urse
. 0.
55#
0.61
9 –0
.385
–0
.41
0.59
0.
576
0.68
15.
I am
abl
e to
form
dis
tinct
impr
essi
ons o
f som
e co
urse
par
ticip
ants
0.
55
0.47
3 –0
.440
–0
.40
0.58
0.
423
0.41
16.
Onl
ine
or w
eb-b
ased
com
mun
icat
ion
is a
n ex
celle
nt m
ediu
m fo
r so
cial
inte
ract
ion
0.63
0.
674
–0.6
07
–0.5
0 0.
62
0.56
2 0.
49
17.
I fel
t com
forta
ble
conv
ersi
ng th
roug
h th
e on
line
med
ium
0.
83
0.81
4 –0
.837
–0
.81
0.85
0.
789
0.68
18.
I fee
l com
forta
ble
parti
cipa
ting
in th
e co
urse
dis
cuss
ions
0.
79
0.78
8 –0
.854
–0
.87
0.86
0.
781
0.80
19.
I fee
l com
forta
ble
inte
ract
ing
with
oth
er c
ours
e pa
rtici
pant
s 0.
81
0.70
1 –0
.850
–0
.94
0.91
20.
I fee
l com
forta
ble
disa
gree
ing
with
oth
er c
ours
e pa
rtici
pant
s whi
le
still
mai
ntai
ning
a se
nse
of tr
ust
0.
620
–0.8
15
–0.7
8 0.
77
0.62
0 0.
59
21.
I fee
l tha
t my
poin
t of v
iew
is a
ckno
wle
dged
by
othe
r cou
rse
parti
cipa
nts
0.71
0.
556
–0.6
90
–0.7
8 0.
78
0.61
3 0.
67
22.
Onl
ine
disc
ussi
ons h
elp
me
to d
evel
op a
sens
e of
col
labo
ratio
n
0.56
1 –0
.643
–0
.75
0.77
0.
509
0.70
206
Ori
gina
l CoI
Inst
rum
ent I
tem
Pu
blis
hed
Loa
ding
s*
Cog
nitiv
e Pr
esen
ce
A
B
C
D
E F
G
23.
Prob
lem
s pos
ed in
crea
se m
y in
tere
st in
cou
rse
issu
es
–0
.785
0.
545
0.67
0.
72
0.79
1 0.
74
24.
Cou
rse
activ
ities
piq
ue m
y cu
riosi
ty
–0
.712
0.
671
0.75
0.
80
0.75
5 0.
77
25.
I fee
l mot
ivat
ed to
exp
lore
con
tent
rela
ted
ques
tions
–0.7
70
0.70
2 0.
79
0.81
0.
825
0.77
26.
I util
ize
a va
riety
of i
nfor
mat
ion
sour
ces t
o ex
plor
e pr
oble
ms p
osed
in
this
cou
rse
–0
.759
0.
681
0.72
0.
72
0.39
8 0.
50
27.
Bra
inst
orm
ing
and
findi
ng re
leva
nt in
form
atio
n he
lps m
e re
solv
e co
nten
t rel
ated
que
stio
ns
–0
.794
0.
751
0.74
0.
75
0.59
7 0.
66
28.
Onl
ine
disc
ussi
on w
ere
valu
able
in h
elpi
ng m
e ap
prec
iate
diff
eren
t pe
rspe
ctiv
es
–0
.699
0.
426
0.44
0.
72
0.55
9 0.
70
29.
Com
bini
ng n
ew in
form
atio
n he
lps m
e an
swer
que
stio
ns ra
ised
in
cour
se a
ctiv
ities
–0.7
16
0.69
8 0.
74
0.84
0.
654
0.71
30.
Lear
ning
act
iviti
es h
elp
me
cons
truct
exp
lana
tions
/sol
utio
ns
–0
.732
0.
717
0.76
0.
84
0.65
5 0.
73
31.
Ref
lect
ion
on c
ours
e co
nten
t and
dis
cuss
ions
hel
ped
me
unde
rsta
nd
fund
amen
tal c
once
pts i
n th
is c
lass
–0.6
40
< 0.
3 0.
75
0.85
0.
590
0.76
32.
I can
des
crib
e w
ays t
o te
st a
nd a
pply
the
know
ledg
e cr
eate
d in
this
co
urse
0.
77
–0.6
19
0.77
4 0.
81
0.78
0.
534
0.57
33.
I hav
e de
velo
ped
solu
tions
to c
ours
e pr
oble
ms t
hat c
an b
e ap
plie
d in
pra
ctic
e
–0.6
53
0.79
7 0.
84
0.78
0.
587
0.51
34.
I can
app
ly th
e kn
owle
dge
crea
ted
in th
is c
ours
e to
my
wor
k or
ot
her n
on-c
lass
rela
ted
activ
ities
0.
82
–0.6
87
0.74
5 0.
74
0.75
0.
689
0.58
207
Rel
atio
nshi
ps A
mon
g C
oI P
rese
nce
Fact
ors
Publ
ishe
d C
orre
latio
ns o
r St
anda
rdiz
ed P
aths
*
A
B
C
D
E
F G
H
Teac
hing
pre
senc
e an
d so
cial
pre
senc
e 0
–0.3
18
–0
.49
0.52
0.52
0.
81
Soci
al p
rese
nce
and
cogn
itive
pre
senc
e 0
–0.5
68
–0
.70
0.52
0.40
0.
30
Teac
hing
pre
senc
e an
d co
gniti
ve p
rese
nce
0 –0
.479
–0.6
9 0.
49
0.
51
0.65
Rel
atio
nshi
ps B
etw
een
CoI
Pre
senc
e Fa
ctor
s an
d St
uden
t Out
com
es
Teac
hing
pre
senc
e an
d sa
tisfa
ctio
n 0.
16
0.24
Soci
al p
rese
nce
and
satis
fact
ion
0.47
Cog
nitiv
e pr
esen
ce a
nd sa
tisfa
ctio
n 0.
04
0.26
Teac
hing
pre
senc
e an
d pe
rcei
ved
lear
ning
0.
51
Soci
al p
rese
nce
and
perc
eive
d le
arni
ng
0.19
Cog
nitiv
e pr
esen
ce a
nd p
erce
ived
lear
ning
0.
55
*A
= E
FA fr
om A
rbau
gh (2
008)
; B =
PC
A fr
om A
rbau
gh e
t al.
(200
8); C
= P
CA
from
Arb
augh
et a
l. (2
010)
; D
= P
AF
from
She
a &
Bid
jera
no (2
009)
; E =
SEM
from
She
a &
Bid
jera
no (2
009)
; F =
EFA
from
Gar
rison
et a
l. (2
010)
; G
= S
EM fr
om G
arris
on e
t al.
(201
0); H
= S
EM fr
om Jo
o et
al.
(201
1)
# Item
had
slig
ht w
ordi
ng d
iffer
ence
in th
is st
udy,
but
the
gene
ral i
nten
t was
sim
ilar
+ Som
e ite
ms w
ere
liste
d tw
ice
the
Gar
rison
et a
l. (2
010)
resu
lts w
ith d
iffer
ent l
oadi
ng v
alue
s
208
App
endi
x B
– A
TI R
evis
ed fo
r Pi
lot S
tudy
D
irec
tions
: Thi
s inv
ento
ry is
des
igne
d to
exp
lore
a d
imen
sion
of t
he w
ay th
at a
cade
mic
s go
abou
t tea
chin
g in
a sp
ecifi
c co
ntex
t or
subj
ect o
r cou
rse.
Thi
s may
mea
n th
at y
our r
espo
nses
to th
ese
item
s in
one
cont
ext m
ay b
e di
ffer
ent t
o th
e re
spon
ses y
ou m
ight
m
ake
on y
our t
each
ing
in o
ther
con
text
s or s
ubje
cts.
Pl
ease
con
side
r the
lect
ure
porti
on th
e 10
0-le
vel c
hem
istry
cou
rse
you
have
taug
ht m
ost r
ecen
tly a
t CU
A. U
se th
at c
ours
e as
a
refe
renc
e fo
r com
plet
ing
this
surv
ey. P
leas
e re
frai
n fr
om st
atin
g th
e na
me
of th
e co
urse
or y
our n
ame
alou
d.
For e
ach
item
ple
ase
circ
le o
ne o
f the
num
bers
(1–5
). Th
e nu
mbe
rs st
and
for t
he fo
llow
ing
resp
onse
s:
1.
this
item
was
onl
y ra
rely
or n
ever
true
for m
e in
this
cou
rse.
2.
th
is it
em w
as so
met
imes
true
for m
e in
this
cou
rse.
3.
th
is it
em w
as tr
ue fo
r me
abou
t hal
f the
tim
e in
this
cou
rse.
4.
th
is it
em w
as fr
eque
ntly
true
for m
e in
this
cou
rse.
5.
th
is it
em w
as a
lmos
t alw
ays o
r alw
ays t
rue
for m
e in
this
cou
rse.
Plea
se a
nsw
er e
ach
item
. Do
not s
pend
a lo
ng ti
me
on e
ach:
you
r fir
st r
eact
ion
is p
roba
bly
the
best
one
.
Only rarely
Sometimes
About half the time
Frequently
Almost always
1.
In th
is c
ours
e st
uden
ts sh
ould
focu
s the
ir st
udy
on w
hat I
pro
vide
them
.
1
2
3
4
5
2.
It is
impo
rtant
that
this
cou
rse
shou
ld b
e co
mpl
etel
y de
scrib
ed in
term
s of s
peci
fic
obje
ctiv
es th
at re
late
to fo
rmal
ass
essm
ent i
tem
s.
1
2
3
4
5
3.
In m
y in
tera
ctio
ns w
ith st
uden
ts in
this
cou
rse
I try
to d
evel
op a
con
vers
atio
n w
ith
them
abo
ut th
e to
pics
we
are
stud
ying
.
1
2
3
4
5
4.
It is
impo
rtant
to p
rese
nt a
lot o
f fac
ts to
stud
ents
so th
at th
ey k
now
wha
t the
y ha
ve
to le
arn
for t
his c
ours
e.
1
2
3
4
5
5.
I set
asi
de so
me
teac
hing
tim
e so
that
the
stud
ents
can
dis
cuss
, am
ong
them
selv
es,
key
conc
epts
and
idea
s in
this
cou
rse.
1
2
3
4
5
209
Only rarely
Sometimes
About half the time
Frequently
Almost always
6.
In th
is c
ours
e I c
once
ntra
te o
n co
verin
g th
e in
form
atio
n th
at m
ight
be
avai
labl
e fr
om k
ey te
xts a
nd re
adin
gs.
1
2
3
4
5
7.
I enc
oura
ge st
uden
ts to
rest
ruct
ure
thei
r exi
stin
g kn
owle
dge
in te
rms o
f the
new
w
ay o
f thi
nkin
g ab
out t
he su
bjec
t tha
t the
y w
ill d
evel
op.
1
2
3
4
5
8.
In te
achi
ng se
ssio
ns fo
r thi
s sub
ject
, I d
elib
erat
ely
prov
oke
deba
te a
nd d
iscu
ssio
n.
1
2
3
4
5
9.
I s
truct
ure
my
teac
hing
in th
is c
ours
e to
hel
p st
uden
ts to
pas
s the
form
al a
sses
smen
t ite
ms.
1
2
3
4
5
10. I
thin
k an
impo
rtant
reas
on fo
r run
ning
teac
hing
sess
ions
in th
is c
ours
e is
to g
ive
stud
ents
a g
ood
set o
f not
es.
1
2
3
4
5
11. I
n th
is c
ours
e, I
prov
ide
the
stud
ents
the
info
rmat
ion
they
will
nee
d to
pas
s the
fo
rmal
ass
essm
ents
.
1
2
3
4
5
12. I
shou
ld k
now
the
answ
ers t
o an
y qu
estio
ns th
at st
uden
ts m
ay p
ut to
me
durin
g th
is
cour
se.
1
2
3
4
5
13. I
mak
e av
aila
ble
oppo
rtuni
ties f
or st
uden
ts in
this
cou
rse
to d
iscu
ss th
eir c
hang
ing
unde
rsta
ndin
g of
the
subj
ect.
1
2
3
4
5
14. I
t is b
ette
r for
stud
ents
in th
is c
ours
e to
gen
erat
e th
eir o
wn
note
s rat
her t
han
copy
m
ine.
1
2
3
4
5
15. A
lot o
f tea
chin
g tim
e in
this
cou
rse
shou
ld b
e us
ed to
que
stio
n st
uden
ts’ i
deas
.
1
2
3
4
5
16. I
n th
is c
ours
e m
y te
achi
ng fo
cuse
s on
the
good
pre
sent
atio
n of
info
rmat
ion
to
stud
ents
.
1
2
3
4
5
17. I
see
teac
hing
as h
elpi
ng st
uden
ts d
evel
op n
ew w
ays o
f thi
nkin
g in
this
subj
ect.
1
2
3
4
5
18
. In
teac
hing
this
cou
rse
it is
impo
rtant
for m
e to
mon
itor s
tude
nts’
cha
nged
un
ders
tand
ing
of th
e su
bjec
t mat
ter.
1
2
3
4
5
19. M
y te
achi
ng in
this
cou
rse
focu
ses o
n de
liver
ing
wha
t I k
now
to th
e st
uden
ts
1
2
3
4
5
20
. Tea
chin
g in
this
cou
rse
shou
ld h
elp
stud
ents
que
stio
n th
eir o
wn
unde
rsta
ndin
g of
th
e su
bjec
t mat
ter.
1
2
3
4
5
21. T
each
ing
in th
is c
ours
e sh
ould
incl
ude
help
ing
stud
ents
find
thei
r ow
n le
arni
ng
reso
urce
s.
1
2
3
4
5
22. I
pre
sent
mat
eria
l to
enab
le st
uden
ts to
bui
ld u
p an
info
rmat
ion
base
in th
is su
bjec
t.
1
2
3
4
5
210
App
endi
x C
– S
tude
nt S
urve
y It
ems U
sed
in P
ilot S
tudy
D
irec
tions
: Ple
ase
cons
ider
the
lect
ure
porti
on o
f you
r firs
t und
ergr
adua
te c
hem
istry
cou
rse
at C
UA
. Use
that
cou
rse
as a
re
fere
nce
for c
ompl
etin
g th
is su
rvey
. Ple
ase
refr
ain
from
stat
ing
the
nam
e of
the
cour
se o
r the
cou
rse
inst
ruct
or a
loud
. Whe
n yo
u ar
e re
ady,
read
eac
h st
atem
ent a
loud
, the
n se
lect
the
num
ber t
hat b
est d
escr
ibes
you
r lev
el o
f agr
eem
ent w
ith e
ach
stat
emen
t. A
fter y
ou se
lect
a n
umbe
r, br
iefly
des
crib
e w
hy y
ou se
lect
ed th
at n
umbe
r. I m
ay st
op y
ou d
urin
g th
is p
roce
ss to
ask
fo
llow
-up
ques
tions
abo
ut th
e w
ay y
ou in
terp
rete
d an
item
.
Stro
ngly
A
gree
A
gree
U
nsur
e D
isag
ree
Stro
ngly
D
isag
ree
Not
A
pplic
able
Q
1. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt c
ours
e to
pics
5
4
3
2
1
0
Q
2. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt c
ours
e go
als
5
4
3
2
1
0
Q3.
The
inst
ruct
or p
rovi
ded
clea
r ins
truct
ions
on
how
to p
artic
ipat
e in
co
urse
lear
ning
act
iviti
es
5
4
3
2
1
0
Q4.
The
inst
ruct
or c
lear
ly c
omm
unic
ated
impo
rtant
due
dat
es/ti
me
fram
es fo
r lea
rnin
g ac
tiviti
es
5
4
3
2
1
0
Q5.
The
inst
ruct
or w
as h
elpf
ul in
iden
tifyi
ng a
reas
of a
gree
men
t and
di
sagr
eem
ent o
n co
urse
topi
cs th
at h
elpe
d m
e to
lear
n 5
4
3
2
1
0
Q6.
The
inst
ruct
or w
as h
elpf
ul in
gui
ding
the
clas
s tow
ards
un
ders
tand
ing
cour
se to
pics
in a
way
that
hel
ped
me
clar
ify m
y th
inki
ng
5
4
3
2
1
0
Q7.
The
inst
ruct
or h
elpe
d to
kee
p co
urse
par
ticip
ants
eng
aged
and
pa
rtici
patin
g in
pro
duct
ive
dial
ogue
5
4
3
2
1
0
Q8.
The
inst
ruct
or h
elpe
d ke
ep th
e co
urse
par
ticip
ants
on
task
in a
way
th
at h
elpe
d m
e to
lear
n 5
4
3
2
1
0
Q9.
The
inst
ruct
or e
ncou
rage
d co
urse
par
ticip
ants
to e
xplo
re n
ew
conc
epts
in th
is c
ours
e 5
4
3
2
1
0
Q10
. The
inst
ruct
or re
info
rced
the
deve
lopm
ent o
f a se
nse
of
com
mun
ity a
mon
g co
urse
par
ticip
ants
5
4
3
2
1
0
Q11
. The
inst
ruct
or h
elpe
d to
focu
s dis
cuss
ion
on re
leva
nt is
sues
in a
w
ay th
at h
elpe
d m
e to
lear
n 5
4
3
2
1
0
Q12
. The
inst
ruct
or p
rovi
ded
feed
back
that
hel
ped
me
unde
rsta
nd m
y st
reng
ths a
nd w
eakn
esse
s rel
ativ
e to
the
cour
se’s
goa
ls a
nd
obje
ctiv
es
5
4
3
2
1
0
Q13
. The
inst
ruct
or p
rovi
ded
feed
back
in a
tim
ely
fash
ion
5
4
3
2
1
0
Q14
. Get
ting
to k
now
oth
er c
ours
e pa
rtici
pant
s gav
e m
e a
sens
e of
be
long
ing
in th
e co
urse
. 5
4
3
2
1
0
211
St
rong
ly
Agr
ee
Agr
ee
Uns
ure
Dis
agre
e St
rong
ly
Dis
agre
e N
ot
App
licab
le
Q15
. I w
as a
ble
to fo
rm d
istin
ct im
pres
sion
s of s
ome
cour
se p
artic
ipan
ts
5
4
3
2
1
0
Q16
. Fac
e-to
-fac
e co
mm
unic
atio
n is
an
exce
llent
med
ium
for s
ocia
l in
tera
ctio
n 5
4
3
2
1
0
Q17
. I fe
lt co
mfo
rtabl
e co
nver
sing
face
-to-f
ace
in c
lass
5
4
3
2
1
0
Q
18. I
felt
com
forta
ble
parti
cipa
ting
in th
e co
urse
dis
cuss
ions
5
4
3
2
1
0
Q
19. I
felt
com
forta
ble
inte
ract
ing
with
oth
er c
ours
e pa
rtici
pant
s 5
4
3
2
1
0
Q
20. I
felt
com
forta
ble
disa
gree
ing
with
oth
er c
ours
e pa
rtici
pant
s whi
le
still
mai
ntai
ning
a se
nse
of tr
ust
5
4
3
2
1
0
Q21
. I fe
lt th
at m
y po
int o
f vie
w w
as a
ckno
wle
dged
by
othe
r cou
rse
parti
cipa
nts
5
4
3
2
1
0
Q22
. In-
clas
s dis
cuss
ions
hel
ped
me
to d
evel
op a
sens
e of
col
labo
ratio
n 5
4
3
2
1
0
Q
23. P
robl
ems p
osed
incr
ease
d m
y in
tere
st in
cou
rse
issu
es
5
4
3
2
1
0
Q24
. Cou
rse
activ
ities
piq
ued
my
curio
sity
5
4
3
2
1
0
Q
25. I
felt
mot
ivat
ed to
exp
lore
con
tent
rela
ted
ques
tions
5
4
3
2
1
0
Q
26. I
util
ized
a v
arie
ty o
f inf
orm
atio
n so
urce
s to
expl
ore
prob
lem
s po
sed
in th
is c
ours
e 5
4
3
2
1
0
Q27
. Bra
inst
orm
ing
and
findi
ng re
leva
nt in
form
atio
n he
lped
me
reso
lve
cont
ent r
elat
ed q
uest
ions
5
4
3
2
1
0
Q28
. Ple
ase
sele
ct “
Dis
agre
e” fo
r thi
s ite
m
5
4
3
2
1
0
Q29
. In-
clas
s dis
cuss
ions
wer
e va
luab
le in
hel
ping
me
appr
ecia
te
diff
eren
t per
spec
tives
5
4
3
2
1
0
Q30
. Com
bini
ng n
ew in
form
atio
n he
lped
me
answ
er q
uest
ions
rais
ed in
co
urse
act
iviti
es
5
4
3
2
1
0
Q31
. Lea
rnin
g ac
tiviti
es h
elpe
d m
e co
nstru
ct e
xpla
natio
ns/s
olut
ions
5
4
3
2
1
0
Q
32. R
efle
ctio
n on
cou
rse
cont
ent h
elpe
d m
e un
ders
tand
fund
amen
tal
conc
epts
in th
is c
lass
5
4
3
2
1
0
Q33
. Ref
lect
ion
on d
iscu
ssio
ns h
elpe
d m
e un
ders
tand
fund
amen
tal
conc
epts
in th
is c
lass
5
4
3
2
1
0
Q34
. I c
an d
escr
ibe
way
s to
test
and
app
ly th
e kn
owle
dge
crea
ted
in th
is
cour
se
5
4
3
2
1
0
Q35
. I h
ave
deve
lope
d so
lutio
ns to
cou
rse
prob
lem
s tha
t can
be
appl
ied
in p
ract
ice
5
4
3
2
1
0
Q36
. I c
an a
pply
the
know
ledg
e cr
eate
d in
this
cou
rse
to m
y w
ork
or
othe
r non
-cla
ss re
late
d ac
tiviti
es
5
4
3
2
1
0
212
Dir
ectio
ns: A
gain
con
side
r jus
t the
lect
ure
porti
on o
f you
r firs
t und
ergr
adua
te c
hem
istry
cou
rse
at C
UA
and
sele
ct th
e nu
mbe
r tha
t re
pres
ents
you
r lev
el o
f agr
eem
ent w
ith e
ach
stat
emen
t. Pl
ease
read
the
stat
emen
t and
you
r cho
ice
alou
d, a
nd b
riefly
des
crib
e yo
ur
reas
on fo
r sel
ectin
g th
at re
spon
se. A
s bef
ore,
I m
ay st
op y
ou to
ask
a fo
llow
-up
ques
tion
abou
t you
r int
erpr
etat
ion
of a
par
ticul
ar
item
.
St
rong
ly
Agr
ee
Agr
ee
Uns
ure
Dis
agre
e St
rong
ly
Dis
agre
e S1
. I w
as sa
tisfie
d w
ith th
e pa
cing
of t
he c
ours
e.
5
4
3
2
1
S2. I
was
satis
fied
with
the
leve
l of e
ffor
t thi
s cou
rse
requ
ired.
5
4
3
2
1
S3
. My
inte
rest
in th
e su
bjec
t mat
ter i
ncre
ased
bec
ause
of t
his
cour
se.
5
4
3
2
1
S4. I
was
satis
fied
with
my
lear
ning
in th
is c
ours
e.
5
4
3
2
1
S5. I
was
hap
py w
ith m
y fin
al g
rade
in th
is c
ours
e.
5
4
3
2
1
D
irec
tions
: A li
st o
f opp
osin
g w
ords
app
ears
bel
ow. R
ate
how
wel
l the
se w
ords
des
crib
ed y
our f
eelin
gs a
bout
the
lect
ure
porti
on
of y
our f
irst c
hem
istry
cou
rse
at C
UA
. For
eac
h lin
e, c
hoos
e a
posi
tion
betw
een
the
two
wor
ds th
at d
escr
ibes
exa
ctly
how
you
felt.
Th
e m
iddl
e po
sitio
n is
if y
ou a
re u
ndec
ided
or h
ave
no fe
elin
gs re
late
d to
the
term
s on
that
line
. As b
efor
e, p
leas
e re
ad th
e tw
o w
ords
and
you
r cho
ice
alou
d an
d br
iefly
des
crib
e yo
ur re
ason
for s
elec
ting
that
resp
onse
. Aga
in, I
may
stop
you
to a
sk a
follo
w-u
p qu
estio
n ab
out y
our r
espo
nses
.
TH
E C
HE
MIS
TR
Y C
OU
RSE
WA
S…
Mid
dle
S6
.
Com
forta
ble
5
4
3
2
1
Unc
omfo
rtabl
e S7
.
S
atis
fyin
g 5
4
3
2
1
Fr
ustra
ting
S8.
P
leas
ant
5
4
3
2
1
Unp
leas
ant
S9.
Cha
otic
5
4
3
2
1
O
rgan
ized
213
App
endi
x D
– A
TI R
evis
ions
Afte
r Pi
lot S
tudy
Tabl
e 21
AT
I Ite
ms U
sed
in P
ilot S
tudy
, Rev
ised
ATI
Item
s, an
d Ra
tiona
le fo
r Rev
isio
n A
TI i
tem
s use
d in
pilo
t stu
dy
AT
I Ite
ms r
evis
ed a
fter
pilo
t stu
dy
Rat
iona
le fo
r re
visi
on
1a.
In th
is c
ours
e st
uden
ts sh
ould
focu
s th
eir s
tudy
on
wha
t I p
rovi
de th
em.
1b. I
n th
is c
ours
e st
uden
ts sh
ould
focu
s th
eir s
tudy
on
mat
eria
ls li
sted
in
the
sylla
bus a
nd p
rovi
ded
by th
e in
stru
ctor
such
as t
he te
xtbo
ok a
nd
lect
ure
note
s.
Cha
nged
“w
hat I
pro
vide
them
” to
incl
ude
all
cour
se m
ater
ials
.
2a.
It is
impo
rtant
that
this
cou
rse
shou
ld b
e co
mpl
etel
y de
scrib
ed in
te
rms o
f spe
cific
obj
ectiv
es th
at
rela
te to
form
al a
sses
smen
t ite
ms.
2b. T
his c
ours
e is
com
plet
ely
desc
ribed
in
term
s of s
peci
fic o
bjec
tives
that
re
late
to c
ours
e as
sess
men
ts.
Rem
oved
bel
ief c
ompo
nent
, “fo
rmal
as
sess
men
t ite
ms”
mos
t fre
quen
tly in
terp
rete
d to
mea
n co
urse
ass
essm
ents
, not
ext
erna
l as
sess
men
ts
3a.
In m
y in
tera
ctio
ns w
ith st
uden
ts in
th
is c
ours
e I t
ry to
dev
elop
a
conv
ersa
tion
with
them
abo
ut th
e to
pics
we
are
stud
ying
.
3b.
In m
y in
tera
ctio
ns w
ith st
uden
ts in
th
is c
ours
e I t
ry to
dev
elop
a
conv
ersa
tion
with
them
abo
ut th
e to
pics
we
are
stud
ying
.
No
chan
ge. I
nstru
ctor
inte
rpre
tatio
n of
co
nver
satio
ns in
clud
e in
cla
ss a
nd o
ut o
f cla
ss
(off
ice
hour
s)
4a.
It is
impo
rtant
to p
rese
nt a
lot o
f fa
cts t
o st
uden
ts so
that
they
kno
w
wha
t the
y ha
ve to
lear
n fo
r thi
s co
urse
.
4b.
In th
is c
ours
e fa
cts a
re p
rese
nted
to
stud
ents
so th
at th
ey k
now
wha
t th
ey h
ave
to le
arn.
Rem
oved
bel
ief c
ompo
nent
. Rem
oved
“a
lot”
du
e to
impl
ied
nega
tive
asso
ciat
ion.
5a.
I set
asi
de so
me
teac
hing
tim
e so
th
at th
e st
uden
ts c
an d
iscu
ss,
amon
g th
emse
lves
, key
con
cept
s an
d id
eas i
n th
is c
ours
e.
5b.
I pro
vide
opp
ortu
nitie
s so
that
the
stud
ents
can
dis
cuss
, am
ong
them
selv
es, k
ey c
once
pts a
nd
idea
s in
this
cou
rse.
Rem
oved
“te
achi
ng ti
me”
to to
avo
id
rest
rictin
g qu
estio
n to
onl
y co
nsid
erin
g le
ctur
e tim
e.
6a.
In th
is c
ours
e I c
once
ntra
te o
n co
verin
g th
e in
form
atio
n th
at m
ight
be
ava
ilabl
e fr
om k
ey te
xts a
nd
read
ings
.
6a.
In th
is c
ours
e I c
once
ntra
te o
n co
verin
g th
e in
form
atio
n th
at is
av
aila
ble
from
ass
igne
d re
adin
gs.
Cha
nged
“ke
y te
xts a
nd re
adin
gs”
to a
lign
with
m
ost f
requ
ent i
nter
pret
atio
n as
ass
igne
d te
xtbo
ok re
adin
g, re
mov
ed “
mig
ht”
to m
ake
ques
tion
mor
e co
ncre
te.
214
Tabl
e 21
, con
tinue
d AT
I Ite
ms U
sed
in P
ilot S
tudy
, Rev
ised
ATI
Item
s, an
d Ra
tiona
le fo
r Rev
isio
n
AT
I ite
ms u
sed
in p
ilot s
tudy
A
TI I
tem
s rev
ised
aft
er p
ilot s
tudy
R
atio
nale
for
revi
sion
7a
. I e
ncou
rage
stud
ents
to re
stru
ctur
e th
eir e
xist
ing
know
ledg
e in
term
s of
the
new
way
of t
hink
ing
abou
t th
e su
bjec
t tha
t the
y w
ill d
evel
op.
7b.
I enc
oura
ge st
uden
ts to
rest
ruct
ure
thei
r exi
stin
g kn
owle
dge
in te
rms
of th
e ne
w w
ay o
f thi
nkin
g ab
out
the
subj
ect t
hat t
hey
will
dev
elop
.
No
chan
ge. R
estru
ctur
ing
know
ledg
e is
rela
ted
to c
onst
ruct
ivis
t ide
a of
mod
ifyin
g pr
eexi
stin
g kn
owle
dge.
Inst
ruct
ors d
isag
reei
ng e
ither
be
lieve
d th
e st
uden
ts e
nter
ed w
ith n
o kn
owle
dge
or e
xist
ing
know
ledg
e w
as g
ood
and
requ
ired
no c
hang
es.
8a.
In te
achi
ng se
ssio
ns fo
r thi
s sub
ject
, I d
elib
erat
ely
prov
oke
deba
te a
nd
disc
ussi
on.
8b. I
n th
is c
ours
e, I
enco
urag
e de
bate
an
d di
scus
sion
. R
emov
ed “
teac
hing
sess
ions
” an
d ch
ange
d “d
elib
erat
ely
prov
oke”
to b
e m
ore
neut
ral.
9a.
I stru
ctur
e m
y te
achi
ng in
this
co
urse
to h
elp
stud
ents
to p
ass t
he
form
al a
sses
smen
t ite
ms.
9b.
I stru
ctur
e m
y te
achi
ng in
this
co
urse
to h
elp
stud
ents
pas
s cou
rse
asse
ssm
ents
.
Cha
nged
“fo
rmal
ass
essm
ents
” to
cou
rse
asse
ssm
ents
.
10a.
I th
ink
an im
porta
nt re
ason
for
runn
ing
teac
hing
sess
ions
in th
is
cour
se is
to g
ive
stud
ents
a g
ood
set
of n
otes
.
10b.
Cla
ss ti
me
in th
is c
ours
e is
use
d to
gi
ve st
uden
ts a
set o
f not
es.
Rem
oved
bel
ief c
ompo
nent
. Rem
oved
teac
hing
se
ssio
ns. R
emov
ed “
good
”.
11a.
In th
is c
ours
e, I
prov
ide
the
stud
ents
the
info
rmat
ion
they
will
ne
ed to
pas
s the
form
al
asse
ssm
ents
.
11b.
In th
is c
ours
e, I
prov
ide
the
stud
ents
the
info
rmat
ion
they
will
ne
ed to
pas
s the
cou
rse
asse
ssm
ents
.
Cha
nged
“fo
rmal
ass
essm
ents
” to
cou
rse
asse
ssm
ents
.
12a.
I sh
ould
kno
w th
e an
swer
s to
any
ques
tions
that
stud
ents
may
put
to
me
durin
g th
is c
ours
e.
12b.
I sh
ould
kno
w th
e an
swer
s to
any
ques
tions
abo
ut c
ours
e co
nten
t th
at st
uden
ts m
ay a
sk.
Nar
row
ed fo
cus t
o qu
estio
ns o
ver c
ours
e co
nten
t.
13a.
I m
ake
avai
labl
e op
portu
nitie
s for
st
uden
ts in
this
cou
rse
to d
iscu
ss
thei
r cha
ngin
g un
ders
tand
ing
of th
e su
bjec
t.
13b.
I m
ake
avai
labl
e op
portu
nitie
s for
st
uden
ts in
this
cou
rse
to d
iscu
ss
thei
r cha
ngin
g un
ders
tand
ing
of
the
subj
ect.
No
chan
ge. O
ppor
tuni
ties i
nter
pret
ed to
mea
n in
cla
ss a
nd o
utsi
de o
f cla
ss in
off
ice
hour
s or
othe
r inf
orm
al d
iscu
ssio
ns.
14a.
It is
bet
ter f
or st
uden
ts in
this
co
urse
to g
ener
ate
thei
r ow
n no
tes
rath
er th
an c
opy
min
e.
14b.
Stu
dent
s are
enc
oura
ged
to
gene
rate
thei
r ow
n no
tes o
r mak
e an
nota
tions
on
min
e ra
ther
than
co
py m
ine
verb
atim
.
Rem
oved
bel
ief c
ompo
nent
. Add
ed id
ea o
f an
nota
ting
note
s.
215
Tabl
e 21
, con
tinue
d AT
I Ite
ms U
sed
in P
ilot S
tudy
, Rev
ised
ATI
Item
s, an
d Ra
tiona
le fo
r Rev
isio
n
AT
I ite
ms u
sed
in p
ilot s
tudy
A
TI I
tem
s rev
ised
aft
er p
ilot s
tudy
R
atio
nale
for
revi
sion
15
a. A
lot o
f tea
chin
g tim
e in
this
co
urse
shou
ld b
e us
ed to
que
stio
n st
uden
ts’ i
deas
.
15b.
Stu
dent
s’ id
eas a
re d
iscu
ssed
in
this
cou
rse.
R
emov
ed “
a lo
t” a
nd “
shou
ld”
to fo
cus o
n pr
actic
e no
t bel
ief,
“que
stio
ning
” se
emed
to
impl
y ju
dgm
ent a
nd w
as c
hang
ed to
“di
scus
s”,
rem
oved
“te
achi
ng ti
me”
to a
void
rest
rictin
g qu
estio
n to
onl
y co
nsid
erin
g le
ctur
e tim
e.
16a.
In th
is c
ours
e m
y te
achi
ng fo
cuse
s on
the
good
pre
sent
atio
n of
in
form
atio
n to
stud
ents
.
16b.
In th
is c
ours
e m
y te
achi
ng fo
cuse
s on
the
pres
enta
tion
of in
form
atio
n to
stud
ents
.
Rem
oved
“go
od”.
17a.
I se
e te
achi
ng a
s hel
ping
stud
ents
de
velo
p ne
w w
ays o
f thi
nkin
g in
th
is su
bjec
t.
17b.
My
teac
hing
in th
is c
ours
e he
lps
stud
ents
dev
elop
way
s of t
hink
ing
in th
is su
bjec
t tha
t are
new
to
them
.
Rem
oved
bel
ief c
ompo
nent
. Cla
rify
“new
” w
ays o
f thi
nkin
g as
new
to st
uden
ts.
18a.
In te
achi
ng th
is c
ours
e it
is
impo
rtant
for m
e to
mon
itor
stud
ents
’ cha
nged
und
erst
andi
ng o
f th
e su
bjec
t mat
ter.
18b.
In te
achi
ng th
is c
ours
e, I
mon
itor
stud
ents
’ cha
nged
und
erst
andi
ng
of th
e su
bjec
t mat
ter.
Rem
oved
bel
ief c
ompo
nent
.
19a.
My
teac
hing
in th
is c
ours
e fo
cuse
s on
del
iver
ing
wha
t I k
now
to th
e st
uden
ts.
19b.
My
teac
hing
in th
is c
ours
e fo
cuse
s on
del
iver
ing
wha
t I k
now
to th
e st
uden
ts.
No
chan
ge.
20a.
Tea
chin
g in
this
cou
rse
shou
ld h
elp
stud
ents
que
stio
n th
eir o
wn
unde
rsta
ndin
g of
the
subj
ect m
atte
r.
20b.
My
teac
hing
in th
is c
ours
e he
lps
stud
ents
que
stio
n th
eir o
wn
unde
rsta
ndin
g of
the
subj
ect
mat
ter.
Rem
oved
bel
ief c
ompo
nent
.
21a.
Tea
chin
g in
this
cou
rse
shou
ld
incl
ude
help
ing
stud
ents
find
thei
r ow
n le
arni
ng re
sour
ces.
21b.
My
teac
hing
in th
is c
ours
e in
clud
es h
elpi
ng st
uden
ts fi
nd
thei
r ow
n le
arni
ng re
sour
ces.
Rem
oved
bel
ief c
ompo
nent
.
22a.
I pr
esen
t mat
eria
l to
enab
le
stud
ents
to b
uild
up
an in
form
atio
n ba
se in
this
subj
ect.
22b.
I pr
esen
t mat
eria
l to
enab
le
stud
ents
to b
uild
up
a ba
se o
f co
nten
t kno
wle
dge
and
skill
s in
this
subj
ect.
Cla
rifie
d in
form
atio
n as
incl
udin
g co
nten
t kn
owle
dge
and
skill
s sin
ce th
is w
as a
freq
uent
in
stru
ctor
inte
rpre
tatio
n.
216
App
endi
x E
– S
tude
nt S
urve
y R
evis
ions
Afte
r Pi
lot S
tudy
D
irec
tions
: Ple
ase
com
plet
e th
is su
rvey
by
cons
ider
ing
only
the
lect
ure
porti
on o
f the
che
mis
try c
ours
e in
whi
ch y
ou a
re e
nrol
led.
Pl
ease
do
not c
onsi
der t
he la
bora
tory
por
tion
of th
e co
urse
.
Stro
ngly
D
isag
ree
Dis
agre
e N
eutr
al
Agr
ee
Stro
ngly
A
gree
Q
1. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt c
ours
e to
pics
1
2
3
4
5
Q
2. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt c
ours
e go
als
1
2
3
4
5
Q3.
The
inst
ruct
or p
rovi
ded
clea
r ins
truct
ions
on
how
to p
artic
ipat
e in
cou
rse
lear
ning
act
iviti
es
1
2
3
4
5
Q4.
The
inst
ruct
or c
lear
ly c
omm
unic
ated
impo
rtant
due
dat
es/ti
me
fram
es fo
r co
urse
lear
ning
act
iviti
es
1
2
3
4
5
Q5.
The
inst
ruct
or w
as h
elpf
ul in
faci
litat
ing
disc
ussi
ons o
n co
urse
topi
cs th
at
help
ed m
e to
lear
n
1
2
3
4
5
Q6.
The
inst
ruct
or w
as h
elpf
ul in
gui
ding
the
clas
s tow
ards
und
erst
andi
ng
cour
se to
pics
in a
way
that
hel
ped
me
clar
ify m
y th
inki
ng
1
2
3
4
5
Q7.
The
inst
ruct
or h
elpe
d to
kee
p co
urse
par
ticip
ants
eng
aged
and
pa
rtici
patin
g in
pro
duct
ive
dial
ogue
1
2
3
4
5
Q8.
The
inst
ruct
or h
elpe
d ke
ep th
e co
urse
par
ticip
ants
on
task
in a
way
that
he
lped
me
to le
arn
1
2
3
4
5
Q9.
The
inst
ruct
or e
ncou
rage
d co
urse
par
ticip
ants
to e
xplo
re n
ew c
once
pts i
n th
is c
ours
e 1
2
3
4
5
Q10
. The
inst
ruct
or re
info
rced
the
deve
lopm
ent o
f a se
nse
of c
omm
unity
am
ong
cour
se p
artic
ipan
ts
1
2
3
4
5
Q11
. The
inst
ruct
or h
elpe
d to
focu
s dis
cuss
ion
on re
leva
nt is
sues
in a
way
that
he
lped
me
to le
arn
1
2
3
4
5
Q12
. The
inst
ruct
or p
rovi
ded
feed
back
that
hel
ped
me
unde
rsta
nd m
y st
reng
ths
and
wea
knes
ses r
elat
ive
to th
e co
urse
’s g
oals
and
obj
ectiv
es
1
2
3
4
5
Q13
. The
inst
ruct
or p
rovi
ded
feed
back
in a
tim
ely
fash
ion
1
2
3
4
5
Q14
. Get
ting
to k
now
oth
er c
ours
e pa
rtici
pant
s gav
e m
e a
sens
e of
bel
ongi
ng
in th
e co
urse
. 1
2
3
4
5
Q15
. I w
as a
ble
to fo
rm d
istin
ct im
pres
sion
s of s
ome
cour
se p
artic
ipan
ts
1
2
3
4
5
Q16
. Fac
e-to
-fac
e co
mm
unic
atio
n is
an
exce
llent
med
ium
for s
ocia
l int
erac
tion
1
2
3
4
5
Q17
. I fe
lt co
mfo
rtabl
e co
nver
sing
face
-to-f
ace
in c
lass
1
2
3
4
5
Q
18. I
felt
com
forta
ble
parti
cipa
ting
in th
e co
urse
dis
cuss
ions
1
2
3
4
5
217
St
rong
ly
Dis
agre
e D
isag
ree
Neu
tral
A
gree
St
rong
ly
Agr
ee
Q19
. I fe
lt co
mfo
rtabl
e in
tera
ctin
g w
ith o
ther
cou
rse
parti
cipa
nts
1
2
3
4
5
Q20
. I fe
lt co
mfo
rtabl
e di
sagr
eein
g w
ith o
ther
cou
rse
parti
cipa
nts w
hile
still
m
aint
aini
ng a
sens
e of
trus
t 1
2
3
4
5
Q21
. I fe
lt th
at m
y po
int o
f vie
w w
as a
ckno
wle
dged
by
othe
r cou
rse
parti
cipa
nts
1
2
3
4
5
Q22
. In-
clas
s dis
cuss
ions
hel
ped
me
to d
evel
op a
sens
e of
col
labo
ratio
n 1
2
3
4
5
Q
23. P
robl
ems p
osed
incr
ease
d m
y in
tere
st in
cou
rse
issu
es
1
2
3
4
5
Q24
. Cou
rse
lear
ning
act
iviti
es p
ique
d m
y cu
riosi
ty
1
2
3
4
5
Q25
. I fe
lt m
otiv
ated
to e
xplo
re c
onte
nt re
late
d qu
estio
ns
1
2
3
4
5
Q26
. I u
tiliz
ed a
var
iety
of i
nfor
mat
ion
sour
ces t
o ex
plor
e pr
oble
ms p
osed
in
this
cou
rse
1
2
3
4
5
Q27
. Bra
inst
orm
ing
help
ed m
e re
solv
e co
nten
t rel
ated
que
stio
ns
1
2
3
4
5
Q28
. Fin
ding
rele
vant
info
rmat
ion
help
ed m
e re
solv
e co
nten
t rel
ated
que
stio
ns
1
2
3
4
5
Q29
. Ple
ase
sele
ct “
Dis
agre
e” fo
r thi
s ite
m
1
2
3
4
5
Q30
. In-
clas
s dis
cuss
ions
wer
e va
luab
le in
hel
ping
me
appr
ecia
te d
iffer
ent
pers
pect
ives
1
2
3
4
5
Q31
. Com
bini
ng n
ew in
form
atio
n he
lped
me
answ
er q
uest
ions
rais
ed in
cou
rse
lear
ning
act
iviti
es
1
2
3
4
5
Q32
. Cou
rse
lear
ning
act
iviti
es h
elpe
d m
e co
nstru
ct e
xpla
natio
ns/s
olut
ions
1
2
3
4
5
Q
33. R
efle
ctio
n on
cou
rse
cont
ent h
elpe
d m
e un
ders
tand
fund
amen
tal c
once
pts
in th
is c
lass
1
2
3
4
5
Q34
. Ref
lect
ion
on d
iscu
ssio
ns h
elpe
d m
e un
ders
tand
fund
amen
tal c
once
pts i
n th
is c
lass
1
2
3
4
5
Q35
. I c
an d
escr
ibe
way
s to
test
and
app
ly th
e kn
owle
dge
crea
ted
in th
is c
ours
e 1
2
3
4
5
Q
36. I
hav
e de
velo
ped
solu
tions
to c
ours
e pr
oble
ms t
hat c
an b
e ap
plie
d in
pr
actic
e
1
2
3
4
5
Q37
. I c
an a
pply
the
know
ledg
e cr
eate
d in
this
cou
rse
to m
y w
ork
or o
ther
no
n-cl
ass r
elat
ed a
ctiv
ities
1
2
3
4
5
---
surv
ey c
ontin
ues o
n th
e ne
xt p
age
-- 218
D
irec
tions
: A li
st o
f opp
osin
g w
ords
app
ears
bel
ow. R
ate
how
wel
l the
se w
ords
des
crib
ed y
our f
eelin
gs a
bout
the
lect
ure
porti
on
of y
our c
hem
istry
cou
rse.
For
eac
h lin
e, c
hoos
e a
posi
tion
betw
een
the
two
wor
ds th
at d
escr
ibes
exa
ctly
how
you
felt.
The
mid
dle
posi
tion
is if
you
are
und
ecid
ed o
r hav
e no
feel
ings
rela
ted
to th
e te
rms o
n th
at li
ne.
T
HE
CH
EM
IST
RY
CO
UR
SE W
AS…
M
iddl
e
S1.
C
omfo
rtabl
e 1
2
3
4
5
Unc
omfo
rtabl
e S2
.
S
atis
fyin
g 1
2
3
4
5
Frus
tratin
g S3
.
Ple
asan
t 1
2
3
4
5
Unp
leas
ant
S4.
Cha
otic
1
2
3
4
5
Org
aniz
ed
219
220
Appendix F – R Program for Calculating Coefficient H
coeff.H<-function(x){
denom <- 0
list<-length(x)
for (i in 1:list){
denom <- denom + ((x[i]^2)/(1-x[i]^2))
}
H <- 1/(1+(1/denom))
return(H)
}
221
Appendix G – Path Tracing and Matrix Determination for Hypothesized Research Model Table 22 Algebraic Statements from Path Tracing
Variables Trace
“V1”/”V2” Teaching presence and social presence
!F1bF2F1 !F2
“V1”/”V3” Teaching presence and cognitive presence
!F1bF3F1 !F3 + !F1bF2F1bF3F2 !F3
“V1”/”V4” Teaching presence and student satisfaction
!F1bF4F1 !F4 + !F1bF2F1bF4F2 !F4 + !F1bF3F1bF4F3 !F4 + !F1bF2F1bF3F2bF4F3 !F4
“V1”/V5 Teaching presence and math ability
0
“V1”/V6 Teaching presence and ACS exam score
!F1bF6F1 + !F1bF3F1bF6F3 + !F1bF2F1bF3F2bF6F3
“V1”/V7 Teaching presence and final course grade
!F1bF7F1 + !F1bF6F1bF7F6 + !F1bF3F1bF7F3 + !F1bF3F1bF6F3bF7F6 + !F1bF2F1bF3F2bF6F3bF7F6 +
!F1bF2F1bF3F2bF7F3
“V2”/”V3” Social presence and cognitive presence
!F2bF3F2 !F3 + !F2bF2F1bF3F1 !F3
“V2”/”V4” Social presence and student satisfaction
!F2bF4F2 !F4 + !F2bF3F2bF4F3 !F4 + !F2bF2F1bF4F1 !F4 + !F2bF2F1bF3F1bF4F3 !F4
“V2”/V5 Social presence and math ability
0
“V2”/V6 Social presence and ACS exam score
!F2bF3F2bF6F3 + !F2bF2F1bF6F1 + !F2bF2F1bF3F1bF6F3
“V2”/V7 Social presence and final course grade
!F2bF3F2bF7F3 + !F2bF3F2bF6F3bF7F6 + !F2bF2F1bF7F1 + !F2bF2F1bF6F1bF7F6 + !F2bF2F1bF3F1bF7F3 +
!F2bF2F1bF3F1bF6F3bF7F6
“V3”/”V4” Cognitive presence and student
satisfaction
!F3bF4F3 !F4 + !F3bF3F1bF4F1 !F4 + !F3bF3F2bF4F2 !F4 + !F3bF3F1bF2F1bF4F2 !F4 +
!F3bF3F2bF2F1bF4F1 !F4
222
Table 22, continued Algebraic Statements from Path Tracing
Variables Trace
“V3”/V5 Cognitive presence and math ability
0
“V3”/V6 Cognitive presence and ACS exam score
!F3bF6F3 + !F3bF3F1bF6F1 + !F3bF3F2bF2F1bF6F1
“V3”/V7 Cognitive presence and final course grade
!F3bF7F3 + !F3bF6F3bF7F6 + !F3bF3F1bF7F1 + !F3bF3F1bF6F1bF7F6 + !F3bF3F2bF2F1bF6F1bF7F6 +
!F3bF3F2bF2F1bF7F1
“V4”/V5 Student satisfaction and math ability
0
“V4”/V6 Student satisfaction and ACS exam score
!F4bF4F3bF6F3 + !F4bF4F1bF3F1bF6F3 + !F4bF4F1bF6F1 + !F4bF4F2bF3F2bF6F3 + !F4bF4F2bF2F1bF6F1 +
!F4bF4F2bF2F1bF3F1bF6F3 + !F4bF4F3bF3F1bF6F1 + !F4bF4F3bF3F2bF2F1bF6F1 + !F4bF4F1bF2F1bF3F2bF6F3
“V4”/V7 Student satisfaction and final course grade
!F4bF4F2bF3F2bF7F3 + !F4bF4F2bF3F2bF6F3bF7F6 + !F4bF4F2bF2F1bF7F1 + !F4bF4F2bF2F1bF6F1bF7F6 +
!F4bF4F2bF2F1bF3F1bF6F3bF7F6 + !F4bF4F2bF2F1bF3F1bF7F3 + !F4bF4F1bF7F1 + !F4bF4F1bF6F1bF7F6 +
!F4bF4F1bF2F1bF3F2bF6F3bF7F6 + !F4bF4F1bF2F1bF3F2bF7F3 + !F4bF4F1bF3F1bF6F3bF7F6 +
!F4bF4F1bF3F1bF7F3 + !F4bF4F3bF6F3bF7F6 + !F4bF4F3bF7F3 + !F4bF4F3bF3F2bF2F1bF6F1bF7F6 +
!F4bF4F3bF3F2bF2F1bF7F1 + !F4bF4F3bF3F1bF6F1bF7F6 + !F4bF4F3bF3F1bF7F1 + !F4cF4F7
V5/V6 Math ability and ACS exam score
bF6F5
V5/V7 Math ability and final course grade
bF7F5 + bF6F5bF7F6
V6/V7 ACS exam score and final course grade
bF7F6 + bF6F5bF7F5 + bF6F3bF7F3 + bF6F3bF3F1bF7F1 + bF6F3bF3F2bF2F1bF7F1 + bF6F1bF7F1 + bF6F1bF3F1bF7F3 +
bF6F1bF2F1bF3F2bF7F3
223
R code for generating the model-implied matrix from spreadsheet containing literature loading
library(openxlsx)
loadings<-read.xlsx("Power Analysis.xlsx", 2)
min<-lapply(abs(loadings), min, na.rm=T)
loadings<-rbind(loadings, min)
## F1: Teaching Presence
TP.H<-coeff.H(loadings[9, 1:13])
H1 <-sqrt(TP.H)
## F2: Social Presence
SP.H<-coeff.H(loadings[9, 14:22])
H2 <-sqrt(SP.H)
## F3: Cognitive Presence
CP.H<-coeff.H(loadings[9, 23:34])
H3 <-sqrt(CP.H)
## F4: Student Satisfaction
SATIS.H<-coeff.H(loadings[9, 35:38])
H4 <-sqrt(SATIS.H)
matrix<-matrix(NA, nrow=7, ncol=7)
b21<-0.52
b31<-0.49
b32<-0.3
b41<-0.24
b42<-0.30
b43<-0.26
b61<-0.30
b63<-0.35
b71<-0.30
b73<-0.35
b65<-0.414
b75<-0.414
b76<-0.18
224
c47<-0.38
cor12<-(H1*b21*H2)
cor13<-(H1*b31*H3 + H1*b21*b32*H3)
cor14<-(H1*b41*H4 + H1*b21*b42*H4 + H1*b31*b43*H4 +
H1*b21*b32*b43*H4)
cor15<-0
cor16<-(H1*b61 + H1*b31*b63 + H1*b21*b32*b63)
cor17<-(H1*b71 + H1*b61*b76 + H1*b31*b73 + H1*b31*b63*b76 +
H1*b21*b32*b63*b76 + H1*b21*b32*b73)
cor23<-(H2*b32*H3 + H2*b21*b31*H3)
cor24<-(H2*b42*H4 + H2*b32*b43*H4 + H2*b21*b41*H4 +
H2*b21*b31*b43*H3)
cor25<-0
cor26<-(H2*b32*b63 + H2*b21*b61 + H2*b21*b31*b63)
cor27<-(H2*b32*b73 + H2*b32*b63*b76 + H2*b21*b71 + H2*b21*b61*b76
+ H2*b21*b31*b73 + H2*b21*b31*b63*b76)
cor34<-(H3*b43*H4 + H3*b31*b41*H4 + H3*b32*b42*H4 +
H3*b31*b21*b42*H4 + H3*b32*b21*b41*H4)
cor35<-0
cor36<-(H3*b63 + H3*b31*b61 + H3*b32*b21*b61)
cor37<-(H3*b73 + H3*b63*b76 + H3*b31*b71 + H3*b31*b61*b76 +
H3*b32*b21*b61*b76 + H3*b32*b21*b71)
cor45<-0
cor46<-(H4*b43*b63 + H4*b41*b31*b63 + H4*b41*b61 + H4*b42*b32*b63
+ H4*b42*b21*b61 + H4*b42*b21*b31*b63 +
H4*b43*b31*b61 + H4*b43*b32*b21*b61 +
H4*b41*b21*b32*b63)
cor47<-(H4*b42*b32*b73 + H4*b42*b32*b63*b76 + H4*b42*b21*b71 +
H4*b42*b21*b61*b76 + H4*b42*b21*b31*b63*b76
225
+ H4*b42*b21*b31*b73 + H4*b41*b71 + H4*b41*b61*b76 +
H4*b41*b21*b32*b63*b76 + H4*b41*b21*b32*b73 +
H4*b41*b31*b63*b76 + H4*b41*b31*b73 +
H4*b43*b63*b76 + H4*b43*b73 + H4*b43*b32*b21*b61*b76 +
H4*b43*b32*b21*b71 + H4*b43*b31*b61*b76 +
H4*b43*b31*b71 + H4*c47)
cor56<-(b65)
cor57<-(b75 + b65*b76)
cor67<-(b76 + b65*b75 + b63*b73 + b63*b31*b71 + b63*b32*b21*b71 +
b61*b71 + b61*b31*b73 + b61*b21*b32*b73)
matrix[1,]<-c(1, cor12, cor13, cor14, cor15, cor16, cor17)
matrix[2,]<-c(cor12, 1, cor23, cor24, cor25, cor26, cor27)
matrix[3,]<-c(cor13, cor23, 1, cor34, cor35, cor36, cor37)
matrix[4,]<-c(cor14, cor24, cor34, 1, cor45, cor46, cor47)
matrix[5,]<-c(cor15, cor25, cor35, cor45, 1, cor56, cor57)
matrix[6,]<-c(cor16, cor26, cor36, cor46, cor56, 1, cor67)
matrix[7,]<-c(cor17, cor27, cor37, cor47, cor57, cor67, 1)
R code to print generated matrix for use in LISREL > print(matrix, digits = 5)
[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,] 1.00000 0.44660 0.55940 0.48275 0.00000 0.49189 0.58043
[2,] 0.44660 1.00000 0.47200 0.47921 0.00000 0.32167 0.37957
[3,] 0.55940 0.47200 1.00000 0.49306 0.00000 0.50365 0.59431
[4,] 0.48275 0.47921 0.49306 1.00000 0.00000 0.34123 0.75055
[5,] 0.00000 0.00000 0.00000 0.00000 1.00000 0.41400 0.48852
[6,] 0.49189 0.32167 0.50365 0.34123 0.41400 1.00000 0.69956
[7,] 0.58043 0.37957 0.59431 0.75055 0.48852 0.69956 1.00000
226
Appendix H – LISREL Syntax and Output for Power Analysis OBSERVED VARIABLES V1-V7 Covariance Matrix 1 0.44660 1 0.55940 0.47200 1 0.48275 0.47921 0.49306 1 0.00000 0.00000 0.00000 0.000 1 0.49189 0.32167 0.50365 0.34123 0.41400 1 0.58043 0.37957 0.59431 0.75055 0.48852 0.69956 1 SAMPLE SIZE IS 1001 LATENT VARIABLES F1-F7 RELATIONSHIPS V1 = .935*F1 V2 = .919*F2 V3 = .926*F3 V4 = .916*F4 V5 = 1*F5 V6 = 1*F6 V7 = 1*F7 F6 = .30*F1 .35*F3 .414*F5 F7 = .30*F1 .35*F3 .414*F5 .18*F6 F2 = .52*F1 F3 = .49*F1 .30*F2 F4 = .24*F1 .30*F2 .26*F3 LET THE ERRORS OF F4 and F7 COVARY SET THE COVARIANCE OF F1 and F5 to 0 SET THE VARIANCE OF F1 to 1 SET THE VARIANCE OF F5 to 1 SET ERROR VARIANCE OF V1 TO .126 SET ERROR VARIANCE OF V2 TO .156 SET ERROR VARIANCE OF V3 TO .142 SET ERROR VARIANCE OF V4 TO .162 SET ERROR VARIANCE OF V5 TO 0 SET ERROR VARIANCE OF V6 TO 0 SET ERROR VARIANCE OF V7 TO 0 Options: ND=3 RS PATH DIAGRAM END OF PROBLEM Sample Size = 1001 CoI Model- Power Analysis
227
Covariance Matrix V2 V3 V4 V6 V7 V1 -------- -------- -------- -------- -------- ------- V2 1.000 V3 0.472 1.000 V4 0.479 0.493 1.000 V6 0.322 0.504 0.341 1.000 V7 0.380 0.594 0.751 0.700 1.000 V1 0.447 0.559 0.483 0.492 0.580 1.000 V5 - - - - - - 0.414 0.489 - - Covariance Matrix V5 -------- V5 1.000 Total Variance = 7.000 Generalized Variance = 0.0126 Largest Eigenvalue = 3.617 Smallest Eigenvalue = 0.047 Condition Number = 8.767 W_A_R_N_I_N_G : Both LX( 1, 1) and PH( 1, 1) are fixed non-zero values.
LISREL is unable to generate Starting Values for this model. The model will be estimated using the NS option.
W_A_R_N_I_N_G : Both LX( 3, 2) and PH( 2, 2) are fixed non-zero values.
LISREL is unable to generate Starting Values for this model. The model will be estimated using the NS option.
CoI Model- Power Analysis Number of Iterations = 2 LISREL Estimates (Maximum Likelihood) Measurement Equations V2 = 0.919*F2, Errorvar.= 0.156, R² = 0.844 V3 = 0.926*F3, Errorvar.= 0.142, R² = 0.858 V4 = 0.916*F4, Errorvar.= 0.162, R² = 0.838 V6 = 1.000*F6,, R² = 1.000 V7 = 1.000*F7,, R² = 1.000
228
V1 = 0.935*F1, Errorvar.= 0.126, R² = 0.874 V5 = 1.000*F5,, R² = 1.000 Structural Equations F2 = 0.520*F1, Errorvar.= 0.730 , R² = 0.270 Standerr (0.0414) Z-values 17.628 P-values 0.000 F3 = 0.300*F2 + 0.490*F1, Errorvar.= 0.517 , R² = 0.483 Standerr (0.0319) Z-values 16.211 P-values 0.000 F4 = 0.300*F2 + 0.260*F3 + 0.240*F1, Errorvar.= 0.542 , R² = 0.458 Standerr (0.0343) Z-values 15.803 P-values 0.000 F6 = 0.350*F3 + 0.300*F1 + 0.414*F5, Errorvar.= 0.480 , R² = 0.520 Standerr (0.0226) Z-values 21.268 P-values 0.000 F7 = 0.350*F3 + 0.180*F6 + 0.300*F1 + 0.414*F5, Errorvar.= 0.261 , R² = 0.739 Standerr (0.0129) Z-values 20.241 P-values 0.000 NOTE: R² for Structural Equations are Hayduk's (2006) Blocked-Error R² Error Covariance for F7 and F4 = 0.380 (0.0196) 19.407 Reduced Form Equations F2 = 0.520*F1 + 0.0*F5, Errorvar.= 0.730, R² = 0.270 F3 = 0.646*F1 + 0.0*F5, Errorvar.= 0.583, R² = 0.417 F4 = 0.564*F1 + 0.0*F5, Errorvar.= 0.681, R² = 0.318 F6 = 0.526*F1 + 0.414*F5, Errorvar.= 0.552, R² = 0.448 F7 = 0.621*F1 + 0.489*F5, Errorvar.= 0.376, R² = 0.624
229
Correlation Matrix of Independent Variables Note: This matrix is diagonal. F1 F5 -------- -------- 1.000 1.000 Covariance Matrix of Latent Variables F2 F3 F4 F6 F7 F1 -------- -------- -------- -------- -------- ------- F2 1.000 F3 0.555 1.000 F4 0.569 0.582 0.999 F6 0.350 0.544 0.373 1.000 F7 0.413 0.642 0.820 0.700 1.000 F1 0.520 0.646 0.564 0.526 0.621 1.000 F5 - - - - - - 0.414 0.489 - - Covariance Matrix of Latent Variables F5 -------- F5 1.000 W_A_R_N_I_N_G: Matrix above is not positive definite Log-likelihood Values Estimated Model Saturated Model --------------- --------------- Number of free parameters(t) 6 28 -2ln(L) 2631.209 2631.206 AIC (Akaike, 1974)* 2643.209 2687.206 BIC (Schwarz, 1978)* 2672.661 2824.651 *LISREL uses AIC= 2t - 2ln(L) and BIC = tln(N)- 2ln(L) Goodness of Fit Statistics Degrees of Freedom for (C1)-(C2) 22 Maximum Likelihood Ratio Chi-Square (C1) 0.00297 (P = 1.0000) Browne's (1984) ADF Chi-Square (C2_NT) 0.00297 (P = 1.0000) The Fit is Perfect ! Time used 0.060 seconds
23
0
App
endi
x I –
CoI
and
Sat
isfa
ctio
n In
stru
men
t Use
d fo
r St
uden
t Dat
a C
olle
ctio
n D
irec
tions
: Ple
ase
writ
e an
d bu
bble
in y
our G
Num
ber o
n th
e sc
antro
n, n
o ot
her i
nfor
mat
ion
is n
eces
sary
abo
ut th
e co
urse
, etc
…
Plea
se c
onsi
der t
he le
ctur
e po
rtion
of y
our C
HM
115
cou
rse
as y
ou a
nsw
er th
e fo
llow
ing
ques
tions
. Bub
ble
in th
e ap
prop
riate
cho
ice
on y
our
scan
tron.
Tha
nk y
ou fo
r you
r hel
p in
this
dat
a co
llect
ion.
St
rong
ly
Dis
agre
e D
isag
ree
Neu
tral
A
gree
St
rong
ly
Agr
ee
1. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt c
ours
e to
pics
A
B
C
D
E
2.
The
inst
ruct
or c
lear
ly c
omm
unic
ated
impo
rtant
cou
rse
goal
s A
B
C
D
E
3.
The
inst
ruct
or p
rovi
ded
clea
r ins
truct
ions
on
how
to p
artic
ipat
e in
cou
rse
lear
ning
act
iviti
es
A
B
C
D
E
4. T
he in
stru
ctor
cle
arly
com
mun
icat
ed im
porta
nt d
ue d
ates
/tim
e fr
ames
for
cour
se le
arni
ng a
ctiv
ities
A
B
C
D
E
5. T
he in
stru
ctor
was
hel
pful
in fa
cilit
atin
g di
scus
sion
s on
cour
se to
pics
that
he
lped
me
to le
arn
A
B
C
D
E
6. T
he in
stru
ctor
was
hel
pful
in g
uidi
ng th
e cl
ass t
owar
ds u
nder
stan
ding
cou
rse
topi
cs in
a w
ay th
at h
elpe
d m
e cl
arify
my
thin
king
A
B
C
D
E
7. T
he in
stru
ctor
hel
ped
to k
eep
cour
se p
artic
ipan
ts e
ngag
ed a
nd p
artic
ipat
ing
in
prod
uctiv
e di
alog
ue
A
B
C
D
E
8. T
he in
stru
ctor
hel
ped
keep
the
cour
se p
artic
ipan
ts o
n ta
sk in
a w
ay th
at h
elpe
d m
e to
lear
n A
B
C
D
E
9. T
he in
stru
ctor
enc
oura
ged
cour
se p
artic
ipan
ts to
exp
lore
new
con
cept
s in
this
co
urse
A
B
C
D
E
10. T
he in
stru
ctor
rein
forc
ed th
e de
velo
pmen
t of a
sens
e of
com
mun
ity a
mon
g co
urse
par
ticip
ants
A
B
C
D
E
11. T
he in
stru
ctor
hel
ped
to fo
cus d
iscu
ssio
n on
rele
vant
issu
es in
a w
ay th
at
help
ed m
e to
lear
n
A
B
C
D
E
12. T
he in
stru
ctor
pro
vide
d fe
edba
ck th
at h
elpe
d m
e un
ders
tand
my
stre
ngth
s an
d w
eakn
esse
s rel
ativ
e to
the
cour
se’s
goa
ls a
nd o
bjec
tives
A
B
C
D
E
13. T
he in
stru
ctor
pro
vide
d fe
edba
ck in
a ti
mel
y fa
shio
n A
B
C
D
E
14
. Get
ting
to k
now
oth
er c
ours
e pa
rtici
pant
s gav
e m
e a
sens
e of
bel
ongi
ng in
th
e co
urse
. A
B
C
D
E
15. I
was
abl
e to
form
dis
tinct
impr
essi
ons o
f som
e co
urse
par
ticip
ants
A
B
C
D
E
16
. Fac
e-to
-fac
e co
mm
unic
atio
n is
an
exce
llent
med
ium
for s
ocia
l int
erac
tion
A
B
C
D
E
17. I
felt
com
forta
ble
conv
ersi
ng fa
ce-to
-fac
e in
cla
ss
A
B
C
D
E
18. I
felt
com
forta
ble
parti
cipa
ting
in th
e co
urse
dis
cuss
ions
A
B
C
D
E
230
23
1
St
rong
ly
Dis
agre
e D
isag
ree
Neu
tral
A
gree
St
rong
ly
Agr
ee
19. I
felt
com
forta
ble
inte
ract
ing
with
oth
er c
ours
e pa
rtici
pant
s A
B
C
D
E
20
. I fe
lt co
mfo
rtabl
e di
sagr
eein
g w
ith o
ther
cou
rse
parti
cipa
nts w
hile
still
m
aint
aini
ng a
sens
e of
trus
t A
B
C
D
E
21. I
felt
that
my
poin
t of v
iew
was
ack
now
ledg
ed b
y ot
her c
ours
e pa
rtici
pant
s A
B
C
D
E
22
. In-
clas
s dis
cuss
ions
hel
ped
me
to d
evel
op a
sens
e of
col
labo
ratio
n A
B
C
D
E
23
. Pro
blem
s pos
ed in
crea
sed
my
inte
rest
in c
ours
e is
sues
A
B
C
D
E
24
. Cou
rse
lear
ning
act
iviti
es p
ique
d m
y cu
riosi
ty
A
B
C
D
E
25. I
felt
mot
ivat
ed to
exp
lore
con
tent
rela
ted
ques
tions
A
B
C
D
E
26
. I u
tiliz
ed a
var
iety
of i
nfor
mat
ion
sour
ces t
o ex
plor
e pr
oble
ms p
osed
in th
is
cour
se
A
B
C
D
E
27. B
rain
stor
min
g he
lped
me
reso
lve
cont
ent r
elat
ed q
uest
ions
A
B
C
D
E
28
. Fin
ding
rele
vant
info
rmat
ion
help
ed m
e re
solv
e co
nten
t rel
ated
que
stio
ns
A
B
C
D
E
29. P
leas
e se
lect
“D
isag
ree”
for t
his i
tem
A
B
C
D
E
30
. In-
clas
s dis
cuss
ions
wer
e va
luab
le in
hel
ping
me
appr
ecia
te d
iffer
ent
pers
pect
ives
A
B
C
D
E
31. C
ombi
ning
new
info
rmat
ion
help
ed m
e an
swer
que
stio
ns ra
ised
in c
ours
e le
arni
ng a
ctiv
ities
A
B
C
D
E
32. C
ours
e le
arni
ng a
ctiv
ities
hel
ped
me
cons
truct
exp
lana
tions
/sol
utio
ns
A
B
C
D
E
33. R
efle
ctio
n on
cou
rse
cont
ent h
elpe
d m
e un
ders
tand
fund
amen
tal c
once
pts i
n th
is c
lass
A
B
C
D
E
34. R
efle
ctio
n on
dis
cuss
ions
hel
ped
me
unde
rsta
nd fu
ndam
enta
l con
cept
s in
this
cl
ass
A
B
C
D
E
35. I
can
des
crib
e w
ays t
o te
st a
nd a
pply
the
know
ledg
e cr
eate
d in
this
cou
rse
A
B
C
D
E
36. I
hav
e de
velo
ped
solu
tions
to c
ours
e pr
oble
ms t
hat c
an b
e ap
plie
d in
pra
ctic
e
A
B
C
D
E
37. I
can
app
ly th
e kn
owle
dge
crea
ted
in th
is c
ours
e to
my
wor
k or
oth
er n
on-
clas
s rel
ated
act
iviti
es
A
B
C
D
E
Dir
ectio
ns: A
list
of o
ppos
ing
wor
ds a
ppea
rs b
elow
. Rat
e ho
w w
ell t
hese
wor
ds d
escr
ibed
you
r fee
lings
abo
ut th
e le
ctur
e po
rtion
of y
our
chem
istry
cou
rse.
For
eac
h lin
e, c
hoos
e a
posi
tion
betw
een
the
two
wor
ds th
at d
escr
ibes
exa
ctly
how
you
felt.
The
mid
dle
posi
tion
is if
you
are
un
deci
ded
or h
ave
no fe
elin
gs re
late
d to
the
term
s on
that
line
. T
HE
CH
EM
IST
RY
CO
UR
SE W
AS…
M
iddl
e
38.
C
omfo
rtabl
e A
B
C
D
E
Unc
omfo
rtabl
e 39
.
S
atis
fyin
g A
B
C
D
E
Frus
tratin
g 40
.
Ple
asan
t A
B
C
D
E
Unp
leas
ant
41.
Cha
otic
A
B
C
D
E
Org
aniz
ed
231
232
Appendix J – Interview Transcript and Course Syllabus Coding Rubric Scoring system: 0 = no evidence of indicator in either interview or syllabus 1 = some evidence of indicator, possibly indirect 2 = definite evidence of indicator in either interview transcript or syllabus 3 = definite evidence of indicator in both interview transcript or syllabus, or multiple mentions in either interview transcript or syllabus Instructor Code:
Indicator of student-centered constructivist learning environment Score Evidence/Quote
Instructor plans to spend some portion of class time not lecturing so that students can participate in some type of learning activity/problem solving
Instructor describes his or her role as providing guidance for or facilitating students’ own learning
Instructor understands student learning to be a process that involves students actively engaging with material and/or constructing their own knowledge
Instructor asks (or requires) students to work in groups during learning activities/problem solving
Instructor emphasizes students understanding underlying concepts in addition to being able to solve mathematical problems
Instructor incorporates authentic problem solving tasks
Learning activities/problem solving tasks require students to test or apply their knowledge
Instructor utilizes discussions as a way to probe student understanding
233
Appendix K – Mplus Model Syntax
One-factor teaching presence model with 13 items and auxiliary variables TITLE: One factor model for teaching presence with 13 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999);
AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T1-T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES; One-factor teaching presence model with 13 items, auxiliary variables, and error covariance TITLE: One factor model for teaching presence with 13 items add AUX variables and T12 T13 error covariance DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T1-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;
234
Two-factor teaching presence model with 13 items, auxiliary variables, and error covariance TITLE: Two factor model for teaching presence with 13 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: PRECOURSE BY T1-T4; INCURS BY T5-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES; One-factor teaching presence model with 9 items, auxiliary variables, and error covariance TITLE: One factor model for teaching presence with 9 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T5-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;
235
Three-factor CoI model with auxiliary variables and error covariance terms TITLE: Three factor model for CoI instrument with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T5-T13 S1-S9 C1-C14; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;
236
Measurement model with error covariance terms TITLE: Measurement model with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;
237
Structural model with error covariance terms TITLE: Structural model with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; SOCIAL ON TEACH; COGNITIVE ON TEACH SOCIAL; SATISF ON TEACH COGNITIVE SOCIAL; CHEM ON TEACH COGNITIVE MATH; GRADE ON TEACH COGNITIVE MATH CHEM; SATISF WITH GRADE; TEACH WITH MATH@0; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;
238
Structural model requesting bootstrapped confidence intervals for indirect effects TITLE: Structural model with 9 item teaching presence and bootstrapped CIs DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS:
BOOTSTRAP = 250; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; SOCIAL ON TEACH; COGNITIVE ON TEACH SOCIAL; SATISF ON TEACH COGNITIVE SOCIAL; CHEM ON TEACH COGNITIVE MATH; GRADE ON TEACH COGNITIVE MATH CHEM; SATISF WITH GRADE; TEACH WITH MATH@0; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; MODEL INDIRECT: CHEM IND TEACH; CHEM IND SOCIAL; GRADE IND TEACH; GRADE IND SOCIAL; GRADE IND COGNITIVE; SATISF IND TEACH; SATISF IND SOCIAL; COGNITIVE IND TEACH; OUTPUT:
SAMPSTAT STANDARDIZED MODINDICES CINTERVAL(bootstrap);
239
Appendix L – Student Variable Correlations from Mplus Table 23 Correlation Matrix from Mplus COURSE ACS ACT T5 T6 T7 T8 T9 T10 T11 T12 T13 COURSE 1.000
ACS 0.856 1.000 ACT 0.450 0.496 1.000
T5 0.230 0.160 0.062 1.000 T6 0.313 0.252 0.188 0.718 1.000 T7 0.140 0.084 -0.005 0.589 0.550 1.000 T8 0.288 0.203 0.114 0.623 0.684 0.644 1.000 T9 0.185 0.109 -0.019 0.474 0.415 0.474 0.446 1.000
T10 0.034 -0.030 -0.061 0.439 0.353 0.465 0.380 0.511 1.000 T11 0.198 0.110 0.057 0.558 0.649 0.503 0.617 0.462 0.386 1.000 T12 0.270 0.179 0.051 0.507 0.544 0.411 0.471 0.362 0.266 0.445 1.000 T13 0.215 0.169 0.125 0.471 0.568 0.425 0.460 0.400 0.334 0.454 0.553 1.000
S1 0.064 0.010 -0.008 0.283 0.291 0.310 0.289 0.286 0.441 0.314 0.205 0.319 S2 0.056 -0.007 -0.022 0.183 0.207 0.191 0.214 0.263 0.341 0.265 0.187 0.216 S3 0.115 0.063 0.019 0.295 0.317 0.259 0.334 0.275 0.325 0.291 0.261 0.401 S4 0.167 0.126 0.052 0.333 0.393 0.283 0.387 0.286 0.291 0.398 0.273 0.325 S5 0.194 0.166 0.041 0.517 0.487 0.405 0.505 0.322 0.359 0.475 0.361 0.347 S6 0.147 0.081 0.005 0.351 0.326 0.278 0.316 0.320 0.348 0.331 0.161 0.233 S7 0.178 0.134 0.054 0.322 0.301 0.292 0.332 0.346 0.342 0.415 0.228 0.172 S8 0.261 0.215 0.104 0.413 0.354 0.338 0.413 0.336 0.348 0.479 0.280 0.257 S9 0.209 0.148 -0.005 0.499 0.408 0.452 0.481 0.415 0.430 0.495 0.375 0.258 C1 0.304 0.254 0.112 0.424 0.455 0.390 0.427 0.459 0.323 0.405 0.369 0.379 C2 0.279 0.233 0.075 0.345 0.374 0.345 0.372 0.442 0.309 0.396 0.297 0.363 C3 0.300 0.241 0.072 0.390 0.415 0.381 0.424 0.463 0.347 0.397 0.413 0.336 C4 0.194 0.060 -0.099 0.236 0.236 0.223 0.266 0.377 0.276 0.228 0.233 0.236 C5 0.207 0.111 -0.046 0.371 0.362 0.310 0.299 0.384 0.337 0.313 0.308 0.347 C6 0.287 0.189 0.047 0.350 0.393 0.323 0.355 0.343 0.320 0.390 0.356 0.391 C7 0.170 0.066 0.002 0.459 0.459 0.462 0.419 0.382 0.402 0.437 0.314 0.391 C8 0.251 0.161 0.002 0.443 0.438 0.399 0.456 0.428 0.387 0.454 0.377 0.379 C9 0.275 0.196 0.094 0.543 0.570 0.469 0.510 0.473 0.380 0.507 0.457 0.461
C10 0.319 0.238 0.109 0.461 0.508 0.427 0.470 0.468 0.388 0.462 0.432 0.352 C11 0.269 0.175 0.025 0.479 0.576 0.469 0.505 0.495 0.414 0.551 0.472 0.453 C12 0.358 0.304 0.165 0.499 0.607 0.435 0.498 0.451 0.345 0.493 0.498 0.522 C13 0.368 0.295 0.164 0.422 0.508 0.420 0.482 0.467 0.332 0.456 0.448 0.465 C14 0.293 0.252 0.126 0.381 0.464 0.374 0.421 0.422 0.317 0.454 0.390 0.385 SS1 -0.521 -0.433 -0.219 -0.410 -0.482 -0.267 -0.453 -0.286 -0.190 -0.435 -0.364 -0.349 SS2 -0.501 -0.447 -0.222 -0.472 -0.565 -0.320 -0.495 -0.307 -0.202 -0.453 -0.384 -0.365 SS3 -0.479 -0.418 -0.237 -0.469 -0.505 -0.332 -0.457 -0.348 -0.238 -0.424 -0.391 -0.400 SS4 0.242 0.249 0.052 0.402 0.395 0.300 0.339 0.253 0.221 0.312 0.319 0.347
Note. COURSE = final course grade; ACS = ACS exam scores; ACT = ACT math scores; T = teaching presence item; S = social presence item; C = cognitive presence item; SS = satisfaction item
240
Table 23, continued Correlation Matrix from Mplus S1 S2 S3 S4 S5 S6 S7 S8 S9 C1 C2 C3 C4
S1 1.000 S2 0.626 1.000 S3 0.415 0.403 1.000 S4 0.467 0.433 0.625 1.000 S5 0.459 0.398 0.441 0.631 1.000 S6 0.575 0.540 0.484 0.671 0.574 1.000 S7 0.481 0.452 0.304 0.515 0.524 0.628 1.000 S8 0.435 0.421 0.318 0.489 0.541 0.561 0.636 1.000 S9 0.479 0.399 0.346 0.360 0.556 0.469 0.506 0.565 1.000 C1 0.398 0.264 0.267 0.305 0.430 0.338 0.310 0.378 0.422 1.000 C2 0.396 0.261 0.258 0.238 0.387 0.335 0.314 0.332 0.421 0.691 1.000 C3 0.400 0.303 0.279 0.313 0.413 0.338 0.359 0.338 0.415 0.706 0.715 1.000 C4 0.240 0.195 0.281 0.154 0.214 0.244 0.240 0.238 0.296 0.341 0.387 0.471 1.000 C5 0.367 0.302 0.276 0.250 0.326 0.362 0.331 0.325 0.400 0.444 0.487 0.486 0.452 C6 0.405 0.297 0.255 0.298 0.364 0.350 0.394 0.402 0.380 0.440 0.415 0.480 0.477 C7 0.458 0.341 0.268 0.311 0.411 0.442 0.422 0.498 0.509 0.455 0.425 0.422 0.305 C8 0.333 0.331 0.304 0.326 0.411 0.381 0.386 0.473 0.473 0.482 0.462 0.481 0.394 C9 0.413 0.293 0.308 0.351 0.398 0.380 0.360 0.442 0.464 0.538 0.505 0.557 0.363
C10 0.336 0.289 0.294 0.347 0.388 0.349 0.382 0.400 0.458 0.566 0.518 0.598 0.362 C11 0.409 0.318 0.325 0.418 0.454 0.377 0.371 0.460 0.529 0.521 0.512 0.580 0.356 C12 0.323 0.241 0.347 0.377 0.418 0.314 0.288 0.345 0.427 0.566 0.524 0.609 0.323 C13 0.330 0.262 0.319 0.352 0.391 0.342 0.321 0.363 0.440 0.543 0.498 0.590 0.388 C14 0.315 0.273 0.216 0.260 0.331 0.305 0.277 0.361 0.393 0.521 0.496 0.576 0.341 SS1 -0.297 -0.236 -0.224 -0.320 -0.416 -0.323 -0.349 -0.387 -0.412 -0.406 -0.391 -0.418 -0.155 SS2 -0.200 -0.132 -0.137 -0.214 -0.334 -0.205 -0.218 -0.277 -0.332 -0.471 -0.438 -0.482 -0.181 SS3 -0.232 -0.172 -0.165 -0.277 -0.379 -0.271 -0.275 -0.357 -0.365 -0.504 -0.429 -0.493 -0.219 SS4 0.208 0.083 0.209 0.218 0.284 0.236 0.200 0.223 0.227 0.318 0.234 0.304 0.082 Note. S = social presence item; C = cognitive presence item; SS = satisfaction item
241
Table 23, continued Correlation Matrix from Mplus
C5 C6 C7 C8 C9 C10 C11 C12 C13 C4 SS1 SS2 SS3 C5 1.000 C6 0.605 1.000 C7 0.408 0.456 1.000 C8 0.458 0.514 0.580 1.000 C9 0.481 0.547 0.560 0.664 1.000
C10 0.444 0.508 0.482 0.554 0.673 1.000 C11 0.506 0.489 0.524 0.572 0.674 0.710 1.000 C12 0.482 0.437 0.437 0.529 0.627 0.598 0.673 1.000 C13 0.505 0.528 0.444 0.560 0.647 0.603 0.617 0.727 1.000 C14 0.372 0.416 0.416 0.516 0.535 0.554 0.591 0.658 0.644 1.000 SS1 -0.315 -0.337 -0.360 -0.403 -0.409 -0.446 -0.401 -0.441 -0.474 -0.414 1.000 SS2 -0.325 -0.290 -0.317 -0.411 -0.466 -0.446 -0.441 -0.502 -0.463 -0.482 0.712 1.000 SS3 -0.325 -0.310 -0.367 -0.429 -0.429 -0.457 -0.469 -0.516 -0.491 -0.518 0.726 0.792 1.000 SS4 0.196 0.224 0.253 0.341 0.345 0.400 0.388 0.383 0.315 0.327 -0.499 -0.454 -0.505 Note. C = cognitive presence item; SS = satisfaction item
242
References Abraham, M. R. (2008). Importance of a theoretical framework for research. In D. M. Bunce &
R. S. Cole (Eds.), Nuts and bolts of chemical education research (pp. 47–66). Washington,
DC: American Chemical Society. doi:10.1021/bk-2008-0976.ch005
ACT. (2015). Understand your scores. Retrieved October 21, 2015, from
http://www.actstudent.org/scores/understand/
American Educational Research Association, American Psychological Association, & National
Council on Measurement in Education. (2014). Standards for educational & psychological
testing. Washington, DC: American Educational Research Association.
Arbaugh, J. B. (2000). Virtual classroom characteristics and student satisfaction with internet-
based MBA Courses. Journal of Management Education, 24(1), 32–54.
doi:10.1177/105256290002400104
Arbaugh, J. B. (2007). An empirical verification of the Community of Inquiry framework.
Journal of Asynchronous Learning Networks, 11(1), 73–85.
Arbaugh, J. B. (2008). Does the community of inquiry framework predict outcomes in online
MBA courses? International Review of Research in Open and Distance Learning, 9(2), 1–
14.
Arbaugh, J. B., Bangert, A. W., & Cleveland-Innes, M. (2010). Subject matter effects and the
Community of Inquiry (CoI) framework: An exploratory study. The Internet and Higher
Education, 13(1-2), 37–44. doi:10.1016/j.iheduc.2009.10.006
243
Arbaugh, J. B., & Benbunan-Fich, R. (2007). The importance of participant interaction in online
environments. Decision Support Systems, 43(3), 853–865. doi:10.1016/j.dss.2006.12.013
Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., &
Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of
the Community of Inquiry framework using a multi-institutional sample. Internet and
Higher Education, 11(3-4), 133–136. doi:10.1016/j.iheduc.2008.06.003
Arjoon, J. A., Xu, X., & Lewis, J. E. (2013). Understanding the state of the art for measurement
in chemistry education research: Examining the psychometric evidence. Journal of
Chemical Education, 90(5), 536–545. doi:10.1021/ed3002013
Ausubel, D. P. (1960). The use of advance organizers in the learning and retention of meaningful
verbal material. Journal of Educational Psychology, 51(5), 267–272. doi:10.1037/h0046669
Bächtold, M. (2013). What do students “construct” according to constructivism in science
education? Research in Science Education, 43, 2477–2496. doi:10.1007/s11165-013-9369-7
Bangert, A. W. (2008). The development and validation of the student evaluation of online
teaching effectiveness. Computers in the Schools, 25(1-2), 25–47.
doi:10.1080/07380560802157717
Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal
of Educational and Behavioral Statistics, 28(2), 135–167. doi:10.3102/10769986028002135
Bernal, P. J. (2006). Addressing the philosophical confusion regarding constructivism in
chemical education. Journal of Chemical Education, 83(2), 324–326.
doi:10.1021/ed083p324
244
Bodner, G. M. (1986). Constructivism: A theory of knowledge. Journal of Chemical Education,
63, 873–878.
Bodner, G. M., Klobuchar, M., & Geelan, D. (2001). The many forms of constructivism. Journal
of Chemical Education, 78(8), 1107. doi:10.1021/ed078p1107.4
Bolliger, D. U., & Wasilik, O. (2012). Student satisfaction in large undergraduate online courses.
Quarterly Review of Distance Education, 13(3), 153–165.
Brandriet, A., Reed, J. J., & Holme, T. (2015). A historical investigation into item formats of
ACS Exams and their relationships to science practices. Journal of Chemical Education,
150817104855004. doi:10.1021/acs.jchemed.5b00459
Bunce, D. M. (2001). Does Piaget still have anything to say to chemists? Journal of Chemical
Education, 78(8), 1107. doi:10.1021/ed078p1107.2
Chambers, K., & Blake, B. (2007). Enhancing student performance in first-semester general
chemistry using active feedback through the World Wide Web. Journal of Chemical
Education, 84(7), 1130–1135.
Coll, R. K., France, B., & Taylor, I. (2005). The role of models/and analogies in science
education: implications from research. International Journal of Science Education, 27(2),
183–198. doi:10.1080/0950069042000276712
Coll, R. K., & Taylor, T. G. N. (2001). Using constructivism to inform tertiary chemistry
pedagogy. Chemistry Education Research and Practice, 2(3), 215. doi:10.1039/b1rp90024b
245
Committee on Developments in the Science of Learning. (2000). How people learn: Brain, mind,
experience, and school. (J. D. Bransford, A. L. Brown, & R. R. Cocking, Eds.).
Washington, DC: National Academy Press.
Conway, C. J. (2014). Effects of guided inquiry versus lecture instruction on final grade
distribution in a one-semester organic and biochemistry course. Journal of Chemical
Education, 91, 480–483. doi:10.1021/ed300137z
Craney, C. L., & Armstrong, R. W. (1985). Predictors of grades in general chemistry for allied
health students. Journal of Chemical Education, 62(2), 127–129.
Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five
approaches (3rd ed.). Los Angeles: SAGE.
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods
approaches (4th ed.). Los Angeles: SAGE.
Creswell, J. W., & Clark, V. L. P. (2007). Choosing a mixed methods design. In Designing and
conducting mixed methods research (pp. 53–106). Thousand Oaks, CA: Sage.
DeFever, R. S., Bruce, H., & Bhattacharyya, G. (2015). Mental rolodexing: Senior chemistry
majors’ understanding of chemical and physical properties. Journal of Chemical Education,
92(3), 415–426. doi:10.1021/ed500360g
Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and
delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational
communications and technology (pp. 170–198). New York: Macmillan Reference USA.
246
Enders, C. K. (2013). Analyzing structural equation models with missing data. In G. R. Hancock
& R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 493–
519). Charlotte, NC: Information Age Publishing.
Ferguson, R. L. (2007). Constructivism and social constructivism. In G. M. Bodner & M. Orgill
(Eds.), Theoretical frameworks for research in chemistry/science education (pp. 28–49).
Upper Saddle River, NJ: Pearson Education.
Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics (4th ed.). Los Angeles:
SAGE.
Finney, S. J., & DiStefano, C. (2013). Non-normal and categorical data in structural equation
modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A
second course (2nd ed., pp. 439–492). Charlotte, NC: Information Age Publishing.
doi:10.1038/sj.bjc.6602399
Flynn, A. B., & Ogilvie, W. W. (2015). Mechanisms before reactions: A mechanistic approach to
the organic chemistry curriculum based on patterns of electron flow. Journal of Chemical
Education, 92(5), 803–810. doi:10.1021/ed500284d
Fosnot, C. T., & Perry, R. S. (2005). Constructivism: A psychological theory of learning. In C.
T. Fosnot (Ed.), Constructivism: Theory, perspectives, and practice (2nd ed., pp. 8–33).
New York: Teachers College Press.
Fox, R. (2001). Constructivism examined. Oxford Review of Education, 27(1), 23–35.
247
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., &
Wenderoth, M. P. (2014). Active learning increases student performance in science,
engineering, and mathematics. Proceedings of the National Academy of Sciences of the
United States of America, 111(23), 8410–8415. doi:10.1073/pnas.1319030111
Gardner, P. L. (1975). Attitudes to science: A review. Studies in Science Education, 2(1), 1–41.
doi:10.1080/03057267508559818
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based
environment: Computer conferencing in higher education. The Internet and Higher
Education, 2(2-3), 87–105.
Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework:
Review, issues, and future directions. Internet and Higher Education, 10, 157–172.
doi:10.1016/j.iheduc.2007.04.001
Garrison, D. R., Cleveland-Innes, M., & Fung, T. (2004). Student role adjustment in online
communities of inquiry: Model and instrument validation. Journal of Asynchronous
Learning Network, 8(2), 61–74.
Garrison, D. R., Cleveland-Innes, M., & Fung, T. S. (2010). Exploring causal relationships
among teaching, cognitive and social presence: Student perceptions of the community of
inquiry framework. The Internet and Higher Education, 13(1-2), 31–36.
doi:10.1016/j.iheduc.2009.10.002
248
Garrison, J. (1995). Deweyan pragmatism and the epistemology of contemporary social
constructivism. American Educational Research Journal, 32(4), 716–740.
doi:10.3102/00028312032004716
Good, R. G., Wandersee, J. H., & St. Julien, J. (1993). Cautionary notes on the appeal of the new
“ism” (constructivism) in science education. In K. Tobin (Ed.), The practice of
constructivism in science education (pp. 71–87). Hillsdale: NJ: Lawrence Erlbaum
Associates.
Gosser, D. K., Kampmeier, J. A., & Varma-Nelson, P. (2010). Peer-led team learning: 2008
James Flack Norris award address. Journal of Chemical Education, 87(4), 374–380.
doi:10.1021/ed800132w
Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of
student ratings. The American Psychologist, 52(11), 1209–1217. doi:10.1037/0003-
066X.52.11.1209
Gupta, T., Burke, K. A., Mehta, A., & Greenbowe, T. J. (2015). Impact of guided-inquiry-based
instruction with a writing and reflection emphasis on chemistry students’ critical thinking
abilities. Journal of Chemical Education. doi:10.1021/ed500059r
Hall, D. M., Curtin-Soydan, A. J., & Canelas, D. A. (2014). The science advancement through
group engagement program: Leveling the playing field and increasing retention in science.
Journal of Chemical Education, 91(1), 37–47. doi:10.1021/ed400075n
249
Hancock, G. R. (1997). Structural equation modeling methods of hypothesis testing of latent
variable means. Measurement and Evaluation in Counseling and Development, 30(2), 91–
105.
Hancock, G. R. (2001). Effect size, power, and sample size determination for structured means
modeling and MIMIC approaches to between-groups hypothesis testing of means on a
single latent construct. Psychometrika, 66(3), 373–388. doi:10.1007/BF02294440
Hancock, G. R. (2006). Power analysis in covariance structure modeling. In G. R. Hancock & R.
O. Mueller (Eds.), Structural equation modeling: A second course (1st ed., pp. 69–115).
Charlotte, NC: Information Age Publishing.
Hancock, G. R., & French, B. F. (2013). Power analysis in structural equation modeling. In G. R.
Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed.,
pp. 117–159). Charlotte, NC: Information Age Publishing.
Hancock, G. R., & Mueller, R. O. (Eds.). (2006). Structural equation modeling: A second course
(1st ed.). Greenwich, CT: Information Age Publishing.
Hanson, D. M. (2006). Instructor’s guide to process-oriented guided-inquiry learning. Lisle, IL:
Pacific Crest.
Hanson, D. M. (2008). A cognitive model for learning chemistry and solving problems:
Implications for curriculum design and classroom instruction. In R. S. Moog & J. N.
Spencer (Eds.), Process oriented guided inquiry learning (POGIL) (Vol. 994, pp. 14–25).
Washington, DC: American Chemical Society. doi:10.1021/bk-2008-0994.ch002
250
Herron, J. D. (1975). Piaget for chemists. Journal of Chemical Education, 52(3), 146–150.
Howard, G. S., & Maxwell, S. E. (1982). Do grades contaminate student evaluations of
instruction? Research in Higher Education, 16(2), 175–188. doi:10.1007/BF00973508
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling: A
Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118
Huta, V. (2014). When to use hierarchical linear modeling. The Quantitative Methods for
Psychology, 10(1), 13–28.
Hyslop-Margison, E. J., & Strobel, J. (2008). Constructivism and education: Misunderstandings
and pedagogical implications. The Teacher Educator, 43, 72–86.
doi:10.1080/08878730701728945
Indiana University Center for Postsecondary Research. (n.d.). The Carnegie Classification of
Institutions of Higher Education, 2015 edition. Retrieved October 15, 2015, from
http://carnegieclassifications.iu.edu/
Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical
paradigm? Educational Technology Research and Development, 39(3), 5–14.
doi:10.1007/BF02296434
Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online university students’ satisfaction and
persistence: Examining perceived level of presence, usefulness and ease of use as predictors
in a structural model. Computers & Education, 57(2), 1654–1664.
doi:10.1016/j.compedu.2011.02.008
251
Jöreskog, K. G., & Sörbom, D. (2015). LISREL. Skokie, IL: Scientific Software International,
Inc. [Computer Software].
Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New
York: The Guilford Press.
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J.
D. Wright (Eds.), Handbook of survey research (2nd ed.). Bingley, UK: Emerald Group
Publishing Limited.
Lewis, S. E. (2014). Examining evidence for external and consequential validity of the first term
general chemistry exam from the ACS Examinations Institute. Journal of Chemical
Education. doi:10.1021/ed400819g
Lewis, S. E., & Lewis, J. E. (2005). Departing from lectures: An evaluation of a peer-led guided
inquiry alternative. Journal of Chemical Education, 82(1), 135–139.
doi:10.1021/ed082p135
Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to
parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9(2),
151–173.
Lund, T. J., Pilarz, M., Velasco, J. B., Chakraverty, D., Rosploch, K., Undersander, M., & Stains,
M. (2015). The best of both worlds: Building on the COPUS and RTOP observation
protocols to easily and reliably measure various levels of reformed instructional practice.
CBE Life Sciences Education, 14(2). doi:10.1187/cbe.14-10-0168
252
Matthews, M. R. (1993). Constructivism and science education: Some epistemological problems.
Journal of Science Education and Technology, 2(1), 359–370. doi:10.1007/BF00694598
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–
103). New York: Macmillan.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’
responses and performances as scientific inquiry into score meaning. American
Psychologist. doi:10.1037/0003-066X.50.9.741
Mitchell, Y. D., Ippolito, J., & Lewis, S. E. (2012). Evaluating Peer-Led Team Learning across
the two semester General Chemistry sequence. Chemistry Education Research and
Practice, 13(3), 378–383. doi:10.1039/c2rp20028g
Mueller, R. O., & Hancock, G. R. (2008). Best practices in structural equation modeling. In J. W.
Osborne (Ed.), Best practices in quantitative methods (pp. 488–508). Thousand Oaks, CA:
Sage Publications, Inc. doi:10.4135/9781412995627
Mueller, R. O., & Hancock, G. R. (2010). Structural equation modeling. In G. R. Hancock & R.
O. Mueller (Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp.
371–383). New York: Routledge.
Mugaloglu, E. Z. (2014). The problem of pseudoscience in science education and implications of
constructivist pedagogy. Science and Education, 23, 829–842. doi:10.1007/s11191-013-
9670-x
Muthén, L. K., & Muthén, B. O. (2010). Mplus User’s Guide. Seventh Edition. Los Angeles, CA:
Muthén & Muthén.
253
Nordstrom, B. (1990). Predicting performance in freshman chemistry. In National Meeting of the
American Chemical Soceity. Boston.
Norman, D. A. (1973). Cognitive organization and learning. La Jolla, CA.
Norman, D. A. (1980). What goes on in the mind of the learner. New Directions for Teaching
and Learning, (2), 37–49. doi:10.1002/tl.37219800205
Novak, J. (1984). Application of advances in learning theory and philosophy of science to the
improvement of chemistry teaching. Journal of Chemical Education, 61(7), 607–612.
doi:10.1021/ed061p607
Nurrenbern, S. C. (2001). Piaget’s theory of intellectual development revisited. Journal of
Chemical Education, 78(8), 1107–1110.
Partlow, K. M., & Gibbs, W. J. (2003). Indicators of constructivist principles in internet-based
courses. Journal of Computing in Higher Education, 14(2), 68–97.
doi:10.1007/BF02940939
Phillips, D. C. (1995). The good, the bad, and the ugly: The many faces of constructivism.
Educational Researcher, 24(7), 5–12. doi:10.3102/0013189X024007005
Piaget, J. (1973). To understand is to invent: The future of education. New York: Grossman.
Piaget, J. (1997). Development and learning. In M. Gauvain & M. Cole (Eds.), Readings on the
development of children (2nd ed., pp. 19–28). New York: W. H. Freeman (Reprinted from
Piaget Rediscovered, pp. 7-20, by R. E. Ripple & V. N. Rockcastle, Eds., 1964).
Preacher, K. J., & Coffman, D. L. (2006). Computing power and minimum sample size for
RMSEA [Computer software].
254
Prosser, M., & Trigwell, K. (2006). Confirmatory factor analysis of the Approaches to Teaching
Inventory. British Journal of Educational Psychology, 76, 405–419.
doi:10.1348/000709905X43571
Prosser, M., Trigwell, K., & Taylor, P. (1994). A phenomenographic study of academics’
conceptions of science learning and teaching. Learning and Instruction, 4(3), 217–231.
doi:10.1016/0959-4752(94)90024-8
R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria:
R Foundation for Statistical Computing [Computer Software].
Revelle, W. (2015). psych: Procedures for psychological, psychometric, and personality
research. Evanston, IL: Northwestern University.
Ruder, S. M., & Hunnicutt, S. S. (2008). POGIL in chemistry courses at a large urban university:
A case study. In R. S. Moog & J. N. Spencer (Eds.), Process oriented guided inquiry
learning (POGIL). doi:10.1021/bk-2008-0994.ch012
Scerri, E. R. (2003). Philosophical confusion in chemical education research. Journal of
Chemical Education, 80(5), 468–474. doi:10.1021/ed080p468
Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster
“epistemic engagement” and “cognitive presence” in online education. Computers and
Education, 52, 543–553. doi:10.1016/j.compedu.2008.10.007
Shea, P., Sau Li, C., & Pickett, A. (2006). A study of teaching presence and student sense of
learning community in fully online and web-enhanced college courses. The Internet and
Higher Education, 9(3), 175–190. doi:10.1016/j.iheduc.2006.06.005
255
Shin, N. (2003). Transactional presence as a critical predictor of success in distance learning.
Distance Education, 24(1), 69–86. doi:10.1080/01587910303048
Smith, J., Wilson, S. B., Banks, J., Zhu, L., & Varma-Nelson, P. (2014). Replicating Peer-Led
Team Learning in cyberspace: Research, opportunities, and challenges. Journal of Research
in Science Teaching, 51(6), 714–740. doi:10.1002/tea.21163
Spencer, H. E. (1996). Mathematical SAT test scores and college chemistry grades. Journal of
Chemical Education, 73(12), 1150–1153. doi:10.1021/ed073p1150
Spencer, J. N. (1992). General chemistry course content. Journal of Chemical Education, 69(3),
182–186.
Stains, M., Pilarz, M., & Chakraverty, D. (2015). Short and long-term impacts of the Cottrell
Scholars Collaborative New Faculty Workshop. Journal of Chemical Education, 92(9),
1466–1476. doi:10.1021/acs.jchemed.5b00324
Staver, J. R. (1998). Constructivism: Sound theory for explicating the practice of science and
science teaching. Journal of Research in Science Teaching, 35(5), 501–520.
Stoyanovich, C., Gandhi, A., & Flynn, A. B. (2015). Acid–base learning outcomes for students
in an introductory organic chemistry course. Journal of Chemical Education, 92(2), 220–
229. doi:10.1021/ed5003338
Suchting, W. A. (1992). Constructivism deconstructed. Science & Education, 1(3), 223–254.
doi:10.1007/BF00430275
256
Swan, K. P. (2003). Learning effectiveness online: What the research tells us. In J. Bourne & J.
C. Moore (Eds.), Elements of quality online education, practice and direction (pp. 13–45).
Needham, MA: Sloan Center for Online Education.
Swan, K. P., Garrison, D. R., & Richardson, J. C. (2009). A constructivist approach to online
learning: The Community of Inquiry framework. In C. R. Payne (Ed.), Information
technology and constructivism in higher education: Progressive learning frameworks (pp.
43–57). Hershey, PA: IGI Global. doi:10.4018/978-1-60566-654-9.ch004
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn
& Bacon/Pearson Education. doi:10.1037/022267
Tai, R. H., Sadler, P. M., & Mintzes, J. J. (2006). Factors influencing college science success.
Journal of College Science Teaching, 36(1), 56–60.
Talanquer, V. (2015). Threshold concepts in chemistry: The critical role of implicit schemas.
Journal of Chemical Education, 92(1), 3–9. doi:10.1021/ed500679k
Talanquer, V., & Pollard, J. (2010). Let’s teach how we think instead of what we know. Chem.
Educ. Res. Pract., 11(2), 74–83. doi:10.1039/C005349J
Tien, L. T., Roth, V., & Kampmeier, J. a. (2002). Implementation of a peer-led team learning
instructional approach in an undergraduate organic chemistry course. Journal of Research
in Science Teaching, 39(7), 606–632. doi:10.1002/tea.10038
Tobin, K. (1999). Constructivism in science education: Moving on. In D. C. Phillips (Ed.),
Constructivism in education: Opinions and second opinions on controversial issues (pp.
227–253). Chicago: National Society for the Study of Education.
257
Toto, J., & Booth, K. (2008). Effects and implications of mini-lectures on learning in first-
semester general chemistry. Chemistry Education Research and Practice, 9(3), 259.
doi:10.1039/b812415a
Trigwell, K., & Prosser, M. (1996). Congruence between intention and strategy in university
science teachers’ approaches to teaching. Higher Education, 32(1), 77–87.
Trigwell, K., & Prosser, M. (2004). Development and use of the approaches to teaching
inventory. Educational Psychology Review, 16(4), 409–424.
Trigwell, K., Prosser, M., & Ginns, P. (2005). Phenomenographic pedagogy and a revised
Approaches to teaching inventory. Higher Education Research & Development, 24(4), 349–
360. doi:10.1080/07294360500284730
Trigwell, K., Prosser, M., & Taylor, P. (1994). Qualitative differences in approaches to teaching
first year university science. Higher Education, 27(1), 75–84.
Varma-Nelson, P., & Banks, J. (2013). PLTL: Tracking the trajectory from face-to-face to online
environments. In T. Holme, M. M. Cooper, & P. Varma-Nelson (Eds.), Trajectories of
chemistry education innovation and reform (Vol. 1145, pp. 95–110). Washington, DC:
American Chemical Society. doi:10.1021/bk-2013-1145.ch007
Varma-Nelson, P., & Coppola, B. P. (2005). Team learning. In N. J. Pienta, M. M. Cooper, & T.
J. Greenbowe (Eds.), Chemists’ guide to effective teaching (pp. 155–169). Saddle River, NJ:
Pearson.
258
Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor analysis:
Some issues in selecting an appropriate procedure. Multivariate Behavioral Research,
25(1), 1–28. doi:10.1207/s15327906mbr2501_1
von Glasersfeld, E. (1974). Piaget and the radical constructivist epistemology. In C. D. Smock &
E. von Glasersfeld (Eds.), Epistemology and education (pp. 1–15). Athens, GA: Follow
Through Publications.
von Glasersfeld, E. (1984). An introduction to radical constructivism. In P. Watzlawick (Ed.),
The invented reality: How do we know what we believe we know? (pp. 17–40). New York:
Norton (English translation of work originally published in 1981).
von Glasersfeld, E. (1989). Cognition, construction of knowledge and teaching. Synthese:
History, Philosophy and Science Teaching, 80(1), 121–140.
von Glasersfeld, E. (1993). Questions and answers about radical constructivism. In K. Tobin
(Ed.), The practice of constructivism in science education (pp. 23–38). Hillsdale: NJ:
Lawrence Erlbaum Associates.
Vrasidas, C. (2000). Constructivism versus objectivism: Implications for interaction, course
design, and evaluation in distance education. International Journal of Educational
Telecommunications, 6, 339–362. doi:10.1080/0031383890330103
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. M.
Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Cambridge, MA: Harvard
University Press. doi:10.1007/978-3-540-92784-6
259
Williams, J., & MacKinnon, D. P. (2008). Resampling and distribution of the product methods
for testing indirect effects in complex models. Structural Equation Modeling: A
Multidisciplinary Journal, 15(1), 23–51. doi:10.1080/10705510701758166
Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An
analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers.
Review of Educational Research, 72(2), 131–175. doi:10.3102/00346543072002131
Wink, D. J. (2014). Constructivist frameworks in chemistry education and the problem of the
“Thumb in the Eye.” Journal of Chemical Education, 91(5), 617–622.
doi:10.1021/ed400739b
Wren, D., & Barbera, J. (2013). Gathering evidence for validity during the design, development,
and qualitative evaluation of thermochemistry concept inventory items. Journal of
Chemical Education, 90(12), 1590–1601. doi:10.1021/ed400384g
Wright, S. (1934). The method of path coefficients. The Annals of Mathematical Statistics, 5(3),
161–215.
Xu, X., & Lewis, J. E. (2011). Refinement of a chemistry attitude measure for college students.
Journal of Chemical Education, 88(5), 561–568. doi:10.1021/ed900071q