Complex Systems Research and Evaluation in Engineering EducationDr. Jonathan C. Hilpert, Georgia Southern University
Dr. Jonathan C. Hilpert is an Associate Professor of Educational Psychology in the Department of Cur- riculum Reading and Foundations in the College of Education at Georgia Southern University. His re- search interests include student motivation, engagement, and interactive learning; emergent and self- organizing properties of educational systems; and knowledge construction of complex scientific phe- nomena. He teaches courses in learning theories, research methods, and assessment and statistics. His research findings have been disseminated in national and international engineering education and psy- chological journals (including Journal of Engineering Education, Journal of Women and Minorities in Science and Engineering, Educational Psychology, Psychological Inquiry, and Motivation and Emotion) as well as presented at national and internal conferences (including American Education Research As- sociation, American Society for Engineering Education; European Association for Research in Learning and Instruction; and various symposiums organized by the National Science Foundation). He has a son named Gray and a dog named Argyle.
Gwen C. Marchand, University of Nevada, Las Vegas
Dr. Gwen Marchand is an associate professor of Educational Psychology and the Director of the Center for Research, Evaluation, and Assessment (CREA) at the University of Nevada, Las Vegas. Dr. Marchand received her doctorate in 2008 from Portland State University in Systems Science: Psychology. Her re- search investigates the development of student engagement from a complex systems perspective, focusing on the interaction of individual and contextual factors.
c©American Society for Engineering Education, 2017
Complex Systems Research and Evaluation in Engineering Education
Abstract
The purpose of this theory-to-practice paper is to discuss complex systems research needs within
engineering education. We provide a comprehensive definition of complex systems educational research
(Hilpert & Marchand, under review; Jacobson et al., 2016) and an overview of methods specific to the
approach (Hollenstein, 2013; Koopsman & Stavalomsis, 2016; Strogatz, 1994). After this, we delineate a
research-based framework that can be used to develop and conduct complex systems research and
evaluation. We identify two areas within the field of engineering education where complex systems
research can be useful: 1) educational research focused on student interaction and cognition and 2)
assessment and evaluation of collaboratives such as grant funded projects and communication/
publication networks. We discuss existing literature in these spaces, and then outline the critical research
needs for engineering education. We address each of these critical needs with an eye on theory as well as
methodological and analytic techniques that can be used to design and conduct complex systems research
and evaluation in engineering education settings and contexts. The result is a set of specific guidelines
that researchers can use to move complex systems research forward in engineering education. This
material is based upon work supported by the National Science Foundation under Grant Number NSF
DUE #1245018 and partial support was also provided by the National Institute of General Medical
Sciences, Grant No. P20GM109025.
Complex Systems Research and Evaluation in Engineering Education
Definition of Complex Systems Research
Complex systems research in education is focused on the complex, emergent, and dynamic
interactions among people, tasks, and settings (Hilpert & Marchand, under review). Complex systems are
collections of micro-level variables (often referred to as agents, components, or elements) that interact to
produce macro-level patterns and outcomes (Kauffman, 1993). Complexity is the mutually causal,
multiple interactions among components of a system (Mitchell, 2009). Emergence is the arisal of non-
reducible, macro level patterns from micro level interaction among systems components (Holland, 2000).
Dynamics is change in micro level variables in relation to each other, or in relation to the macro level,
across time (Richardson et al., 2014). Complex systems are considered interaction-dominant, as opposed
to component-dominant, because the focus of analysis is on the dynamic interaction among system
components as opposed to the replication of components and their nomothetic relationships (Holden, Van
Orden, Turvey, 2009).
In contrast to mechanical systems that presuppose linear mechanisms, complex systems assume
nonlinear relationships among variables and multiple levels of analysis. Whereas mechanical systems are
often described using stable, left-to-right style diagrams that portray their underlying causal mechanism, a
complex system contains an underlying network structure, with mutually causal relationships among
system components that produce emergent patterns at a higher level of analysis (Mitchell, 2009). The
network structure implies self-organizing properties, meaning there is little centralized control and system
homeostasis is produced through bottom-up interaction of system components (Dale, et al., 2014). The
complexity of a system, which is often measured as a balance between ordered and random interaction of
system micro components, is the necessary condition for the dynamic emergence of self-organization at
the macro level (Mitchell, 2009). Complex systems research is focused on the processes of stability and
change in a system, the critical values and initial conditions that lead to stability and change, and system
response to perturbation (Koopsman, 2016).
For example, recently researchers have begun to examine educational phenomena including
classroom learning (Conner et al., 2015; Hilpert & Husman, 2016), student-to-teacher interaction
(Hollenstein, 2013; Pennings & Mainhard, 2016), student collaborative groups (Hilpert, 2016;
Stamovlosis, 2016), and student cognition and engagement (Skinner et al., 2008; Stamovlosis & Sideridis,
2014) from a complex systems perspective. Moreover, complex systems thinking has found its way into
the evaluation literature as well, including the examination of how educational innovations diffuse among
educators and practitioners (Borrego, et al., 2010), the establishment of faculty to faculty collaborative
networks (Borrego & Newsander, 2008), publications networks for research and teaching (Madhavan, et
al., 2011), institution to institution networks (Berggren et al., 2003) for initiating and examining cultural,
institutional, and faculty change, and innovative methods for assessing research collaboratives (Marchand
& Hilpert, 2017). In each of these examples, key features of complex systems such as emergent
relationships between multiple levels of analysis, underlying network structures of information sharing
and collaboration, and dynamic mutually causal change have provided new research insights.
What these studies have to offer is unique conclusions about the adaptivity and functioning of a
bounded system. Much education research, both at large and within engineering education, is dominated
by a deductive-nomothetic paradigm that seeks to provide evidence for the replicability of linear
structures, mediating and moderating effects, or test interventions. Other qualitative approaches provide
their own unique insights as well. Complex systems research offers researchers methods for analyzing the
complex, dynamic, and emergent qualities of systems, allowing for conclusions about how adaptive a
systems is, how well information flows through a system, and under what conditions a system is likely to
undergo shifts in behavior that influence its functioning. Complex systems research, and the conclusions
that can be drawn from it, can complement existing research but require that researchers understand the
methodology (theory, philosophy, and assumptions of research), method (techniques and procedures to
gather evidence), and practice (analytic techniques and takeaways) related to the approach (i.e. Harding,
1987; Schwandt, 2001; Twining, Heller, Nussbaum, & Tsai, 2017).
Complex Systems Research Methodology and Method
Because complex systems are interaction-dominant, they require research methods that account
for the dynamic interplay among variables. Any system that competes, collaborates, or interferes contains
dynamic interplay that produces non-linear behavior (Strogatz, 1994); accordingly, nonlinear methods, or
a combination of qualitative, linear, and nonlinear methods, are used in educational research to study
complex systems (i.e. Hilpert & Marchand, under review; Jacobson, et al., 2016; Koopsman &
Stavalomsis, 2016). In contrast to the use of linear statistical techniques (i.e. Fisher, 1925) which assume
independent observations, homogeneity of variance, and an underlying normal distribution, nonlinear
techniques can be used to study sequential dependence, highly variable change, and underlying power law
distributions. Hilpert and Marchand (under review) categorize the nonlinear techniques useful for
complex systems research into three categories: time series analysis, nonlinear dynamical modeling, and
network analysis. Koopsman and Stavalomsis (2016) and Jacobson and colleagues (2016) also contain
descriptions and examples of these techniques in educational contexts.
Typically the goal of complex systems research is to produce evidence for adaptive system
functioning, or evidence for stability and change within systems and the conditions and critical values that
produce stability or change. For example, nonlinear time series analysis is often used to test the change in
complexity hypotheses (CICH; Sleimen-Malkoun, et al., 2015), where time series signals are analyzed to
produce evidence for the deterministic, random, and stochastic characteristics of a data stream (Gao, et
al., 2007; Riley & van Orden, 2005). Results can be used to determine if a system is losing complexity, or
becoming overly ordered or overly complex in its dynamic interplay. Or, nonlinear dynamic modeling (of
both simulated or observed data) may be used to produce evidence for attractor states (Hollenstein, 2013;
Strogatz, 1994), which are the tendency for systems to settle into patterned homeostasis. Both nonlinear
time series analysis and nonlinear dynamic modeling can be used to determine if there are critical values
or contextual factors that produce attractor states, or phase transitions between attractor states.
Additionally, network analysis can be used to model the underlying network structure of a bounded
system, producing evidence for endogenous and exogenous effects, and well as influence and selection
effects, among system components (Lusher, et al., 2014; Krackhardt, 1987). Below we discuss some
relevant complex systems approaches and the application of related analytic techniques to research and
evaluation in engineering education.
Complex Systems Research Domains in Engineering Education
Engineering education is a broad field of study, replete with its own cultural value system and
ways of conducting research. It is probably reasonable to suggest that the prevailing research paradigms
in engineering education are largely characterized by deductive-nomothetic, phenomenological, and
critical perspectives. However, despite the predominance of these approaches, within the larger milieu of
engineering education much work is devoted to the study of complex, emergent, and dynamic phenomena
but investigated using methods that do not assess these characteristics. Complex systems research
approaches are uniquely positioned to provide improved understanding in two engineering education
domains: 1) educational research focused on classroom interaction/ learning and student cognition, and 2)
evaluation and assessment of collaboratives such as grant funded projects and communication/
publication networks.
Many research, evaluation, and assessment pursuits in engineering education target interaction-
dominant phenomena, and thus may also lend themselves to complex systems approaches. These methods
don’t serve to replace existing research traditions and paradigms, but rather complement them by
providing new insights and forms of evidence that can be used for improved understanding and decision
making (i.e. Carver & Schiere, 2001; Davis & Sumara, 2006). There are a variety of domains where
complex systems thinking can provide new insights and ways of thinking about research. Below we
examine some possibilities by focusing on existing research where elements of complex systems have
been incorporated, or show promise for examination using complex systems methodologies. Each section
begins with a discussion of the interaction-dominant nature of the domain, and is followed by a brief
discussion of existing research or examples relevant to engineering education.
Educational research. Educational phenomena such as classroom learning and interaction,
student collaboration, and motivation, cognition, and engagement (Ohland, et al., 2008; Benson, Kirn, &
Faber, 2013; Felder & Brent, 2016; Vogt, 2008) all contain central features of interaction-dominant
complex systems. These features include complex, dynamic qualities that produce emergent outcomes
(Kaplan, et al., 2012; Mitchell, 2009; Richardsen, et al., 2014). Research conducted within learning
environments (i.e. classrooms, laboratories, etc.) necessarily involves the interaction of settings, tasks,
teachers, and students (Schwab, 1971) and the study of motivation and engagement involves competing
intraorganismic and extraorganismic factors (Deci & Ryan, 2002). Because cooperation, competition, and
interference are ever present features of these areas of study, changes in any system variables results in
changes to another and so on that reverberate throughout the learning context or the organism under study
creating dynamic change. Many engineering studies suggest that teacher to student and student to student
interaction produce complex, dynamic outcomes.
For example, Shekhar and colleagues (2015) demonstrate the conditions under which Research
Based Instructional Strategies (RBIS) (i.e. strategies that have demonstrated evidence of improved student
engagement) can lead to different forms of student resistance and disengagement in engineering contexts.
Husman and Hilpert (2016) show how different aggregate levels of engagement within engineering
classroom contexts can shift the direction and strength of motivational influence and student desire to
pursue a career in engineering. Brown and colleagues (2014) demonstrate the role of peer networks and
social capital in interactive engineering classrooms. And Hilpert and Husman (2016) developed a measure
of interactive engagement that describes the complex and adaptive ways students rely upon the social and
intellectual capital of their peers to develop innovative solutions to engineering problems. These studies
provide points of leverage for complex systems research to be integrated into studies of classroom
interaction and collaboration, instructional strategy use, and how environment shapes student learning.
Similar to the nature of learning environments, motivation, affect, and engagement are widely
accepted to be composed of complex combinations of self-organizing physiological and psychological
substratum (Ellis & Newton, 2000; Fried, et al., 2016; Reeve, 2014). The study of goal directed behavior
(Kirn & Benson, 2013), academic engagement (Ohland, et al., 2008), and affective/ motivated perceptions
of academic tasks and achievement related pursuits (Carberry, Lee, & Ohland, 2010; Nelson, et al., 2015;
Villanueva et al., 2016) are all areas of research common in engineering education with underlying
dynamic complexity that have not been thoroughly examined in engineering contexts. One common
strand in these studies is that the unique qualities of engineering culture and context are important to
student perceptions and motivational expression (i.e. Benson & Kirn, 2013), suggesting a complex and
dynamic interaction between context and learner that shapes the emergence of relevant affective and
motivational processes related to learning and knowledge construction in engineering education. The
underlying stability of student affect/ motivation and engagement, the environmental factors that
contribute to its dynamic change over time, and the meaningful levels of analysis and time frames of
study are all points of leverage for future research.
Evaluation and assessment. The establishment of faculty collaborative networks for research and
teaching (Madhavan, et al., 2011) and the diffusion of educational innovations among engineering
education faculty (Borrego, et al., 2010) possess underlying network structures that are the hallmark of
complex systems (Lusher, et al., 2012). Mutually causal network structures create the push and pull that
produces nonlinear dynamic change. What is not well understood are how these network structures
develop, spread, and exceed the initial boundaries of the founding collaborative group. Evaluation
approaches rooted in complex systems research methods can provide useful information about the spread
of information among collaborative partners, offering insight into how interactions that take place during
the course of collaboration contribute to the quality of products derived from collaborations (Welch,
Feeney, & Siciliano, 2016). Moreover, complex systems approaches to the evaluation of the diffusion of
innovation amongst faculty or research collaborations move the focus from discrete products or outcomes
to the creation of human and social capital from whence emerges new knowledge (Bozeman, Dietz &
Gaughan, 2001).
Collaborative work in the classroom, on grants, and across institutions offers unique demands,
particularly as much work in engineering education and other STEM fields brings together faculty and
students from multiple disciplines (Borrego & Newswander, 2008). Developing networks of actors from
diverse disciplinary groups may pose challenges due to specific localized definitions and meaning of
terms, paradigmatic differences in research philosophy and approaches, and discipline-specific norms for
conducting scholarly inquiry. Such differences often constitute a significant barrier to effective cross-
disciplinary scholarship. A shared language among collaborators is needed to coordinate ideas and action
when individuals interact to solve a problem (Molinari, et al., 2009). Creation of successful
interdisciplinary research teams that lead to information-information interaction (Qin, et al., 1997) require
explicit attention to the development of competencies needed for such a collaborative (Gebbie et al.,
2008). When members work together successfully to develop shared conceptual and methodological
frameworks, they transcend any one disciplinary perspective (Stokols et al., 2008).
These qualities of collaboration inherently draw upon complex systems concepts in that the
underlying structure of the collaborative is a network, actors within a network interact in a dynamic ways
around research and innovation, and hallmarks of successful and sustainable collaborations are emergent,
shared goals. Typical evaluation designs take a component-centered approach and apply designs that
measure anticipated outcomes (e.g., student performance, grant submissions) prior and subsequent to the
educational innovation or formation of collaboration. While these designs may be somewhat useful for
measuring a change in discrete, bounded outcomes they provide limited utility for understanding socially
embedded learning and change processes. Traditional evaluation designs provide no evidence for how ties
between actors (e.g., instructors or collaborators) form around the innovation or collaboration, whether
particular attributes of actor in the system are related to the development of networks, or how patterns of
dynamic change within the innovation or collaboration may be variable and when the system is
particularly open to change (accelerating growth, etc.) (Bozeman et al., 2001).
The number and type of funding mechanisms for collaboration that are designed to support
engineering education practice and research is on the rise (National Science Board, 2016; Xian &
Madhavan, 2012). For instance, multi-country and institution collaboratives have formed to reform
engineering education (e.g., Berggren et al., 2003); faculty and students from colleges of education &
engineering have formed research collaboratives to address particular issues in engineering and STEM
(Lohani, et al. 2004); and centers and institutes have emerged to support outreach and science in
engineering education (Bozeman & Boardman, 2004; Brophy, et al., 2008). These collaboratives offer a
second point of entry for complex systems research and evaluation in engineering education to take hold.
Complex systems approaches lead to evaluation that is designed to provide evidence of system
functioning and information diffusion. This evidence can be used to make changes to a program to disrupt
maladaptive patterns that might limit collaborative growth or spread of innovation; enhance and further
support systems with adaptive functioning; and track how changes amongst system components lead to
desirable emergent properties or how adjustments to environment (e.g., more funds or support) influence
the interaction of system components.
Critical Research Needs for Complex Systems Research
The success of complex systems research in engineering education hinges on several critical
research needs for any given project or evaluation. Alignment of methodology and method not only
requires understanding the philosophical and theoretical nature of complex systems (methodology), but
also addressing the practical concerns of research context, data collection, and analysis (method). These
concerns include consideration of factors such as the appropriate context for research, the identification of
system critical variables, possible methods for data collection, and tools and software for analyzing data.
The following sections outline aspects of complex systems research that need to be considered within the
context of engineering education in order for complex systems research to gain traction. We provide
accessible examples of complex systems research from the natural sciences to illustrate each of the
critical needs. In the discussion/conclusion we relate these examples to education research to
demonstrate their application to engineering education.
Context of Educational Systems. To produce findings that can be leveraged by stakeholders, use
inspired educational problems (i.e. Stokes, 1997; Sloane, 2006) should be researched that are likely to
produce context specific conclusions relevant to practitioners in engineering education. Furthermore,
complex systems research is inherently an interaction-dominant approach. Interaction-dominant systems
in education (i.e. those related to classrooms and research collaboratives) are open, biological,
thermodynamic systems that exchange heat, matter, and information. Although they have natural
boundaries (i.e. in the same way that organs are composed of cells) there is no place where the diffusion
of energy, matter, or information stops. Thus, researchers must make theoretically driven choices about
the contextual boundaries of a given system, often based on use-inspired and feasibility needs. For
example, when a person is ill and winds up in the hospital, cardiac monitoring of heart activity, or
electrocardiography, is used to continuously assess a patient's condition by drawing conclusions about
their cardiac rhythm. The monitor will be used under some conditions and not others, depending upon
severity of the patient's condition, risk assessment, and the feasibility and cost of using the monitor.
Contextual factors drive the use-inspired decisions about what data to collect and when. Also of interest is
the notion that the heart is a system critical variable worth monitoring, as opposed to some other measure
taken from the body (which is a complex system). This illustrates our next critical need: choosing system
critical variables that provide evidence of holistic system functioning.
Critical System Variables. Critical systems variables and/or appropriate families of indicators
must be specified to satisfy the demands of system modeling and produce meaningful outcomes.
Although complex systems research is not deductive, it is reductive in the sense that researchers have to
make choices about what variables may provide the best evidence for system functioning. Because
interaction-dominant systems are open, researchers not only have to place arbitrary contextual boundaries
on the system, but also choose system critical variables at the center of a system that are subject to its
push and pull. For example, overly ordered or overly random characteristics of a heart rhythm allow for
robust conclusions about the complex functioning of the body. Taking dynamic, times series measures of
system critical variables can provide evidence of how a complex system is functioning, without having to
know the entire system. For example, in the famous Lorenz system, convention can be modeled using
three system critical variables: fluid viscosity/ thermal conductivity, temperature difference between top
and bottom of a vessel, and deviation of linear change in temperature. Each of these variables has their
own dynamics, and certain parameter values produce chaotic solutions (i.e. to coupled differential
equations) with smooth attractor states indicative of stable functioning. Interestingly, Takens (1981)
showed how a measuring a single variable from the center of a convention system can be used to recover
the smooth attractor state produced by all three variables by reconstructing the phase space. Although
Taken’s work simplifies the measurement demands for studying Lorenz systems, the example leads us to
the next critical need: methods for gathering data at proper granularity to capture phenomena of interest.
Methods for Systems Data Collection. Because the underlying structure of a complex system is a
mutually causal network, and the push and pull of components within a network produce dynamic
behavior over time, new methods for data collection that can assess 1) the underlying network structure of
a bounded system and 2) the dynamic behavior of system critical variables over time, are required to
move complex systems research forward. Methods that can produce adequate data to routinely monitor
and assess psychological and human systems must be developed and validated. For example, social
physics researchers from the media lab at the Massachusetts Institute of Technology have developed
sociometric badges (see Petland, 2014) that can used for experience sampling for social networks. About
the size of a cell phone, the badges provide finely grained information about social networks in physical
spaces (i.e. organizations and collaborative groups) that emerge through human interaction. Worn around
the neck, they provide real-time data about face-to-face interaction, human to computer interaction,
physical proximity, speech characteristics, nonverbal cues, physical activity, and conversation analysis.
The badges are specifically designed for longitudinal social network data collection, and have been used
primarily to study teamwork and organizational behavior. Novel solutions to improved data collection
such as these, however, place increasing demands on data analysis and modeling, our final critical need.
Tools for Data Analysis and System Modeling. Off-the-shelf tools for analyzing complex
systems data must be developed and vetted for completing complex systems projects. Although there are
a variety of software packages for analyzing and modeling data using linear statistics, there are relatively
few options available for complex systems researchers in education that can be used to examine nonlinear
data. This trend may be partly due to the fact that many researchers in the natural sciences “speak math”
and thus do much of their analysis using matlab or other general mathematics suites, but there are many
popular software packages offered to researchers in these spaces (e.g. Anylogic, 2016). Also, there are a
variety of packages available in R that can be used for nonlinear analysis such as time series analysis,
dynamic modeling, and network analysis, certainly too many to provide an exhaustive account here.
However, the development of new software packages that make it easier for researchers to analyze
nonlinear data specific to education research would almost certainly help complex systems research
become more widespread in educational research contexts. One example is time series analysis software
that has been developed by the researchers at the Center for Cognition, Action, and Perception at the
University of Cincinnati (CAP, 2016). Using Matlab code and a basic GUI, they have produced software
specific to their own purposes that can conduct batched time series analysis. For example, patients with
and without Parkinson’s may participate in a lab study, their gross motor movements measured by body
sensors during the study. The CAP software can conduct batched time series analyses of the sensor data
streams, preparing it for hypothesis testing about statistical differences between qualities of the time
series between study groups.
Discussion
To conclude, we will provide some examples of how the illustrations described in the critical
needs section above are being translated into complex systems research in engineering education. Similar
to the heart monitoring example, Villanueva and colleagues (2014; 2016) have taken skin response
measures to examine students’ emotional states during a laboratory examination activity. Their work is an
example of how researchers are conducting complex systems research to understand student motivation
and emotion. They integrate salivary and survey measures with the galvanic skin response data collection.
They are also developing their own algorithms to examine the qualities of the signals. Analyzing the time
series skin response data in conjunction with the survey and salivary data allows them to draw
conclusions at multiple levels of analysis: 1) the underlying biophysical substrata of the cognitive system
and 2) how students are experiencing and regulating their emergent emotional states.
Similar to the Lorenz system example, Hilpert and colleagues (2013, 2014) have used differential
equation modeling to produce simulations of how students plan for a future career in engineering as they
enter young adulthood. Their work is an example of how dynamic modeling can be used to examine
students planning, self-regulation, and problem solving. They integrate interviews, surveys, and student
drawings of timelines of their lives to produce dynamic models for how students’ goals shift with regard
to 1) what they value in the future and when and 2) gender differences in these shifts that influence work
life balance and motivation in the present. Instead of focusing on stable components of a cognitive
system, these researches assume an interaction-dominant system of cognition, demonstrating how
dynamic shifts in future goals shape adaptive or maladaptive functioning.
Similar to the social physics network example, Martinez and colleagues (2003) drew upon
multiple data sources, including questionnaires, interviews, and computer log files to evaluate computer
supported collaborative learning projects in an engineering course with social network analyses, yielding
the insight that collaborative learning was characterized by sharing information. The network approach
allowed evaluators to understand social roles and structural patterns related to participation, which could
then be used formatively to intervene when collaboratives were not functioning optimally. Researchers
interested in the growth and change of Engineering Education Research (EER) as a field also applied
network mapping of large scale data with time-series analyses to document the development and impact
of research collaborations (Madhavan et al., 2011). Madhavan and colleagues introduced a tool called
iKNEER that could be applied to publication and presentation data and to illustrate impact, showed how a
specific conference fostered the growth of academic networks.
These are examples of how complex systems research is beginning to take shape in engineering
education. In education research at large, the approach is also gaining some momentum, but has been an
important paradigm for some time. Davis and Sumara (2006) provide a wonderful description of
complexity research in education, detailing an extensive review of the methodological underpinnings.
More recently, Jacobson and colleagues (2016) have provided a theoretical framework for complex
dynamic systems research in educational psychology. And Koopsman and Stavamolsis (2016) have edited
an excellent book that contains examples of empirical complex systems research in education. These
resources may be useful for engineering education researchers interested in further exploration. By
utilizing methodological approaches that assume underlying complex systems, conducting research in
engineering education domains focused on interaction-dominant phenomena, and meeting critical data
collection and analysis needs, complex systems research can provide important insights for the
engineering education community.
Anylogic (2016) Retrieved from: http://www.anylogic.com/
Benson, L., Kirn, A., & Faber, C. (2013, June). CAREER: Student motivation and learning in
engineering. In ASEE Annual Conference Proceedings.
Berggren, K. F., Brodeur, D., Crawley, E. F., Ingemarsson, I., Litant, W. T., Malmqvist, J., & Östlund, S.
(2003). CDIO: An international initiative for reforming engineering education. World Transactions
on Engineering and Technology Education, 2(1), 49-52.
Borrego, M., & Newswander, L. K. (2008). Characteristics of successful cross-disciplinary engineering
education collaborations. Journal of Engineering Education, 97(2), 123.
Borrego, M., Froyd, J. E., & Hall, T. S. (2010). Diffusion of engineering education innovations: A survey
of awareness and adoption rates in US engineering departments. Journal of Engineering Education,
99(3), 185-207.
Bozeman, B., & Boardman, C. (2004). The NSF Engineering Research Centers and the university–
industry research revolution: a brief history featuring an interview with Erich Bloch. The Journal of
Technology Transfer, 29(3-4), 365-375.
Bozeman, B., Dietz, J. S., & Gaughan, M. (2001). Scientific and technical human capital: an alternative
model for research evaluation. International Journal of Technology Management, 22(7-8), 716-740.
Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P12
classrooms. Journal of Engineering Education, 97(3), 369-387.
Brown, S., Street, D. & Martin, J. P. (2014) Engineering Student Social Capital in an Interactive Learning
Environment, International Journal of Engineering Education, 30(4), 813-821.
CAP (2016) Retrieved from: https://www.uc.edu/cap.html
Carberry, A. R., Lee, H. S., & Ohland, M. W. (2010). Measuring engineering design selfefficacy. Journal
of Engineering Education, 99(1), 71-79.
Carver, C. S., & Scheier, M. F. (2004). Self-regulation of action and affect. Handbook of self-regulation:
Research, theory, and applications, 13-39.
Connor, C. M., Spencer, M., Day, S. L., Giuliani, S., Ingebrand, S. W., McLean, L., & Morrison, F. J.
(2014). Capturing the complexity: Content, type, and amount of instruction and quality of the
classroom learning environment synergistically predict third graders’ vocabulary and reading
comprehension outcomes. Journal of educational psychology, 106(3),
762.http://dx.doi.org/10.1037/a0035921
Dale, R., Fusaroli, R., Duran, N., & Richardson, D. C. (2013). The self-organization of human interaction.
Psychology of learning and motivation, 59, 43-95. http://dx.doi.org/10.1016/B978-0-12-407187-
2.00002-2
Davis, B., & Sumara, D. J. (2006). Complexity and education: Inquiries into learning, teaching, and
research. Psychology Press.
Deci, E. L., & Ryan, R. M. (2002). Overview of self-determination theory: An organismic dialectical
perspective. Handbook of self-determination research, 3-33.
Ellis R.D. & Newton N. (eds.) (2000). The Caldron of Consciousness: Motivation, Affect and Self-
Organization--An Anthology. Amsterdam: John Benjamins. http://dx.doi.org/10.1075/aicr.16
Felder, R. M., & Brent, R. (2016). Teaching and Learning STEM: A Practical Guide. John Wiley & Sons.
Fisher, R. A. (1925). Theory of statistical estimation. In Mathematical Proceedings of the Cambridge
http://dx.doi.org/10.1017/S0305004100009580
Fried, E. I., Bockting, C., Arjadi, R., Borsboom, D., Amshoff, M., Cramer, A. O. & Stroebe, M. (2015).
From loss to loneliness: The relationship between bereavement and depressive symptoms. Journal of
abnormal psychology, 124(2), 256. http://dx.doi.org/10.1037/abn0000028
Gao, J., Cao, Y., Tung, W. W., & Hu, J. (2007). Multiscale analysis of complex time series: integration of
chaos and random fractal theory, and beyond. John Wiley & Sons.
http://dx.doi.org/10.1002/9780470191651
Gebbie, K. M., Mason Meier, B., Bakken, S., Carrasquillo, O., Formicola, A., Aboelela, S. W., ... &
Larson, E. (2008). Training for Interdisciplinary Health Research Defining the Required
Competencies. Journal of allied health,37(2), 65-70.
Harding, S. G. (1987). Feminism and methodology: Social science issues. Indiana University Press.
Hilpert, J. & Husman, J. (2016). A Systems Approach to a Self-Report Measure of Interactive Student
Engagement. Paper presented at the Envisioning the Future of Undergraduate STEM Education
(EnFUSE): Research and Practice Symposium, Washington DC.
Hilpert, J. & Marchand, G. Complex Systems Research in Educational Psychology. (Theoretical Article
Under Review)
Hilpert, J., Carrion, M. L., Husman, J. (2014). Gender differences in engineering students’ imagined
futures. Journal of Women and Minorities in Science and Engineering. DOI:
10.1615/JWomenMinorScienEng.2014007079 (Journal Impact Factor: None).
Hilpert, J., & Hyppolite, J. (2013, October). Cognitive Pathways to Engineering. Proceeding published
in the annual proceedings of the Frontiers in Education Conference, Oklahoma City, OK.
Holden, J. G., Van Orden, G. C., & Turvey, M. T. (2009). Dispersion of response times reveals cognitive
dynamics. Psychological Review, 116(2), 318. http://dx.doi.org/10.1037/a0014849
Holland, J. H. (2000). Emergence: From chaos to order. OUP Oxford.
Hollenstein, T. (2013). State space grids. Springer. http://dx.doi.org/10.1007/978-1-4614-5007-8
I. Villanueva, A. Raikes, N. Ruben, S. Schaefer, and J. Gunther. The use of physiological tools to identify
changes in affective responses for graduate students recently admitted into a scientific discipline.
2014 FIE Conference under the ‘Student Beliefs, Motivation, and Persistence Through the College
Years’ session, Madrid, Spain (2014).
Jacobson, M. J., Kapur, M., & Reimann, P. (2016). Conceptualizing Debates in Learning and Educational
Research: Toward a Complex Systems Conceptual Framework of Learning. Educational
Psychologist, 1-9. http://dx.doi.org/10.1080/00461520.2016.1166963
Kaplan, A., Katz, I., & Flum, H. (2012). Motivation theory in educational practice: Knowledge claims,
challenges, and future directions. APA educational psychology handbook, 2, 165-194.
Kauffman, S. A. (1993). The origins of order: Self organization and selection in evolution. Oxford
University Press, USA. http://dx.doi.org/10.1007/978-94-015-8054-0_8
Kirn, A., & Benson, L. (2013, October). Quantitative assessment of student motivation to characterize
differences between engineering majors. In Frontiers in Education Conference, 2013 IEEE (pp. 69-
74). IEEE.Reeve, J. (2014). Understanding motivation and emotion. John Wiley & Sons.
Koopmans, M. (2016). Mixed Methods in Search of a Problem Perspectives From Complexity Theory.
Journal of Mixed Methods Research, 1558689816676662.
Koopmans, M., & Stamovlasis, D. (Eds.). (2016). Complex Dynamical Systems in Education: Concepts,
Methods and Applications. Springer. http://dx.doi.org/10.1007/978-3-319-27577-2
Krackhardt, D. (1988). Predicting with networks: Nonparametric multiple regression analysis of dyadic
data. Social networks, 10(4), 359-381. http://dx.doi.org/10.1016/0378-8733(88)90004-4
Lazer, D., Pentland, A. S., Adamic, L., Aral, S., Barabasi, A. L., Brewer, D., ... & Jebara, T. (2009). Life
in the network: the coming age of computational social science. Science (New York, NY), 323(5915),
721.
Lohani, V. K., Sanders, M., Knott, T., Griffin, H., Bull, E., McIntyre, S., ... & Magliaro, S. (2004,
October). Engineering-Education Collaboration-Virginia Tech. In Proc. International Conference on
random graph models for social networks: Theory, methods, and applications. Cambridge University
Press. http://dx.doi.org/10.1017/CBO9780511894701
Madhavan, K., Xian, H., Vorvoreanu, M., Jesiek, B., Johri, A., Wang, A., Wankat, P. (2010) Interactive
Knowledge Networks for Engineering Education Research (iKNEER). Supported by NSF Grant
EEC-0957015, EEC-0935109, EEC-0935124.
Martinez, A., Dimitriadis, Y., Rubia-Avi, B., Gomez-Sanchez, E., De La Fuente, P. (2003). Combining
qualitative evaluation and social network analysis for the study of classroom social interactions.
Computers and Education, 41, pp.353-368. <10.1016/j.compedu.2003.06.001>.
Mitchell, M. (2009). Complexity: A guided tour. Oxford University Press.
Molinari, G., Sangin, M., Dillenbourg, P., & Nüssli, M. A. (2009). Knowledge interdependence with the
partner, accuracy of mutual knowledge model and computer-supported collaborative learning.
European journal of psychology of education, 24(2), 129-144.
National Science Board. 2016. Science and Engineering Indicators 2016. Arlington, VA: National
Science Foundation (NSB-2016-1).
Nelson, K. G., Shell, D. F., Husman, J., Fishman, E. J., & Soh, L. K. (2015). Motivational and self regulated learning profiles of students taking a foundational engineering course. Journal of
Engineering Education, 104(1), 74-100.
Ohland, M. W., Sheppard, S. D., Lichtenstein, G., Eris, O., Chachra, D., & Layton, R. A. (2008).
Persistence, engagement, and migration in engineering programs. Journal of Engineering Education,
97(3), 259-278.
Pennings, H. J., & Mainhard, T. (2016). Analyzing Teacher–Student Interactions with State Space Grids.
In Complex Dynamical Systems in Education (pp. 233-271). Springer International Publishing.
http://dx.doi.org/10.1007/978-3-319-27577-2_12
Pentland, A. (2014). Social physics: How good ideas spread-the lessons from a new science. Penguin.
Qin, J., Lancaster, F. W., & Allen, B. (1997). Types and levels of collaboration in interdisciplinary
research in the sciences. JASIS, 48(10), 893-916.
Richardson, M. J., Dale, R., & Marsh, K. L. (2014). Complex dynamical systems in social and personality
psychology. Handbook of research methods in social and personality psychology, 253.
http://dx.doi.org/10.1017/CBO9780511996481.015
Riley, M. A., & Van Orden, G. C. (2005). Tutorials in contemporary nonlinear methods for the behavioral
sciences. Retrieved from, http://www.nsf.gov/sbe/bcs/pac/nmbs/nmbs.jsp
Schwab, J. J. (1973). The practical 3: Translation into curriculum. The School Review, 81(4), 501-522.
http://dx.doi.org/10.1086/443100
Schwandt, T. A. (2001). Dictionary of qualitative inquiry (Vol. 31, pp. 439-448). Thousand Oaks, CA:
Sage Publications.
Shekhar, P., Demonbrun, M., Borrego, M., Finelli, C., Prince, M., Henderson, C., & Waters, C. (2015).
Development of an observation protocol to study undergraduate engineering student resistance to
active learning. International Journal of Engineering Education, 31(2), 597-609.
Skinner, E., Furrer, C., Marchand, G., & Kindermann, T. (2008). Engagement and disaffection in the
classroom: Part of a larger motivational dynamic?. Journal of Educational Psychology, 100(4), 765.
http://dx.doi.org/10.1037/a0012840
Sleimen-Malkoun, R., Temprado, J. J., & Hong, S. L. (2015). Aging induced loss of complexity and
dedifferentiation: consequences for coordination dynamics within and between brain, muscular and
behavioral levels. From Brain to Body: The Impact of Nervous System Declines on Muscle
Performance in Aging. http://dx.doi.org/10.3389/fnagi.2014.00140
Sloane, F. (2006). Normal and design sciences in education: Why both are necessary. Educational design
research, 19-44.
Analysis with Orbital Decomposition. In Complex Dynamical Systems in Education (pp. 273-297).
Springer International Publishing. http://dx.doi.org/10.1007/978-3-319-27577-2_13
90.
Stokes, D. E. (1997). Pasteur’s Quadrant: Basic Science and Technological Innovation. Brookings
Institution Press
Strogatz, S. H. (1994). Nonlinear systems and chaos. Perseus Publishing
Takens, F. (1981). Detecting strange attractors in turbulence. In Dynamical systems and turbulence,
Warwick 1980 (pp. 366-381). Springer Berlin Heidelberg.
Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C. C. (2017). Some Guidance on Conducting and
Reporting Qualitative Studies.Computers & Education, 106, A1-A9.
Villanueva, I., Valladares, M., & Goodridge, W. (2016). Use of Galvanic Skin Responses, Salivary
Biomarkers, and Self-reports to Assess Undergraduate Student Performance During a Laboratory
Exam Activity. JoVE (Journal of Visualized Experiments), (108), e53255-e53255.
Vogt, C. M. (2008). Faculty as a critical juncture in student retention and performance in engineering
programs. Journal of Engineering Education, 97(1), 27-36.
Welch, E., Feeney, M. & Siciliano, M. (2016). Social network analysis for science evaluation.
Presentation at the 2016 Science of Team Science conference.
Xian, H., & Madhavan, K. (2012). A quantitative study of collaboration patterns of engineering education
researchers. In American Society for Engineering Education. American Society for Engineering
Education.