Embed Size (px)
Citation preview
Paper ID #33445
Evaluation of Targeted Systems Thinking and Systems EngineeringAssessments in a Freshmen-Level Mechanical Engineering Course
Dr. Cassandra M. Birrenkott, South Dakota School of Mines and Technology
Dr. Cassandra (Degen) Birrenkott received her B.S. degree in Metallurgical Engineering from the SouthDakota School of Mines and Technology in 2007. She received her Ph.D. in Materials Science andEngineering in 2012 from the University of Illinois at Urbana-Champaign, studying mechanochemicalreactions of a spiropyran mechanophore in polymeric materials under shear loading. She is currentlyan Assistant Professor in the Mechanical Engineering department at the South Dakota School of Minesand Technology where her research interests include novel manufacturing and characterization techniquesof polymer and composite structures and the incorporation of multifunctionality by inducing desired re-sponses to mechanical loading.
Dr. Karim Heinz Muci-Kuchler, South Dakota School of Mines and Technology
Dr. Karim Muci-Kuchler is a Professor of Mechanical Engineering and Director of the Experimental andComputational Mechanics Laboratory at the South Dakota School of Mines and Technology (SDSMT).Before joining SDSMT, he was an Associate Professor of Mechanical Engineering at the University ofDetroit Mercy. He received his Ph.D. in Engineering Mechanics from Iowa State University in 1992.His main interest areas include Computational Mechanics, Solid Mechanics, and Product Design andDevelopment. He has taught several different courses at the undergraduate and graduate level, has over50 publications, is co-author of one book, and has done consulting for industry in Mexico and the US. Hecan be reached at [email protected].
Dr. Mark David Bedillion, Carnegie Mellon University
Dr. Bedillion received the BS degree in 1998, the MS degree in 2001, and the PhD degree in 2005, allfrom the mechanical engineering department of Carnegie Mellon University. After a seven year careerin the hard disk drive industry, Dr. Bedillion was on the faculty of the South Dakota School of Minesand Technology for over 5 years before joining Carnegie Mellon as a Teaching Faculty in 2016. Dr. Be-dillion’s research interests include distributed manipulation, control applications in data storage, controlapplications in manufacturing, and STEM education.
Dr. Marsha Lovett, Carnegie Mellon University
Dr. Marsha Lovett is Associate Vice Provost of Teaching Innovation, Director of the Eberly Center forTeaching Excellence and Educational Innovation, and Teaching Professor of Psychology – all at CarnegieMellon University. She applies theoretical and empirical principles from learning science research toimprove teaching and learning. She has published more than fifty articles in this area, co-authored thebook How Learning Works: 7 Research-Based Principles for Smart Teaching, and developed severalinnovative, educational technologies, including StatTutor and the Learning Dashboard.
Dr. Laura Ochs Pottmeyer, Carnegie Mellon University
Laura Pottmeyer is a Data Science Research Associate at Carnegie Mellon University’s Eberly Centerfor Teaching Excellence and Educational Innovation. She consults with faculty members and graduatestudents on implementing educational research projects. She assists with study design, data collection,and data analysis. Laura’s training includes a Ph.D. in Science Education and M.Ed. in EducationalPsychology from the University of Virginia, where she studied the impact of engineering design integratedscience on student learning.
c©American Society for Engineering Education, 2021
Evaluation of Targeted Systems Thinking and Systems
Engineering Assessments in a Freshmen-Level Mechanical
Engineering Course
Abstract
Developing high performing, cutting edge products and systems requires engineers that, in
addition to being proficient in their specific discipline, have a solid background in product
development, systems engineering (SE), and systems thinking (ST). Introducing ST/SE skills
gradually throughout a traditional mechanical engineering curriculum has the potential to produce
graduates ready to participate in multidisciplinary teams working to develop complex products or
systems. To explore how to progressively develop the ST and SE skills of mechanical engineering
students, a freshman-level introduction to mechanical engineering course was utilized to
incorporate basic ST/SE concepts and evaluate the gains in the knowledge, skills, and abilities
(KSAs) of students in these areas.
Educational materials in the form of lectures, homework assignments, and a short project specific
to freshman-level appropriate ST/SE topics were developed and implemented in an introduction
to mechanical engineering course during both the Fall of 2019 and the Fall of 2020. Sample student
work and scores were collected from homework assignments containing ST/SE questions. Rubrics
for the targeted homework assignments were developed to effectively compare student work
across 3 different semesters: Fall 2018 (a control group, no ST/SE intervention materials
presented), Fall 2019 (first implementation of ST/SE materials), and Fall 2020 (second
implementation of ST/SE materials). In this paper, the rubric development process and analysis
technique will be described. In addition, the analysis of students’ KSAs in the selected ST/SE
topics is used to highlight the impact of the newly implemented educational materials on students’
ST/SE understanding. Finally, the analysis is used to suggest improvement of the materials for
future implementations.
Introduction
Engineers participating in the design and development of new products and systems have to deal
with the rapid increase in complexity that products and systems are experiencing over time. In
addition, the level of interaction between products and systems independently developed by
different companies and organizations has grown substantially. Preparing undergraduate
engineering students that have the competencies required to work in such an environment is a
challenging task. To successfully participate in the different activities that occur over the life cycle
of a product or system, engineering graduates need much more than a solid foundation in the
traditional mathematics, science, and engineering subjects related to their discipline. Furthermore,
the new Accreditation Board for Engineering and Technology (ABET) accreditation criteria for
engineering programs that took effect in the 2019–2020 accreditation cycle [1] reflect an increased
emphasis in having engineering graduates that are prepared to participate in the development of
complex products and systems.
The wide array of knowledge, skills, and abilities (KSAs) desired in engineering professionals is
evident in references such as the Engineering Competency Model jointly developed by the
American Association of Engineering Societies (AAES) and the US Department of Labor (DoL),
the CDIO (Conceive Design Implement Operate) Syllabus 2.0 proposed by the CDIO organization,
and the U. S. Department of Defense Systems Engineering Career Competency Model [2-5]. Some
of the listed KSAs [2-5] highlight the need for incorporating systems thinking (ST) and basic
systems engineering (SE) concepts in the undergraduate curriculum of all engineering majors.
Efforts aimed at identifying ST and SE KSAs that should be included in all undergraduate
engineering programs [2, 6-16] have made progress towards the long-term goal of introducing ST
and SE concepts in engineering curricula in a gradual fashion, beginning in the freshman year and
culminating with the major capstone design experience that takes place in the senior year.
The curriculum of many mechanical engineering undergraduate programs includes a freshman-
level introduction to mechanical engineering course. Although the number of credits and the
course content can vary between academic institutions, in general the purpose of the class is to
familiarize freshmen with the mechanical engineering profession, to provide a brief introduction
to the technical subjects that students will cover during their mechanical engineering
undergraduate studies, and to develop some basic skills that will be useful in multiple courses. In
the case of the South Dakota School of Mines and Technology (SDSM&T), the course is ME-110
Introduction to Mechanical Engineering and it is a two-credit class that is listed in the first semester
of the freshman year.
A freshman-level introduction to mechanical engineering course is an attractive choice to present
an overview of the product development process (PDP) and to expose students to some basic ST
and SE concepts. The PDP provides an excellent context to motivate the need for ST/SE, and
introducing ST/SE concepts during the freshman year may help students to start developing a
holistic mindset before a reductionist problem solving approach is engrained in the various
discipline-specific technical courses that they will take during their undergraduate education.
Unfortunately, most engineering students think that they do not need to be involved in the initial
activities of the product development process (PDP) and that their work begins when someone else
provides them a list of target specifications. In part, this attitude could be the result of the type of
problems that engineering students are usually asked to solve in most of their technical courses:
well-posed problems where they are given some information and data in the problem statement,
and then are asked to perform calculations to find some desired results. Introducing freshman
students to the chosen ST/SE topics may help to alleviate this situation.
The focus of this work was to introduce basic ST and SE concepts in SDSM&T’s Introduction to
Mechanical Engineering course and to evaluate the targeted assessments. This paper provides a
description of the instructional materials developed for the course and explains the intervention
approach that was used, provides an analysis of student work pertaining to the ST/SE topics, and
highlights student changes in knowledge about selected ST/SE KSAs. Student work across three
different semesters was compared: Fall 2018 (a control group, no ST/SE intervention materials
presented), Fall 2019 (first implementation of ST/SE materials), and Fall 2020 (second
implementation of ST/SE materials). It should be noted that some of the materials were also
implemented in Spring 2020, but due to the switch to remote learning mid-semester and the
challenges associated with the COVID-19 pandemic, no conclusive results arose from Spring
2020, and the results were not included in this paper. Additionally, for the course where the
materials were implemented, the Fall 2020 semester was held completely in-person following
CDC guidelines, mirroring the semesters pre-pandemic.
Implementation of Materials Pertaining to Selected ST/SE Topics
A detailed description of the unmodified course as well as the process used for selection of
appropriate ST/SE topics to include in the intervention has been presented previously [17]. In
addition, the new educational materials developed for the intervention were described [18] and
include: a presentation providing an overview of the PDP, a presentation focusing on the selected
ST and SE topics, six homework assignments each dealing with one traditional course topic and
one ST/SE topic, and a team project.
Regarding the six homework assignments developed for the intervention, the objective was to
reinforce concepts covered in the ST/SE presentation later in the semester. Each assignment started
with a brief background about a product or system. Then, that product or system was used to briefly
illustrate one ST/SE concept and one or more questions related to that topic were included. Finally,
one or two problems corresponding to a traditional course topic were provided. For example, one
homework assignment used mountain bicycles to illustrate the idea of internal and external
interfaces and included three problems. In the first problem, students were asked to identify the
external interfaces between a mountain bike and the biker. In the remaining two, students needed
to solve simple static equilibrium problems involving a mountain bike. Table 1 provides general
information about the six homework assignments developed for the intervention. For ease of
reference, the homework assignments were labeled using numbers based on the order in which
they were implemented in the course. However, it is important to mention that the assignments
developed as part of this work were only a subset of all the homework assignments given during
the course.
Table 1. General information about the six homework assignments developed for the intervention in the
Introduction to Mechanical Engineering course.
Homework Traditional
Topic
ST/SE
Topic
Product or System
Considered
1 Unit Systems and Conversions System Verification The Mars Climate Orbiter [19]
2 Equilibrium: Concurrent
System of Forces System Elements Rock Climbing Gear
3 Equilibrium: General
Coplanar Forces
Identification of
Interfaces Mountain Bicycle
4 Engineering Materials Stakeholders,
Customer Needs Prosthetic Arm
5 Axial Stress and Factor of
Safety Subsystems
Off-Road Utility Vehicle
(UTV)
6 Buoyancy Force System Life Cycle The Ocean Cleanup Project
[20]
The Introduction to Mechanical Engineering course also offered students a team project in which
they were tasked with designing and building a “seaworthy” and exceptionally buoyant small-scale
boat. Across the two implementations, the project varied slightly while maintaining the same
overall goal (for students to design and build an exceptionally buoyant boat). The learning activity
for both Fall 2019 and Fall 2020 made use of the small wave tank shown in Figure 1. For both
semesters, the students were given limitations on the allowed maximum overall dimensions of the
boat, but there were no constraints on the shape of the boat.
For the Fall 2019 implementation, students were given a kit containing items such as varying small
pieces of balsa wood and a hot glue gun to manufacture their boats by hand. For the Fall 2020
implementation, the students designed their boat in a CAD program and manufactured the boats
using 3D printing. The use of 3D printing reduced manufacturing constraints allowing students the
possibility to create more interesting designs. In both the Fall 2019 and the Fall 2020
implementations, the students were not told in advance that the testing tank was a wave tank (they
were only told that testing would take place in a small aquarium tank) and were encouraged to use
the PDP discussed earlier in the semester. With the assumption that students would not consider
that the tank had waves, student groups had the opportunity to test and redesign their boats once
throughout the semester before the final testing occurred.
Figure 1. Small wave tank used to test boat designs proposed by the students.
Rubric Development and Assessment of ST/SE Materials
The first implementation of the modified course took place during Fall 2019 in one section of the
Introduction to Mechanical Engineering course offered at SDSM&T. On the first day of class 60
students were enrolled in that section; towards the end of the semester, the class size was 41
students, but one of them was not attending the lectures or doing course assignments. The decrease
in the number of students was not related to the modifications made to the course. For the course
project, the students worked in teams of three to five students and the total number of teams was
ten: three teams of three students each, four teams of four students each, and three teams of five
students each.
The second implementation of the modified course took place during Fall 2020 in one section of
the Introduction to Mechanical Engineering course offered at SDSM&T. This course was taught
by a different instructor than the Fall 2019 implementation. The enrollment on the first day of class
was 22 students which was significantly smaller than the Fall 2019 implementation due to reduced
classroom capacity with COVID-19 restrictions. One student never attended class, and the course
ended with 21 students enrolled. For the final project, students worked in teams of three and there
was a total of 7 teams.
To assess the benefits of each implementation, data related to the performance of the students in
the ST/SE related homework problems was collected. In addition, the team project implemented
in the course provided a context to apply some of the ST/SE ideas presented in class. In the case
of the homework assignments, six homework questions were selected for analysis. Table 2
provides general information about the homework questions that were selected and indicates the
semesters in which the questions were assigned. Regarding the items listed in Table 2, Items 1 to
3 are questions from the course textbook [21], and Items 4 to 6 are from homework assignments
developed as part of this work. None of the six questions were part of a team assignment.
The course textbook includes information about the mechanical design process as well as questions
related to that topic that can be assigned as homework. In the unmodified course, the content about
product design was based on the material presented in the textbook. In the modified course, a more
comprehensive introduction to product design was provided using the lecture about the PDP that
was developed for the intervention. Since product design was covered at different levels of detail
in the unmodified and the modified course, the authors thought that it would be interesting to
assign three textbook questions about the topic to students enrolled in one section of the
unmodified course in Fall 2018, in one section of the modified course in Fall 2019, and in one
section of the modified course in Fall 2020.
Table 2. Items from homework assignments for which a detailed analysis was conducted.
Item General Information About the Homework Item Semesters
1 Identifying the stakeholders for a dishwasher. Fall 2018, Fall 2019, Fall 2020
2
Determining the global, environmental, and economic issues
for a dishwasher that need to be considered at the beginning of
the design process.
Fall 2018, Fall 2019, Fall 2020
3 Generating concepts for capturing and storing the kinetic
energy from students walking around campus. Fall 2018, Fall 2019, Fall 2020
4 Identifying stakeholders and customer needs for a low-cost, 3-
D printed prosthetic arm. Fall 2019, Fall 2020
5
Dividing a bicycle into subsystems based on function and
indicating which bicycle parts/components are part of each
subsystem.
Fall 2019, Fall 2020
6 Identifying the external interfaces between a mountain bike and
the biker during a ride. Fall 2019, Fall 2020
Homework Analysis
As was mentioned earlier, each of the six homework assignments that were developed for the
intervention included one or more questions related to a traditional course topic and one or more
questions related to one of the ST/SE topics presented in the modified course. Due to time
constraints, only three questions from these homework assignments were considered for detailed
analysis.
The approach used to classify student responses into categories varied by Item in Table 2. In the
case of Items 1 to 5, three categories were used: “below expectations”, “meets expectations”, and
“exceeds expectations”. In the case of Item 6, given the limited range of possible student answers,
only two categories were used: “below expectations” and “meets expectations”. The results of the
analysis for each Item are shown graphically below.
As can be seen in Figure 2, the intervention had a positive effect on the skill of identifying product
stakeholders, which was the focus of Item 1. This was not surprising since the topic of
product/system stakeholders was addressed in the lecture about basic ST/SE concepts that was
developed for the intervention.
Figure 2. Percentage of students that were below, met, or exceeded expectations for Item 1.
In the case of Item 2, the question required students to think about global, environmental, and
economic issues to be considered while designing a product. As part of the intervention, students
were told about the importance of systems thinking but, due to the level of the course and the
amount of class time available for the intervention, the topic of systems thinking was only
addressed at the lowest level of the revised Bloom’s taxonomy (i.e., remembering) [22, 23]. Based
on the results presented in Figure 3, it is fair to conclude that the level at which the topic was
covered in class did not have an impact on student performance.
Figure 3. Percentage of students that were below, met, or exceeded expectations for Item 2.
The question corresponding to Item 3 dealt with concept generation and the intervention did not
target that topic. The activities that are carried out during the concept development phase of the
PDP were mentioned in one of the lectures developed for the intervention, but no additional
information was given. As expected, the results presented in Figure 4 indicate that student
performance was relatively similar between the section of the unmodified and the sections of the
modified course.
Figure 4. Percentage of students that were below, met, or exceeded expectations for Item 3.
In the case of the question corresponding to Item 4, the answer had two parts: one which dealt with
the identification of stakeholders and one which dealt with the identification of customer needs.
The two parts of the answer were analyzed separately. The part which dealt with identifying
product stakeholders was similar to Item 1 and, as was mentioned earlier, corresponds to a topic
addressed in the intervention. A key difference between Item 1 and Item 4 was when the homework
that included the question was assigned. Item 1 was assigned at the beginning of the semester,
right after the overview of the PDP and the introduction to basic ST/SE topics was given in class.
In contrast, Item 4 was assigned around mid-semester. Comparing the Fall 2019 and Fall 2020
results for Item 1 shown in Figure 2 and the results for the identification of product stakeholders
part of Item 4 shown in Figure 5, there was a decrease in the number of students that were in the
“meets expectations” category accompanied by a relatively small increase in the number of
students in the “below expectations” category and a pronounced increase in the number of students
in the “exceeds expectations” category. These changes could be an artifact of the rubric used for
each item. However, the increase in the number of students in the “exceeds expectations” category
could also be because the practice gained with Item 1 helped some students improve their skills
related to identifying product stakeholders. Also, the slight increase in the number of students in
the “below expectation” category could be because some students did not retain over time what
they learned in class about product stakeholders. Finally, the results for Items 1 and 4 could
indicate that some students may have not acquired the skill at the desired level, even after receiving
feedback for Item 1.
As for the part of the answer for Item 4 corresponding to the identification of customer needs, the
results presented in Figure 6 show that the percentage of students that performed “below
expectations” was 24% for Fall 2019 and 39% for Fall 2020. However, the percentage of students
that “exceeded expectations” in Fall 2019 and Fall 2020 were 50% and 39%, respectively. Taking
into consideration that the intervention only highlighted the importance of identifying customer
needs and that Item 4 was assigned around mid-semester, the results obtained can be viewed as
positive.
Figure 5. Percentage of students that were below, met, or exceeded expectations for the portion of Item 4
corresponding to the list of stakeholders.
Figure 6. Percentage of students that were below, met, or exceeded expectations for the portion of Item 4
corresponding to the list of customer needs.
Finally, the results for Item 5 and Item 6 are presented in Figure 7 and Figure 8, respectively. The
graphs presented in the figures show that the majority of the students performed well (i.e., met or
exceeded expectations) in both items. Consequently, it is reasonable to say that the concepts
involved in the assignment were sufficiently covered in the intervention.
Figure 7. Percentage of students that were below, met, or exceeded expectations for Item 5.
Figure 8. Percentage of students that were below or met expectations for Item 6.
Overall, the results of the analysis conducted on Items 1 to 6 show the potential benefits of
interventions that incorporate ST/SE concepts in freshman-level courses. More implementations
of the instructional materials and a detailed analysis of a larger number of homework assignments
will be required to draw firm conclusions.
Course Team Project Analysis
Selected items from the team project report that the student teams turned in at the end of the
semester were considered for analysis. The team project was implemented in one section of the
unmodified course during the Fall 2018 semester, in one section of the modified course during the
Fall 2019 semester, and in one section of the modified course during the Fall 2020 semester, and
there were several differences between the three implementations. Most notably, in the Fall 2018
implementation the boats were not tested in a wave tank and only one test was conducted at the
end of the semester.
Taking into consideration the level of the course, the scope of the intervention, the nature of the
project, and the differences between the implementations, the assessment was based on three
aspects of the project reports submitted by the student teams: the concept generation and selection
approach used by the students (all the implementations), the physical prototyping and testing
activities conducted by the teams (Fall 2019 implementation only), and the reflections that the
students made after observing the performance of their boat design during the final test (all the
implementations). The aspects assessed were rather general and did not point to a specific ST/SE
skill emphasized during the implementation. To assess the performance of the teams, for each of
the three aspects mentioned above a rubric was developed and used to classify each team in one
of three categories: “below expectations”, “meets expectations”, or “exceeds expectations”.
The assessment corresponding to the physical prototyping and testing activities in Fall 2019
showed that half of the teams fell in the “below expectation” category. This was not surprising
since building and testing physical prototypes take time and the students were only required to
build a physical prototype for the two tests that were included in the class schedule. In this regard,
since the Introduction to Mechanical Engineering course is a 2-credit class and the team project
was only one of the many learning activities that the students needed to complete, it is not
reasonable to expect students to do extensive prototyping and testing.
Figure 10 compares the performance of the Fall 2018, Fall 2019 and Fall 2020 student teams in
the concept generation and selection approach that the team members followed. Figure 11 shows
a similar comparison but for the reflections that the team members made after observing the
performance of their boat design during the final test. The differences observed in the graphs could
be tied to the differences between the projects assigned during the Fall 2018, Fall 2019, and Fall
2020. Also, the results for Fall 2020 could have been significantly affected by all the difficulties
and limitations that the COVID-19 pandemic imposed on team activities. Limiting the comparison
to Fall 2018 and Fall 2019, it is reasonable to say that the intervention had a positive impact on the
aspects considered.
Figure 10. Percentage of teams that were below, met, or exceeded expectations for the concept
generation and selection team project rubric.
Figure 11. Percentage of teams that were below, met, or exceeded expectations for the reflections about
the proposed boat design performance team project rubric.
Impact on Student KSAs
The revised Systems Thinking Skills Survey (STSS) that was developed in parallel with this work
was applied as a pre-test at the start of the course and as a post-test during the last week of classes
in the sections in which the intervention took place. The STSS has two parts: one with self-efficacy
questions and one with ST/SE-related technical questions [24]. Unlike the homework assignments
and exam questions corresponding to the intervention, the technical questions in the STSS are not
consistent with the scope of the intervention since they target the knowledge level expected of
graduating seniors with more exposure to ST and SE concepts. Only results obtained with the
revised STSS instrument are reported here and STSS results collected during the Fall 2018 are not
included since they were based on a prior version of the survey.
For both the Fall 2019 and Fall 2020 intervention, a matched pairs analysis was conducted,
comparing pre-test and post-test data on the self-efficacy portion of the revised STSS. Results for
the Fall 2019 and Fall 2020 intervention show that students reported significantly higher self-
efficacy scores at the conclusion of the course (Figure 12). Similarly, significant increases were
found across the various self-efficacy subscales (Table 3).
Figure 12. Overall mean self-efficacy scores for the Introduction to Mechanical Engineering course.
Table 3. Mean self-efficacy scores by category for the Introduction to Mechanical Engineering course.
Fall 2019 Fall 2020
Category Pre Mean
(std dev)
Post Mean
(std dev) ta df p
Pre Mean
(std dev)
Post Mean
(std dev) Zb N p
Overall 1.61 (.82) 2.80 (.50) -8.02 33 <0.001* 2.13 (.64) 3.03 (.60) -3.41 15 0.001*
ST/ SE 1.43 (.87) 2.79 (.55) -8.16 33 <0.001* 2.11 (.77) 3.11 (.55) -3.30 15 0.001*
Concept
Selection 1.70 (.87) 2.79 (.52) -7.26 33 <0.001* 2.27 (.65) 2.96 (.76) -2.90 15 0.004*
Identify
Customer
Needs
1.91 (.81) 3.13 (.48) -8.00 33 <0.001* 2.30 (.65) 3.10 (.54) -3.11 15 0.002*
Set Target
Specifications 1.50 (.92) 2.56 (.73) -5.96 33 <0.001* 1.96 (.84) 2.91 (.76) -3.31 15 0.001*
Concept
Generation 1.99 (.83) 2.87 (.58) -5.28 33 <0.001* 2.25 (.62) 2.96 (.67) -3.34 15 0.001*
Systems
Architecture 1.55 (.91) 2.90 (.51) -8.16 33 <0.001* 2.07 (.64) 3.09 (.68) -3.08 15 0.002*
a repeated measures paired t-test b non-parametric Wilcoxon-sign rank test
*significant at the 0.05 level
Results of paired t-test indicated that the overall increase in self-efficacy scores over the course of
the Fall 2019 semester (pre M=1.61, SD=0.82; post M=2.80, SD=0.50) was statistically significant
t(33)=-8.02, p<0.001. Additionally, the increase in self-efficacy scores in Fall 2020 (pre M=2.13,
SD=0.64; post M=3.03, SD=0.60) was also statistically significant Z=-3.41, p<0.001.
Overall, these self-efficacy results demonstrate significant improvement in students’ beliefs that
they can apply various topics/skills from ST and SE to develop a product or system as part of an
engineering project. While it is possible that this change occurred naturally (e.g., as a result of
maturation or from studying engineering in general), a reasonable interpretation is that this change
was fostered by the intervention introduced in the course – an encouraging result. In any case,
these results confirm that students exit the Introduction to Mechanical Engineering course with
more positive beliefs about their ability to tackle ST and SE problems.
As for the other half of the STSS instrument that measured students’ actual performance on SE
problems, the analogous matched-pairs analysis was conducted for data collected during the Fall
2019 and Fall 2020 interventions. The results of the analysis are presented in Figure 13 and Table
4. The change in the overall scores was not statistically significant. Particular sub-topics
represented in the technical questions showed some change pre- to post-test, but these differences
are not statistically significant. It is important to note that there are three fewer technical questions
included in the Fall 2019 analysis compared to Fall 2020. Changes in these three questions from
pre- to post-test in Fall 2019 prevented them from being able to be compared.
Figure 13. Overall mean technical scores for the Introduction to Mechanical Engineering course.
Results of the non-parametric Wilcoxon-sign rank test indicated that the overall decrease in Fall
2019 pre-scores (M=0.53, SD=0.13) to post-scores (M=0.49, SD=0.17) was not statistically
significant Z=-1.66, p=0.10. No additional pre- to post-score comparisons were statistically
significant for either the Fall 2019 or Fall 2020 semesters.
In interpreting the technical question results, there are several explanations for the lack of change
in students’ pre- to post- performance. A good number of technical questions in the STSS
correspond to ST/SE concepts that were not addressed in the intervention, which was designed for
the limited amount of class time available to add the new course content. Furthermore, the
technical questions in the STSS are at a level that is above the scope of the intervention that took
place in the Introduction to Mechanical Engineering course. In general, the technical questions are
geared towards what would be expected of mechanical engineering students at the time of
graduation. Thus, the strength of the intervention in the course might have been insufficient to
show an effect, the technical questions could have relied on systems thinking skills that were not
included in the intervention or not included at the appropriate level, and/or students might have
lacked sufficient motivation when completing the technical questions at post-test because it did
not affect their final grade. Ongoing work to analyze the alignment between the instructional
intervention and the technical questions promises to identify further adjustments that could be
productive – both in improving the measurement qualities of the technical questions and in
enhancing the impact of the interventions. Those adjustments may involve the development of
separate sets of technical questions that target interventions in specific courses.
Table 4. Mean technical scores by category for the Introduction to Mechanical Engineering course.
F19 (N=34) F20 (N=15)
Category
Mean
(std dev) Za p
Mean
(std dev) Za p
Pre Post Pre Post
Overall .53 (.13) .49 (.17) -1.66 .10 .52 (.16) .57 (.16) -1.36 .17
Identify and define system
boundaries and external interfaces .61 (.27) .60 (.25) -1.16 .25 .60 (.23) .69 (.26) -1.02 .31
Identify major stakeholders and
understand that stakeholders must
be involved early in the project
lifecycle
.30 (.47) .41 (.50) -1.16 .25 .60 (.51) .53 (.52) -.38 .71
Identify possible technical
performance measures
(specifications) for determining a
system’s success
.45 (.24) .52 (.18) -1.24 .22 .46 (.22) .48 (.14) -.63 .53
Understand the different types of
architecture .69 (.33) .65 (38) -.62 .54 .83 (.31) .80 (.32) -1.00 .32
Understand the need to explore
alternative and innovative ways of
satisfying the need
.55 (.26) .45 (.29) -1.45 .15 .48 (.24) .62 (.23) -1.81 .07
Define selection criteria, weightings
of the criteria and assess/evaluate
potential solutions against selection
criteria
.48 (.51) .41 (.50) -.83 .41 .60 (.51) .47 (.52) -.71 .48
Concept selection .46 (.31) .49 (.36) -.41 .68 .50 (.38) .47 (.23) -.45 .66
anon-parametric Wilcoxon-sign rank test
*significant at the 0.05 level
Conclusions and Future Work
As product complexity increases, a holistic approach coupled with systems engineering tools for
managing complex systems are increasingly needed by engineers from traditional engineering
majors. Components can no longer be designed in isolation but must be conceived considering
system-level impacts. Educators teaching in traditional disciplines such as mechanical engineering
must embrace tools from outside the conventional design process for products of low to moderate
complexity and train their students in the use of those tools.
Prior work focusing on a sophomore design class for mechanical engineering students [11, 25]
showed that a single exposure to ST/SE concepts is insufficient to significantly change students’
technical proficiency. The interventions considered in this paper broadened the perspective by
planting the seeds of ST and SE in freshman students.
The analysis of student artifacts collected during the interventions showed progress, but not at the
desired level. This result is not surprising since the intervention occurred in a single course rather
than as a curriculum-wide effort, which the authors hypothesize is needed for significant
improvements. In general, however, the progress observed shows promise and points to the need
for a more holistic, curriculum-wide approach to training.
Similar to results from a prior effort focused on incorporating ST/SE in a sophomore design course
[11, 25], the STSS showed substantial improvements in students’ ST/SE self-efficacy as a result
of the interventions. However, the STSS did not show significant improvement in students’ ST/SE
technical skills. Since the technical questions of the STSS were not designed for a specific
intervention in a particular course, it is difficult to reach final conclusions regarding a student’s
progress in ST/SE technical skills based on STSS scores alone.
Based on observations made in this work and previous efforts, future directions include a broad
study to determine how ST/SE skills are currently developed in the conventional curriculum across
courses both within and outside of engineering courses as well as co-curricular activities.
Additionally, the students involved in the courses where the developed materials were
implemented could be tracked to assess their success in future courses involving ST/SE skills (i.e.
following the freshmen at SDSM&T to their capstone design experience). The authors ultimately
envision a mechanical engineering program that progressively builds ST/SE skills throughout the
curriculum. By increasing the level of holistic thinking in all courses, we hope to build graduates
that begin to bridge the gap between systems engineers and conventional mechanical engineers.
Acknowledgements
This work was supported by the Office of Naval Research under Award No. N00014-18-1-2733.
The views and conclusions contained in this document are those of the authors and should not be
interpreted as representing the official policies, either expressed or implied, of the Office of Naval
Research or the U.S. Government. The U.S. Government is authorized to reproduce and distribute
reprints for government purposes notwithstanding any copyright notation hereon.
The authors want to acknowledge the help of Mr. Ranjun Singh, an undergraduate student at
SDSM&T, in the development of the homework assignments, the modified course project used for
the intervention, and the analysis of student work. The authors would also like to thank the students
that took the Introduction to Mechanical Engineering course at SDSM&T and were exposed to
this work.
References
1. The Engineering Accreditation Commission (EAC). General Criteria for Baccalaureate
Programs. 2017; Available from: http://www.abet.org/wp-content/uploads/2017/12/E001-18-
19-EAC-Criteria-11-29-17-FINAL_updated1218.pdf
2. American Association of Engineering Societies and United States Department of Labor,
Engineering Competency Model, Employment and Training Administration, Editor. 2015,
United States Department of Labor.
3. Crawley, E.F., et al., Rethinking engineering education: the CDIO approach. 2nd ed. ed. 2014,
New York, NY: Springer. xiv, 286 p.
4. Whitcomb, C., et al., A Description of the Defense Systems Engineering Career Competency
Model, in 14 th Annual Acquisition Research Symposium, April. 2017. p. 26-27.
5. Whitcomb, C.A., et al., The US Department of Defense Systems Engineering Competency
Model, in INCOSE International Symposium. 2017, Wiley Online Library. p. 214-228.
6. Cattano, C., T. Nikou, and L. Klotz, Teaching systems thinking and biomimicry to civil
engineering students. Journal of Professional Issues in Engineering Education & Practice,
2010. 137(4): p. 176-182.
7. Gelosh, D.S., et al., Developing the Next Generation of the INCOSE Systems Engineering
Competency Framework. INCOSE International Symposium, 2014. 24(1): p. 635-642.
8. International Council on Systems Engineering, Systems engineering handbook: a guide for
system life cycle processes and activities. 4th ed. ed. 2015, Hoboken, New Jersey: John Wiley
& Sons Inc.
9. Lesh, R., Modeling students modeling abilities: The teaching and learning of complex systems
in education. The Journal of the Learning Sciences, 2006. 15(1): p. 45-52.
10. Muci-Kuchler, K.H., et al., Incorporating basic systems thinking and systems engineering
concepts in a sophomore-level product design and development course, in ASME International
Mechanical Engineering Congress and Exposition, . 2016: Phoenix, AZ.
11. Muci-Kuchler, K.H., et al., Incorporating basic systems thinking and systems engineering
concepts in a mechanical engineering sophomore design course, in ASEE Annual Conference,
2017: Columbus, Ohio.
12. Squires, A. and W. Larson, Improving systems engineering curriculum using a competency-
based assessment approach. International Journal of Intelligent Defence Support Systems,
2009. 2(3): p. 184-201.
13. Squires, A.F., et al., Investigating an innovative approach for developing systems engineering
curriculum: The Systems Engineering Experience Accelerator, in ASEE Annual Conference.
2011, American Society for Engineering Education.
14. Valenti, M., Teaching tomorrow's engineers. Mechanical Engineering, 1996. 118(7): p. 64-69.
15. Whitcomb, C., R. Khan, and C. White, Development of a System Engineering Competency
Career Development Model: an analytical approach using Bloom’s Taxonomy. 2014,
Monterey, California. Naval Postgraduate School.
16. Whitcomb, C.A., et al., The Department of the Navy Systems Engineering Career Competency
Model. 2015, Monterey, California. Naval Postgraduate School.
17. Muci-Kuchler, K.H., et al., Extending Systems Thinking Skills to an Introductory Mechanical
Engineering Course, in ASEE Annual Conference,. 2019: Tampa, Florida.
18. Muci-Kuchler, K.H.B., Cassandra M.; Bedillion, Mark D., Lovett, Marsha; Whitcomb,
Clifford;. Incorporating Systems Thinking and Systems Engineering Concepts in a Freshman-
Level Mechanical Engineering Course. in ASEE's Virtual Conference at Home with
Engineering Education. 2020. Virtual.
19. Isbell, D.a.S., Don. Mars Climate Orbiter Failure Board Releases Report, Numerous NASA
Actions Underway in Response. 1999; Available from:
https://mars.jpl.nasa.gov/msp98/news/mco991110.html.
20. The Ocean Clean Up. Available from: https://theoceancleanup.com/.
21. Wickert, J. and K. Lewis, An introduction to mechanical engineering. 2012: Cengage learning.
22. Krathwohl, D.R., A revision of Bloom's taxonomy: An overview. Theory into practice, 2002.
41(4): p. 212-218.
23. Krathwohl, D.R. and L.W. Anderson, A taxonomy for learning, teaching, and assessing: A
revision of Bloom's taxonomy of educational objectives. 2009: Longman.
24. Muci-Kuchler, K.H., et al., Incorporating basic systems thinking and systems engineering
concepts in a mechanical engineering sophomore design course, in ASEE Annual Conference,
2017: Columbus, Ohio.
25. Degen, C.M., et al., Measuring the Impact of a New Mechanical Engineering Sophomore
Design Course on Students’ Systems Thinking Skills, in ASME International Mechanical
Engineering Congress and Exposition, 2018: Pittsburgh, PA.
