How does Academic Preparation Influence How Engineering Students Solve Problems?

Sarah J. Grigg and Lisa C. Benson Department of Engineering and Science Education

Clemson University Clemson, SC USA

lbenson@clemson.edu

Abstract—In first year engineering courses, students possess a wide range of academic preparation such as their exposure to various mathematics courses and pre-engineering programs. Additionally, students bring perceptions about their abilities, and have already begun practicing preferred methods of analysis and documentation. Understanding how students with different backgrounds develop problem solving skills in first year engineering programs is of critical importance in order to close achievement gaps between diverse populations.

This study examines how students solve engineering problems and identifies variations based on student factors of gender, ethnicity, mathematics preparation and achievement, and prior engineering experience. Solutions for three problems from 27 students were analyzed (n = 68 solutions). Students worked out problems using custom-designed software that digitally records ink strokes and allows researchers to associate codes to the problem solution at any point, even in portions of the problem solution that would not have been available without the use of this technology, such as in erased work.

Differences in how students solve problems were assessed based on the prevalence (or absence) of elements and errors in the problem solving process, which were evaluated using task analysis. Results indicated that pre-engineering experience did not have a significant impact on successfully solving problems; however, having completed a calculus course was significantly related to successful problem solving. Future research will expand the study population to a larger sample of first year engineering students across multiple semesters, identifying key strategies that are absent for students with low mathematics preparation, and to investigate relationships between prior academic preparation and indicators of metacognition.

Keywords- problem solving; first year engineering; academic preparation

I. INTRODUCTION

Engineering students must apply basic mathematical skills and reasoning to solve problems, ranging from arithmetic manipulations to analysis of variables. However, the level of mathematic and engineering preparation they bring to their first-year courses vary widely. Often instructors find that students do not have the prerequisite knowledge needed or

have strong enough analytical skills to learn new concepts successfully. The instructor may even feel it is necessary to review prerequisite material to the entire class before continuing to new concepts, which can restrict the amount of material that can be covered in a course.

In view of the one-way migration pattern from engineering majors [1], it is important to identify factors that cause students to withdraw from or fail to succeed in engineering courses. Here we examine problem solving proficiency as a potential factor. Engineering students from underrepresented groups such as females and minorities have been shown to have distinctly different engineering education experiences [2]. Research indicates that males seem to exhibit more advanced problem solving performances than females [3] and that females doubt their problem solving abilities more than males [4]. Research on the mathematical problem solving of minority students has shown that they suffer a larger dropout rate from engineering than all other students [5] and exhibit a lower success rate solving non-routine problems, even though their solutions indicate proportional skills levels [6]. Malloy defined non-routine problems as those “ that could be solved with multiple strategies, could be solved with holistic or analytical strategies, and required inferential, deductive, or inductive reasoning [6]. This definition of non-routing problems describes contextual “story” problems in a typical first year engineering course. If this trend is evident in first year engineering problem solving, it could shed light on a potential factor attributing to the higher than average withdrawal rate for under-represented minorities. Understanding how students with different backgrounds develop problem-solving skills in first year engineering programs is of critical importance in order to close achievement gaps between diverse populations.

When students work through problems, they construct an interpretation of the concepts being taught using pre-existing knowledge [7]. For meaningful learning to occur, a learner must make sense out of the information presented and have relevant conceptual knowledge to anchor new ideas [8]. A learner’s framework of relevant concepts allows him or her to solve problems efficiently and successfully. When this prior knowledge is lacking or inappropriate, the learner has difficulty solving the problem in the intended manner [9].

This study investigates the relationship between how students solve problems and their prior academic experiences,

978-1-4673-1352-0/12/$31.00 ©2012 IEEE

specifically prior mathematics courses and any pre-engineering experience such as involvement in FIRST or Project Lead the Way. FIRST is a program that encourages students ages 6-18 to build science, engineering, and technology skills through designing, building, and programing robots [10]. Research suggests that students who participated in FIRST Lego League showed increases in confidence and overall technological problem solving performance [11]. Project Lead the Way is a Science, Technology, Engineering, and Mathematics (STEM) education curricular program geared toward middle and high school students. Its goal is to develop critical-reasoning and problem-solving skills [12]. Research suggests that students who participate in Project Lead the Way have higher achievement in reading, mathematics, and science [13].

II. METHODS

A. Engineering Problems

This study examines problem solving solutions from students enrolled in a first-year undergraduate engineering course. Solutions were captured for three different problems typical of those given in a first year engineering course. Three problems covered topics of 1) efficiency, 2) circuits, and 3) pressure. All problems had 1) a constrained context, including pre-defined elements (problem inputs), 2) allowed multiple predictable procedures or algorithms, and 3) had a single correct answer [14]. All three problems were story problems, in which the student is presented with a narrative that embeds the values needed to obtain a final answer [15].

The first problem involved a multi-stage solar energy conversion system and required calculation of the efficiency of one stage given input and output values for the other stages [16]. The second problem required students to solve for values of components in a given electrical circuit. This problem, developed by the project team, also contained a Rule-Using/Rule Induction portion (a problem having one correct solution but multiple rules governing the process [15]), where students were asked to determine an equivalent circuit based on a set of given constraints. The third problem involved total pressure calculations and required students to solve for values within the system, and convert between different unit systems [16]. None of the problems required the use of calculus to solve. Example solutions for each problem are shown in Figures 1-3. Sixty-eight solutions were analyzed.

B. Participants

There were 27 participants, 6 females and 21 males. Twenty-three of the participants were Caucasian. Seven of the students had prior engineering experience through extracurricular activities. In terms of mathematics preparation, 4 of the students’ highest mathematics course was Pre-Calculus, 3 had taken AP Statistics but no Calculus, 11 had taken AB Calculus, and 9 had taken BC Calculus.

Figure 1: Solution for Solar Efficiency Problem

Figure 2: Solution for Equivalent Circuits Problem

Figure 3: Solution for Total Pressure Problem

C. Technology used to Capture Problem Solving Processes

Problem solving data was obtained via students’ completed in-class exercises using a program called MuseInk, developed at Clemson University [17, 18]. This software was used in conjunction with tablet computers that were made available to all students during the class period. Students worked out problems in the MuseInk application, which digitally records ink strokes and allows researchers to associate codes to the problem solution at any point, even in portions of the problem solution that would not have been available without the use of this technology, such as in erased work. Solutions were coded using the coding scheme developed by Grigg and Benson to describe cognitive and metacognitive processes, errors, and strategies revealed in student work [19].

D. Statistical Analysis Methods

To investigate differences in how students solve problems, statistical analyses were conducted to assess differences in 1) the presence of problem solving elements, and 2) the frequency of use of problem solving elements. Evaluations assessed student factors including gender, ethnicity, and prior academic experiences such as pre-engineering experience and highest level of mathematic course taken.

Repeated measures ANOVAs were conducted to determine if significant differences existed in the presence of task elements, errors, strategies, or answer state that are attributable to student factors of gender, ethnicity, pre-engineering experience, or mathematics preparation. Repeated measures ANOVAs were also conducted to evaluate differences in the frequency of task usage among problem solutions to further explore how problem solving solutions vary between populations. Only significant effects of student factors are reported. P values of 0.05 were utilized as the significance level.

III. RESULTS

Analysis was conducted on 54 codes of process elements, errors, strategies, and solution accuracy. For codes related to process elements, the basic structure set forth in the coding scheme by Wong, Lawson, and Keeves was used within the categories of knowledge access, knowledge generation, self-management [20]. For codes relating to errors, a structure derived from error detection literature in accounting, was used to classify errors as conceptual and mechanical errors [21, 22] with an added classification of management errors to capture errors in metacognitive processes. Strategy codes were obtained from a subset of strategies that appeared most applicable to story problems from the compilation described in “Thinking and Problem Solving” [23].

A. Assessment of students’ use of problem solving features

Results were evaluated for all codes of process elements, errors, strategies, and solution accuracy for the collective sum

of problem solutions, hence repeated measures ANOVAs were utilized to account for variations attributed to the problem. If the code was associated to a problem solution at least once, that code was determined as present, even if the work was later modified to eliminate its presence in the final solution.

The level of academic preparation of female students was quite different from that of male students. Of the female students in the study, only 33% had calculus experience compared to 86% of male students. Additionally, none of the female students had pre-engineering experience compared to 33% of male students. Therefore, it should be noted that differences found in terms of gender may be due (at least in part) to differences in level of academic preparation. This will be explored in future data analysis that includes a larger sample population of students.

Findings revealed significant differences that are attributed to gender. Females were more likely than males to explicitly write out equations and then plug in values in separate steps (p = 0.035), and had a higher occurrence of incorrectly deriving unit (p=0.002). Additionally, females’ solutions were more likely to indicate the use of lower level strategies such as a “guess and check strategy” (p = 0.030) or a “plug and chug strategy” (p=0.53) while males’ solutions were more likely to indicate the usage of advanced approached such as a “chunking strategy” (p = 0.048). Males exhibited a higher level of mastery, as they were also more likely to obtain a correct solution. Full results are shown in Table 1.

TABLE 1: SIGNIFICANT EFFECTS OF GENDER BASED ON REPEATED MEASURES ANOVAS

Process Analysis Measure F ratio p value Mean

(Female) Mean (Male)

Plugged values in equation 4.62 0.035 1.00 0.77

Incorrect unit derivation 10.75 0.002 0.20 0.04 Plug and chug 5.23 0.026 0.53 0.23

Guess and check 4.95 0.030 0.33 0.11 Chunking 4.08 0.048 0.07 0.30

Correct Answer 13.23 0.001 0.13 0.64

The level of academic preparation of minority students was roughly equivalent to that of the remainder of the students. Of the minority students in the study, 75% had calculus experience compared to 78% of non-minority students. Additionally, 25% of minority students had pre-engineering experience compared to 26% of non-minority students. Therefore, it is reasonable to assume that differences found based on ethnicity are not attributable to prior academic experiences of completing a calculus class or participating in a pre-engineering program.

Very few significant results were found based on ethnicity. The only significant findings were that minority students had a larger number of solutions that did not utilize units throughout the problem solving solutions (p<0.001) and with correct answers but with incorrect units (p=0.003). Full results are shown in Table 2.

Supported the National Science Foundation Award # EEC-0935163

TABLE 2: SIGNIFICANT EFFECTS OF ETHNICITY BASED ON REPEATED MEASURES ANOVAS

Process Analysis Measure

F ratio

p value

Mean (Caucasians)

Mean (Minorities)

Missing Units Throughout 15.87 0.000 0.00 0.20

Correct but Incorrect Units 9.28 0.003 0.05 0.20

In terms of prior academic preparation, there were only a

few significant findings that supported claims in the literature that pre-engineering programs enhance problem solving performance. There were no significant differences found for success in the classroom overall, or for correct problem solving solutions. Significant differences revealed that differences were mainly based on format of solving problems; students that had pre-engineering experience had a larger number of solutions where they identified prior knowledge (p=0.018), documented algebraic steps (p=0.018) and explicitly identified their final answers either by boxing in their answer or writing out the conclusion in sentence form (p=0.030). Full results are shown in Table 3.

TABLE 3: SIGNIFICANT EFFECTS OF PRE-ENGINEERING EXPERIENCE

BASED ON REPEATED MEASURES ANOVAS

Process Analysis Measure

F ratio

p value

Mean (With pre-engineering Experience)

Mean (Without pre-engineering Experience)

Identify prior knowledge 5.89 0.018 0.12 0.02

Document math 5.91 0.018 0.94 0.65 Identify final

answer 4.93 0.030 0.88 0.61

A more extensive set of differences were revealed between

students based on mathematics preparation as distinguished between students with and without calculus experience. It was revealed that students who had not yet taken a calculus class beyond pre-calculus committed more errors and had more rework than students with calculus experience. Solutions from students without calculus experience were more likely to utilize labeling or renaming (p = 0.012), have incorrect unit

TABLE 4: SIGNIFICANT EFFECTS OF CALCULUS EXPERIENCE

BASED ON REPEATED MEASURES ANOVAS

Process Analysis Measure

F ratio

p value

Mean (With

Experience)

Mean (Without

Experience) Labeling / Renaming 6.75 0.012 0.38 0.75

Incorrect unit derivation

5.94 0.018 0.04 0.19

Incorrect unit assignment

4.34 0.041 0.04 0.19

Using incorrectly generated

information 4.69 0.034 0.17 0.44

Missing units throughout

6.74 0.011 0.00 0.13

Correct Answer 7.61 0.008 0.62 0.25 Correct but Incorrect

Units 6.45 0.014 0.04 0.18

derivations (p = 0.018), have incorrect unit assignment (p = 0.041), use incorrectly generated equations (p =0.034), have missing units throughout the entire attempt (p = 0.011), and have answers that were correct but in incorrect units ( p = 0.014). Similarly, students with calculus experience were more likely to obtain correct answers. Full results are shown in Table 4.

B. Assessment of frequency of use of problem solving features

As a means of validating the results found based on the presence of codes, the same evaluation was conducted based on differences in the frequency of codes. When assessed by frequency of codes, fourteen of the eighteen significant differences remained with eight additional significant effects. Most differences were found for effects based on gender, with one discrepancy and four additional effects. Based on frequency of use, there was not a significant difference in the number of times males or females plugged values into equations This may be explained by males iterating through more instances of plugging in values within the same problem than females (eg. trying more combinations of possible solutions.) Additional effects found that females solved for intermediate values more frequently than males (p = 0.035), identified more known values, (p = 0.029), and identified more errors (p = 0.006) though this may be due to committing more error such as more frequently using incorrectly generated information (p = 0.034). These additional effects support the results indicating that females showed a tendency of using lower level strategies. Full results are shown in Table 5.

TABLE 5: SIGNIFICANT EFFECTS OF GENDER

FOR FREQUENCY OF USE OF PROBLEM FEATURES Process Analysis

Measure F ratio p value Mean

(Female) Mean (Male)

Solve intermediate values

4.65 0.035 3.27 2.17

Identify known value 5.00 0.029 2.80 1.42 Identify errors 8.17 0.006 6.20 3.87 Incorrect unit

derivation 10.75 0.002 0.20 0.04

Using incorrectly generated information

4.70 0.034 0.80 0.32

Plug and chug 5.23 0.026 0.53 0.23 Guess and check 4.95 0.030 0.33 0.11

Chunking 4.09 0.048 0.07 0.30 Correct Answer 4.51 0.038 0.33 0.85

Only one additional significant effect was revealed based

on ethnicity with no discrepancies; minority students more frequently assigned units incorrectly (p=0.030). This finding also supports the original analysis by revealing another significant effect related to the use of units. Full results are shown in Table 6.

TABLE 6: SIGNIFICANT EFFECTS OF ETHNICITY FOR FREQUENCY OF USE OF PROBLEM FEATURES

Process Analysis Measure

F ratio

p value

Mean (Caucasians)

Mean (Minorities)

Incorrect unit assignment

4.93 0.030 0.07 0.30

Missing units throughout

15.87 0.000 0.00 0.20

Correct but incorrect units

9.27 0.003 0.05 0.20

For students with pre engineering experience, there was

one discrepancy and one additional effect uncovered. While the relationship to documenting math was not evident when assessed based on frequency, it was revealed that students with pre-engineering experience more frequently ignored problem constraints (p=0.004). While this finding is surprising, it reiterates that pre-engineering experience does not ensure that students’ solutions will be procedurally or conceptually correct. Full results are shown in Table 7.

TABLE 7: SIGNIFICANT EFFECTS OF PRE-ENGINEERING EXPERIENCE

FOR FREQUENCY OF USE OF PROBLEM FEATURES

Process Analysis Measure

F ratio

p value

Mean (With

Experience)

Mean (Without

Experience) Identify prior knowledge

5.90 0.018 0.18 0.02

Identify final answer 6.54 0.013 1.29 0.76 Ignored problem

constraints 8.77 0.004 0.47 0.10

For students with calculus experiences, significant effects

for labeling / renaming and incorrect unit assignment disappeared when assessed by frequency; however, two additional significant effects of calculus experience emerged. Students without calculus experience both erased work (p = 0.024) and identified errors (p = 0.003) more often than students with calculus experience, though this may be due to the higher frequency of specific errors. Therefore, these results also support the original analysis. Full results are shown in Table 8.

TABLE 8: SIGNIFICANT EFFECTS OF CALCULUS EXPERIENCE

FOR FREQUENCY OF USE OF PROBLEM FEATURES

Process Analysis Measure

F ratio

p value

Mean (With

Experience)

Mean (Without

Experience) Erasing work 5.33 0.024 1.33 2.53 Identify errors 9.67 0.003 3.71 6.41 Incorrect unit

derivation 6.46 0.014 0.03 0.18

Using incorrectly generated

information 5.26 0.025 0.27 0.88

Missing units throughout

5.74 0.020 0.00 0.12

Correct Answer 6.16 0.016 0.86 0.35 Correct but Incorrect

Units 5.69 0.020 0.04 0.18

IV. DISCUSSION

This analysis revealed several significant differences in how students solve problems and revealed that differences of male gender and calculus experience were related to a higher probability of successful problem solving attempts, though none of the populations tested showed significant difference in terms of overall course grade.

While course grades did not indicate a difference in achievement between male and female students, females seemed to struggle with problem solving more than males. Solutions completed by females were more likely to contain instances of incorrectly derived units and lower level strategies to approach the problems and solutions completed by males were more likely to result in a correct final answer. Units seemed to present more problems for students who did not have calculus experience (with more instances of incorrect unit assignment, missing units throughout, and having answers with incorrect units) and minority students (with more instances of missing units, and answers with incorrect units). However, it seems that females and students without calculus experience are at least aware of their performance deficiencies, as both had a higher frequency of identifying errors.

Therefore, this analysis revealed some targeted needs of specific student populations within this particular first year engineering course. While these results do have similarities to those found in the literature, these results are not necessarily generalizable to the entire population of first year engineering students. However, other instructors could utilize the methodology described in this paper as a means of identifying areas of instructional needs of students in their own classes.

V. CONCLUSIONS

This study supports current literature that shows significant differences based on gender in terms of how students approach problem solving, and supports the literature that shows a lack of significant differences in how students from different ethnic backgrounds solve problems (with the exception that students in this particular group struggled with consistent use of units). While research conducted on middle school students showed that minority students struggle with non-routine problems, this was not evident in this study. It is possible that engineering students (or at least the engineering students that participated in this study) have strengthened their problem-solving skills through advanced mathematics courses. Future research will investigate the interconnectivity of predictor factors using regression models to control for certain effects (such as gender and ethnicity) while exploring the impact of academic preparation.

The results of this study suggest that while academic preparation in the form of pre-engineering experience significantly influences how students solve problems, it did not indicate a significant impact of the probability of success in this course overall or in successfully solving problems. However, academic preparation in the form of previously completing a calculus class (but not necessarily grade) was associated with successful problem solving. This brings into question whether

there is something about going through a calculus class that has bearing on problem solving success, such as a familiarity with conducting more complicated sets of procedures, or whether it is the students’ natural mathematical abilities that have enabled them to get into a calculus class at an earlier age that has the greatest impact. Future analyses will compare results of a mathematics placement test that is given to all incoming freshmen at our institution in order to test whether mathematical skill level is a better predictor of success than the highest level of mathematics course completed.

Future research will also expand the study population to a larger sample of first year engineering students across multiple semesters, identifying key strategies that are absent for students with low mathematics preparation, and to investigate relationships between prior academic preparation and indicators of metacognition.

ACKNOWLEDGMENT

This work could not have been completed without the support of the National Science Foundation, the willingness of the students to participate in the study, as well as other researchers who had a hand in the research process including Michelle Cook, Catherine McGough, Brandon Olszewski, Jennifer Parham-Mocello, and David Bowman.

REFERENCES

[1] Ohland, M.W., Sheppard, S.D., Lichenstein, G., Eris, O., and Charchra, D., Persistence, engagement, and migration in engineering programs. Journal of Engineering Education, 2008. 97(3): p. 259-278.

[2] Adelman, C., Women and men of the engineering path: a model for analysis of undergradute careers. 1998, Office of Educational Research and Improvement.: Washington, D.C.

[3] Zhu, Z., Gender differences in mathematical problem solving patterns: A review of literature. International Education Journal, 2007. 8(2): p. 187-203.

[4] Felder, R.M., Felder, G.N., and Mauney, M., A longitudinal study of engineering student performance and retention. III. Gender differences in student performance and attitudes. Journal of Engineering Education, 1995. 84(2): p. 151-163.

[5] National Research Council . Retention Task Force and National Research Council . Committee on Minorities in Engineering, Retention of minority students in engineering: a report of the Retention Task Force. 1977: National Academy of Sciences.

[6] Malloy, C.E. and Jones, M.G., An Investigation of African American Students' Mathematical Problem Solving. Journal for Research in Mathematics Education, 1998. 29(2): p. 143-163.

[7] Bruner, J.S., Going beyond the information given, in Beyond the information given: Studies in the psychology of knowing, J.M. Anglin, Editor. 1973, Norton: New York. p. 218-238.

[8] Novak, J.D. and Gowin, D.B., Learning how to learn. 1984, Cambridge, United Kingdom: Cambridge University Press.

[9] Chi, M.T., Feltovich, P.J., and Glaser, R., Categorization and representation of physics problems by experts and novices. Cognitive Science, 1981. 5: p. 121-152.

[10] FIRST. 2010 April 9, 2012]; Available from: http://www.usfirst.org/. [11] Varnad, T.E., The effects of a technological problem solving activity on

FIRST LEGO League Participants' problem solving style and performance, in Curriculum and Instruction. 2005, Virginia Polytechnic Institute and State University: Blacksburg, Virginia. p. 135.

[12] PLTW, S. What is PLTW? 2012 [cited April 9, 2012; Available from: http://www.southcarolinapltw.com/About_PLTW.html.

[13] Bottoms, G. and Anthony, K. (2005) Project Lead the Way: A pre-engineering curriculum that works.

[14] Jonassen, D.H., Learning to Solve Problems. 2004, San Fransisco, CA: Pfeiffer.

[15] Jonassen, D.H. Research Issues in Problem Solving. in 11th International Conference on Education Research. 2010.

[16] Stephan, E.A., Park, W.J., Sill, B.L., Bowman, D.R., and Ohland, M.H., Thinking Like an Engineer: An Active Learning Approach. 2010: Pearson Education.

[17] Bowman, D. and Benson, L. MuseInk: Seeing and hearing a freshman engineering student think. in 2010 ASEE Annual Conference. 2011.

[18] Grigg, S. and Benson, L., Work in Progress: Robust Engineering Problems for the Study of Problem Solving Strategies, in Proceedings of the Frontiers in Education Conference. 2011.

[19] Grigg, S. and Benson, L., Design and validation of a coding scheme for analyzing engineering problem solving. 2012, Clemson University: Submitted to JEE.

[20] Wong, R.M.F., Lawson, M.J., and Keeves, J., The effects of self-explanation training on students' problem solving in high-school mathematics. Learning and Instruction, 2002. 12(1): p. 233-262.

[21] Owhoso, V.E., Messier, W.F., and Lynch Jr., J.G., Error detection by industry-specialized teams during sequential audit review. Journal of Accounting Research, 2002. 40(3): p. 883-900.

[22] Ramsay, R.J., Senior/Manager difference in audit workpaper review performance. Journal of Accounting Research, 1994. 32(1): p. 127-135.

[23] Nickerson, R.S., The teaching and thinking of problem solving, in Thinking and Problem Solving, R.J. Sternberg, Editor. 1994, Academic Press: San Diego. p. 409-449.