iSTEM and Learning Outcomes

Anthony Petrosino

University of Texas, Austin

NATIONAL ACADEMY OF ENGINEERING

NATIONAL RESEARCH COUNCILBOARD ON SCIENCE EDUCATION

Committee on Integrated STEM Education

Second Meeting

Keck Center, Room 204

Washington, D.C.

January, 10-12, 2012

iSTEM and Learning Outcomes

Acknowledgements

Leona Schauble (Vanderbilt)

Rick Duschl (Penn St)

Lupita Carmoa (University of Texas, Austin)

Candace Walkington (University of Wisconsin)

A typical integrated STEM Activity

Beyond Fun and Engagement

How high did the rocket go?

How did you figure that out?

Doing Deeper

Scaffolding the Experimentation Process

Sources of Variance Data Modeling

Discipline vs Integration

John Dewey

Dewey criticized the presentation of information in isolated separate subjects:

We do lie in a stratified earth, one of which is mathematical, another physical, another historical, and so on. We should not be able to live very long in any one taken by itself. We live in a world where all sides are bound together. All studies grow out of relations in the one great common world.

The Uniqueness of the Engineer

Engineers use science, but distinguish themselves from scientists. They do math but do not identify themselves as mathematicians. They use and invent technology but typically reject the title of technician. As a profession, engineers enjoy a complex relationship with the other STEM fields, having to demonstrate mastery with each of them, yet acting in a manner wholly distinct from any of them. Walkington et al, in press

Candace Walkington

Candace A. Walkington, Mitchell J. Nathan, Matthew Wolfgram, Martha W. Alibali, and Rachaya Srisurichan-Bridges and Barriers to Constructing Conceptual Cohesion Across Modalities and Temporalities: Challenges of STEM Integration in the Precollege Engineering Classroom

7

STEM: Integrated or Separated

Integrated STEM: The principles of science and the analysis of mathematics are combined with the design process of technology and engineering in the classroom.

Separated S.T.E.M.: Each subject is taught separately with the hope that the synthesis of disciplinary knowledge will be applied. This may be referred to as STEM being taught as Silos

Overarching Questions

1) What are the stated goals/objectives of integrated STEM projects?

2) In what ways have these goals/objectives been measured?

3) What evidence from research or evaluation do we have that integrated STEM projects are achieving these goals/objectives? Do we know what characteristics of integrated STEM are associated with positive outcomes?

4) Does the broader research literature on science learning provide insights about why and how integrated STEM projects are effective for supporting these goals/objectives?

5) Where evidence is not available, what kinds of research or evaluation would be needed in order to show that iSTEM supports these goals/objectives?

1) What are the stated goals/objectives of integrated STEM projects?

Using an interdisciplinary or integrated curriculum provides opportunities for more relevant, less fragmented, and more stimulating experiences for learners

When done properly, integration of STEM brings together overlapping concepts and principles in a meaningful way and enriches the learning context.

Learning situated in such enriched (macro) contexts often lead to meaningful learning experiences.

2) In what ways have these goals/objectives been measured?

Affective measures

Standardized tests

Project specific assessments

Detailed qualitative analysis

However.

However.

Studies showing advantages of integrated curricula on student performance typically show only relative benefits over business-as-usual models rather than an explication of how actions and reasoning processes have changed.

How integration happens, why it succeeds or fails, and the manner in which it improves performance and learning all remain fairly underspecified in the literature. (Nathan, M. J. Srisurichan, R., Walkington, C., Wolfgram, M., Williams, C. & Alibali, M. W. (under review)

Without a clearly defined construct of iSTEM, scholars and policy makers cannot be certain that studies purported to show benefits (or those that dont) are all testing comparable interventions and outcomes.

Nathan, M. J. Srisurichan, R., Walkington, C., Wolfgram, M., Williams, C. & Alibali, M. W. (under review) Cohesion as a Mechanism of STEM Integration.Journal of Engineering Education,Special Issue on The Role of Representations in Engineering Learning & Practice.JEE iSTEM Integration Repns-Rev1 Submit.pdf

12

3a) What evidence from research or evaluation do we have that integrated STEM projects are achieving these goals/objectives?

There is some empirical support for positive effects on learning with curricula that provide iSTEM (Burghardt and Hacker, 2002; Fortus, Krajcik, Dershimer, Marx, & Mamlok-Naaman, 2005; Hartzler, 2000; Kolodner, et al., 2003; Satchwell & Loepp, 2002; Phelps, Camburn, & Durham, 2009; Wang, Moore, Roehrig, & Park, in press).

Burghardt, D., & Hacker, M. (2007, February). Engineering professional development. Paper presented at the

National Symposium to Explore Effective Practices for Professional Development of K-12 Engineering and

Technology Teachers, Dallas, TX.

Fortus, Krajcik, Dershimer, Marx, & Mamlok-Naaman, (2005). Design-based science and real-world problemsolving.

Howard M. Glasser, Knowles Science Teaching Foundation, Moorestown, NJ

PHYSICS FIRST A two-sample t-test related the pre-inversion and postinversion

sets, and the rightmost column of Table I shows that

for the class of 2003 the results were significant at the p < 0.1

level and the data for the classes of 2004 and 2005 were significant

at the p < 0.005 level. Therefore, a statistically significant

difference between pre-inversion and post-inversion scores

was found, indicating a strong association between physics in

ninth grade and improved test scores on the mathematics portion

of the PSATs for these students.

Phelps, L.A., Camburn, E., & Durham, J. (2009, June). Engineering the math performance gap. Research Brief.

Madison: Center on Education and Work, University of Wisconsin-Madison.

Wang, H.-H., Moore, T.J., Roehrig, G.H., & Park, M.S. (in press). STEM integration: The impact of professional

development on teacher perception and practice. Journal of Pre-College Engineering Edutcation Research.

International Journal of Science Education, 27(7), 855879.

Hartzler, D. (2000). A meta-analysis of studies conducted on integrated curriculum programs and their effects on

student achievement. Doctoral Dissertation. Indiana University.

Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., & Ryan, N. (2003).

Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting

Learning by Design into practice. The Journal of the Learning Sciences, 12( 4), 495-547.

13

Continued

This is also supported by learning sciences research on transfer of knowledge (e.g. Pellegrino et al., 2001; Sheppard, Pellegrino, & Olds, 2008).

provided there is explicit attempts to integrate

Pellegrino, J.W., Chudowsky, N., & Glaser, R. (2001), Knowing what students know: The science and design of

educational assessment. Washington, DC: National Academy Press.

Sheppard, S.D., J.W. Pellegrino, and B.M. Olds. 2008. On becoming a 21st century engineer. Journal of

Engineering Education, 97(3), 231-234.

14

Findings Show Uneven and Difficult

Yet the effects of iSTEM on student learning are uneven (Hartzler, 2000; Prevost et al., in press; Tran & Nathan, 2010a, 2010b)

And high quality implementations of iSTEM are not commonplace (Katehi et al., 2009; Nathan et al., 2008; Prevost et al., 2009; Prevost et al., 2010; Welty, Katehi, Pearson & Feder, 2008).

Hartzler, D. (2000). A meta-analysis of studies conducted on integrated curriculum programs and their effects on

student achievement. Doctoral Dissertation. Indiana University.

Prevost, A. C., Nathan, M. J., Phelps, L. A., Atwood, A. K., Tran, N. A., Oliver, K., & Stein, B. (in press).

Academic connections in precollege engineering contexts: The intended and enacted curricula of Project

Lead the Way and beyond. In Strobel, J., Purzer, S. & Cardella, M. (Eds.) Engineering in Pre-College

Settings: Research into Practice. Sense Publishers, Rotterdam, Netherlands.

Tran, N. A. & Nathan, M. J. (2010a). An investigation of the relationship between pre-college engineering studies

and student achievement in science and mathematics. Journal of Engineering Education, 99(2), 143-157.

Tran, N. A. & Nathan, M. J. (2010b). Effects of pre-college engineering studies on mathematics and science

achievements for high school students. International Journal of Engineering Education, 26(5), 1049-1060.

Katehi, L., Pearson, G., & Feder, M. (Eds.) (2009). Engineering in K-12 education: Understanding the status and

improving the prospects. Washington, D.C.: The National Academies Press.

Nathan, M. J. Tran, N., Phelps, L. A., & Prevost, A. (2008). The structure of high school academic and preengineering

curricula: Mathematics. Proceedings of the American Society of Engineering Education

Prevost, A., Nathan, M. J., Stein, B., Tran, N., & Phelps, L. A. (2009). Integration of mathematics in pre-college

engineering: The search for explicit connections. Proceedings of the American Society of Engineering

Education (ASEE) 2009 (Paper no. AC 2009-1790, pp. 1-27). Austin, TX: ASEE Publications

Prevost, A., Nathan, M. J., Stein, B., & Phelps, L. A. (2010). The enacted curriculum: A video based analysis of

instruction and learning in high school engineering classrooms. Proceedings of the American Society of

Engineering Education (ASEE) 2010 (Paper no. AC 2010-1121). Louisville, KY: ASEE Publications.

Welty, K., Katehi, L., & Pearson, G. and Felder (2008). Analysis of K12 engineering education curricula in the United States:

A preliminary report. In Proceedings of the American Society for Engineering Education Annual

Conference and Exposition. Pittsburgh, PA

15

3b) Do we know what characteristics of integrated STEM are associated with positive outcomes?

We seem to think we do

Base integration on how students experience, organize, and think about science and math.

Take advantage of patterns as children from the day they are born are looking at patterns and trying to make sense of the world.

Collect and use data in problem-based integrated activities that invoke process skills.

Integrate where there is an overlapping content in math and science.

Be sensitive to what students believe and feel about math and science, their involvement and the confidence in their ability to do science and math.

Use instructional strategies that would bridge the gap between students classroom experiences and real-life experiences outside the classroom.

White and Berlin (1992)

Sunal and Furner (1995)

Base integration on how students experience, organize, and think about science and math.

Take advantage of patterns as children from the day they are born are looking at patterns and trying to make sense of the world.

Collect and use data in problem-based integrated activities that invoke process skills.

Integrate where there is an overlapping content in math and science.

Be sensitive to what students believe and feel about math and science, their involvement and the confidence in their ability to do science and math.

Use instructional strategies that would bridge the gap between students classroom experiences and real-life experiences outside the classroom.

16

Cont

An understanding of the nature of subject field and the need for teachers, for example, single subject field/single teacher; single subject field/multiple teachers; multiple subject fields/single teacher; or multiple subject fields/multiple teachers.

A deeper knowledge of methods of interdisciplinary subject matter correlation (unified subject field, theme, topic, problem-based, etc.)

Strategies for motivating students to use process skills, such as reading, writing, reporting, research, problem solving, mathematical application, data collection, data analysis, an drawing conclusions.

Robinson, L. (1994). Interdisciplinary planning and instruction. Handout made for Interdisciplinary Teaching and Learning Workshop on May 30, 1994 for Tuscaloosa City High Schools.

17

So, Where Are We?

Mitch Nathan

Yet, our review of iSTEM research and policy reveals an important and noticeable gap; namely that the integration process, while touted for its benefits and broad appeal, remains somewhat mysterious. How integration occurs, whether integration is chiefly about instructional practices or the knowledge states of students, and what demonstrable impact it has on performance, all remain largely underspecified.

Nathan and colleagues offer A Possible Path: However, cohesion production specifically directed at making connections across modal engagements and linking ideas and inscriptions that share key conceptual structure across the silos provide a process account of iSTEM.

A modal engagement is defined by Hall & Nemirovsky (2010, p. 1) as an activity someone participates in, with others, tools, and symbols.

18

4) Does the research literature provide insights on effective integrated STEM projects?

A situated posits that knowing in a domain involves the adoption and reorganization of appropriate participation practices in social systems of activity

knowledge of mathematical and scientific ideas is not separable from the practices through which these ideas arise, or the context of the learning environment

The context of an activity system like school includes learners, teachers, curriculum materials, and the physical environment, as well as representational, material, informational, and conceptual resources.

Communities and groups have the power to shape what counts as knowledge, including the meanings of terminology, concepts, and principles, and how these can be applied in practice by community members.

Viewing learning as a trajectory of participation in activity systems leads to a conceptualization of transfer as the ways in which participation in activity in one social setting contributes to growth as a learner and future participation in other activity systems of value

(Cobb & Bowers, 1999; Driver, Asoko, Leach, Mortimer, & Schott, 1994; Greeno, 2006).

19

Big P---Little p/Strong P---Weak p

Project-based instruction

Model-eliciting activities

Agent-based modeling and simulations, etc.

Challenge Based Instruction

These activities have in common that they require students to solve a particular problem, usually in teams, and document the development of how they arrived to their solution. Because of the way these activities are designed, they usually require students to bring to bear and develop knowledge in some or all of the STEM areas.

20

The Legacy Cycle

Schwartz, D., Lin, X., Brophy S., Bransford, J. (1999). Toward the development of flexibly adaptive instructional designs. In C. Reigeluth (Ed.) Instructional-Design Theories and Models: A New Paradigm of Instructional Theory (Pp. 183 214). Mahwah, NJ: Erlbaum.

The STAR Legacy Cycle (Schwartz, Lin, Brophy, & Bransford, 1999), a challenge-based approach to instruction based on learning science research, also directs students to both look ahead and reflect back on project activity. In this way, students may benefit from opportunities to understand where project activity is leading, and how invariant relations will be instantiated in future stages of the design cycle. Students also may benefit from interventions that allow them to reflect on previous stages of project activity, integrating mathematical and scientific ideas across the classrooms lived history of modal engagements. This may be especially important as participants become increasingly oriented towards motivating and highly salient capstone events, and risk getting caught up in activity for activitys own sake.

21

5) What kinds of research or evaluation would be needed in order to show that iSTEM supports these goals/objectives?

21st Century CompetenciesiSTEM Perspective Instruction Focus on the completion of a goal or design. Students are successful when the design meets the goal. (e.g., the airplane is able to fly).The focus is on finding generalizations that help understand (the conditions of) what works and what doesnt work and why. Assessment Dichotomous perspective/RubricA more non-judgmental characterization that focuses on student understanding of STEM and iSTEM content. Validation Rather than beginning with a pre-determined set of standards and/or competencies The focus needs to be on experts validating the knowledge that emerges or is generated by students engaged in iSTEM rich activities. Psychometrics Move beyond psychometrics that emphasize dichotomous or hierarchical variables (e.g. IRT 1-PL, 2-PL, 3-PL)Use some implementations of Cognitive Diagnostic Modeling that allow for multivariate representations and characterizations of student thinking.

An example is the knowledge wheel (Carmona) that has been developed specifically to characterize student understanding of math as elicited in their responses to a modeling activity.

23

Leona Schauble

Many projects have attempted to "integrate," but mostly they are problematic because they are not guided either by strong long-term views of the development of the disciplines they are trying to integrate, much less by any long-term sense of how student knowledge co-develops across disciplines.

Most "integration" involves the clumsy smushing together of disciplines, usually by treating one as the handmaiden to the other. Think about it--without really serious long-term planning, how would you know that a particular mathematical idea would be available at a particular time to serve as a resource in science unless you had already carefully developed it? This, therefore, turns out to be a really fundamentally difficult issue, and in my view, there has been no CLEAR HEADED thinking about it. It is certainly true that real problems do not appear in disciplinary boxes, but this does not mean that abandoning disciplinary structure is the way to prepare people to solve problems like this.

Without really serious long-term planning, how will we know that a particular mathematical idea will/can be available at a particular time to serve as a resource in science unless we had already carefully developed it? This, therefore, turns out to be a really fundamentally difficult issuethere has been very little CLEAR HEADED thinking about it. It is certainly true that real problems do not appear in disciplinary boxes, but this does not mean that abandoning disciplinary structure is the way to prepare people to solve real world problems

24

Conclusion

Although these competencies might be of great importance (especially from an economics perspective for its connection to the job market), not enough is emphasized about the STEM content addressed and the development of student thinking of STEM. Even less attention is provided to the development of student thinking of iSTEM. I argue that if iSTEM education is to become a reality in K-12, there is an imminent need to put less weight in student development of skills and competencies for the 21st Century and begin prioritizing student understanding of STEM and iSTEM content.