1

Evaluating Complex System Interventions

Evaluation 2009 Professional Development Workshop

Beverly Parsons and Meg Hargreaves

2

What is a System?

A group of interacting, interrelated, and interdependent elements forming a complex whole

A configuration of parts connected and joined together by a web of relationships

The whole is different from, and greater than, the sum of its parts

3

Parts of an Elephant

4

Systems Thinking

A way of understanding reality that emphasizes the relationships among a system’s parts, rather than the parts themselves.

Concerned about interrelationships among parts and their relationship to a functioning whole

Sees underlying patterns and structures

5

Foundations of Systems Theory

Cybernetics: system feedback, information; differences (that make a difference); human – machine analogy; inclusion of the observer and the observed in the system

General systems theory: open systems; system integrity; nested system hierarchy, boundaries, webs, emergence (sum greater than parts)

6

Systems Theories

Soft and critical systems: human systems - multiple perspectives, power issues, intractable problems without simple solutions

Systems dynamics: systems have reinforcing and balancing feedback loops, circularity, system archetypes, mental models, unintended consequences

7

More Systems Theories

Complexity theory: complex adaptive systems; semi-independent, interacting agents; self-organization; emergence; nonlinearity; co-evolution; past is irreversible; future is unpredictable

Learning systems: the way that people learn and the systems in which they learn

8

System Boundaries

Shows what is inside and outside of the system Geographical (location) Organization (department, unit or function) Physical (money, material, information) Conceptual (goals, mission, purpose, rules) Intangibles (perceptions, awareness, models) Natural or man-made

9

System Relationships (Interconnections)

Connections and exchanges among system parts, parts and the whole, and the whole and its environment Flows of information Flows of funding Client referrals Collaborative partnerships Family, community, and social networks

10

System Perspectives

Stakeholders’ worldviews and purposes

System agents who have different perspectives may pursue different purposes within a given situation

Patterns of (mis)alignment of purposes and processes within and across system levels

11

System Change

System differences generate creative tension or energy within a system

Positive or negative, energy provides potential for system change

System change: shifts in patterns (similarities and differences) of system relationships, boundaries, focus, timing, events and behaviors over time and space

12

System Dynamics

Random (unorganized) Organized (simple or complicated) Adaptive (organic, self-organizing) All three system dynamics can be

present in a complex situation

13

Random System Attributes

Random activity – no pattern Unconnected collection of parts No cause-effect relationships Turbulence – no stability or equilibrium Answers are unknowable No purpose or direction – people react

blindly in a war zone or natural disaster

14

Random System: Hurricane Katrina

15

Organized (Simple) System Attributes

Stable, static pattern Parts tightly connected machines Predictable cause-effect relationships System can be reduced to parts and

processes and replicated Directive leadership, designed change Answers are knowable, with recipes or

prescriptions for action

16

Single Organized System: Ring-Around the Rosie

17

Simple Organized System:Riding a Bicycle

18

Organized (Complicated) System Attributes

Dynamic patterns of feedback loops with many interrelated parts within and across subsystem levels

Recursive, non-linear cause-effect relationships; reinforcing and balancing feedback loops maintain equilibrium

Expert analysis can identify causal loops, deep structural causes to actions

19

Insider Trading: A Tangled Web of

Tips and Trades

20

Adaptive (Complex) System Attributes

Dynamical patterns – parts adapting, co-evolving with each other and environment

Parts are massively entangled and interdependent; nested webs, networks

Parts self-organize, learn, and change Equilibrium in flux, sensitive to initial

conditions; system change emerges through interactions among parts

21

Ecological View of an Elephant

22

Complex Interdependencies

23

Alignment of Context, Program, and Evaluation Dynamics

Context can be random, organized, adaptive, or combination of dynamics

Program design uses random, organized (entity-based), or adaptive (paradigm-based) or a combination of dynamics

Evaluation design (content and process) can be entity-focused (organized), paradigm-focused (adaptive) or a combination of both

24

System Dynamics of Family Nutrition

25

Simple Organized Dynamics of Family Nutrition

Context: hungry family Intervention: buy ingredients, bake a

cake, serve family at dinner Evaluation: quality of cake, family

satisfaction

26

Complicated Organized Dynamics of Family Nutrition

Context: hungry family with different tastes and preferences

Intervention: ask for family preferences, create optional dishes, serve family multiple dishes at dinner

Evaluation: quality and variety of dinner options, matching of dishes to tastes

27

Complex, Adaptive Dynamics of Family Nutrition

Context: hungry family with different tastes, schedules, and cooking ability

Intervention: Buy and store meal options, make dishes for non-cooks, agree on dinner schedule, adapt shopping patterns to use of food and supplies

Evaluation: trends, patterns of food use, meals, family nutrition, overall health

28

System Dynamics of H1N1 Flu

29

Simple Organized Dynamics of H1N1 Flu

Context – everyone should be protected through vaccination

Program design – universal flu shot clinics

Evaluation design - How many clinics were conducted, how many people were vaccinated, how many people contracted the H1N1 flu virus

30

Complicated Organized Dynamics of H1N1 Flu

Context – people are at different risk levels for contracting the H1N1 flu

Program design – allocate, administer flu shots by risk level, triage patients by level of risk

Evaluation design - What proportion of people with high/medium/low risk receive the vaccine? What proportion of people at each risk level contract the H1N1 flu? How many deaths and hospitalizations are avoided as result of shots?

31

Complex Adaptive Dynamics of H1N1 Flu

Context – Timing of two interacting epidemics (H1N1 and seasonal flu) is ahead of current vaccine production

Program design – Multi-level intervention: national media messages, provider triage by risk, populations self-organize multiple responses

Evaluation design – What are changing patterns of twin epidemics? How are governments, providers, populations reacting and interacting in response to situation? Population health impacts?

32

System Dynamics Discussion

What are the situations’ boundaries, focus, interconnections, perspectives, power, timing, and dynamics?

What are the risks of not understanding the system attributes and dynamics?

What are the benefits of understanding the system attributes and dynamics?

33

System Dynamics of Child Abuse Prevention – Home

Visiting

34

U.S. Child Abuse and Neglect Trends

35

Context, Program Design of Child Abuse Prevention

Context: Many programs exist but child abuse and neglect rates are increasing

Program design: AFC funding for 17 grants for the adaptation, implementation, spread, and sustainability of evidence-based home visiting programs through infrastructure development and system change

36

Evaluation Design of Evidence-based Home Visiting Initiative Program evaluation – tracking of cross-site

cost, implementation, fidelity, and child and family outcomes of 17 EBHV programs

System evaluation – tracking of cross-site and grantee-specific system infrastructure, theories of action, measures of system change, partner collaboration and network analysis; system unit of analysis

37

Open Space Technology: System Dynamics Exercise

What are the dynamics (i.e., the nature and balance of types of system dynamics) of the situation as a whole?

What are the system dynamics of the intervention?

What are the implications for the evaluation design and process?

38

Three Dynamics of a Social System and its Context

39

Match of Evaluation Designs to Dynamics

of Social Systems and Their Context

Exploratory Design

Predictive Design

Init

iati

ve

Re

ne

wa

l D

esig

n

Organic Design

40

Complex Adaptive Systems and Adaptive (Self-organizing)

Dynamics1. Self-organizing/adaptive/organic2. Sensitivity to initial conditions3. Emergence4. Macro pattern

41

Complex Adaptive Systems and Adaptive (Self-organizing)

Dynamics (cont.)5. Feedback6. Co-evolution7. Pattern formation and points of

influence

42

Implications for Evaluation and Action

1. Small differences can create large effects.

2. The past influences but does not predict the future.

3. Many points of influence exist.4. Boundaries, differences, and

relationships are levers of influence toward a purpose.

43

Implications for Evaluation and Action

5. Simple rules underlie patterns.6. Pattern-based feedback and

actions are iterative.7. Tensions are not resolved.8. Patterns are outcomes.

44

Four Stages of Evaluation

Design Evaluation

Shape Practice

Collect Data

Make Meaning from

Data

45

46

Example: LEAP

Learning through Engineering Design and Practice

47

Example: LEAP Research Design

Quasi-experimental design embedded in curriculum development process

Pre-post assessments of Content knowledge

Perceptions of engineers at work

Tinkering

Self-efficacy

Engineering notebooks

Career behaviors survey

48

External Evaluation Design

The external evaluation focused on: Confirmation of effectiveness Scale-up Sustainability

49

Conceptual Shifts

The fundamental conceptual shift in this project was from:

teacher-directed de-contextualized learning to student-engaged project-based learning

fixed skills and knowledge as learning outcomes to the desired outcomes being that students are actively engaged; develop the capacity to explore and figure things out; and act like an engineer.

50

Confirmation of Effectiveness

Knowledge and skills related to project topics and STEM concepts

Enjoyment and pride in project work Development of teamwork,

collaboration and workplace skills Interest in STEM courses and pursuit of

STEM career and educational pathways

51

Scale-Up

Scale-up involved two tracks: Greater use of the curriculum within

the school system Transfer of the curriculum to Boys

and Girls Clubs

52

Sustainability

Program sustainability Sustainability of learning of

participating students Sustainability of collaborations Sustainability of teaching capacity

53

Program Sustainability

Maintain relationships with the district Professional development for teachers Shape research related to the project Explore ways to continue project at 9th grade Track STEM course selection of project students

in high school Collaborate on additional community

dissemination and funding

54

Sustainability of Learning of Students

Classroom Extracurricular activities Career-related activities Focused attention through high school Continued involvement of university

faculty and students Continued contact with science center

55

Sustainability of Collaborations

Use current collaborations to spur others over time

56

Sustainability of Teaching Capacity

Build capacity through formal and informal professional development approaches Training for new teachers Coaching by master teachers Summer professional development

activities Technology enhanced training

57

Example 2

Communities of Learning, Inquiry, and Practice

(CLIPs)

(video at www.insites.org/clip)

58

Example 3

Strengthening Families

59

Strengthening Families Protective Factors

Parental resilience Social connections Concrete support in time of need Knowledge of parenting and child

development Social and emotional competence of

child

60

61

World Café Exercise

What data gathering and/or analysis tools have you found helpful in gaining a deeper understanding of complex systems or interventions?

What practices help you develop your capacity to recognize patterns?

Do certain practices seem more related to finding surface patterns and others more related to finding deep patterns?

62

Contact Information

Beverly Parsons, Ph.D. bparsons@insites.org (360) 638-1442

Meg Hargreaves, Ph.D. mhargreaves@mathematica-mpr.com (617) 301-8994