Evaluation of Efforts to Broaden STEM Participation:

Results from A Two-Day Workshop

Planning Committee:

Bernice AndersonElmima Johnson

Beatriz Chu ClewellNorman Fortenberry

Presenters:

Patricia B. Campbell & Veronica Thomas  

Evaluation of Efforts to Broaden STEM Participation: Workshop

GoalsTo develop and validate a strategy by which to demonstrate the value of NSF's investment in broadening participation (BP).

To negotiate answers to two questions:1. What metrics should be used for project

monitoring? 2. What designs and indicators should be

used for program evaluation?

Evaluation of Efforts to Broaden STEM Participation: The Workshop

ReportThe Policy Context for NSF Programs for

Broadening Participation (Fortenbury)

Measuring Success and Effectiveness in NSF’s Broadening Participation Programs (Clewell)

Outcomes and Indicators Related to Broadening Participation (Campbell, Thomas, & Stoll)

Evaluating Efforts to Broaden Participation (Campbell, Stoll, & Thomas)

Implications of the NSF Broader Impacts Statement (Nelson & Bramwell)

The Policy Context: Historically

NSF’s goal of broadening participation has been shaped through a variety of policy actions by the legislative and executive branches of government.

Within the agency itself, policies articulated by the National Science Board (NSB) and the Committee on Equal Opportunity in Science and Engineering (CEOSE) have informed the NSF approach and strategy to address this goal, as referenced in major policy documents issued by NSF.

The Policy Context: Currently

Broadening Participation at the National Science Foundation: A Framework for Action (May 2008), outlines the NSF-wide broadening participation plan.

It provides guidelines for broadening participation both externally and internally, through:• Expanding the reviewer pool• Training NSF staff and reviewers• Enforcing accountability for NSF staff and

principal investigators• Communicating promising practices• Maintaining and monitoring a portfolio of

relevant programs

The Policy Context: A Core Value and A Strategic Goal

Broadly Inclusive: seeking and accommodating contributions from all sources while reaching out especially to groups that have been under-represented; serving scientists, engineers, educators, students, and the public across the nation; and exploring every opportunity for partnerships, both nationally and internationally. Investing in America's Future: Strategic Plan FY 2006-2011, National Science Foundation, NSF 06-48, National Science Foundation, Arlington, VA, 2006.

Measuring Success: NSF – BP Programs

Broadening Participation Focused Programs (28 Programs; 17 Require Evaluations) Programs with Emphasis on Broadening Participation (17 Programs; 8 Require Evaluation) Programs with Broadening Participation Potential (16 Programs; 9 Require Evaluation) Other Broadening Participation Efforts (5 Programs).

Measuring Success: Suggested Monitoring Metrics

Institution Focused Targeted Programs

Goal: Increase research capability and teaching effectiveness

Baseline data: Collaborative relationships, Funding distribution, % URM students, Total enrollment

Follow-up: Collaborative relationships established, Funding support obtained, Teaching reforms effected

Measuring Success: Recommendations

It is recommended that the NSF:•Conduct periodic evaluations, including external reviews ranging from the program level to larger cross-sections of the portfolio•Develop a common framework requiring that BP projects collect uniform data •Review all funded programs to determine:

If program funds serve a representative proportion of members of under-represented groups or institutions;

If positive outcomes of programs are distributed equitably among all groups of participants or institutions.

Broadening Participation (BP) : Critical Issues Related to Indicators and Outcomes

• Developing shared understanding and clarifying meaning

• Addressing ‘‘success” at multiple levels

Important Distinctions

• Inputs

• Outputs

• Process

• Outcomes

Inputs

Resources, contributions, investments that gointo the project

Input indicators measure resources,contributions and investments such as:

Staff Volunteers Funding Materials Facilities Investments made to support BP

Outputs

Units of services and goods provided by the project

Output indicators measure things such as the scope/size of activities, services, events, and products reachingunderrepresented

Numbers of students served Numbers of workshops

Process

Ways in which project services andgoods are provided

Process indicators measures extent towhich BP projects, programs, andstrategies delivered as intended(alignment)

Outcomes

Things project hopes to achieve; actualbenefits, impact, or changes

Outcomes indicators expressed in terms of changes forindividuals, groups, communities, institutions, and system :

Knowledge, attitude, and skill changes Behavior changes Value changes Policy, procedural, and practice changes

Considering BP Success at Multiple Levels

Level 1: Having access to the benefits of STEM knowledge

Level 2: Having access to STEM knowledge

Level 3: Studying STEM

Level 4: Working in STEM areas

Level 5: Generating STEM knowledge

Problems in Determining “Success”

Defining in terms of increase in absolute number or

percentage

Defining in terms of increase in both number and percentage

Defining in terms of the end point being “parity” (absolute number)

Other Considerations in Defining Success

Defining “parity” as a range

Achieving parity, as more participate overall

Considering discipline/field size to which definition of success apply

Integrating qualitative indicators (e.g., broadening and transforming perspectives)

Other Indicators of Success Broadening Participation

Individual level indicators

Institutional level indicators

Foundation level indicators

Individual (Student) Level Indicators

Participation

Retention, persistence, and success

Experiences

Attitudes

Institutional Level Indicators

• Staffing

• Policies, programs, and institutional commitment

• Accountability and rewards

• Monitoring, tracking, and using data for improvement

• Collaborations

Foundation Level Indicators

• Inclusion of information about importance of BP

• Review and monitoring of foundation policies/practices in terms of potential to broaden participation

• Diversity of professional involved with the Foundation

• Foundation resources devoted to BP

• Improvements to knowledge base about broadening participation

• Implementation of effective strategies at Foundation level to BP

Evaluating BP: Research vs. Evaluation

GoalResearch: To move the knowledge base forward.Evaluation: To assess quality/effectiveness.

OutcomeResearch: Why something does or doesn’t.Evaluation: If something does or doesn’t work.

FocusResearch: The research.Evaluation: The program/intervention.

Designs/measures/analysis: No difference

Evaluating BP: Longitudinal Tracking

Being able to follow students longitudinally is the key to any sophisticated understanding of how colleges are doing and what's happening to students. - Thomas R. Bailey, 2008 Without longitudinal data, the generation and testing of causal models tied to successful participation in STEM for diverse populations will be difficult if not impossible.

Evaluating BP: Using Comparison Groups

Evaluating BP: Selecting Designs

The appropriateness of the fit between the design of the program or “intervention” and the requirements of more rigorous evaluation methodologies.

The timing of the evaluation.

The balance between the level of investment in the evaluation and the level of investment in and the intensity of the intervention.

The level of evidence expected given the nature of the intervention.

The strength of rival hypotheses.

Evaluating BP: Selecting Designs

Study Type DesignRepresentati

on

Typical questions

answered by the

design

Quantitativ

e Case

Study

One-shot

Post-test

only

Design

X O

After attending a

preview weekend are at

least 50% of the

students planning to

apply to the institution?

Quasi-

experiment

al Study

One-shot

Pre-test-

Post-test

Design

Oa X Ob

Does working with a role

model increase girls’

interest in science

careers?