Implementing Outcomes Assessment: An Approach Based on Competencies Steve Mickelson Agricultural and Biosystems Engineering

Implementing Implementing Outcomes Assessment:Outcomes Assessment:

An ApproachAn ApproachBased on CompetenciesBased on Competencies

Steve MickelsonAgricultural and Biosystems

Engineering

22

Agricultural & Biosystems Agricultural & Biosystems Engineering (ABE)Engineering (ABE) Two undergraduate majors

- Agricultural Engineering (~150 students)- Agricultural Systems Technology (~150

students)- Industrial Technology (~220 students)

Two graduate majors (~90) In two colleges: Engineering and Agriculture 33 faculty

- 22 involved in undergraduate teaching- Faculty are academic advisors- No teaching assistants

Agricultural & Biosystems Engineering

33

Presentation Road MapPresentation Road Map Our path to competencies Our outcomes assessment plan Implications and implementation Now what?


44

The Path to CompetenciesThe Path to Competencies


55

The Buzz: Outcomes Assessment!The Buzz: Outcomes Assessment!

ISU university-wide accreditation in 2005- Evidence of active outcomes

assessment programs is required. ISU College of Engineering

- ABET in 2000 & 2006 ABE has been thinking and working a lot on this with mostly good faculty buy-in


66

Our Outcomes via ABETOur Outcomes via ABETa) an ability to apply knowledge of mathematics, science, engineering

and technologyb) an ability to design and conduct experiments, as well as to analyze

and interpret datac) an ability to design a system, component, or process to meet desired

needsd) an ability to function on multi-disciplinary teamse) an ability to identify, formulate, and solve engineering problemsf) an understanding of professional and ethical responsibilityg) an ability to communicate effectivelyh) the broad education necessary to understand the impact of

engineering solutions in a global and societal contexti) a recognition of the need for, and an ability to engage in life-long

learningj) a knowledge of contemporary issuesk) an ability to use the techniques, skills, and modern engineering tools

necessary for engineering and technology practiceWe don’t know how to measure an We don’t know how to measure an ability.ability.


*

**

*

* Alverno “ABET” Outcomes

77

OutcomesOutcomesAbilitiesAbilitiesCompetenCompetenciescies

A complex ability cannot be observed directly; it must be inferred from performance.

Abilities are complex combinations of competencies.

Competencies are the application of behavior and motivation to knowledge, understanding and skill.

Key actions that demonstrate workplace competencies can be measured.


88

Identifying Workplace Identifying Workplace CompetenciesCompetencies Cooperative effort with Development

Dimensions International, Inc. (DDI) Stakeholder dialogue, faculty analysis

- “Critical incident” stories of workplace success

- Stories identified and validated 14 competencies mapped back to outcomes

Each competency has Key Actions that are definable and measurable


99

ISU Engineering CompetenciesISU Engineering CompetenciesEngineering Knowledge

Innovation Teamwork

General Knowledge

Cultural Adaptability

Integrity

Continuous Learning

Analysis & Judgment

Professional Impact

Quality Orientation

Planning Customer

Focus

Initiative Communication Leadership and

Safety?


1010

ABET Outcomes – ISU Competency ABET Outcomes – ISU Competency MatrixMatrix

1111

Competencies Have MeaningCompetencies Have Meaning

“Transparent” to all participants A great match with:

- employers’ goals - student and faculty understanding

Provides a map and tools for students to achieve their goals in a meaningful way


1212

Our Outcomes Assessment PlanOur Outcomes Assessment Plan


1313

How Should We Assess How Should We Assess Outcomes?Outcomes? Actual student work is the primary

evidence. Exit interviews, post-graduate surveys,

etc. are opinions and are only supporting evidence

ABE will use:- ePortfolios- Internship evaluations- Employer evaluations of alumni- FE exam


1414

What are ePortfolios?What are ePortfolios? Collection of artifacts for certain

purposes displayed on the web Our purpose: developing and

demonstrating student competencies It’s all about learning!

- Reflection- Demonstration


1515

Why ePortfolios? (Students)Why ePortfolios? (Students) Student learning

- Building an ePortfolio requires reflectionReflection → Learning!

- Faculty can have access to student work for comment, review

Adds depth to resume for career search success- Highlight skills in ways not

previously possible- Potential employers are interested


1616

Why ePortfolios? (Faculty)Why ePortfolios? (Faculty) Outcomes assessment A tool to better manage, review,

reflect, and comment on student work (class & advising)

Faculty learning- Building an ePortfolio requires

reflectionReflection → Learning!

- Adding access to student work to document teaching excellence


1717

On-line Competency AssessmentOn-line Competency Assessment

OPAL by DDI, Inc.- On-line Performance and Learning- Web-based assessment and learning

tool built on competencies Multiple uses for us

- Internship evaluations by employers- Alumni evaluations by employers- Classroom (self and 360º)


1818

Mission, Goals

& Objectives

Stakeholders

Ind Advisory Board

Faculty

ABET / ASAE

Outcomes &

Competencies

Classes mapped to outcomes & competencies

Key Assignments

Co-op & Internships

Extra-curricularactivities

ePortfolios

ABE Outcomes Assessment Map

Annually

6 years

INDIRECT MEASURES

DIRECT MEASURES

Curriculum

Committee

Recommendations

Ind Advisory Board

• Senior exit surveys

• SEI

• 5 year post grad survey

• CSREES review

• Advisor evaluations

• Placement statistics

• Co-op & internship evaluations

• ePortfolios

• Employer evaluation of 5- year grads

• FE exam

Faculty(Individuals

and as a Whole)

Strengths &

Weaknesses

1919

TTYP – Turn to Your PartnerTTYP – Turn to Your Partner What are the barriers to

implementing an outcomes assessment plan?


2020

Our BarriersOur Barriers Faculty and administrative buy-in Financial resources Expertise Clear vision Time

- Faculty development- Competing demands

Rewards Champion(s)


2121

Implications and Implications and ImplementationImplementation


2222

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

EngineeringWorkplace

Internship Classroom Laboratory Capstonedesign

Professionalactivities

Non-professional

activities

Stakeholders’ View of the Probability of Stakeholders’ View of the Probability of Developing and Demonstrating the Developing and Demonstrating the

Communication CompetencyCommunication Competency

50%

2323

Competency-based Learning Competency-based Learning (CBL)(CBL) Learning outcomes in terms of

competencies- curricula focus on competency

development “Transparent” to all participants Great match with employers’ goals and

student understanding Provides a map and tools for students to

achieve their goals in a meaningful way


2424

Faculty must think, teach and assess in terms of competencies.

Our curriculum must evolve.- It can no longer be a series

of requirements (or hurdles).

- Opportunity to integrate previously disparate parts of the undergraduate experience.

CBL demands changeCBL demands change


2525

A CBL ModelA CBL Model

Voorhees et al, U.S. Dept. of Education, 2001


Demonstrations

Competencies

Skills, Abilities and Knowledge

Traits and Characteristics

Assessment ofPerformance

Acquired Skills,Abilities, andKnowledge

Developed in theLearning Process

Foundation

ASSESSM

ENTIntegrative Learning Experiences

Learning Experiences

2626

Implementing CBLImplementing CBL Avoid “extra” work for faculty Classes already mapped to outcomes

- Map competencies to classes Identify key assignments for student

portfolios Help faculty learn the “language” of

competencies Provide common rubrics


2727

Implementing CBL with Implementing CBL with TechnologyTechnology ePortfolios to develop and

demonstrate competencies On-line case simulations to simulate

the workplace On-line assessment for internships,

alumni and in the classroom


2828

ePortfolio ImplementationePortfolio Implementation

Freshman Learning Community- introduce competencies - start collecting and reflecting on artifacts

Sophomore and Juniors seminars- Start ePortfolios- Feedback from instructor and advisor

Senior seminar- Final ePortfolio required for graduation- Assessed by faculty and professionals


2929

ABE ePortfolio SystemABE ePortfolio System Database for holding multi-modal

artifacts, reflections, and assessments

Student-designed, student-owned interface to display artifacts


3030

On-line Case SimulationsOn-line Case Simulations Case Studies

- Promotes competencies and reflection English 314 (Technical Writing)

- Communication - the weakest competency


3131

One Instance: Omega MolecularOne Instance: Omega Molecular Students are employees of consulting

firm Client: Omega Molecular

- Engagement with virtual biotech organization

- Analyzing and producing recommendations and communications

Feeding the world, better.


3232

On-line Case SimulationOn-line Case Simulation


3333

Now What?Now What? Our status today:

- Students are collecting artifacts.- Pilot group is creating ePortfolios.- Faculty are designating key

assignments and learning about CBL.- We’re assessing internships & alumni.

Just Completed:- Implement seminar classes (S’06)- Do a complete cycle (S’06)- Conducted ABET visit (F’06)


3434


Documents

Implementing Outcomes Assessment: An Approach Based on Competencies Steve Mickelson Agricultural and Biosystems Engineering