PROGRAM ASSESSMENT: DESIGNING AND DOCUMENTING LEARNING

Welcome!

March 29, 2010Forum Session 5-6 p.m.League for Innovation in the Community College ConferencePresenter: Robin Nickel, Ph.D.

AGENDA

Explore a process and tool for designing program assessment

Examine sample programs Identify criteria for program

assessment Connect program assessment

to performance-based curriculum

QUOTE OF THE DAY

“As long as you’re going to be thinking anyway, think big.

--Donald Trump quoted in Sky Magazine March 2010

Informed judgments about achievement of intended learning outcomes

Based on evidence

Assessments are valid, reliable, and fair

Assessment is built into the plan for teaching

Data-driven evaluation that is actively used for continual improvement of teaching and learning

Assessment of Learning

WHY DO WE HAVE TO DO THIS?

What are the drivers?

DRIVERS

Accreditation (NCA, AQIP, industry)

Increased accountability to improve teaching and learning

DRIVERS

Carl D. Perkins IV legislation and funding

Objectively measure student attainment of industry recognized skills upon graduation

ASSESSMENT IN COMMUNITY AND TECHNICAL COLLEGES

Focus on industry aligned skill setsStudents know they are acquiring

skills that have value and portability

Increases odds that students will be successful in their employment

Ensures that your college is educating students to meet the needs of employers

QUESTIONS ABOUT OUTCOMES

Who cares about outcomes?Which outcome is which?How are outcomes related?What is the desired outcome of

outcomes?What does this mean for students

and instructors?

QUESTIONS ABOUT OUTCOMES Who cares about outcomes? Which outcome is which? How are outcomes related?

What is the desired outcome of outcomes?

What does this mean for students and teachers

Stakeholders Types of outcomes Relationship of institutional,

program, and course level outcomes

Learning, assessment, documentation, etc.

Impact on course design, learning and assessment

WHO ARE OUR STAKEHOLDERS?

Accrediting agencies and funding sources…

 “Assessment of student learning is a participatory, iterative process that:

Provides data/information you need on your students’ learning

Engages you and others in analyzing and using this data/information to confirm and improve teaching and learning

Produces evidence that students are learning the outcomes you intended

Guides you in making educational and institutional improvements

Evaluates whether changes made improve/impact student learning, and documents the learning and your efforts.”

NCA Higher Learning Commission

From “Student Learning, Assessment and Accreditation: Criteria and Contexts”, presented at Making a Difference in Student Learning: Assessment as a Core Strategy, a workshop from the Higher Learning Commission, July 26-28, 2006. 

Assessment of Learning

 

HLC/AQIP asks 5 fundamental questions: How are your stated learning outcomes appropriate to

your mission, programs, students and degrees? How do you ensure shared responsibility for student

learning & assessment of student learning? What evidence do you have that students achieve your

stated learning outcomes? In what ways do you analyze and use evidence of student

learning? How do you evaluate and improve the effectiveness of

your efforts to assess and improve student learning?

From “Student Learning, Assessment and Accreditation: Criteria and Contexts”, presented at Making a Difference in Student Learning: Assessment as a Core Strategy, a workshop from the Higher Learning Commission, July 26-28, 2006. 

Assessment of Learning

Curriculum and SACS

Core Requirement 2.5 Review of Programs Systematic Results in continuing improvement Demonstrates you accomplish mission

Curriculum and SACS

Core Requirements2.7.3 General Education component2.12 Quality Enhancement Plan relates to student learning

Curriculum and SACS

Comprehensive Standards 3.3.1 “institution identifies expected outcomes … assesses whether it achieves the outcomes; and provides evidence of improvement based on analysis of those results.”

3.4.1 “establish and evaluate program and learning outcomes

Curriculum and SACS

Comprehensive Standards 3.5.11 “general education core… provide evidence that graduates have attained these competencies”

Curriculum and SACS

Federal Requirements 4.2 “The institution maintains a curriculum that is directly related and appropriate to its purpose and goals….”

BENEFITS OF PROGRAM ASSESSMENT

• Feedback about skills needed on the job

• Added value for educationStudents

• Improvement of teaching and learning

• Supports overall missionColleges

• Assurance that your graduates have attained technical skills needed on the jobEmployers

WHAT DOES THAT MEAN FOR ME?

Indirect Measures Data based on

inferences about why achievement is high or low: Enrollment Retention rates Course completion Student/graduate/

employer satisfaction surveys

Placement statistics

Direct Measures Data that provides

evidence that students have achieved the outcomes: Performance

assessments with rubrics Portfolios Artifacts Performances Outcome referenced

tests

Increased Accountability for Direct Measurement of Learning

Sound Assessment

Valid Outcomes based on standards

(industry) Measures intended outcomes Measures application and critical

thinking

Reliable Performance assessment based on

consistent rubrics, scoring guides, and rating scales

Fair Valid Reliable Learners informed of expectations

up front! Feedback to learners

Program Level or End of Course

ProgramQuality

Student Credentials

Course LevelFeedback to

studentImprovement of

Learning/Teaching

Summative/Formative Assessment

Continuous

Im

pro

vem

ent

Summative Assessment

Based on Program

Outcomes

PROGRAM OUTCOMES Measurable, observable, and field-specific

skills – major outcomesNumber 5-7 per program (guideline not

rule) Originate from:

Current DACUMsAccrediting AgenciesNational (or other) Skill StandardsAdvisory Committees

Threaded through courses Performance verified with summative

assessment of skill performance

RADIOGRAPHY PROGRAM OUTCOMES

A. Carry out the production and evaluation of radiographic images

B. Adhere to quality management processes in radiography

C. Apply computer skills in the radiographic clinical setting

D. Practice radiation safety principlesE. Provide quality patient careF. Model professional and ethical behavior

consistent with the A.R.R.T. Code of EthicsG. Apply critical thinking and problem solving

skills in the practice of diagnostic radiography

Your own Assessment External Assessment Indirect Assessments

OPTIONS

Balance of College Assessments and External Assessments

College A

ssessment

External A

ssessment

Indirect

2012-2013

New Accountability for Direct Measurement of Learning

INDIRECT ASSESSMENT STANDARDS

EXTERNAL ASSESSMENT STANDARDS

COLLEGE ASSESSMENT STANDARDS

is valid and reliableis approved by industry (advisory

committee or standards)meets quality guidelines for 3rd party

assessment if no rubric is used such as…

Program Assessment

See Handout

2. EXTERNAL ASSESSMENT

Third party exam (i.e. NCLEX, Barb Cosmetology)

Required for job placement or minimal/no cost to student

Valid reliable Linked to technical skills required on the job

CONSIDERATIONS FOR SELECTING 3RD PARTY EXTERNAL ASSESSMENTS

Can we obtain the results/data? Are cost and administration

feasible? Does it add value or control entry to

the student/occupation? Is it valid, reliable, reputable,

recognized? Is it summative, cumulative or a

partial measurement of skills?

Concerns: Only measures part of the program outcomes-

“All Code-No Application”

Does not indicate job performance

Does it help employers?

Focused only on technical skills, what about academic skills and soft skills?

INDIRECT ASSESSMENT

GPA Course Completion Teacher developed exams that do not meet

the larger state or system standards

Proven processProgram and Course

Software ToolExpertise

What is WIDS?What is WIDS?Curriculum DesignCurriculum Design

For any discipline or delivery mode

Institutionally-defined

Instructor-defined

Exit Learning OutcomesProgram Outcomes Core Abilities Gen Ed Outcomes

WHAT CAN YOU CREATE WITH WIDS?

Program Documentation Outcome assessment charts Program profiles

Official Course Documentation Syllabi Assessments (rubrics and

checklists) Learning Plans/Teaching Plans Reporting Matrices

OFFICIAL PROGRAM DOCUMENTATION

Program Profile

Assessment Rubric

Improvement Plan

See Handout

Then… let’s see how they are generated!

See Handout

Then… let’s see how they are generated!

QUESTIONS?

Contact:Presenter: Robin Nickel, Ph.D.(608) 849-2411nickelr@wids.org