Assessing Leadership in Short-Duration Experiential Learning Programs:Pilot Results and Future Directions

Linda McGarvey, Ph.D.Anita Barrett, M.Ed.

Hillary Michaels, Ph.D.

Discussant: Alan Sheinker, Ed.D.

June 22, 2010

Experiential Leadership Education

1.Seminars2.Simulations3.Site visits4.Speakers

FrameworkLiteratureReview

ProgramContent

Purpose of Assessment

Ongoing, embedded performance assessment• Program evaluation• Instructional improvement

Goals1. Classroom-level assessment

• Web-based • Non-intrusive • Organic

2. Measurement of gains3. 360 view of students4. Timeline for development

• Pilot: Spring 2010• Field Test: Summer & Fall 2010• Operational: Spring 2011

Challenges

• Short duration• Fast pace• Time to administer/observe• Training

Rubric for Content UnderstandingLevel Name Content Support

1 No Understanding The student does not understand even the basic foundations of the content.

Requires a lot of support and scaffolding.

2 Novice The student can recall or remember information.

Requires a lot of support and scaffolding.

3 Basic The student can explain ideas and concepts. Requires support, scaffolding and probing.

4 Proficient The student can apply the information; use it in a new way.

Requires little support or scaffolding.

5 Advanced The student can analyze information, distinguishing different ideas, methods and concepts.

Requires no support or scaffolding.

6 Distinguished The student can justify decisions and positions and can hypothesize new processes, products, ideas or points of view.

Requires no support or scaffolding.

7 Leave Early Scholar left voluntarily or involuntarily from the conference.

8 No Show Scholar never arrived to the conference.9 No Evidence Insufficient evidence.

Scholar was not engaged during the workshop.Scholar has temporarily checked out (sick, excused absence).

Faculty Advisor Training 90-minute session focused on:• Framework overview• Rubric introduction • Calibration exercises

Used classroom video clips to calibrate

More time and materials are needed, particularly for our diverse and transient pool of instructors

Session concluded with a demonstration of the Student Assessment Application (SAA)

Psychometric Challenges

• Little time to train on content standards & rubric

• Need more video of each level for training and calibration

• Based on observations and products• Embed into short, experiential learning

programs

Debrief

• Intuitive to use• Liked using the Netbook over the iPad• Felt the information would be useful

for students and the program• Needed more training• Felt that the assessment would

challenge them and make them more aware

Data• Full assessment

– Early: 11 FAs, 92 students rated– Post: 5 FAs, 46 students rated

• 1 FA rated all students early and post at the same time• 1 FA rated 9 students in less than one minute

• Random assessment• 13 FAs, 119 students• 612 valid scores (792 opportunities -180 missing scores)

• Time– Full (n~8)

• Average time 15 min, Range 5 – 20 min • 1.5 min per student early and 1.8 min per student post

– Random (n=6) • Average time 2 min, Range .45 – 11

Next Steps

• Field test Summer and Fall 2010– Deleted random assessment– FAs will do full assessment every evening– Field test with more training through Fall

• Design reports for students and parents• Consider revisions to content and content

expectations• Operational Spring 2011

– 360 assessment: mentors, peers, self, FAs– Reports to parents

Alan SheinkerCTB/McGraw-Hill

“I think that the world is flat.” — Thomas Friedman

“Around here, however, we don’t look backwards for very long. We keep moving forward, opening up new doors and doing new things, because we’re curious… and curiosity keeps leading us down new paths.” — Walt Disney

It’s a Brave New World

• 70% of jobs today require specialized knowledge & skills

• Students will hold more than 10 jobs by the time they are 40 years old

• Many of the jobs they will hold don’t exist • So what skills allow for this flexibility

What are the Skills that Ensure Success

• Critical Thinking & Problem Solving

• Collaboration• Agility & Adaptability• Initiative &

Entrepreneurialism• Oral & Written

Communication• Accessing & Analyzing

Information• Curiosity & Imagination

(Wagner, 2008)

• Critical & Creative Thinking• Social & Cross Cultural

Capabilities • Self Direction• Communication• Self-Awareness and Self-

Efficacy(Envision, 2009)

Comments

• Web-based tool• Formative assessment to improve instruction &

curriculum • What are the implications of the real-life

simulations for the general education world?• How do you know what students bring into the

program?• What could follow-up look like?

• Learning communities• Self-Evaluation tool

Comments• Increase training with instructors to increase

score reliability• Video taping can be intrusive, but it is a good

way to validate the use of scoring guides/rubrics

• Random checks through observations & videos might also be useful for monitoring raters

“Opportunity is missed by most people because it is dressed in overalls and looks like work.” — Thomas Alva Edison

Envision EMI (2010). Developing the skills for success: What we know about learning for the 21st century. http://www.envisionemi.com/about/education.php

Envision EMI (2010) Experiential leadership education: Building the foundation for a lifetime of success. http://www.envisionemi.com

Friedman, T. L. (2005). The world is flat: A brief history of the twenty-first century. New York: Farrar, Straus and Giroux.

Thomas A. Edison. (n.d.). BrainyQuote.com. Retrieved June 14, 2010, from BrainyQuote.com Web site: http://www.brainyquote.com/quotes/quotes/t/thomasaed104931.html

Wagner, T. (2008). The global achievement gap. New York: Basic Books.

Walt Disney. (n.d.). BrainyQuote.com. Retrieved June 14, 2010, from BrainyQuote.com Web site: http://www.brainyquote.com/quotes/quotes/w/waltdisney132637.html

References