Learning Good Employee Skills: Maximizing Internship Program
Effectiveness
NSEE Annual Conference, September 2016
Abby Trout, Career CenterCarol Trosset, Institutional Research and Assessment
Carleton College
1Institutional Research and Assessment
Overview
Carleton College = a highly selective liberal arts college in Minnesota, with 2000 undergraduates.
• Introduce the Carleton College Career Center’s internship process and its focus on learning goals
• Review Trosset’s research at Bennington College on which skills are most valued by interns’ employers
• Present new research at Carleton on how students become aware of and learn these skills
• Discuss implications for Career Center programming
Institutional Research and Assessment 2
Institutional Research and Assessment 3
Career Center mission:
Institutional Research and Assessment 4
Carleton Career Center learning goals:• Self-assessment • Career field awareness• Transferable skills • Market selves • Experience• Job search• Graduate degrees• Access to networks• Effective networking
Institutional Research and Assessment 5
Career Tracks
Institutional Research and Assessment 6
Internships at Carleton
• 488 (33%) of rising Carleton sophomores, juniors, and seniors did internships in Summer 2016
• 113 of these internships were funded through the Career Center
• Career Center provides funding to cover expenses (i.e., housing, transportation, food)
• $386K awarded for Summer 2016• Internships located all over the world (all over US and 29
countries) and in a variety of industries
Institutional Research and Assessment 7
What makes for a rewarding internship?
• A strong relationship with a supervisor• Learning• Goals• Self-discovery
At Carleton, students specify learning goals:
Institutional Research and Assessment
Research on Learning Outcomes
Institutional Research and Assessment 9
• What are the actual learning outcomes?• What are the most important/beneficial outcomes,
that internships should foster?• How can we assess (a) the degree to which students
achieve these outcomes, and (b) the quality of their performance as interns?
• Can we identify how beneficial learning took place, so we can design experiences that maximize these effects for other students?
Bennington College Research: Assessing Internship Performance
Liberal arts college in southwest Vermont700 undergraduates7-week internship required every winterAll students complete 4 internshipsResearch in 2013-14 and 2014-15Holly McCormack, Dean of Field Work Term
Institutional Research and Assessment 10
What are internships for?
•Students want to explore possible career paths, and to start building a resume that will lead to jobs after college
• Chronicle of Higher Education and NPR’s Marketplace 2012 study “The Role of Higher Education in Career Development: Employer Perceptions” found that
–Employers place more weight on experience, particularly internships and employment during school vs. academic credentials including GPA and college major when evaluating a recent graduate for employment.
–Among all industry segments, an internship is the single most important credential for recent college graduates to have on their resume.
–60% of employers practice intern-to-permanent hiring.
Institutional Research and Assessment 11
Liberal Arts Learning vs. Job Training
•AACU 2013 study “It Takes More than a Major: Employer Priorities for College Learning and Student Success” found that employers endorsed a blended model of liberal and applied learning—practices that involve students in active, effortful work.
• Bennington Field Work Term’s goal: Students will apply their academic learning in a broader context outside the classroom.
• Same AACU study asked employers whether they want employees to have the skills liberal arts colleges already value, like writing and critical thinking. Employers said “yes.”
Institutional Research and Assessment 12
But what do interns need to learn?
Supervisors of Bennington students wrote open-ended evaluations, listing interns’ strengths and areas needing improvement.What did they write about without being prompted?• “She was always on time and willing to step in and help with anything.”• “He asked great questions and responded to feedback gracefully.”• “She could pay more attention to her body language during meetings and
while working. At times she seemed a bit distracted and disinterested.” • “He tended to drift off task to check Facebook.”
Institutional Research and Assessment 13
Categories employers mentioned:
Institutional Research and Assessment 14
• Work ethic• Engagement with work• Quality of work• Organization/efficiency• Punctuality • Professional conduct• Takes direction• Teamwork
• Interpersonal skills• Independence/initiative• Learns quickly• Confidence• Creativity• Writing skills• Critical thinking skills• Other job skills
Note: Most of these are neither liberal arts skills nor job-specific skills. Instead, they are qualities of any good employee in any job sector.
Qualities of a good student?(taken from Bennington Faculty narrative evaluations)
• Work ethic – “works hard,” “diligent,” “spends time texting during class”• Engagement – “excellent work when focused but not consistently engaged”• Quality – “does high quality work,” “should focus on producing high-quality work
the first time around”• Organization – “sometimes unprepared,” “talented but disorganized”• Punctuality – “late to class several times,” “some absences”• Takes direction – “responded to feedback,” “needs to learn to follow directions”• Teamwork – “listens and contributes well in discussion,” “good team member”• Initiative – “challenges himself,” “does the minimum to get by”• Learns quickly – “learns from mistakes,” “learns quickly”• Confidence – “should have more confidence,” “wish would speak up more”
Institutional Research and Assessment 15
Evaluating internship performance
Institutional Research and Assessment
Unacceptable Needs Improvement
Good Exceptional
Focused work ethic
Too laid-back, easily distracted, takes frequent breaks
Mostly works hard, but some
tendency to lose focus and drift off
task
Productive and conscientious
Works harder than others and is much
more productive
Engagement with the work
Seems unmotivated, avoids or resists unwanted tasks
Engagement and motivation vary
with the task
Engaged in the work, willing to help as needed
Seizes every opportunity to
learn as much as possible
Organization and Efficiency
Inefficient, forgets about assignments,
may not follow through to completion
Still learning how to manage time
and priorities when working
alone
Good time management, follows up on assigned tasks
Unusually efficient, intuitive sense of priorities
Takes direction Fails to incorporate feedback into work
products; rarely checks in with
supervisor
Some tendency to put own agenda ahead of others’
needs
Checks in regularly, responds
appropriately to feedback
Frequently seeks out feedback and uses it to improve
the work
16
Benefits of the rubric:• All employers are asked to comment on a consistent group of skills • Students have a consistent definition of what the College means by
workplace readiness• Students can:
–compare their self-rating using the same rubric to that of their employers
–track their rubric ratings over four years–use employer evaluations as part of a job application packet
• Employers have asked us for copies of the rubric to use with their staff
Institutional Research and Assessment 17
Challenges for using rubrics with employers:
•Colleges need to compare the students in aggregate, and over time.
•But students all have different jobs and employers.•The students start at different stages of development.•The employers have different expectations.•We cannot norm the employers to consistent standards.•The evaluation process cannot be blind or unbiased.
Institutional Research and Assessment 18
Carleton College Research:Employee Skills Metacognition
Focus was on tracking and increasing student awareness of these skills.
90 of the funded interns in Summer 2016 allowed us to use their materials for this research.
Materials included:Application essays Blog postsLearning contracts Reflective essays
Rachel Leatham, Program Director
Institutional Research and Assessment 19
20
Student Motives and Preparation
Motives
Support the mission – 44%Interest in activity – 39%Personal connection – 19%Learn skills – 17%Explore a career – 13%Apply academics – 9%Broaden my horizons – 6%
Preparation
Coursework – 52%Previous “job” – 39%Specific skills – 33%Studying the organization – 17%Off-campus study – 10%Local knowledge – 10%Extracurricular activity – 9%
Institutional Research and Assessment
Students’ Learning Goals• Explore a possible career – 73%• Gain academic learning – 47%• Learn job-specific skills – 41%• Improve research skills – 22%• Improve communication skills – 13%• Improve time management skills – 13%• Improve foreign language skills – 12%• Gain cross-cultural competency – 11%• Improve interpersonal interaction skills – 11%• Gain life skills – 7%
32% had only job-specific or academic goals.
Institutional Research and Assessment 21
Learning Outcomes from Final Reflections
Institutional Research and Assessment 22
Goal Outcome Goal Outcome
Work ethic 0 3 Takes direction 1 12
Engagement 0 6 Teamwork 8 9
Quality of work 0 9 Interpersonal skills 1 7
Time management 12 10 Initiative 0 4
Flexibility 1 6 Confidence 0 6
Professionalism 3 3 Communication skills 12 13
What They Learned about Employee Skills
Work Ethic• The importance of persistence• How much work is required to develop a good product• How to time breaks to be more productive
Engagement• That feeling engaged with one’s work improves its quality• That personal growth happens when you commit yourself to
something
Institutional Research and Assessment 23
What They Learned, cont.
Quality of Work• The importance of doing high-quality work• Attention to detail• The value of being well-prepared
Time Management• Improved time management skills• Became better organized• The importance of managing one’s time• How to set priorities
Institutional Research and Assessment 24
What They Learned, cont.
Flexibility• The importance of flexibility• The value of patience• That being flexible can improve the quality of the work
Taking Direction• The importance of asking questions• The importance of taking notes• How to accept critique
Institutional Research and Assessment 25
What They Learned, cont.
Teamwork• The importance of teamwork• The importance of learning from co-workers• Teamwork and facilitation skills• Project management and leadership experience
Interpersonal Skills• The value of interacting and building relationships with
co-workers
Institutional Research and Assessment 26
Those with a goal more likely to learn;Most who learned did not have that goal.
Institutional Research and Assessment 27
0%
10%
20%
30%
40%
50%
60%
Learned target skill Learned different skill No employee skills reported
With Goal
No Goal
What leads to the learning when it’s not motivated by a goal?
• Specific on-the-job events• The difference between a job and a class• Directed reflection
Institutional Research and Assessment 28
Specific events• “One of the biggest lessons I learned early in my internship was to not be afraid to
ask for help, or be inexperienced in any particular skill. The first project I was assigned involved using Excel. Initially I was frustrated and embarrassed that it took me longer than expected to learn how to use that program and that I didn’t know how to use it before. However, as I reflected with my supervisor about this anecdote at the end of my internship, she was surprised that I didn’t ask her for help rather than dealing with it on my own. At the time I didn’t know that was an option, I believed the expectation was to already be equipped to complete the projects assigned.”
• “Staying organized was something that I struggled with sometimes. My supervisor would send me an e-mail of something that would be coming up in two weeks and because it was two weeks away I’d say, ‘Well, I’ll read it next week when I have more time.’ I would then lose my e-mail in my constantly overflowing inbox and I’d have to ask my supervisor to resend it. Early on, at meetings I wouldn’t take notes, thinking my memory would be good enough, but when it became clear that the little nuances and wordings mattered just as much as the main idea we were trying to convey I realized that I had to start taking some notes or become lost.”
Institutional Research and Assessment 29
Events, cont.
• “My supervisor was an extremely busy woman – as a result she usually sent tasks to me and my two fellow interns and left it up to us to work together or divvy up the work as we thought best. This was an important exercise in teamwork: I learned the importance of giving credit where credit was due was essential for a successful team. Even small things like saying ‘Ashley and I edited this report’ rather than ‘here’s the report you wanted edited’ goes a long way in creating a healthy group dynamic.”
• “I’ve learned to keep my collected data clean. Although this project had been going on for three years now, there was no standardized manner of storing or analyzing the data. Thus, the last two weeks of my internship were largely spent going through thousands of haphazardly sorted photos, essentially doing other interns’ work for a second time. This was incredibly frustrating. Acknowledging that I have been lax in storing my data neatly during past projects, I learned the hard way that it pays to stay organized from the beginning!”
Institutional Research and Assessment 30
Difference between a job and a class
• “Through this project I developed my willingness to pay attention to detail as I encountered many specific comments of constructive criticism, a type of conversation I rarely have in an academic setting as little of my work ever reaches beyond my professor grading a specific assignment. This experience helped me understand the kind of work put into public education initiatives and the importance of detail-oriented review.”
• “Usually I was very engaged in my work because I had to be extremely attentive when processing information and working on my projects. In the field that I am interning for, mistakes must be recognized and fixed before it is sent in.”
• “Here, giving up means letting over 40 children down. Giving up means sending the message to those children that they aren’t worth it. Giving up meant impacting someone else’s life just as much as my own. In school, giving up in a class, for example, only really affects me. I guess in other words I learned a greater sense of responsibility, because I have wanted to just quit so many times.”
Institutional Research and Assessment 31
Directed Reflection via Weekly Blog Prompts
• Observe others’ work ethic• How you stay engaged with the work• What you’ve learned by making a mistake• Standards of punctuality• Observed nature of professional conduct• Difference between a supervisor and a professor• How people work in teams• Relevance of interpersonal skills• Working independently and taking initiative• Accomplishments and skill building• Observe effective communication
Institutional Research and Assessment 32
Among those who reflected on learning non-goal employee skills, 3/4 wrote about employee skills in response to summer blog prompts.
• “The skill that I’m most proud of is my growing sense of flexibility. I tend to be a rather stubborn person, and I don’t like to change things or let on when I am lost. This internship, however, has really challenged me to have to change this. Every time I think that I am finally done with creating my stimuli, my supervisor thinks of something I can change. My original plans and expectations for my experiments have certainly changed a lot over the course of the past five weeks, and I think that’s absolutely awesome. Though it’s certainly frustrating to feel as if I’m not making progress at times, it’s really cool seeing just how much of a process experiment design is.”
• “Even doing boring tasks (sharpening pencils, printing tickets, folding programs), I am surrounded by friendly and engaging people. Many of my bosses are on teams that put on shows here, so seeing their shows serves as motivation and a reminder that grunt work is almost always necessary.”
Institutional Research and Assessment 33
Supervisor Evaluations
Institutional Research and Assessment 34
Positive Improvement Needs Work
Work ethic 22 2 0
Engagement 10 0 1
Quality work 9 2 1
Time management 6 2 1
Flexibility 5 3 0
Professionalism 3 0 1
Take direction 8 2 2
Teamwork 6 7 2
Interpersonal 15 3 4
Initiative 11 5 2
Confidence 2 12 4
Communication 5 10 0
Differential Salience
• Students start out knowing the importance of time management and communication skills.
• They become aware of the importance of asking questions, attention to detail, and relationships with co-workers.
• They remain unaware of most dimensions of professionalism (other than dress code).
Institutional Research and Assessment 35
Next Steps•Conduct focus groups with some of the 2016 interns about
what facilitated this learning, and how to get other students to see its importance.
• Summer Interns 2017–Require one of the three learning goals to be a generalizable skill.– Continue to adjust blog prompts to include features of employee skills we know students are encountering, and that they should think about.
– Find ways to expand their awareness of the less salient skills (like professionalism).
–Use metrics to show incentive to other interns to register with our office
Institutional Research and Assessment 36
Next Steps
•Broader application of specific findings and the usage of structured reflection pre-, during, and post-experience to other preparation programs
–Post-grad fellows –On-campus jobs and interns–Externships
•Structure alumni-student programming–Build alumni profile questions around employee skills prompts–30 Minutes program questions
Institutional Research and Assessment 37
Questions?