A learning intervention for work engagement using mixed methods: In-person classroom

and synchronous online learning

Annual State MeetingHattiesburg, MS

June 4, 2014

John Kmiec, Ph.D. Certified ROI Professionalwww.linkedin.com/in/jkmiec/

Consulting EmployeeTraining and Simulation Services and Solutions

Science Applications International Corporationjohn.j.kmiec.jr@saic.com / 228.365.2559 / www.saic.com

Overview• Background• Opportunities• Needs• Program Objectives• Intervention• Data Collection Plan• Data/ROI Analysis Plan• Results• Lessons• Resources

Background• Bulk manufacturer of lightweight food and medical grade plastics• Complex, massive, hazardous operational environment• Highly skilled, extensively trained, well paid, disciplined workforce• Rigorous hygienic, chemical and environmental safety standards• Very expensive raw materials frequently in short supply• Product waste and rework impeding business performance

(Source: Phillips & Phillips, 2012)

Opportunities• Product Waste and Rework: $272,850/month ($3.27M/year)

– Every 1% of total product produced as waste costs $245K– Every 1% of total product produced as rework costs $35K

• Stressed employees under the gun to cut costs and improve quality • Mediocre employee satisfaction and work engagement levels noted• Most immediate managers lacked supervisory training/experience

(Source: Phillips & Phillips, 2012)

NeedsProgram Needs• Product Waste ($245K/1%) and Rework ($35K/1%) costs

$272,850/month ($3.27M/year)• Stressed work environment barrier to Work Engagement• Immediate managers unskilled and inexperiencedEvaluation Needs• Effectiveness of learning intervention?• Increase Work Engagement?• Reduce Product Waste and Rework?• Rollout Decision?

(Source: Phillips & Phillips, 2012)

Program ObjectivesReaction (1)• Relevance, importance and intent to use on the jobLearning (2)• Successful application of 5 self-coaching skills• Can create and sustain motivational environmentOn-the-Job Application (3)• Effectively, continuously apply 5 self-coaching skills• Create and sustain motivational work environmentBusiness Impact (4)• Reduce % Product Waste and Rework • Increase Work EngagementROI (5): 15% Target ROI

(Source: Phillips & Phillips, 2012)

InterventionPurpose: To prepare immediate managers to more effectively

create and sustain motivational work environments that positively impact work engagement and organizational performance.– 90-Day (7-Session) Learning Intervention based on the 5-

Skills of Savvy Managers – Blended learning format (classroom, online, on-the-job skills

practice, journaling, and peer-interaction)– Participants: Four frontline managers from Production unit– Test Group (Production Unit) and Control Group

(Maintenance Unit) research design

(Source: Phillips & Phillips, 2012)

InterventionTwo In-person Sessions: Introduction/overview and finale:

– Course Introduction and Overview (1 day): Introduce the course, overview key concepts, and familiarize participants with the virtual learning environment

– Course Finale (0.5 day): Summarize the course, discuss challenges and lessons learned, complete course assessments and planned actions

Five Skills of Savvy Managers

Motivational Work Environment

Increased Work Engagement

Improved Organizational Performance

To prepare immediate managers to more effectively create and sustain

motivational work environments that positively impact work engagement

and organizational performance.

(Source: Phillips & Phillips, 2012)

InterventionFive On-line Sessions (Blackboard): Summarize previous session content, discuss any assignments, discuss current topic, and preview next session:

– Self-managing (1.5 hours)– Reflecting (1.5 hours)– Acting consciously (1.5 hours)– Collaborating (1.5 hours)– Evolving (1.5 hours)

InterventionSavvy Manager Learning Process

Set learning and

performance goals

Learn new skills

(Plan Actions)

Practice new skills on the job

(Implement)

Reflect on the outcomes (Journal)

Seek feedback on the

experience (Discuss)

Make enhancements

(Adjust)

Monitor progress

(Improve)

InterventionTheory to Practice Assignments

• Use It or Lose It – Planned Action(s)• Skill Development Journal• Group Discussion and Feedback• Learn, Practice, Adjust, Monitor Progress and Improve

Five Skills of Savvy Managers

Motivational Work Environment

Increased Work Engagement

Improved Organizational Performance

To prepare immediate managers to more effectively create and sustain

motivational work environments that positively impact work engagement

and organizational performance.

(Source: Phillips & Phillips, 2012)

InterventionUse It or Lose It Planned Actions

Write down one idea you “got” as a takeaway from today’s learning, reading, experience. Then, write down what you plan to do starting tomorrow to integrate this idea, practice it, and make it a habit.

1. What I got: 2. How I will use it tomorrow: 3. Which savvy skills I am developing:

InterventionSkill Development Journal

Savvy Skill Coaching QuestionsSelf Manage What do you need to uncover about this

performance issue? Where will you apply your discipline/control?

Reflect What becomes clear and more apparent? What is the new picture that emerges?

Act Consciously What specific actions will you take to help you reach your goal?

Collaborate Who can help you? Where are your resources?

Evolve What did you learn that will help you to grow? How will you implement new strategies for going forward?

Job

Aid

InterventionGroup Discussion and Feedback

What other things might we do to help build our savvy manager skills on the job?

Set learning and performance

goals

Learn new skills (Plan Actions)

Practice new skills on the job

(Implement)

Reflect on the outcomes (Journal)

Seek feedback on the

experience (Discuss)

Make enhancements

(Adjust)

Monitor progress

(Improve)

InterventionPractice, Discuss, Adjust, Monitor and ImproveImpact Measures (Level-4):• Work Engagement: Test/Control Group Arrangement

– 32 line employees from Production unit (test)– 31 line employees from Maintenance unit (control)– Utrecht Work Engagement Scale (Schaufeli & Salanova, 2008)

• Production business unit controllable Waste and Rework – Product Waste & Rework: $272,850/month ($3.27M/year) – Every 1% of total product produced as Waste costs $245K– Every 1% of total product produced as Rework costs $35K

(Source: Phillips & Phillips, 2012)

Data Collection PlanReaction (1)• Reaction questionnaires taken at the end of each of seven sessionsLearning (2)• Pre- and post program skill self-assessment profiles• Facilitator assessment of participant discussions, responses to

questions, and completed assignments during the program• Utrecht Work Engagement Scale at program start, middle and endOn-the-Job Application (3)• Skill self-assessment profiles (3-months)• Utrecht Work Engagement Scale (3-months)Business Impact (4)• Percent Product Waste and Rework• Utrecht Work Engagement Scale (6-months)

(Source: Phillips & Phillips, 2012)

Data/ROI Analysis Plan

Data Items Isolation of Effects Monetary Conversion

Percent Waste Trend analysis

Participant and management estimates

Percentage of waste times $245,000

Percent Rework Trend analysis

Participant and management estimates

Percentage of rework times $35,000

Work Engagement Control Group Not converted

(Source: Phillips & Phillips, 2012)

Results

Reaction (1)• Content relevance (4.16/5.0, 4.0)• Importance (4.07/5.0, 4.0) • Intent to use (4.23/5.0)Learning (2)• Pre program skill self-assessment profile mean (105.0/125)• Post program skill self-assessment profile mean (107.6/125)• Utrecht Work Engagement Scale: Test group higher than control

group by the end of the 90-day learning interventionOn-the-Job Application (3)• Skill self-assessment profile mean (112.6/125) at 3-months• Utrecht Work Engagement Scale: Test group higher than control

(Source: Phillips & Phillips, 2012)

ResultsBusiness Impact (4)

$367,500 Projected Trend - $119,350 Post Average = $248,150 per month $248,150 X 12 = $2,977,800 per year less cost for Waste and Rework

0.50 Impact Estimate X 0.85 Confidence X $2,977,800 = $1,265,565

ROI (5)

Net Program Benefit/Program Costs X 100

($1,265,565 - $253,761)/$253,761 = 3.99 X 100 = 399%

Intangibles• Increased Work Engagement• Employee Satisfaction• Improved Teamwork and Communications • Better Decision-making

(Source: Phillips & Phillips, 2012)

Lessons

• Plan must be based on performance needs – drives purpose • Close coordination with client from start to finish critical• Do your homework, establish relationships, connect the dots• “Stuff” happens – be prepared to adjust the plan• Communicate, communicate, communicate!

– Needs– Issues– Progress– Results

ResourcesMeasuring ROI in Learning & Development: Case Studies

from Global Organizations. Editors: Patricia Pulliam Phillips, Ph.D. and Jack J. Phillips, Ph.D. ASTD Press, Alexandria, VA , 2012.

The Savvy Manager: 5 skills that Drive Optimal Performance. Jane R. Flagello, Ph.D. and Sandra B. Dugas, Ph.D. ASTD Press, Alexandria, VA, 2009.

Enhancing Work Engagement Through the Management of Human Resources. W.B. Schaufeli and M. Salanova. In The Individual in the Changing Working Life (pp. 380–404). Editors: K. Näswall, M. Sverke, and J. Hellgren. Cambrigde University Press, New York, 2008.

ResourcesFor a free copy of this slide deck, please contact John Kmiec at 228.365.2559 or john.j.kmiec.jr@saic.com

A learning intervention for work engagement using mixed methods: In-person classroom

and synchronous online learningJohn Kmiec, Ph.D.

Certified ROI Professionalwww.linkedin.com/in/jkmiec/

Consulting EmployeeTraining and Simulation Services and Solutions

Science Applications International Corporationjohn.j.kmiec.jr@saic.com / 228.365.2559 / www.saic.com

Annual State MeetingHattiesburg, MS

June 4, 2014