Can Essential Skills Training in the Workplace Make a ... · Transfer payments: EI, welfare - 0 + 0 Tax payments - 0 + 0 Other Household Income, Tax Credits +/- 0 0 +/-Firm Effects

Can Essential Skills Training in the Workplace Make a Difference?

• David Gyarmati, Research Director, Social Research and Demonstration Corporation

• Val Lockyer, Executive Director, The Training Group at Douglas College

| BC CENTRE FOR EMPLOYMENT EXCELLENCE 2

Presenters

Establishing the Business Case for Workplace Essential Skills Training

UPSKILL: A Pan-Canadian Demonstration Project

March 7, 2014

A Pan-Canadian research and demonstration project measuring the effects of literacy and Essential Skills training in the workplace

Implemented in 8 provinces with 110 firms and nearly 1500 workers in frontline service occupations

Here in British Columbia –30 firms and 450 workers

Managed by SRDC with funding from the Federal Government’s Office of Literacy and Essential Skills (OLES)

Implemented in partnership with

Goals of the Study

To measure the effects of Literacy and Essential Skills (LES) training in the workplace

To establish a clear business case for LES training by measuring its return on investment for firms

To understand the conditions in which LES training can be most successful and ROI can be maximized

Accommodations sector in the Tourism industry

110 firms joined the study, largely hotels

104 underwent organizational needs assessments

88 firms with clear training needs continued in the study

Only 1 firm withdrew after workers were engaged

Regional distribution:

◦ ~ 30% in BC

◦ ~ 25% in the Prairies (AB, MB, SK)

◦ ~ 20% in ON

◦ ~ 25% in the Atlantic region (NB, NS, NL)

1454 workers joined the study from 88 active firms

Four frontline occupational groups

◦ Administrative, Client Service – Front Desk Agents

◦ Custodial Positions – Housekeeping Room Attendants

◦ Food and Beverage – Servers, Bartenders

◦ Culinary – Line Cooks, Kitchen Assistant/Prep

ES training needs, primarily in oral communication, problem solving, document use, and numeracy

1. Evaluation Design – utilizing a randomized control trial (RCT) to estimate unbiased impacts of ES training

2. Evaluation Framework – comprehensive set of measures to help understand conditions for success

3. Performance Metrics – objective measurement of behaviour change - the “missing link” between Essential Skills and business outcome measures

4. Rigorous Cost-Benefit Analysis – monetizing impacts to estimate ROI for workers, firms, and government

Employers join the project

Complete baseline needs assessment

Employees join the project

Complete baseline skills and performance assessments

Post-training evaluations are completed and compared to

baseline

All employees receive training





Post-training evaluations are completed and compared to

baseline

All employees receive trainingMarket Conditions,

Layoffs

Employees motivation





Post-training evaluations are completed and groups are compared

Program Group receives training

Market Conditions,

Layoffs

Employees motivation

Control Group No training

Random assignment

Long-te

rm

Outc

om

es

Retu

rns

Inte

rmedia

te O

utc

om

es

Inputs

, acti

vit

y,

outp

uts

Everyday

Practices

Workplace

Performance

Individual Factors

Structural Factors

Workplace LES Training

Psycho-

social

Capital

Social

Capital

Human

Capital

Individual ROI Firm ROI

Individual Outcomes

FinancialNon-financial

Firm Outcomes

FinancialNon-financial

Gov’t ROI

Maintaining Service

Excellence

Enhancing Customer Relations

Increasing Productivity

Enhancing Human

Resources

Increasing Sales

Business Needs

Improving Health and

Safety

ES needs

Oral Communication

Maintaining Service

Excellence



Enhancing Human

Resources

Increasing Sales

Business Needs


Safety

Thinking Skills

Document Use

Numeracy

Working with Others

Continuous Learning

Reading Text

Writing

ES needs

Maintaining Service

Excellence



Enhancing Human

Resources

Increasing Sales

Meeting Service Standards

Effective Communication

Increasing Task Efficiency

Health and Safety Practice

Professionalism

Reducing Error

Retention and Absenteeism

Problem Solving

Product Knowledge

Business Needs Performance Gaps


Safety Emergency Preparedness

Up-selling

Success in further training

Oral Communication

Thinking Skills

Document Use

Numeracy

Working with Others

Continuous Learning

Reading Text

Writing

Maintaining Service

Excellence



Enhancing Human

Resources

Increasing Sales



Increasing Task Efficiency

Health and Safety Practice

Professionalism

Reducing Error

Retention and Absenteeism

Problem Solving

Product Knowledge

Business Needs Performance, Industry Standards


Safety Emergency Preparedness

Up-selling

Success in further training

Measured through

objective performance

assessments based on

industry standards

Establish a clear

relationship with

business outcomes

Maintaining Service

Excellence




Professionalism

Problem Solving


1. Service Quality Indices – quality metrics of reliability, responsiveness

2. Guest Satisfaction – various customer rating systems, complaint tracking

3. Customer Loyalty, Retention

e.g. return to hotel, return to brand

4. Promoter Scores

e.g. likelihood to recommend hotel

5. Occupancy Rates, Room Revenue ($)

6. Ancillary Spending ($) on future visits


Organization, Efficiency

Reducing Error


7. Efficiency Metrics – speed of check-in, check-out; room-cleaning

8. Error Rate – room, order, billing, HR

9. Cost Savings in Staff Time ($) – for frontline (e.g. lower time-to-complete) and supervisors (check/correct errors)

10. Cost Savings in Wastage ($) –room revenues, inventories savings, reduced spoilage

11. Room Turn-Around ($) – arising from quicker cleaning, checkout

Participants Firms

Government

Budgets Society

Participant Impacts

Employment and Earnings + - 0 +

Transfer payments: EI, welfare - 0 + 0

Tax payments - 0 + 0

Other Household Income, Tax Credits +/- 0 0 +/-

Firm Effects

Sales Revenue: Occupancy rates, Ancillary spending 0 + 0 +

Productivity: Task Efficiency/Staff Time 0 + 0 +

Changes in Direct Costs: Inventories/Supplies 0 + 0 +

Health and Safety: Sick Leave/Absenteeism 0 + 0 +

HR: Hiring Costs/Retention 0 +/- 0 +/-

Taxes: Sales, Payroll, Corporate 0 - + 0

Program Delivery Costs

UPSKILL Program Delivery 0 0 - -

UPSKILL Release Time 0 - - -

Other Transfer Program Administrative costs 0 0 + +

Benchmark Model: Benefit/Cost per Participant ($) +/- +/- +/- +/-

Return on Investment (ROI) (%) +/- +/- +/- +/-

Financial Components (Fully-monetized)

Perspective

The Offer to Firms and Workers

• Needs Analysis and skills assessments

• Up to 40-hours of on-site LES training

• Customized to workers needs

• Blended delivery – group sessions, on-

to-one, self-paced

• ½ of the release time covered

Sectoral Engagement

Sectoral Performance

Gaps & Training Needs

Analysis

Firm Engagement

Participant

Recruitment

Needs Analysis &

Delivery Plan

Curriculum

Customization

Training Delivery

Core Curriculum

Design

Valerie LockyerMarch 7, 2014

Firm Engagement

Employee Recruitment

Needs Analysis

Curriculum Development

Training Delivery

12/03/2014 24

Job specific

training

Essential

Skills

12/03/2014 25

12/03/2014 26

Training

Employer

Employee

Funder

Thank you

douglascollege.ca

1. Essential Skills

2. Job Performance

3. Employment and Earnings

4. Non-Financial Impacts – Psychosocial, Health

5. Firm Impacts, Business Outcomes

6. Benefit Cost Analysis – Return on Investment

7. Conditions for Success

Essential Skills

Did UPSKILL training produce measurable impacts on

Literacy and Essential Skills?

UPSKILL increased literacy scores on Prose/Document Use

by about 12 points immediately after LES training.

For those assessed more than a year after enrolment, this

increased up to 23 points, or nearly a half level.

Mean Impacts: Change in Document Use Scores

Outcome Group Group Impact Standard Error

Document Use Met or surpassed industry standard (%)Immediate post-training assessment 12.8 2.2 10.6 *** (3.4)

SurpassSecond post-training assessment 16.4 -1.8 18.2 *** (5.7)

Met or surpassed industry standard (%)Mean scores among those with assessments

SurpassLess than 6 months after enrolment 15.8 3.9 11.9 ** (5.1)

% meeting6 to 12 months after enrolment 12.7 4.9 7.8 * (4.3)

% below, but approachingGreater than 12 months after enrolment 16.1 -6.8 22.9 *** (6.6)

Program Control Difference-in-Difference

Participating workers were 22 per cent more likely to have

literacy scores at the level required for their jobs (Level 3).

0.2

-1.2

1.6

-1.4

0.7

-5.6

-6.4

-1.6

-7.9

21.5

-15.0 -10.0 -5.0 0.0 5.0 10.0 15.0 20.0 25.0

Lower Level 1 (<180)

Upper Level 1 (180 to 224)

Lower Level 2 (225-249)

Upper Level 2 (250-274)

Level 3 (275+)

At Enrolment/Baseline

Follow-up at 6 Months orLess

***

*

**

*

Percentage of sample

For the average employer with 15 participants, this translates into

an additional 3-4 employees with literacy levels they need.

Job Performance

Does Essential Skills Training lead to measurable

impacts on Job Performance?

• The Performance measures are based on industry expectations in the national standards for certification

• Observational measures taken on-site by industry assessors

• Very high standards for certification – requires min. of 85%

• Surpassed ( ≈ 95-100 %)

• Met ( ≈ 85-95 %)

• Below, but approaching ( ≈ 65-85%)

• Far below ( ≈ below 65%)

• Standards cover all of the core business needs of interest to employers in the Tourism sector

Participating workers were 20 percentage points more likely to meet industry standards for guest relations

Customer greetings

Addressing questions

Complaint resolution

For the average business with 15 participants, this translates into 3 additional employees performing at high

standards.

***

***

***

***

***

Participating workers were 15 percentage points more likely to surpass industry standards in productivity

Time management

Effective Teamwork

Organizing workspace

For the average business with 15 participants, this translates into 2 additional employees performing at high

standards.

***

**

Improvements in health and safety were achieved through performance gains in safe working practices.

Emergency preparedness

Avoiding injury, safe lifting

Safely handling chemicals

UPSKILL participants were 15 percentage points more likely to surpass standards in work safety after training

*

*

Overall, LES Training has resulted in significantly higher success rates for participants on industry certification.

In addition to positive effects for employers, this may reinforce participants future training goals, career paths,

and employment prospects.

Employment and Earnings

Does UPSKILL training produce financial gains for participants in terms of employment and earnings?

LES Training increased job retention by 8 percentage points

and employment rates by about 6 percentage points

within the year following training.

91%

83%

0% 20% 40% 60% 80% 100%

ProgramGroup

ControlGroup

Working with the sameEmployer as baseline (%)

Working with a different Employer

Unemployment Spell Out of the Labour Force

Impacts on Firms

Do gains in Essential Skills and job performance lead to

measurable effects on business outcomes?

17.30%

33.40%

19.40%

29.90%

2.30%

14.00%

22.50%

61.30%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Large Increase Medium Increase Small Increase No Changes

Guest Satisfaction Ratings: Departmental Services (room cleanliness, check-in/out)

Program Group

Control Group**

*

***

3.03 + % 1.21 to < 3.03% <1.21%

17.65%

41.18%

23.53%17.65%

0.00%

6.67%

20.00%

73.33%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Large Reduction MediumReduction

Small Reduction No Changes

Service Recovery: Guest Complaints

Program Group

Control Group

5+ per week 2-3 per week 1 or less per week

**

***

***

11.40%

19.20% 17.10%

52.30%

0.00%

22.70% 18.00%

60.20%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%


Revenue Change linked to Performance Change:Average Occupancy Rates

Program Group

Control Group

**

(3.19 to 5.11 %) (1.28 to < 3.19 %) (>0 to <1.28%)

4.70%

25.10%21.30%

49.50%

1.00%

17.60%21.10%

60.30%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%


Revenue Change linked to Performance Change: Food and Beverage Revenue

Program Group

Control Group*

(2.9 + %) (1.1 to < 2.9 %) (>0 to <1.1%)

Lower Supervisory costs

Reduction in supervisor time spent monitoring, correcting errors

Mean impact: 6 hours per week saved in program group firms

Distributional:

◦ 35% 12+ hours, 25% 5-8 hours, 40% no changes

Average Savings: $3,510 in cost savings over the follow-up

Lower Hiring costs

From increased retention of participants

Mean impact: 8.5 percent point increase in retention in program group firms

For an average firm of 15 participants this represents 1 foregone hire

Cost Savings: approximately $4,400 in cost savings

Return on Investment

What are the returns from investments in LES training

for workers, firms, and government?

Participants Firms

Government

Budgets Society

Participant Impacts

Employment and Earnings + - 0 +

Transfer payments: EI, welfare - 0 + 0

Tax payments - 0 + 0

Other Household Income, Tax Credits +/- 0 0 +/-

Firm Effects

Sales Revenue: Occupancy rates, Ancillary spending 0 + 0 +

Productivity: Task Efficiency/Staff Time 0 + 0 +

Changes in Direct Costs: Inventories/Supplies 0 + 0 +

Health and Safety: Sick Leave/Absenteeism 0 + 0 +

HR: Hiring Costs/Retention 0 +/- 0 +/-

Taxes: Sales, Payroll, Corporate 0 - + 0


UPSKILL Program Delivery 0 0 - -

UPSKILL Release Time 0 - - -

Other Transfer Program Administrative costs 0 0 + +

Benchmark Model: Benefit/Cost per Participant ($) +/- +/- +/- +/-

Return on Investment (ROI) (%) +/- +/- +/- +/-


Perspective

Participants Firms

Government

Budgets Society

Participant Impacts

Employment and Earnings 948 -862 86

Transfer payments: EI, welfare -165 165 0

Tax payments -191 191 0

Other Household Income, Tax Credits

Firm Effects

Sales Revenue: Occupancy rates, Ancillary revenue 2282 0 2282

Productivity: Task Efficiency/Staff Time 1347 0 1347

Changes in Direct Costs: Inventories/Supplies 0 0 0

Health and Safety: Absenteeism/Presenteeism 0 0 0

HR: Hiring Costs/Retention 234 0 234

Taxes: Sales, Payroll, Corporate -692 692 0


UPSKILL Program Delivery 0 -1781 -1781

UPSKILL Release Time -27 -235 -235 -497

Other Transfer Program Administrative costs 0 0 0

Benchmark Model: Benefit/Cost per Participant ($) 565 2074 -1118 1671

Return on Investment (ROI) (%) 2,093% 883% -54% 73%


Perspective

Participants Firms

Government

Budgets Society

Participant Impacts





Firm Effects








UPSKILL Program Delivery 0 -1781 0 -1781

UPSKILL Release Time -27 -470 0 -497


Benchmark Model: Benefit/Cost per Participant ($) 565 562 544 1671

Return on Investment (ROI) (%) 2,093% 25% + 73%

Perspective


Participants Firms

Government

Budgets Society

Participant Impacts





Firm Effects








UPSKILL Program Delivery 0 -594 -1187 -1781



Benchmark Model: Benefit/Cost per Participant ($) 565 1453 -347 1671

Return on Investment (ROI) (%) 2,093% 137% -29% 73%

Perspective


Participants Firms

Government

Budgets Society

Participant Impacts





Firm Effects








UPSKILL Program Delivery 0 -1056 -725 -1781




Return on Investment (ROI) (%) 2,093% 72% 0 73%

Perspective


Explanatory Analysis

Under what conditions do investments in LES make sense?

How do impacts vary based on contextual and

delivery factors?

Impacts vary by context in several policy relevant ways

Participant Characteristics, at enrolment

- Skills gaps: target where its needed - larger need, larger returns

- Motivation : affects degree of engagement in learning activities

- Trust: influences whether and how new skills are utilized on job

- Social capital: provides more opportunities for learning and support

Impacts vary by context in several policy relevant ways

Business Characteristics, at enrolment

- Breadth of business needs – greater need, greater return

- Link from skills to performance – have a clear transfer plan

- A learning culture – have additional supports for training

Implementation Process, Training Dosage

- Alignment – effectively matching training with business needs

- Dosage – higher number of contact hours, larger gains

Performance impacts are driven by gains among those with low Essential Skills at baseline (Level 1 Document use)

Standard

Performance and Skill GAINS, by Baseline Skill Level Group Group Impact

Job performance

Met or surpassed industry standards (%)

Level 1 10.4 -4.0 14.5 ** (6.9)

Level 2 11.8 9.8 2.0 (6.3)

Document Use Score

First post-training assessment (1-6 months)

Level 1 34.4 20.6 13.7 *** (3.9)

Level 2 0.3 -8.3 8.7 * (4.8)

Second post-training assessment (9-18 months)

Level 1 35.5 28.0 7.5 (8.0)

Level 2 3.0 -10.6 13.6 * (7.2)

Program Control

Error

Performance impacts and longer run skills gains are experienced by only those with higher levels of trust at baseline

Standard

Performance and Skill GAINS, by Levels of Trust Group Group Impact

Workplace performance

Met or surpassed expectations (%)

Low Levels of Trust 6.9 5.1 1.8 (7.0)

High Levels of Trust 18.8 1.0 17.7 *** (6.1)

Document Use / Prose

First post-training assessment (1-6 months)

Low Levels of Trust 15.6 1.4 14.2 *** (4.5)

High Levels of Trust 14.4 -0.8 15.2 *** (3.8)

Second post-training assessment (9-18 months)

Low Levels of Trust 9.5 -1.7 11.2 (7.7)

High Levels of Trust 12.3 -7.1 19.4 *** (6.6)

Program Control

Error

Skills and performance impacts are larger in firms with a pre-existing culture of training

Standard

Performance and Skill GAINS, by Training Culture Group Group Impact

Workplace performance

Met or surpassed expectations (%)

Training Expenditures, low 16.9 -2.9 19.7 ** (9.3)

Training Expenditures, high 15.6 -8.0 23.5 *** (9.0)

Training Expenditures unknown 6.9 4.2 2.7 (5.7)

Document Use Scores

Immediate post-training assessment (1-9) months

Training Expenditures, low 13.5 5.6 8.0 (7.2)


Training Expenditures unknown 10.5 2.9 7.7 ** (3.8)

Second post-training assessment (1-9) months

Training Expenditures, low 14.7 -0.8 15.5 (11.0)


Training Expenditures unknown 15.5 12.3 3.2 (6.3)

Program Control

Error

Participants Firms

Government

Budgets Society

Participant Impacts





Firm Effects






Taxes: Sales, Payroll, Corporate 0 0 0





Benchmark Model: Benefit/Cost per Participant ($) 61 -2,038 54 -1923

Return on Investment (ROI) (%) 225% -91% + -84%

Perspective


Participants Firms

Government

Budgets Society

Participant Impacts





Firm Effects












Return on Investment (ROI) (%) + 113% + 227%

Perspective


Is the offer of workplace LES training attractive to firms?

Yes, there were high take-up rates, low withdrawals, and a very high degree of satisfaction with training

Key to its attractiveness is embedding LES in a familiar performance framework and aligning with business needs

However, hours of training delivered were half (20 hours) of what was offered due to ongoing business constraints

For more intensive training needs, one should supplement an instructor-led model with other approaches

e.g. ES with mentorship training, which lowers release time

Does workplace LES training improve worker outcomes?

Yes, there were large sustainable gains in Essential Skills and job performance

Increased job retention, employment rates, and earnings, are observed at least in the short-term

Improvements in psychological and social capital that help both reinforce skill gains and broader well being

Does workplace LES training improve firm outcomes?

Yes, there were large gains in most business areas of interest to employers

◦ Improved customer relations leading to increased revenue

◦ Increased productivity leading to reduced labour costs

◦ Increased job retention leading to reduced hiring costs

This is accompanied by several gains in non-financial measures related to worker morale and motivation

Does workplace LES training generate positive ROI?

Yes, returns were substantial for employees who bear few costs under this model

Average returns for firms were also high at about 25 percent, even when they bear the FULL costs of training

However, impacts and returns vary based on context and conditions among workers and firms

What are some of the conditions for success, which produce larger impacts and ROI ?

Key factors include depth of business needs, training readiness, and receptivity to learning

Impacts are substantial when performance needs are large i.e. clear performance gaps exist, breadth of business needs

Impacts are larger when training readiness is high i.e. firm has existing culture of learning, plan to utilize training

Impacts are larger with more receptive learners i.e. level of trust and motivation among learners is high

| BC CENTRE FOR EMPLOYMENT EXCELLENCE 79

Q & A

Thank you!

Documents

Can Essential Skills Training in the Workplace Make a ... · Transfer payments: EI, welfare - 0 + 0 Tax payments - 0 + 0 Other Household Income, Tax Credits +/- 0 0 +/-Firm Effects