Measuring the Business Impact of Learning: Lagging indicators to predictive analytics

@leolearning @WatershedLRS

Measuring the Business Impact

of Learning Lagging indicators to predictive analytics

Gareth JonesPiers Lea

Chief Strategy Officer: LEO

Mike Rustici

CEO: Watershed Principal Consultant: LEO

www.leolearning.com www.watershedlrs.com

@leolearning @WatershedLRS

Points we will cover today

• Current state of play

• Why learning analytics matter

• The components of a business impact strategy

• Case Studies- how others have done them

• A strategic framework

@leolearning @WatershedLRS

@leolearning @WatershedLRS

Where are you on the measurement journey?

@leolearning @WatershedLRS

State of PlayPiers Lea

@leolearning @WatershedLRS

Respondents from over 350 companies

in various sectors, from government

to automotive, aviation, defence

and technology

LEO and Watershed Research

I want to use analytics to improve learning

I believe it is possible to demonstrate learning’s impact

@leolearning

I feel executive pressure to measure learning’s impact

@leolearning

The biggest challenge of measuring the impact of

learning in my organisation is...

@leolearning @WatershedLRS

L&D skills gaps … what the research is telling us

@leolearning @WatershedLRS

@leolearning @WatershedLRSGlobal Human Capital Trends 2014: Engaging the 21st-century workforce - A report by Deloitte Consulting LLP and Bersin by Deloitte

@leolearning @WatershedLRS

84% of those using analytics

actively review their programs

against organizational goals

“Making an Impact - How L&D leaders can

demonstrate value” Towards Maturity and

LEO Learning, June 2016

Some are now red hot … we need to increase the impact!

@leolearning @WatershedLRS

L&D skills gaps … the analytics capability curve

@leolearning @WatershedLRS

Big Picture – Why Learning

Analytics Matter Mike

@leolearning @WatershedLRS

“Scalable learning is

the new reason for

large organizations to

exist”

John HagelFounder

Deloitte Center for the Edge Innovation

@leolearning @WatershedLRS

“Without data

you’re just

another person

with an opinion”

W Edward DemingManagement Systems Legend

@leolearning @WatershedLRS

“L&D is responsible for improving our

most important asset, our people. If we

can improve L&D, we can improve the

world”

MeI said that

@leolearning @WatershedLRS

Understanding learning analytics: Complexity

measurement

The simple act of

tracking things and

recording values. Can

be passively or

actively collected.

evaluation

The process of trying to

make meaning from the

data measured.

Descriptive analytics.

Does the data mean

something “good” or

“bad”?

advanced

evaluation

When data sets get large

enough we can use

advanced evaluation

techniques to discover

powerful insights. Data

mining, AI, machine

learning, etc.

predictive and

prescriptive

Make predictions and

decisions based on data

and advanced

algorithms.

Recommendations

engines are the best

example in learning.

@leolearning @WatershedLRS

Learning

ExperienceLearner

Learning

Program

Understanding learning analytics: Categories

Understand an overall

learning program. Is this

initiative helping to meet

business objectives.

Understand a learner or group

of learners. Ensure

organizational readiness and

compliance.

Understand more about a

specific learning activity.

Maximize effectiveness and

spot problems.

@leolearning @WatershedLRS© Watershed Systems, Inc 2017

@leolearning @WatershedLRS

all data

Learning is so much bigger than the

LMS. Track informal and experiential

activities

xAPI and the LRS present a transformative

technology for learning analytics. Seamlessly

bring all your data together in one platform and

perform real-time analytics tailor made for L&D

business impact

Include data about behaviors and

performance in your learning

analytics real time

Create, view and distribute reports

seamlessly in real-time

xAPI & data analytics engine

@leolearning @WatershedLRS

Which category would you like to measure most?

@leolearning @WatershedLRS

Key components of a business impact strategyGareth

@leolearning @WatershedLRS

Royal Institution

@leolearning @WatershedLRS

The Top 10 components

@leolearning @WatershedLRS

1: Partner with the business

and co-create

Components of a business-impact strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

Components of a business-impact strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

Components of a business-impact strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

Components of a business-impact strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

6. Define technical integration

steps and tools

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

6. Define technical integration

steps and tools

7. Run a pilot and refine the reporting

processes and procedures

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

6. Define technical integration

steps and tools

7. Run a pilot and refine the reporting

processes and procedures

8. Document the business

case and strategy

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

6. Define technical integration

steps and tools

7. Run a pilot and refine the reporting

processes and procedures

8. Document the business

case and strategy

9. Define the change programme

@leolearning @WatershedLRS

1: Partner with the business

and co-create

• 2: Define business drivers

• 3. Map and get

stakeholders on board

• 4: Define chain of evidence/existing KPIs

• 5: Define data sources and types

Components of a business-impact strategy

6. Define technical integration

steps and tools

7. Run a pilot and refine the reporting

processes and procedures

8. Document the business

case and strategy

9. Define the change programme

10. Measure your measurement strategy

@leolearning @WatershedLRS

Case Studies Mike & Piers

@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied

@leolearning @WatershedLRS

Comprehension vs. Application

Source: Getting Buy-In From the Boss. Andy Webb, Applied

@leolearning @WatershedLRS

Visibility Through Dashboard, Mapping, Sorting

Source: Getting Buy-In From the Boss. Andy Webb, Applied

@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied

@leolearning @WatershedLRS

Ford of Europe

• Reduced time to competence from 36

to 13 months

• 93% of key messages understood by

learners

• Significant increase in car sales: 2.4

cars/annum per head on average*

* Control groups of more than 1,000 people

@leolearning @WatershedLRS

0

1

2

3

4

5

6

7

8

79% 100%

Average sales per person

Achievement is also

important – those who

passed Fiesta e-

learning with 100%

sold 2 more cars than

those who passed

with 79%

e-learning generates incremental sales

@leolearning @WatershedLRS

Code Blue

When a patient's heart stops

and needs to be resuscitated.

@leolearning @WatershedLRS

Key Metrics

Time to

first drug

Time to

defibrillation

Time to

chest

@leolearning @WatershedLRS

Learning Ecosystem

Mock Code Blue/ In

person ObservationMobile Simulation/

Defibrillator App

LMS Training Lab

Learning Analytics

Platform (Watershed)

@leolearning @WatershedLRS

Does the amount of training activity over time have an impact on Code Blue KPIs?

Time to

defibrillation

Time to

chest

Time to first

drug

Arrival

times

Time hands off

chest

Training

Activities

Jan 2015 Feb 2015 Mar 2015Apr

2015

May

2015Jun

2015

Tra

inin

g

Activitie

s

Code B

lue

KP

Is

Training Effectiveness

@leolearning @WatershedLRS

Creating A StrategyPiers

@leolearning @WatershedLRS

What degree of pressure does your senior

management place on measuring the business

impact of learning?

Close

OUTPUT:Final documents and

lessons learned

A. Finalise all

deliverables

B. Post project

review

C. Next steps

D. Formal close

Change

OUTPUT:Change programme and

presentation

A. Document

enabling

organizational

changes

B. Stakeholder roles

and

responsibilities

C. Change

management

implementation

plan

D. Presentation of

strategy

Create

OUTPUT:Strategy and

Business case

A. Establish costs

B. Define benefits

C. Document

business case

D. Document

strategy

Design

OUTPUT:Measurement

framework

A. Run workshop

with key

stakeholders

B. Create operating

model to measure

business impact

(people, process,

content,

technology)

C. Refine

measurement

methodology and

framework

D. Initial pilot

(optional)

A. Desk research and plan

stakeholder workshops

B. Document chain of

evidence

C. High level data types

(e.g. business KPIs)

D. High level data sources

(learning/business

systems)

E. Technical integration

requirements

F. Dashboard

specification &

prototyping

Inquire

OUTPUT:Prototype measurement

framework

Measuring the business impact of learning: Process for creating a documented strategy

A. Establish business

drivers

B. Prioritize critical

capabilities

C. Map stakeholders

D. Refine inquiry

method

E. Agree output

F. Create plan and

reporting protocols

Initiate

OUTPUT:The plan and schedule

© Copyright LEO Learning 2017

@leolearning @WatershedLRS

What percentage of your L&D budget is

committed to measuring the business impact of

learning solutions?

@leolearning @WatershedLRS

?

@leolearning @WatershedLRS

Look out for our follow up:1. Top 10 components measuring business Impact

2. 5 Steps to get started with Learning Analytics eBook

Ready to get started?