Upload
watershed
View
273
Download
0
Embed Size (px)
Citation preview
@leolearning @WatershedLRS
Measuring the Business Impact
of Learning Lagging indicators to predictive analytics
Gareth JonesPiers Lea
Chief Strategy Officer: LEO
Mike Rustici
CEO: Watershed Principal Consultant: LEO
www.leolearning.com www.watershedlrs.com
@leolearning @WatershedLRS
Points we will cover today
• Current state of play
• Why learning analytics matter
• The components of a business impact strategy
• Case Studies- how others have done them
• A strategic framework
@leolearning @WatershedLRS
@leolearning @WatershedLRS
Where are you on the measurement journey?
@leolearning @WatershedLRS
State of PlayPiers Lea
@leolearning @WatershedLRS
Respondents from over 350 companies
in various sectors, from government
to automotive, aviation, defence
and technology
LEO and Watershed Research
I want to use analytics to improve learning
I believe it is possible to demonstrate learning’s impact
@leolearning
I feel executive pressure to measure learning’s impact
@leolearning
The biggest challenge of measuring the impact of
learning in my organisation is...
@leolearning @WatershedLRS
L&D skills gaps … what the research is telling us
@leolearning @WatershedLRS
@leolearning @WatershedLRSGlobal Human Capital Trends 2014: Engaging the 21st-century workforce - A report by Deloitte Consulting LLP and Bersin by Deloitte
@leolearning @WatershedLRS
84% of those using analytics
actively review their programs
against organizational goals
“Making an Impact - How L&D leaders can
demonstrate value” Towards Maturity and
LEO Learning, June 2016
Some are now red hot … we need to increase the impact!
@leolearning @WatershedLRS
L&D skills gaps … the analytics capability curve
@leolearning @WatershedLRS
Big Picture – Why Learning
Analytics Matter Mike
@leolearning @WatershedLRS
“Scalable learning is
the new reason for
large organizations to
exist”
John HagelFounder
Deloitte Center for the Edge Innovation
@leolearning @WatershedLRS
“Without data
you’re just
another person
with an opinion”
W Edward DemingManagement Systems Legend
@leolearning @WatershedLRS
“L&D is responsible for improving our
most important asset, our people. If we
can improve L&D, we can improve the
world”
MeI said that
@leolearning @WatershedLRS
Understanding learning analytics: Complexity
measurement
The simple act of
tracking things and
recording values. Can
be passively or
actively collected.
evaluation
The process of trying to
make meaning from the
data measured.
Descriptive analytics.
Does the data mean
something “good” or
“bad”?
advanced
evaluation
When data sets get large
enough we can use
advanced evaluation
techniques to discover
powerful insights. Data
mining, AI, machine
learning, etc.
predictive and
prescriptive
Make predictions and
decisions based on data
and advanced
algorithms.
Recommendations
engines are the best
example in learning.
@leolearning @WatershedLRS
Learning
ExperienceLearner
Learning
Program
Understanding learning analytics: Categories
Understand an overall
learning program. Is this
initiative helping to meet
business objectives.
Understand a learner or group
of learners. Ensure
organizational readiness and
compliance.
Understand more about a
specific learning activity.
Maximize effectiveness and
spot problems.
@leolearning @WatershedLRS© Watershed Systems, Inc 2017
@leolearning @WatershedLRS
all data
Learning is so much bigger than the
LMS. Track informal and experiential
activities
xAPI and the LRS present a transformative
technology for learning analytics. Seamlessly
bring all your data together in one platform and
perform real-time analytics tailor made for L&D
business impact
Include data about behaviors and
performance in your learning
analytics real time
Create, view and distribute reports
seamlessly in real-time
xAPI & data analytics engine
@leolearning @WatershedLRS
Which category would you like to measure most?
@leolearning @WatershedLRS
Key components of a business impact strategyGareth
@leolearning @WatershedLRS
Royal Institution
@leolearning @WatershedLRS
The Top 10 components
@leolearning @WatershedLRS
1: Partner with the business
and co-create
Components of a business-impact strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
Components of a business-impact strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
Components of a business-impact strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
Components of a business-impact strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
6. Define technical integration
steps and tools
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
6. Define technical integration
steps and tools
7. Run a pilot and refine the reporting
processes and procedures
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
6. Define technical integration
steps and tools
7. Run a pilot and refine the reporting
processes and procedures
8. Document the business
case and strategy
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
6. Define technical integration
steps and tools
7. Run a pilot and refine the reporting
processes and procedures
8. Document the business
case and strategy
9. Define the change programme
@leolearning @WatershedLRS
1: Partner with the business
and co-create
• 2: Define business drivers
• 3. Map and get
stakeholders on board
• 4: Define chain of evidence/existing KPIs
• 5: Define data sources and types
Components of a business-impact strategy
6. Define technical integration
steps and tools
7. Run a pilot and refine the reporting
processes and procedures
8. Document the business
case and strategy
9. Define the change programme
10. Measure your measurement strategy
@leolearning @WatershedLRS
Case Studies Mike & Piers
@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied
@leolearning @WatershedLRS
Comprehension vs. Application
Source: Getting Buy-In From the Boss. Andy Webb, Applied
@leolearning @WatershedLRS
Visibility Through Dashboard, Mapping, Sorting
Source: Getting Buy-In From the Boss. Andy Webb, Applied
@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied
@leolearning @WatershedLRS
Ford of Europe
• Reduced time to competence from 36
to 13 months
• 93% of key messages understood by
learners
• Significant increase in car sales: 2.4
cars/annum per head on average*
* Control groups of more than 1,000 people
@leolearning @WatershedLRS
0
1
2
3
4
5
6
7
8
79% 100%
Average sales per person
Achievement is also
important – those who
passed Fiesta e-
learning with 100%
sold 2 more cars than
those who passed
with 79%
e-learning generates incremental sales
@leolearning @WatershedLRS
Code Blue
When a patient's heart stops
and needs to be resuscitated.
@leolearning @WatershedLRS
Key Metrics
Time to
first drug
Time to
defibrillation
Time to
chest
@leolearning @WatershedLRS
Learning Ecosystem
Mock Code Blue/ In
person ObservationMobile Simulation/
Defibrillator App
LMS Training Lab
Learning Analytics
Platform (Watershed)
@leolearning @WatershedLRS
Does the amount of training activity over time have an impact on Code Blue KPIs?
Time to
defibrillation
Time to
chest
Time to first
drug
Arrival
times
Time hands off
chest
Training
Activities
Jan 2015 Feb 2015 Mar 2015Apr
2015
May
2015Jun
2015
Tra
inin
g
Activitie
s
Code B
lue
KP
Is
Training Effectiveness
@leolearning @WatershedLRS
Creating A StrategyPiers
@leolearning @WatershedLRS
What degree of pressure does your senior
management place on measuring the business
impact of learning?
Close
OUTPUT:Final documents and
lessons learned
A. Finalise all
deliverables
B. Post project
review
C. Next steps
D. Formal close
Change
OUTPUT:Change programme and
presentation
A. Document
enabling
organizational
changes
B. Stakeholder roles
and
responsibilities
C. Change
management
implementation
plan
D. Presentation of
strategy
Create
OUTPUT:Strategy and
Business case
A. Establish costs
B. Define benefits
C. Document
business case
D. Document
strategy
Design
OUTPUT:Measurement
framework
A. Run workshop
with key
stakeholders
B. Create operating
model to measure
business impact
(people, process,
content,
technology)
C. Refine
measurement
methodology and
framework
D. Initial pilot
(optional)
A. Desk research and plan
stakeholder workshops
B. Document chain of
evidence
C. High level data types
(e.g. business KPIs)
D. High level data sources
(learning/business
systems)
E. Technical integration
requirements
F. Dashboard
specification &
prototyping
Inquire
OUTPUT:Prototype measurement
framework
Measuring the business impact of learning: Process for creating a documented strategy
A. Establish business
drivers
B. Prioritize critical
capabilities
C. Map stakeholders
D. Refine inquiry
method
E. Agree output
F. Create plan and
reporting protocols
Initiate
OUTPUT:The plan and schedule
© Copyright LEO Learning 2017
@leolearning @WatershedLRS
What percentage of your L&D budget is
committed to measuring the business impact of
learning solutions?
@leolearning @WatershedLRS
?
@leolearning @WatershedLRS
Look out for our follow up:1. Top 10 components measuring business Impact
2. 5 Steps to get started with Learning Analytics eBook
Ready to get started?