Learning and Business Impact: Making the Case through Metrics and Analytics

#CLOwebinar

Sponsored by

The presentation will begin at the top of the hour.

A dial in number will not be provided.

Listen to today’s webinar using your computer’s speakers or headphones.

Welcome to the webinar!

#CLOwebinar

Sponsored by

Tools You Can Use Audio Control

–  A dial in number will not be provided.

–  Adjust the volume by sliding the indicator in the Media Player box to the right.

–  Also check your computer’s volume for external speakers or headsets.

#CLOwebinar

Sponsored by

Tools You Can Use

Speaker Bios

Resources

Media Player Q&A

Slides

Twitter

Help

Post Event Evaluation Share This

Group Chat

HRCI & SHRM

CLO Webinars

#CLOwebinar

Sponsored by

1. May I receive a copy of the slides? YES! Click on the resource list located on the top right portion

of your screen.

2. May I review the webinar recording at a later date? YES! You may log in again using today’s link to review the

presentation on-demand.

3. Is this webinar HRCI or SHRM certified? YES! The HRCI and SHRM certification codes will appear in

the box to the right of the slides after the required watch-time has elapsed.

Frequently Asked Ques6ons

#CLOwebinar

Sponsored by

Sco% Schaffer Global Head of Learning Measurement & Analy=cs Ericsson

Learning and Business Impact: Making the Case through Metrics and Analy6cs

Slide title 70 pt

CAPITALS

Slide subtitle minimum 30 pt

Learning and Business Impact: Making the Case through Metrics and Analytics

Scott Schaffer, PhD Twitter: @schaffer3999 Global Head of Learning Measurement & Analytics

Slide title 44 pt

Text and bullet level 1 minimum 24 pt

Bullets level 2-5 minimum 20 pt

Characters for Embedded font: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fifl

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐəәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the

footer area

Scott Schaffer, Learning measurement and analytics, 2017

Slide title 70 pt

CAPITALS


Aligning Measurement with Learning and Business objectives

Scott Schaffer, PhD Global Head of Learning Measurement & Analytics

›  Introduction to Ericsson

› Measuring Business Impact

› Measurement Maturity

› Analytics & Learning Measurement

Slide title 70 pt

CAPITALS


Public | © Ericsson AB 2014 | 2014-12-5 | Page 2

37,000

40%

5 busd

1 Billion

2.5 Billion

65,000

28 busd

180

111,000

Subscribers

managed

Subscribers

supported Services professionals

Net Sales 2016

Countries

Employees

Patents

Mobile traffic

In R&D

Ericsson at a glance

MOBILE INFRASTRUCTURE

OPERATIONS & BUSINESS SUPPORT SYSTEMS

TELECOM SERVICES

TV PLATFORMS

Slide title 44 pt







footer area


Our Strategic direction

FUTURE/ EMERGING

PRESENT/ LARGE

Excel in Core Business

Establish leadership in targeted areas

Expand Business In New Areas 1 2 3

Radio, Core & Transmission Telecom

Services

IP Networks

TV & Media

OSS & BSS

Industry & Society

Cloud

Slide title 44 pt







footer area


CLO

CEO

Ericsson leaders want to know that we are planning to Measure L&D Metrics

Is it money well spent? What is the impact on our business?

What does it cost to achieve the competence shift I need?

Was the learning any good?

How does that compare to other people in the Unit or Firm?

What learning has been completed by my team?

Slide title 44 pt







footer area


What gets Measured gets Managed

Slide title 44 pt







footer area


Xerox Internal Use Only

Learning Measurement & Analytics Maturity What key metrics link learning activities to organization goals, and how will those metrics be captured and reported?

Data Modern Learning

Insights

Slide title 44 pt







footer area


Key Global Business Requirements Key Business Requirements List the desired bottom-line business impact(s)

Key Drivers Describe the actions that must be taken to drive the Key Business Requirement

Key Metrics List the necessary metrics to track the listed Key Drivers.

Growth Growth acceleration Sales Profitability Cost efficiency Profit Margins Order-to-Cash Capital efficiency; Cash flow WCD; high-margin sales and cash collection Pricing Transformation Commercial excellence New – Retained Customers

Key Global Learning Requirements Key Business Requirements List the desired bottom-line business impact(s)

Key Drivers Describe the actions that must be taken to drive the Key Business Requirement

Key Metrics List the necessary metrics to track the listed Key Drivers.

L&D Business Alignment Competency analysis CCRG L&D Spend Efficiency Cost efficiency Avg $ spend per employee

L&D Effectiveness Human capital readiness Success cases; KPI improvement L&D Learning Excellence - Key learning programs

Access to right learning assets, JIT, right format

Value, Quality

Slide title

44 pt

Text and bullet level 1

minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Data Sources and People

Data Description Describe what data is needed to deliver the Key Metrics listed in Section 1

Data Type Describe where the data listed is stored and how it is formatted.

Access Describe how the data will be accessed and credentials needed (if applicable)

Refresh Rate Describe how “real-time” the data needs to be

Prep Needed Describe how the data may need to be prepared, transformed or join with other data described in this table

DRIVERS Describe role(s) of those that drive the Key Metrics listed in Section 1.

QTY LEADERS Describe role(s) of those that review the Key Metrics listed in Section 1

QTY PREPARERS Describe the role of those that prepare the Key Metrics listed in Section 1.

QTY

Data – what metrics and measures are associated with each requirement?

People – Who is accountable for results? Who will verify data?

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Align Business Requirements and Learning Investments

Priority Business Goals-L&D Programs 2016 Actual 2017 Plan

1 Profit: Increase Cost Efficiency by X % Corporate Goal = Order to Cash Business Acumen for Managers

5%

8%

2 Growth: Growth Acceleration of X % Corporate Goal = Sales Negotiation Sample Program

8%

10%

3 Pricing Tr: New / Retained Customers X% Corporate Goal = Consultative Selling Sample Program

10%

20%

Adapted from TDRp, Dave Vance

Slide title 44 pt







footer area


Outcomes are aligned to business goals and separated into individual contributors to success

Business outcome:

What are the contributors to success?

Increase Working Capital Days by 10%

3%

5%

12%

3%

7%

1

Increase cash flow

Increase in customer satisfaction

Increase in billed CR

Reduce WIP

30%

25%

20%

15%

10%

5%

0%

Causal outcome: Increase OTC efficiency by 30%

Increase in process efficiencies

ILLUSTRATIVE

Slide title 44 pt







footer area


Source: Measurement map derived from work of Kaufman and Keller; and R. Smith

Learning Program

Major Objective •  Topic •  Topic •  Topic •  Topic Major Objective •  Topic •  Topic •  Topic

Performance Objectives

Performance 1

Performance 3

Performance 5

Performance 4

Performance 2

Leading Indicators

Improved presentations, calls,

proposals

Reduced errors, increased volume

Business Results

Client Satisfaction

Productivity

Strategic Goals

Cost Reduction

Gross Margins

Individual learning programs are designed to drive business value The impact map identifies links between learning objectives and business objectives

From left to right, ask “How do we achieve these objectives?”

From right to left, ask “Why are we doing this [intervention or measurement]?”

Performance 6

Slide title 44 pt







footer area


Impact MaPs Define Targets

Increase revenue •  <current> • <target>

Financial Objectives Strategic Objectives Tactical Objectives Performance Objectives Learning Objectives

Increase in new and expanded

accounts •  Measure – new accounts sales $

Increase profits from all accounts

• Measure - increment per account$ Maximize deal value

•  Cost of remediation

Develop partner network

• Measure xxxx

Expand sales team footprint

• Measure xxxx

Consultative selling

•  Measure xxx

Better Presentations

•  Measure xxx

Develop accounts •  Measure xxx

Cross-functional selling •  Measure xxx

Negotiate better deals

• Price discounts • Value of concessions

Opportunity ›  Increase 2016 Sales volume

–  Q1 2015 = $154.2 million –  6% percent increase over prior quarter –  28 percent increase over 2014 Q1

Ask why?

Ask how?

Networking Skills •  Level 0,1

Sales Negotiation Skills •  Level 0,1

Slide title 44 pt







footer area


Business Goal

Learning Impact @ the Solution Level

Impact occurs when:

A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal (adapted from R. Brinkerhoff).

Learning

Intervention

S

S

S

S

S

S

S R

Individual or Team Behaviors/Performance

20

Skills are applied while in the role…

Improved KPIs; Correlations

between S & R

Slide title 44 pt







footer area


Business Goal

The Anatomy of Learning Impact: When Learning works, what happens?

Impact occurs when:

A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal

Learning

Intervention

S

S

S

S

S

S

S R

Individual or Team Job This is the Finish Line

Slide title 44 pt







footer area


The Anatomy of Learning Impact: When Learning works, what happens?

Impact occurs when:

A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal

Learning

Intervention

S

S

S

S

S

S

S R

Individual or Team Job This is the Finish Line

Not this!

Business Goal

Slide title 44 pt







footer area


End to end support for L&D that provides reporting, dashboards, and analytic insights into the specific impact of learning on individual performance, business units and the organization.

Learning Measurement and Analytics as a service

Slide title 44 pt







footer area


1.  Completely! We don’t communicate learning impact at the executive level. 2.  Somewhat. We measure consumption but don’t have a compelling impact

story. 3.  We’re Covered. We tie learning metrics to business goals and communicate

our success at the executive level.

Poll 1: 62% of cLO’s feel Executive Pressure to use Learning Analytics to show impact How exposed are you?

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› The seven levels represent the chain by which learning creates value in large organizations.

› Learning is consumed on an individual basis, begins to add value when it is retained, adds further value when the individual takes it back to the workplace, and adds more value when it is integrated into the larger team performance. This process is built from the bottom up.

Impact Measurement and the Learning Value Chain

Business Outcomes

Operational Results

Application

Learning

Learner

Corporate Dashboard

Usage

However, business results are produced when learning interacts with many other factors — factors that are commonly managed from the top down.

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


levels of Measurement maturity across Outcomes

Traffic, Activity, Smart Sheets, Quizzes, Tests Behaviors, KPIs Correlate behaviors with metrics; causal relationships

Measurement Measurement Measurement Measurement Measurement

Application

Consumption Learner Learning

Business & Societal Outcomes Operational Results

Cost Value

Individual Outputs & Outcomes Indiv & Team Outcomes Organizational & Societal Outcomes

•  Types of Learning Products

•  Learning Reach

•  Completions •  Time on task

•  Learner reactions

•  Reflections •  Projects •  Cognitive/

Skills Testing

•  Design Quality

•  Formative and Summative Assessment

•  Learning x Learner Measures

•  Transfer surveys

•  Transfer Interviews

•  Focus Groups •  Observations •  Experiential/

Work-based learning

•  Checklists

KPI Metrics •  Quality •  Productivity •  Customer

Satisfaction/ Retention

•  Margins •  Spend

Business Measures •  Revenue •  Profit •  Growth

Sustainable Development

•  End extreme poverty

•  Fight inequality and injustice

•  Halt climate change •  Tech for Good

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Measurement starts with a Research question

Traffic, Activity, Smart Sheets, Quizzes, Tests Behaviors, KPIs Correlate behaviors with metrics; causal relationships

Measurement Measurement Measurement Measurement Measurement

Application

Consumption Learner Learning

Business & Societal Outcomes Operational Results

Cost Value

Individual Outputs & Outcomes Indiv & Team Outcomes Organizational & Societal Outcomes

What are the most popular learning channels? Are the most expensive courses also highly rated? What percentage of people believe the learning experience was a good investment? What percentage of people would recommend this training to a peer How much do we spend on L&D investments?

What specific behaviors lead to better sales margins? Did the learning solution transfer to the job? Which employees were most successful when attempting to transfer learning to the job? What enhancers/barriers affect learning transfer?

Which learning investments had the highest return? Is learning associated with high performance? What specific learning factors predict increased profits? What role does learning play in sustainability and development goal attainment?

Slide title 44 pt







footer area


An example of a Measurement Strategy Phase Product Will be used to measure… Method/Tool Status

Analyze Align Plan

Impact Map Measurement Plan

Alignment of learning investment with business results; intention to measure

PC completes/Template PC completes/Template

Part of new process Impact maps are part of SCM projects

Readiness Checklist Organization, Program, and Learner Readiness PM completes/Checklist Part of new process

Engage (L 0,1)

Activity/Consumption Completions, activities, log-ins for a particular program ITM, Promote, Domo, Traffic

Too slow. ITM reporting unwieldy. Domo could help. Need use cases.

End of Course Survey Perceived value, relevance, confidence, intent to apply ITM, Promote, Netigate

Awaiting Learning Leadership approval. Needs to be built.

Learning (L 2)

Survey/Tests/ Projects Learning/competence gain. Formative and summative. ITM, Promote Not enough data to

comment.

Transfer & Early

Impact (L 3,4)

Survey, Interview Successes, Unrealized Potential, Non-starters; Barriers; Manager Support

ITM, SCM, Netigate PC completes

Conducting 4 SCM studies through Q3

Business impact case Evaluation Study

Early business impact of the offerings (Qualitative); Learning effect by comparing participant and control groups or trend analysis.

Interview transcripts/Email Evaluator consults Domo, Tableau, Excel,

This is a Q3 project. Could use SSN or SPIN

Predict & Optimize

(L 5,6)

Prescriptive Predictive Optimization Report

Correlations between learning and performance; cause & effect relationships, e.g., increase $ spend predicts increased performance

Evaluator consults Domo, Plateau, (Vestrics)

Q3 – Q4 project dependent upon previous phase. Need analysis tool.

Slide title 44 pt







footer area


Analytics Involves Multiple sources and owners of Data! C

›  Full accountability and governance process ›  Simplified data architecture reduces risk of critical events ›  Cost savings from limited duplication ›  Easy reconciliation and auditing

Single enterprise data warehouse with tools for data visualization and analysis

Data Sources: Operational & Learning Platforms

Downstream Users

Single golden source

EDW Formal

Blended

Informational / Social

Extraction tools: Domo, Excel, SPSS, Tableau

SuccessFactors Pluralsight AdobeConnect

Promote

Traffic Kaltura

Operations

Finance BW HRMS

Slide title 44 pt







footer area


Moving from Data to Reporting to insight is the Essence of the Analytics journey

Analytics Data management

HIGH

LOW

LOW HIGH Degree of Intelligence

Com

petit

ive

Adv

anta

ge

Raw data

Clean data

Standard reports

Ad hoc queries

Selective Drill Down

Alerts

Prescriptive Optimization

Predictive Analysis

Diagnostics

Descriptive Analysis

Advanced Analytics

Standard Reporting

Data

Standard Analytics

Source: McKinsey & Company

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


q Very Mature! We make decisions based on insights gained from evidence based on rigorous data collection.

q Mature. We use statistical analyses to analyze data that was collected in a systematic way.

q Somewhat. We have standard analytics and self-service reporting in place. q We’re just getting started. We struggle with reporting and data management.

Poll 2: Measurement and Analytics Maturity. Where do you fit on the curve?

Slide title 44 pt







footer area


Use Cases for Learning Impact questions

• Core Dashboards for Function, Regions, Business Units • Learning transfer surveys 60-90 days post

How are we doing on key metrics compared to last year? Is Learning impacting performance?

• Top Sales Winners have 20% more learning credits peers • Top performers have significantly more learning hours than others

Does overall learning activity positively relate to customer satisfaction? Promotability?

• High manager engagement is positively associated with better performance

What factors differentiate high performers from others?

Slide title 44 pt







footer area


Dashboards for all

Slide title 44 pt







footer area


Does Learning Consumption correlate with higher performance?

Slide title 44 pt







footer area


60-90 days post-workshop survey

Identify Success and Non-Success Cases

Analyze Data

Success Case Method – Identify top performers and What Differentiates Them from others

Insights & Recommendations

Manager and Employee Interviews

Impact Map*

Collect KPI Measures (if available)

Slide title 44 pt







footer area


Four Impact Measurement Projects were identified as priorities based on their alignment to our business strategy, reach and impact

OTC

WCD

Sales-related

Impact Measurement Projects

Consultative sales related

1 2 3 4

Prioritized

P

P

P

P

P

P

P

P

P

P

HIGH

HIGH

MED

MED

HIGH

MED

HIGH

HIGH P

Slide title 44 pt







footer area


Learning Program

Fundamentals •  Sales Process •  Ericsson Business •  Business Drivers •  Customer Focus •  PGM/PD Roles Project Management •  Key CPM Actions •  Changes that

Maximize Margins •  Assure Customer

Acceptance •  Decision-making

that adds Value •  Establishing

Billing Milestones

Performance Objectives

Analyze customer needs

Agreement to milestones and fulfillment criteria

Identify & negotiate add-on sales /change requests

Release billing milestones early

Consult & negotiate agreement with customers

Leading Indicators

Increased cash flow Increase margins on project work Reduced WIP

Increased project pipeline Improved billed CR Increased cash flow

Business Results

Increased revenues per customer Increased ROC Increased WCD

Improved CPE Increased customer retention Increased operating cash flow

Strategic Goals

Profits

Growth

How (This Global Program) will drive business value Align learning, performance and business objectives

From left to right, ask “How do we achieve these objectives?”

From right to left, ask “Why are we doing this?”

Control project costs

Slide title 44 pt







footer area


Success Case Stories

What I learned How I applied it Results I have created

“Negotiation was the best part for me – tension & concession – give and take – I’m now always doing that! “ “How can I anticipate argumentation on the part of the customer?” “Methods for consciously applying high value to customer, low cost for us.”

“It works really well in Renewal – the challenge is customer price erosion – not reducing the opex; they ask for discounts as efficiency is improved.” “I scope clients using value argumentation. What are the needs? Why do they need it? Once we have this information we can quantify the value of the deal.” “We won an OSS/BSS deal… but the customer wanted to renegotiate focused on price. I now had to use negotiables – to give them something.”

“I avoided a 10% discount in exchange for training: saved $x5K/year for two years – i.e. $xx130K by providing training worth $20K = deal value of $xxx,000.” “By arguing the numbers: What do you want? What can we give away that wont cost much? These projects have become prioritized and we sacrifice others where we make less $.” “Instead of giving a x % saving – we gave training + x% saving that we in turn negotiated from the PPP/supplier, This was a $xxx K deal saving = $xxK”

Slide title 44 pt







footer area


Early Impact: 74% Indicate Learning had Significant-Moderate Impact on Performance

*Survey completed by a small sample of mid-low success cases

Slide title 44 pt







footer area


What Factors Differentiate Higher Performers from others?

Relationship of Manager Engagement to Impact.

Copyright Brinkerhoff Evaluation Institute 2016

Slide title 44 pt







footer area


Prescriptive Analytics: what investments should we Double Down on and which ones should be Discontinued?

41

•  “Evidence of impact” chain links investments to business results

•  “What-‐if” predic=ve modeling both up and down the chain

•  Predic=ve modeling ROI to op=mize investment decisions

Slide title 44 pt







footer area


March 22 – CLO Week Silicon Valley : Driving L&D Leadership Through Measurement: Shaping the Impact Discussion July 17 – Corporate Learning Analytics, Nashville: Solving for Talent: Developing the role of the learning analytics leader

For More on This topic!

#CLOwebinar

Sponsored by

Register for the next webinar! One Left Behind: Driving Performance within

a Deskless Workforce

Tuesday, March 28, 2017

Webinars start at 2 p.m. Eastern / 11 a.m. Pacific

Register for all upcoming Chief Learning Officer Webinars at events.clomedia.com/webinars

OR click the icon on the widget bar!

Technology

Learning and Business Impact: Making the Case through Metrics and Analytics