24
Large Scale, Blended Learning Development: Benchmark A Chapman Alliance, Research Study March 2013 By Bryan Chapman, Chief Learning Strategist [email protected] www.chapmanalliance.com Abstract: In previous studies, we tackled development times and costs for isolated, learning- development tasks (ILT and eLearning). The purpose of this research is to analyze development of large-scale, often complex, blended learning courses with several different delivery formats bundled into a cohesive and unified course. We had 188 respondents who graciously shared information about their development practices including development times, costs, deliverables produced during the process, team structure and itemized information about each task in their work flow. Using these data we isolated a development ration of 49:1 (meaning that it takes 49 hours of development to create 1 finished hour of blended leaning (“seat time”). We also discovered that the per-finished-hour standard cost is $3,938 ($US Dollars). There are many variables that change the development ratio and cost that can be gleaned from the study with data you can use to benchmark your own practice. Now the good news! You are welcome to share this blended learning development research with anyone you’d like. Because we feel that this information is so important to the industry, we have released it under Creative Commons, meaning that you can share the whole deck or pull-out slides/charts and add them to your presentations, white papers, conference presentations, etc. The only restrictions are (1) you must ALWAYS list the source of information as “Chapman Alliance” and (2) you are NOT allowed to sell the results without written permission from Chapman Alliance. That’s it. Enjoy! Source Citation: Chapman, B. (2013). Large Scale, Blended Learning Development: Benchmark [Research Study]. Published by Chapman Alliance LLC. www.chapmanalliance.com

Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Embed Size (px)

Citation preview

Page 1: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Large Scale, Blended Learning Development: Benchmark A Chapman Alliance, Research Study March 2013 By Bryan Chapman, Chief Learning Strategist [email protected] www.chapmanalliance.com

Abstract: In previous studies, we tackled development times and costs for isolated, learning-development tasks (ILT and eLearning). The purpose of this research is to analyze development of large-scale, often complex, blended learning courses with several different delivery formats bundled into a cohesive and unified course. We had 188 respondents who graciously shared information about their development practices including development times, costs, deliverables produced during the process, team structure and itemized information about each task in their work flow. Using these data we isolated a development ration of 49:1 (meaning that it takes 49 hours of development to create 1 finished hour of blended leaning (“seat time”). We also discovered that the per-finished-hour standard cost is $3,938 ($US Dollars). There are many variables that change the development ratio and cost that can be gleaned from the study with data you can use to benchmark your own practice. Now the good news! You are welcome to share this blended learning development research with anyone you’d like. Because we feel that this information is so important to the industry, we have released it under Creative Commons, meaning that you can share the whole deck or pull-out slides/charts and add them to your presentations, white papers, conference presentations, etc. The only restrictions are (1) you must ALWAYS list the source of information as “Chapman Alliance” and (2) you are NOT allowed to sell the results without written permission from Chapman Alliance. That’s it. Enjoy!

Source Citation: Chapman, B. (2013). Large Scale, Blended Learning Development: Benchmark [Research Study]. Published by Chapman Alliance LLC. www.chapmanalliance.com

Page 2: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Research Methodology How did we collect the data

The data contained in this research was collected by surveying 188 organizations involved in blended learning development. It was a daunting survey and we would like to thank those of you who participated, often sharing sensitive information to develop these metrics. The chart, above, shows the flow of the survey.

Page 3: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Demographics Dashboard Information about survey respondents and the blended courses represented in the results

Basic 188 – companies /organizations represented in the survey (courses) 7,025,269 – total # of employees across all companies/organizations 2,174,663 - # of learners reached by represented, blended courses

Located

83% - in the United States 17% - outside the US

By Industry

Information Technology/High Tech 12% Government/Military 10% Healthcare 9% Banking/Financial Services 9% Transportation/Automotive/Aviation 7% Professional Services 6% Energy 5% Industrial/Manufacturing 5% Higher Education 5% Pharmaceutical 5% Telecommunications 5% Educational Provider (sells learning content) 4% Other 3% Insurance 3% Learning Vendor 3% Association 3% Retail 2% General Business 2% NGO 2% K-12 1%

Learning Content Types

Job Specific/Technical Skills 39% Softskills (Leadership, Customer Service, etc.) 23% Computer Application Training 14% New Hire Orientation 10% Other 6% Product Knowledge & Sales 6% Compliance 2%

Other includes: Professional Licensure or Certificate, Mixture of Multiple Types, Academic Courses

Developed…

Internally 72% Outsource (External) 13% Hybrid (some of both) 15%

Duration of Blended Courses

53 hours (6.6 days) Average duration, blended courses in survey

24 hours (3 days) Median duration, blended courses in survey

45 minutes to 800 hours Range of blended courses represented in the survey

SMEs per course 4.3 Average number of

SME's used on blended learning projects

0 - 50 Range of the number of SME's listed on projects

7.3 Interesting fact: average # of SME's used on Software/Application blended Learning dev projects

Page 4: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

How Long Does It Take to Create a Blended Learning Course “development time” per “finished hour” (seat time)

Across all 188 courses evaluated, the overall average development ratio came out at 49:1; in short, 49 hours of development for every 1 finished hour of final, blended course (“seat time”) Next, we’ll show you how this ratio changes based on specific variables…

per finished hour

So how long does it take to create a blended learning course? Because of the vast differences in course durations (ranging from 45 minute courses to 800 hours in seat time) we synthesized the information to one ratio….development time per “finished hour” (or seat time). Here’s what we found…

* $80.37 (US Dollars) estimated hourly rate * NOTE: survey respondents listed “person

hours” and overall cost. Displayed hourly rate is derived through calculation. In short, respondents did NOT provide this number. This is true for all “rates” shown throughout the summary.

Page 5: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

How Long Does It Take to Create a Blended Learning Course Boil it down to “development time” per “finished hour” (seat time)

We cross-tabbed the data on outsource vs. internal vs. hybrid development. We also looked at how the development ratio and cost changes when developing “new” content vs. repurposing existing content.

per finished hour

per finished hour

Factor: Outsourced (Externally Developed)

- 18 + $1,158

per finished hour

Factor: Hybrid (mix of internal and outsource)

- 12 - $1,130

per finished hour

Factor: “New” Content (60% or more of course is “new” content)

+ 20 + 2,397

per finished hour

Factor: Repurposing Content (60% or more existing content)

- 27 - 2,217

Page 6: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

How does “Length of Course” change development ratios? Course length does appear to have an effect on development times and cost

Another observation is that blended learning “course length” also changed the development ratio and cost. Advance to the next slide to see the results.

per finished hour

Page 7: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

How does “Length of Course” change development ratios? Course length does appear to have an effect on development times and cost

Notice how significantly the development ratios and cost changes when creating smaller blended learning course (less than one day in length). We attribute this to economies of scale during development. .

per finished hour

per finished hour

Blended courses 2 to 3 days in length

- 7 - $743

per finished hour

Blended courses less the 1 day in length

+ 32 + $1,629

per finished hour

Blend courses 4 to 5 days in length

- 14 - $1,273

NOTE: the development ratio came out the same as 2 to 3 day courses in the study, with a slightly higher cost. This is not a typo

per finished hour

Blend courses 5 to 10 days in length

- 15 - $1,335

per finished hour

Blended courses longer than 10 days

- 16 - $1,631

Page 8: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Using Benchmark Data for your Projects Simple scenario, using core metrics to create high-level estimates

Setup: you’ve been tasked to create a 5-day, 40 hour, blended learning course

1. Multiply 40 (the number of hours in the course (“seat time”) by 49 (benchmark average, according to this study for blended learning course development)

2. The total is 1,960 “person hours” of overall development time, whether you are producing the course internally, outsourcing development, or using a hybrid approach with both internal and outsource development.

3. Next, apply cost. For default purposes, you are welcome to use the benchmark cost of $3,938 (average cost of development for 188 organizations per finished hour of course time). Multiply 40 (number of hours in course) by $3,938 (or any price adjustment that best matches your scenario).

4. Using benchmark data only, the estimated cost of development is $157,520 Remember, this is a benchmark average, meaning there are many factors that can change pricing including:

• Your choice to produce internally or outsource • New content vs. repurposed content • Development staff pay rate (NOTE: the US dollar amounts listed in this study is an average of

the 188 participants of this study. You’re rate of pay may have a significant impact on overall price)

• Deliverables included with the course (covered later in this study) • Development tasks that may be omitted depending on the overall blend (i.e. your blended

course doesn’t use eLearning) • etc.

Page 9: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Adjusted for

100K or more employees

How Long Does It Take to Create a Blended Learning Course “development time” per “finished hour” (seat time)

Across all 188 courses evaluated, the overall average development ratio came out at 49:1; in short, 49 hours of development for every 1 finished hour of final, blended course (“seat time”) Next, we’ll show you how this ratio changes based on specific variables…

per finished hour

So how long does it take to create a blended learning course? Because of the vast differences in course durations (ranging from 45 minute courses to 800 hours in seat time) we synthesized the information to one ratio….development time per “finished hour” (or seat time). Here’s what we found…

* $76.20 (US Dollars) estimated hourly rate

* NOTE: survey respondents listed “person hours” and overall cost. Displayed hourly rate is derived through calculation. In short, respondents did NOT provide this number. This is true for all “rates” shown throughout the summary.

Page 10: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

How Long Does It Take to Create a Blended Learning Course “development time” per “finished hour” (seat time) Filtered for respondents with 100K or more employees

All Respondents 100K or more employees Time Cost cost per hour Time Cost cost per hour

Overall 49 $ 3,938 $ 80.37 54 $ 4,115 $ 76.20

Outsourced 31 $ 5,096 $ 164.39 50 $ 5,825 $ 116.50

Hybrid 37 $ 2,808 $ 75.89 41 $ 3,116 $ 76.00

New Content 69 $ 4,992 $ 72.35 70 $ 5,291 $ 75.59

Repurposing Content 22 $ 1,721 $ 78.23 23 $ 1,979 $ 86.04

1 day or less 81 $ 5,567 $ 68.73 85 $ 5,811 $ 68.36

2 to 3 days 42 $ 3,246 $ 77.29 43 $ 3,505 $ 81.51

4 to 5 days 35 $ 2,685 $ 76.71 39 $ 2,841 $ 72.85

6 to 10 days 34 $ 2,603 $ 76.56 34 $ 2,748 $ 80.82

10 days + 33 $ 2,307 $ 69.91 31 $ 2,459 $ 79.32

Adjusted for

100K or more employees

Page 11: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Budget Allocation How budget amounts change based on internal development, outsourcing, or using a hybrid approach

Internal Development

Outsourced Development

Hybrid Development

Average across all respondents

Internal Labor Cost = 87% Materials = 7% Outsourced Services = 4% Other = 2%

Internal Labor Cost = 9% Materials = 3% Outsourced Services = 88% Other = 0%

Internal Labor Cost = 48% Materials = 6% Outsourced Services = 46% Other = 0%

Internal Labor Cost = 71% Materials = 6% Outsourced Services = 21% Other = 1%

Page 12: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Lots of Different Kinds of Blends Observation: The study picked up lots of different blends. No two blends are the same. See random sample of blends from the actual survey below (actual listings by some respondents):

[add text here]

ILT

(face to face)

Virtual

Classroom

eLearning Hands-on Lab Mobile on

Smartphone

Mobile on

Tablet

Social

Collaboration

Informal

Learning

On-the-Job

(OJT)

Coaching

/Mentoring

Other

Respondent 1 80% 20%

Respondent 2 10% 5% 10% 5% 5% 60% 5%

Respondent 3 70% 10% 10% 5% 5%

Respondent 4 60% 25% 5% 5%

Respondent 5 15% 15% 60% 10%

Respondent 6 5% 85% 10%

Respondent 7 40% 20% 20% 20%

Respondent 8 60% 15% 5% 15% 5%

Respondent 9 75% 25%

Respondent 10 15% 65% 20%

Respondent 11 30% 5% 5% 5% 10% 15% 10% 10% 10%

Respondent 12 40% 30% 10% 10% 10%

Respondent 13 30% 10% 30% 15% 15%

Respondent 14 20% 60% 20%

Respondent 15 33% 67%

Respondent 16 45% 20% 20% 5% 5% 5%

Respondent 17 70% 20% 10%

Respondent 18 40% 40% 20%

Respondent 19 50% 20% 30%

Respondent 20 80% 10% 10%

Respondent 21 90% 5% 5%

Respondent 22 25% 5% 35% 5% 5% 20% 5%

Respondent 23 50% 50%

Respondent 24 40% 20% 20% 10% 10%

Page 13: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Blended Learning “Deliverables” Most common deliverables in blended learning development (sorted by most frequently used)

End-of-Course Evaluations 92% Instructor Guide (s) 84% Knowledge Test(s) (online or paper-based) 83% PowerPoint Deck(s )for Visuals 79% Student/Participant Guide(s) 78% Self-Paced eLearning (s) 68% Handouts or in class Activity Workbook(s) 62% Video (s) 54% Job Aide(s) 50% Virtual Classroom Material(s) 44% Reference Manual/Text Book (s) 41% Group Participation e-learning(s) 40% Pre-Reading (s) 32% Other 24% Poster(s) 18% Wiki(s) 13% Podcast(s) 12% Blog(s) 8%

Common Answers for “Other” includes: Simulations, Skill Assessments (more than just a knowledge test), Games, Virtual Labs and Lab Guides, Participatory Exercises, Certification Tests, Webinar, SharePoint Discussion Board, Social Learning Site, and Procedural Checklists)

Page 14: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Delivery Formats What modalities are used most often to deliver blended learning courses

In Class (face-to-face) 87% eLearning (self paced) 87% Virtual Classroom 58% Social Collaborative Learning 49% Coaching/Mentoring 49% Assigned On-the-Job Training (OJT) 42% Informal Learning & Knowledgebases 37% Mobile on Tablet 31% Mobile on Smartphone 28% Other 23%

Common Answers for “Other” includes: Self-Paced Video Training, Web Portal Resources, Teleconferences, Skype

Page 15: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Itemized Development Metrics

The next several slides provide detailed summaries of time spent across 30 developmental tasks through the ADDIE process, when creating large-scale blended learning courses. You can use this

information to further adjust the variables when calculating expected development times and costs (i.e. if eLearning is not part of your blend, remove the total hours for eLearning development

from your projections).

Page 16: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Analysis Phase – Development Tasks “Person Hours” spent in each task, organized by course length

Blended Learning Course Duration

Scoping and Project

Initiation

Kick Off Meetings

Front End Analysis and

Discovery Other

One day or less 23 11 37 12 2 - 3 days 39 12 67 12 4 - 5 days 48 17 107 14 6 - 10 days 52 27 134 20 Longer than 10 days 72 41 153 24

Total

83 person hours

130 person hours

186 person hours

233 person hours

290 person hours

* All times shown above are “person hours” spent in each task

Page 17: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Blended Learning Course Duration

Drafting Learning

Objectives

Course Outline

Content Analysis &

Data Mapping

Create Detailed Design

Document

Create Design

Protoype(s) Other

One day or less 16 10 22 69 34 5 2 - 3 days 17 19 35 80 53 7 4 - 5 days 22 23 61 109 117 7 6 - 10 days 36 38 72 129 139 10 Longer than 10 days 57 59 96 176 177 54

Design Phase – Development Tasks “Person Hours” spent in each task during the design phase, organized by course length

Total

156 person hours

211 person hours

339 person hours

424 person hours

619 person hours

* All times shown above are “person hours” spent in each task

Page 18: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Develop Phase – Development Tasks “Person Hours” spent in each task during the develop phase, organized by course length

Total

524 person hours

699 person hours

1228 person hours

2151 person hours

2956 person hours

Blended Learning Course Duration

Create Instructor Guide(s)

Create Student Guide(s)

Create PowerPoint (classroom

Visuals)

Create Handout(s) or In-Class

Activity Workbook

Create job Aide(s)

Create Pre-Reading(s)

Create Self-Paced

eLearning

eLearning Modules

for in-class exercises

Create Video

Create Knowledge

Test(s)

Create Course

Evaluation

Create Other Course

Materials

One day or less 41 38 39 16 23 16 165 76 50 33 15 12 2 - 3 days 63 53 57 18 28 17 204 96 100 35 14 14 4 - 5 days 172 103 79 37 57 18 409 102 140 70 16 25 6 - 10 days 295 109 181 51 75 30 1021 112 147 78 15 37 Longer than 10 days 482 170 232 86 101 36 1312 137 167 139 23 71

* All times shown above are “person hours” spent in each task

Page 19: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Implementation & Evaluation Phases – Development Tasks “Person Hours” spent in each task during implementation and Evaluation, organized by course length

Total

221 person hours

257 person hours

404 person hours

538 person hours

894 person hours

Blended Learning Course Duration

Set Up Virtual

Delivey of Learning

Publish Courses to the LMS

SME Reviews

Team Meetings

Course Walkthrough

Pilot Testing

Revision of Course Materials

Final Release

Debrief /After Action Report

One day or less 23 11 27 30 18 72 20 10 10 2 - 3 days 28 17 31 40 19 74 25 11 12 4 - 5 days 40 18 66 81 24 88 56 11 20 6 - 10 days 41 34 100 84 32 109 100 14 24 Longer than 10 days 58 26 151 130 76 267 116 39 31

* All times shown above are “person hours” spent in each task

Page 20: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Summary – Total Hours of Development This chart includes totals for the entire ADDIE process (sorted by course duration)

Blended Learning Course Duration Analysis Phase Design Phase Develop Phase Implement & Evaluate Phase

One day or less 83 156 524 221

2 - 3 days 130 211 699 257

4 - 5 days 186 339 1228 404

6 - 10 days 233 424 2151 538

Longer than 10 days 290 619 2956 894

Total Development

Time

984 person hours

1297 person hours

2157 person hours

3346 person hours

4759 person hours

Blended Learning Course Duration% % Analysis % Design % Develop % Implement &

Evaluate

One day or less 8% 16% 53% 22% 2 - 3 days 10% 16% 54% 20% 4 - 5 days 9% 16% 57% 19% 6-10 days 7% 13% 64% 16% Longer than 10 days 6% 13% 62% 19%

* All times shown above are “person hours” spent in each task

Page 21: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Development Time by Role (“person hours”) Use this information to benchmark how teams are structured for blended learning development

Project Manager(s) 13%

Lead Instructional Designer(s) 15%

Instructional Designer(s) 19%

Writer(s) 6%

Graphic Artist(s) 6%

Programming/Authoring Specialist(s) 14%

Audio/Video 3%

Quality Assurance 6%

Subject Matter Expert(s) 11%

Other 7%

Page 22: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

Summary of Findings Key takeaways from this research

• On average, the development ratio for blended learning is 49:1, meaning it takes 49 hours to create one finished hour of blended learning (on average)

• Using the data from 188 survey respondents, the average price per finished hour of blended learning is $3,938

• Creating “new “content can increase your development ratio to 69:1

• Outsourcing content takes less time (31:1), but the cost per finished hour goes up by $1,158 per finished hour

• Shorter blended courses (less than 1 day in duration) increase the development ratio average (81:1)

• The median duration of a blended learning course is 3-days in length (24 hours), although they range in the survey from 45 minutes to 800 hours (100 days)

• There is NO STANDARD BLEND. There were literally 188 different blends, using a variety of delivery formats and integrating new, emerging technologies (such as Social, Mobile, and Informal learning)

Page 23: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March

About the Author Bryan Chapman is Chief Learning Strategist at Chapman Alliance; a provider of research-centric consulting solutions that assist organizations in defining, operating and optimizing their strategic learning initiatives. As a veteran in the industry, he has over 20 years experience and has worked with such organizations as American Express, Shell, Kodak, Sprint, Sharp Electronics, Honda, IBM, Microsoft, Avon, UNICEF, American Red Cross, The Food and Drug Administration, U.S. State Department, and many others; to help them optimize learning efficiency through the use of innovative learning techniques and technologies. Click here for a full bio. Please feel free to contact me if you need some assistance…

• working through learning management system (LMS) or any other learning technology selection project

• finding and applying learning industry research; custom research projects

• optimizing or establishing your learning governance approach

• exploring best practices for rapid learning development

• learning strategy “jam sessions” (available as a 2-hour block, full-day onsite, or as a workshop)

• Click here for more available services Contact information: Bryan Chapman Chief Learning Strategist Chapman Alliance (801) 568-7011 [email protected] www.chapmanalliance.com To participate in future research studies, Click here (add you name to our mailing list)

Page 24: Large Scale, Blended Learning Development: … articles/misc/Large Scale, Blended...Large Scale, Blended Learning Development: Benchmark. A Chapman Alliance, Research Study . March