Joint ILS Manager Tools Course Development T.R. “Randy” Pilling Program Director, Acquisition...

Joint ILS Manager ToolsCourse Development

T.R. “Randy” PillingProgram Director, Acquisition Logistics(703) 805-2277 thomas.pilling@dau.mil

September 12, 2005

Joint ILS Manager ToolsCourse Development

• DAU contacted by LOGSA (Bill Edwards) seeking endorsement, partnership, co-teaching in course development and delivery

• DAU also aware of LMI support to Navy regarding development and delivery of ILS courses

• First coordination meeting 4 Aug 05:- Consensus on need, requirements and content - Assignment specific, level 200 course- Agreed to share coordination & control in course development- Level of commitment to get it done- Course objectives and development timelines in work—

potential deployment 1Q FY 07- Manpower and funding provided by LOGSA

ILS Manager Tools CoursePreliminary Schedule

Metrics-That-Matter (MtM)Logistics & Sustainment Course Metrics

T.R. “Randy” PillingProgram Director, Acquisition Logistics(703) 805-2277 thomas.pilling@dau.mil

September 12, 2005

DAU Metrics-That-Matter (MtM)

• Metrics that Matter (MTM) Survey System serves as the standard application DAU-Wide for recording of customer support for both students

and special customer requested projects.• Data is collected at the end of or immediately

following each course• Survey questions cover areas relating to:

- Instructor- Courseware- Learning Effectiveness/On-Line Learning- Job Impact- Environment

Report Card Grading

Other includes: • Courseware• Learning Effectiveness• Environment• Job impact • Online Delivery (where applicable)

MtM 7 Point Scale

HighlySatisfied

HighlySatisfiedDissatisfied

HighlyDissatisfied

HighlyDissatisfied Neutral

1 2 3 4 5 6 7

6.0: DAU Instructor rating5.2: DAU Instructor rating

5.6: DAU other category ratings

5.2: DAU other category ratings

Satisfied

Instructor

Courseware

Environment

Online Delivery

Learning

Effectiveness

Job Impact

All Courses: 2nd Quarter Summary

48 0 0

64 7 0

41 4 3

16 7 0

65 6 0

31 29 11

71 courses: 48 residence, 23 DL

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

101 102 201A 201B 203 204

LOG: 2nd Quarter Summary

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

235A 235B 304

LOG: 2nd Quarter Summary

Program Mgmt CurriculumRatings by Course

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

ACQ ACQ ACQ PMT PMT PMT FE101 201A 201B 250 352A 352B 201

PMT/ACQ/FE: 2nd Quarter Summary

Engineering & Technology Curriculum

Ratings by Course

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

IRM IRM IRM SAM SAM SAM PQM101 201 301 101 201 301 101

E&T: 2nd Quarter Summary

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

PQM PQM PQM PQM PQM PQM STM104 201A 201B 202 212 301 201

E&T: 2nd Quarter Summary

Instructor

Courseware

Environment or Online Delivery

Learning

Effectiveness

Job Impact

STM SYS SYS SYS TST TST TST302 201A 201B 301 101 202 301

E&T: 2nd Quarter Summary

DAU Strategy

• Identify issues/inconsistencies with MtM results• Review survey questions • Determine applicable data filters• Review processes for data collection/allocation

• Develop standardized procedures for collecting, filtering and reporting accurate data

• Continue to review MtM results/progress at executive level and incorporate data into faculty/staff assessment reports

• Implement “Data Mart” tool for fusion of MtM, travel, timekeeping and financial data systems

BACKUP

LOGISTICS CURRICULUMRatings by Course

MTM LOG 101 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=795 n=1,705

Online Delivery n=794 n=1,696

Learning Effectiveness n=768 n=1,654

Job Impact n=801 n=1,681

5.855.89

5.695.70

5.895.90

5.385.36

MTM LOG 102 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=1,043 n=1,594

Online Delivery n=1,039 n=1,588

Learning Effectiveness n=1,009 n=1,541

Job Impact n=1,019 n=1,567

5.855.84

5.755.73

5.905.81

5.335.31

MTM LOG 201A Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=261 n=688

Online Delivery n=261 n=683

Learning Effectiveness n=249 n=654

Job Impact n=255 n=678

5.925.86

5.935.86

5.905.90

5.315.17

MTM LOG 201B Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=343 n=630

Environment n=341 n=622

Courseware n=344 n=630

Learning Effectiveness n=341 n=627

Job Impact n=340 n=624

6.616.64

6.056.12

5.975.97

6.026.02

5.505.48

MTM LOG 203 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=42 n=80

Online Delivery n=14 n=78

Learning Effectiveness n=41 n=78

Job Impact n=42 n=80

6.146.15

6.076.25

6.046.02

5.505.55

MTM LOG 235A Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=266 n=791

Online Delivery n=265 n=588

Learning Effectiveness n=257 n=766

Job Impact n=263 n=778

5.975.85

5.885.72

5.955.82

5.365.26

MTM LOG 235B Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=298 n=508

Environment n=291 n=500

Courseware n=298 n=508

Learning Effectiveness n=296 n=504

Job Impact n=291 n=499

6.606.68

6.046.05

6.046.20

6.096.13

5.475.55

MTM LOG 204 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=46 n=72

Environment n=46 n=72

Courseware n=46 n=72

Learning Effectiveness n=45 n=71

Job Impact n=45 n=71

6.666.74

5.835.41

6.156.24

6.136.19

5.936.11

MTM LOG 304 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=101 n=212

Environment n=99 n=205

Courseware n=101 n=212

Learning Effectiveness n=99 n=209

Job Impact n=100 n=211

6.736.75

6.166.08

6.276.20

6.266.18

5.885.88

Program Mgmt CurriculumRatings by Course

MTM ACQ 101 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=3342 n=6793

Online Delivery n=3330 n=6765

Learning Effectiveness n=3279 n=6672

Job Impact n=3305 n=6746

5.94

5.86

6.07

5.12

6.09

5.95

5.88

5.45

MTM ACQ 201A Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=1883 n=3618

Online Delivery n=1876 n=3600

Learning Effectiveness n=1814 n=3512

Job Impact n=1848 n=3572

5.75

5.54

5.85

5.25

5.84

5.74

5.53

5.24

MTM ACQ 201B Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=1513 n=3309

Environment n=1493 n=3265

Courseware n=1511 n=3307

Learning Effectiveness n=1498 n=3290

Job Impact n=1499 n=3275

6.68

5.82

6.12

6.21

5.53

6.67

5.77

6.10

6.17

5.58

MTM PMT 250 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=32 n=114

Environment n=0 n=44

Courseware n=32 n=114

Online Delivery n=32 n=68

Learning Effectiveness n=30 n=111

Job Impact n=32 n=113

6.536.69

5.14

5.935.79

5.595.62

5.735.57

5.735.29

Engineering & Technology Curriculum

Ratings by Course

MTM STM 201 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=63 n=86

Environment n=63 n=86

Courseware n=63 n=85

Learning Effectiveness n=63 n=86

Job Impact n=63 n=86

6.31

4.91

5.46

5.52

5.35

6.33

4.81

5.42

5.41

5.33

MTM STM 302 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=35 n=51

Environment n=35 n=51

Courseware n=35 n=51

Learning Effectiveness n=35 n=51

Job Impact n=35 n=51

6.40

5.35

5.62

5.81

5.44

6.34

5.63

5.50

5.83

5.32

MTM SYS 201A Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Courseware n=59 n=101

Online Delivery n=59 n=99

Learning Effectiveness n=58 n=97

Job Impact n=59 n=100

5.59

5.33

5.49

5.10

5.52

5.32

5.52

5.09

MTM SYS 201B Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=446 n=879

Environment n=444 n=878

Courseware n=446 n=881

Learning Effectiveness n=439 n=870

Job Impact n=443 n=873

6.46

5.79

5.69

5.62

5.23

6.39

5.74

5.57

5.50

5.11

MTM SYS 301 Report Card2nd Qtr

Category Current Qtr Rating

Cumulative FY Rating

Instructor n=324 n=576

Environment n=322 n=570

Courseware n=324 n=575

Learning Effectiveness n=321 n=573

Job Impact n=321 n=571

6.43

5.60

5.65

5.62

6.40

5.66

5.62

5.60

5.29 5.34

Report Card Grading (survey questions)

Report Card Grading (survey questions)

Report Card Grading (survey questions)

7 6 5 4 3 2 1 n/aThe content was divided into small enough units to allow for sufficient learning.

7 6 5 4 3 2 1 n/aThe electronic user interface was friendly and easy to navigate.

7 6 5 4 3 2 1 n/aThe online delivery had the right level of user interaction/involvement.

7 6 5 4 3 2 1 n/aThe graphics and illustrations used were meaningful and within context.

7 6 5 4 3 2 1 n/aThe instructor was helpful. (web-based only)

7 6 5 4 3 2 1 n/aThis delivery method was an effective way for me to learn the material.

7 6 5 4 3 2 1 n/aThe online help desk was beneficial.

7 6 5 4 3 2 1 n/aThe exercises or knowledge reviews added value to my learning. (facilitated online courses only

On-line Survey Questions