KPIs and Management Report

Report to the Management Board and General Assembly on Q3 2015

KEY PERFORMANCE INDICATORS

Content KPIs (slide 1 of 4)

▪ Request submission for the July 2015 release

KPI Target Achieved

All requests will be reviewed and have their status updated from NEW within 7 days of submission

100% 100%

All requests received before the request close date will be completed, closed or be allocated to content tracker item

100% 98.5%

Authored   3312   62.5%  Request closed not authored   1278   24.1%  Assigned to Content Tracker item   350   6.6%  Awaiting resolution   357   6.7%  

Total   5297   100.0%  

July 2015 release summary

Content (slide 2 of 4) SIRS Backlog

▪  Status as of 23rd, September 2015

▪  Figures exclude CMT related submissions

▪  All requests predating July 2014 have been dealt with or are allocated to Content Tracker item

2015 2014 2013 2012 2011 2010 2009 2008 Reviewed only 484 7 0 0 0 0 0 0 Allocated to content tracker item 326 815 745 132 10 20 20 0

Content KPIs – release summary (concepts) (slide 3 of 4)

0

5000

10000

15000

20000

25000

30000

Changed concepts

New Concepts

Release Date in form of YYYYMMDD

Num

ber o

f con

cept

s

Content KPIs – release summary (descriptions) (slide 4 of 4)

0

20000

40000

60000

80000

100000

120000

140000

160000

Changed descriptions

New descriptions

Num

ber o

f des

crip

tions

Release Date in form of YYYYMMDD

Education KPIs (slide 1 of 3) Presentations available at end of reporting period

Public accessible presentations Course-specific presentations

Q2 38

Q3 5 48

Assessments available at end of reporting period Public accessible assessments Course-specific assessments

1 4

1 7

E-Learning development and delivery during reporting period Presentations developed Assessments developed Interactive webinars delivered as part of E-Learning courses

20 4 8

10 3 24

Usage of public E-learning materials during reporting period Public accessible presentation views Public accessible assessments taken Average mark for assessment taken

143 243 48%

2153 133 68%

Course enrollment and completion during reporting period Students starting Foundation Course Students currently taking Foundation Course Students successfully completing Foundation Course Drop out/did not successfully complete Foundation Course Students starting Implementation Course Students currently taking Implementation Course Students successfully completing Implementation Course Drop out or did not successfully complete Implementation Course

Q2 521 522 275 99 0 0 0 0

Q3 622 905 186 53 51 49 0 2

Course demand in this reporting period New Foundation Course applicants New Implementation Course applicants

890 112

365 132

Unmet course demand at end of reporting period – Students applying for… Foundation Course with confirmed place allocated in next 6 months Foundation Course without a confirmed place in next six months Implementation Course with confirmed place allocated in next 6 months Implementation Course without a confirmed place in next 6 months

700 0 112 0

365 0 162 2

Education KPIs (slide 2 of 3)

Education (slide 3 of 3) Helpdesk tickets and response times and resolution rates

Num

ber o

f tic

kets

and

min

utes

Num

ber o

f tic

kets

and

per

cent

ages

Human Resources KPIs

Q3 2015 total includes 3 fixed term/secondment: Chris Swires, Louise Jones and Jim Case.

0 5

10 15 20 25 30 35 40 45 50

Q1 2014

Q2 Q3 Q4 Q1 2015

Q2 Q3

Number of Departing Employees Number of New Employees during Q Total Number of Employees end of Q

Num

ber o

f em

ploy

ees

2014Q3 Q4 2015 Q1 Q2 Q3

# Members 27 27 27 27 27

# Large Vendor Agreements 0 0 0 0 0

# New Collab Agreements 2 0 0 1 0

Appropriate Representation at External Events

100% 100% 100% 100% 100%

% Survey Results Showing Strong Product/Service Levels

n/a 98% (Showcase survey)**

90.5% (Member survey)

n/a n/a

Member satisfaction is at least 80% for CRM (annual measurement)

n/a n/a Yes 95.2%

n/a n/a

Member satisfaction of IHTSDO is at least 75% (annual measurement)

n/a n/a Yes, 90.5%

n/a n/a

Customer Relations & Collaborations KPIs (slide 1 of 2)

2014 Q3 Q4 2015 Q1 Q2 Q3

Customers receive a response to enquiry with confirmation of assignment in 48 hrs

3h56m avg first

response

5h4m avg first

response

5h26m avg first

response

3h21m avg first response

3h59m avg first response

25% of enquiries are resolved in first contact by assigned agent

Yes, at 67%

Yes, at

65%

Yes, at 68%

Yes, at 64%

Yes, at 62%

Avg time to resolution of enquiries is 5 working days/one week

1.27 days 1.5 days

2.75 days

0,94 days 1.51 days

Completing customer engagement plan 75% targets met

n/a n/a On track On track On track

Collaboration operational plan 2015/16 – 75% targets met

n/a n/a Plan being

updated

Customer Relations & Collaborations KPIs (slide 2 of 2)

Note: Q3 performance still better than targets but affected by particularly challenging tickets and summer holidays of staff and customers.

SNOMED CT delivered to Members on time (all editions) R=SLA missed by more than 2 working days A=SLA missed by 1 day G=SLA met

Green

SNOMED CT delivered without known defect (all editions) R=Critical defects exist A=Non-critical defects exist G=Zero known defects

Amber

Product release schedule documented and is aligned with content roadmap R=Less than 60% of content projects are aligned with roadmap A=60% to 80% of content projects are aligned with roadmap G=80% of content projects are aligned with roadmap

Green (100% of known projects have been scheduled)

Product lifecycle is documented and products are actively managed R=Less than 60% of content projects are documented and actively managed A=60% to 80% of content projects are documented and actively managed G=80% of content projects are documented and actively managed (Actively managed means product development, maintenance and deprecation processes are in use where applicable)

Green (100% of known products are managed)

Delivery KPIs

Technical Services - KPIs

▪  SLAs are detailed here –

▪  https://confluence.ihtsdotools.org/display/IT/Tooling+Key+Performance+Indicators

Technical Services KPIs Customer Service (internal & external)

Tickets On time resolution (%)

First contact resolution (%)

Average first response (mins)

Average response (mins)

Average resolution (mins)

Q4 2014 75 70 71 342 265 3,098 Q1 2015 160 78 75 236 205 5,556 Q2 2015 103 90 73 146 96.82 2,401 Q3 2015 92 80 67 206 123 7,980

Target 85 75 300 240 2880

Note: Q3 higher resolution time was due to a very small number of internal tickets taking longer than usual and Freshdesk being updated after the fact. Removing those tickets brings resolution time from 5 days 13 hours to 3 days 10 hours. Mitigation: enforcing tighter use of Freshdesk.

Num

ber o

f tic

kets

and

per

cent

ages

Num

ber o

f tic

kets

and

min

utes

Technical Services KPIs Infrastructure Uptime (%)

Target Uptime (%) 99.9 The regular downtime on the Mapping tool was mostly planned downtime for upgrades, and necessary patch fixes. This took place generally outside of normal working hours and had no impact on availability to users. Tracking of Collabnet became difficult due to Collabnet network changes, hence the blank cell in Q3.

Per

cent

age

Technical Services KPIs Online Service User Sessions

Num

ber o

f ses

sion

s

Tool

MLDS Affiliate Report 2015 Q3

Work Plan – Work Item Status Update

For Q3 2015 Red = behind schedule and will not complete in 2015 Amber = behind schedule but will complete in 2015 Green: = on track to complete by 31st Dec 2015 For Q3 2015: 1.  I&E - Document Maintenance continues to report Amber due to E-Learning priorities. High demand for E-Learning courses a key

staff absence has forced group to focus on E-Learning. Document updates will be completed in Q4. 2.  Content Request Project (SIRS) is Amber and is delayed due to external resource availability. Product remains scheduled to be

developed in 2015, with implementation in Q1 2016. Template Based Content authoring is reporting as Red as the focus is now on single concept authoring. Template based authoring will be completed in 2016.

3. Release Management - release service enhancements are reporting as Red and have been rescheduled for 2016.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2014 Q3 2014 Q4

2015 Q1 2015 Q2

2015 Q3

57 58 41

30 40

13 0

3

4 2

6 14

0 2 2

Red

Amber

Green

UPDATES BY LINE OF BUSINESS

▪  Functioning and abilities – creation of Expert Reference Group ▪ CMT – process documentation has improved

communication and streamlined authoring activities ▪  LOINC/SNOMED Technology Preview published ▪  Agreement on a clear way forward for the updating of drug

content in the International Release

Content – 2015 Q3 Achievements

▪  Planning work item delivery for July 2016 International Release ▪  Dentistry – content addition following clinical review ▪  Nursing – addition of nursing procedures content and publication of

SNOMED CT/ICNP linkage table ▪  Publication of updated GP/FP subsets and ICPC-2 map following

baseline release ▪  Functioning and abilities – delivery of concept model for functioning and

editorial guidance based on the Observables project ▪  Anatomy – updating of content following University of Washington

review ▪  Substances – updating of substance content in line with agreed

reworking of substance hierarchy ▪  Drugs – meeting of Member Forum subgroup to define the International

SNOMED CT model for drug representation for national drug releases

Content – 2015 Q4 Key Objectives

▪  Potential disruption of authoring productivity due to development and delivery of new authoring tool, also reduced scope of content additions due to restrictions of new tooling

▪  Availability of technical solutions to reconcile the disruption caused by the roll-out of the new authoring tool

▪  Lack of feedback on the alpha releases and technology previews will lead to uncertainty of the ability to implement the solution

Content – 2015 Key Risks

▪  Foundation course ▪  186 people successfully completed course in Q3

▪  Cumulative completions for year so far: 461 ▪  522 people entered course in Q2

▪  Cumulative intake for year so far: 1,518 ▪  Now running on demand

▪  Start within 3 to 8 weeks of application ▪  Self-paced within 4 month timeframe

▪  Implementation course ▪  49 people start course July 6th (6 now delayed)

▪  108 will start in October ▪  60+ will start in January

▪  24 interactive webinars delivered ▪  will double to 16 per month from October

▪  Development delayed slightly ▪  So far only 10 day delay to pilot group progress

▪  Other activities ▪  Four tutorials prepared for Montevideo ▪  Support for CRM activities in AsiaPac ▪  Expression Constraint specification published

Education – 2015 Q3 Achievements

▪  Foundation course ▪ Revision of course based on feedback ▪ Regular monthly intakes – to match demand ▪  Intakes from July and September complete course

▪  Implementation Course ▪ Development of course materials complete ▪  Pilot intake completion (most likely to complete early January) ▪  Second intake starts (108) ▪  Applications remain open for more people to join course

▪ Content Development Theory course ▪  Initial ‘pilot’ intake ready to start in January

▪  SNOMED CT Expo - Montevideo ▪  Tutorials delivered

Education – 2015 Q4 Key Objectives

▪ No contingency in team capacity to … ▪ Meet demand to develop and maintain course materials ▪  Provide interactive support and assessments for advanced courses ▪  Provide resilience in the event of unplanned absence of key staff

▪ Difficulties meeting demand for interactive support and assessment of learners on Implementation course ▪ Unpredictability of future demand for E-Learning services ▪ Delays to guidance documentation updates in 2015 due to

efforts to complete E-Learning development and delivery

Education – 2015 Key Risks

▪ Member budget process to gather priorities and feed to

finance for budgeting purposes for 2016 was completed. ▪  Product and Services Placemat developed ▪ Collaboration handover from Jane Miller to CRM team

completed ▪ Uruguay conference program finalized ▪ Development of marketing material underway ▪ HIMSS APAC, China Health IT event, member event participation, NRC meetings executed, potential member meetings ▪ CRM Vendor Engagement specialist in place

Customer Relations – 2015 Q3 Achievements

▪  Delivery against agreed regional CRM Action Plans ▪  Additional intelligence gathered on non-Member countries

and maturity level with regards to standards ▪  Work on ensuring Member knowledge about translation

policy ▪  SNOMED In Action goes live ▪  New vendor engagement resource in in terms of regional needs and requirements ▪  Member survey 2015 distributed

Customer Relations – 2015 Q4 Key Objectives

▪  The existing core delivery resources that are mainly centralized in Europe would not be scalable to support Members in Asia Pacific and Americas regions ▪  Potential new members requiring more formal contract

process in order to join and submit payment. ▪  Lack of SCT translation into non-member languages and

financial support to do the translations is a barrier to vendor implementation of SCT in non-member territories.

Customer Relations – 2015 Key Risks

▪ Handover of liaison function to CRM Leads ▪ Draft License agreement for free starter set of SNOMED CT

concepts with various SDOs (DICOM as exemplar) ▪  Agreement in principle around a collaboration with the AMA ▪ Discussion around free starter set of concepts with all

significant SDOs was conducted and well received in most cases ▪  ICNP/SNOMED CT Nursing Diagnoses linkage table

baseline release communicated and distributed ▪  EXPO collaboration track preparations in place ▪ Collaboration template completed ▪ Draft OpenEHR agreement being discussed ▪ GP/FP subset and ICPC-2 map released

Collaboration 2015 Q3 Achievements

▪  Finalize collaboration agreement with AMA ▪  Finalize license agreements for free starter set with

DICOM and IHE. ▪  Finalize updated agreement with ADA ▪  Finalize license agreement with OpenEHR ▪  Finalize license agreement with OpenMRS ▪  Finalize development license arrangement with HL7 ▪ Monitoring of progress of joint collaboration activities

Collaboration – 2015 Q4 Key Objectives

▪ Relationship with WHO continues to be strained and requires significant attention.

Collaboration – 2015 Key Risks

▪ Delivered July 2015 International Edition to IHTSDO Members on June 30 ▪ Delivered LOINC Technology Preview phase 2 on Sept 1 ▪ Delivered ICNP Cooperative project Baseline on Sept 4 ▪  Improvements to automated Release service (SRS) ▪ Refinement, and performance-tuning of Release

Validation Framework (RVF) ▪ Release Process & Procedure improvements, including

communication and collaboration with other teams ▪  Enhancement of Service Catalog, and other

Release documentation on Confluence ▪ Creation of Known Issues register to

improve transparency & users’ capabilities

Delivery – 2015 Q3 Achievements

▪ Delivery of Spanish Edition, October 30 ▪ Delivery of January 2016 SNOMED CT International

Edition, December 31 ▪ Delivery of in-development SNOMED CT products ▪  Integration with Single Concept Authoring tool &

Terminology Server ▪ Continued maturing SNOMED Release Management

function ▪  adoption of new SNOMED product development process, focus

on IHTSDO Member priorities & ROI ▪  advanced Release Process and Procedure

improvements ▪  documentation and enforcement of product

licensing requirements

Delivery – 2015 Q4 Key Objectives

▪ Risk to the quality of SNOMED CT releases: ▪ New staff are inexperienced in our domain ▪ Newly developed software tends to be unstable

▪ Reduced capacity in Delivery team results in a high risk of non-delivery of non-prioritized items. This is due to two thirds of the team's technical resources having been moved to other Projects. Mitigation is to manage scope of work until the team returns to full capacity ▪ High number of in-flight content development projects

requiring tooling that is currently unavailable, due to lack of capacity in the technical teams

Delivery – 2015 Key Risks

▪ Continuing use of Members Licensing & Distribution Service (MLDS) live for more Members, including Portugal, Denmark, Belgium, New Zealand, Sweden, Estonia, India, Malta, Chile, and Uruguay.

▪  Phase 1 of Authoring Tool development completed and demonstrated to IHTSDO Authoring Team, and the second phase of development has now started.

▪  Successful RFP for the development of Refset Management and Translation Tooling, awarded to WCI.

▪  Project to move off Collabnet under way and on target

▪ Redesign of SNOMED In Action in final stages.

Technical Services - 2015 Q3 Achievements

▪ Continued documentation improvements across all tooling services ▪ Migration from Collabnet TeamForge to Confluence completed

before end of November, 2015 ▪ Redesign and enhancement to SNOMED In Action site

completed before Uruguay conference ▪  Phase 2 of Single Concept Authoring tool complete, and

authoring cycle for July 2016 will start on the new platform. Workbench will no longer be in use at this point. ▪ Reference Set Management and Translation Tooling ready to be

deployed by end of 2015 ▪ Development complete for integrated Content Request Service

Technical Services - 2015 Q4 Key Objectives

▪  Very tight development and delivery timelines without much room to maneuver (staff illness has already had an impact earlier in 2015) ▪ Costs and timescales affected due to unforeseen

implementation details, especially in areas of new functionality including web based authoring tools and template based authoring. ▪ Reduced service management capability will impact

services as they are introduced to Members. ▪  As the team is following an open and transparent

development, any mistakes made and lessons learnt are public.

Technical Services – 2015 Key Risks