Overview of the mapping pilot process
Final workshop: skills mapping pilot3 May 2018
Objectives of the skills mapping pilot
Process and resource requirements for the creation of the correspondence tables of skills.
Tools and/or services to support the mapping process.
Level of interoperability between ESCO and national skill classifications.
Level of interoperability among the national skill classifications during cross-border matching.
Understand:
Steps Actors
1.Identify the national skill
classifications
Commission services
2. Select the skills subset Commission services
3. Create the mappings Commission services + PES
4. Assess the results Commission services + PES
Overall pilot approach
Step 1: Identify the national skills classifications
Participants Observers
Step 2: Select the skills subset
514 Hairdressers, beauticians and related workers: 285
2432 Public relations professionals: 231
Transversal: 107
Step 3: Create the mappingsPreparatory phase
ISCO unit groups
Occupations Skills
EC transforms the data into
SKOS
EC imports the data into
the tool
PES send data to the
EC
Step 3: Create the mappingsDuring the creation of the mappings
Tool makes suggestions for
mapping concepts
EC and PES create mappings
EC consolidates discrepancies
PES decide on the resolution
Step 4: Assess the results
Assess the level of interoperability which has been achieved between national classification systems after mapping to ESCO
Why?
How? EC and PES will simulate real case scenarios
Step 4: Assess the results
1st use case: exchange of CVs between PES
2nd use case: exchange of job vacancies between PES
Part 1: Receipt and integration of national and cross-border CVs and job vacancies into a national system
Part 2: Matching of national and cross-border CVs to national job vacancies and vice versa
Resources for this pilot
PES Commission services
Human resources to carry out step 3 and 4
Human resources to carry out step 3 and 4
Participation in 2-4 workshops either online or on site
Mapping management platform
Guidelines technical assistance
Organization of workshops
Reimbursement of travel and accommodation expenses of the participants
Documentation and dissemination of results
Summary report
Share with the PES and
Publish on the ESCO portal
Timeline
See concept paper of the skills mapping pilot for the following terms:Mapping: sub-section “During the creation of the mappings”, p.5-6 Annotation: step 3 of the sub-section “Preparation for the assessment”, p.7 Address discrepancies: steps 5-6 of the sub-section “During the creation of the mappings”, p.5Analysis of the results: cases a, b, c, d of the section “Create the mappings”, p.6-7 Compilation of final report: section “Documentation and dissemination of results”, p.12
26/02 Week 1 Week 4 Week 5 Week 8 Week 9Week 2 16/03 06/04 Week 7 04/05 Week 11
Mapping Annotation JV/CV Analysis of the results
Address discrepancies
Compilation of final report
28/0220/03
03/05
Week 12 25/05
Questions?
Creation of mapping relations: results
Final workshop: skills mapping pilot3 May 2018
STEP 3
Note
• Abbreviation:
• E(s) > AUT = Mapping from sample of ESCO to full Austrian classification
Mapper: AUT
• SWE(s) > E = Mapping from sample of Swedish classification to full ESCO
Mapper: EC
610
419
74
0
100
200
300
400
500
600
700
# Concepts
Number of concepts in each sample
ESCO Sample AUT Sample SWE Sample
92%96%
68%72%
0%
20%
40%
60%
80%
100%
120%
% of sample mapped
% of skills mapped in the sample
E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E
% of skills mapped in the sample
• SWE efforts have smaller portion of mapping relations due to differences in granularity and lack of descriptions
• Mapping from either direction has similar coverage
1
2,32,1
2,4
0
0,5
1
1,5
2
2,5
3
Average mapping relations per concept
Average number of relations per concept
E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E
Average number of relations per concept
• AUT colleagues assigned only a single mapping relation to every concept
• Averages of mapping relations are aligned for the remaining mapping efforts
39%
13%
69%
5%
9%
38%
15%
7%
48%
16%
13%
12%
4%
32%
2%
77%
0%
20%
40%
60%
80%
100%
120%
E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E
% distribution of the mapping types used
Exact More specific than ESCO More general than ESCO Close
% distribution of the mapping types used
• Broader/Narrower relations are aligned across mappings (mirroring each other)
• Mappings by ESCO team use close match much more frequently compared to MS. This may be due to miscommunication regarding exact match
Percentage of skills in this intersection
Percentage of skills in this intersection
Portion of skills inside sample of target classification
28%24%
55%
5%
0%
10%
20%
30%
40%
50%
60%
% of skills inside sample of the target classification
Portion of skills inside sample of target classif.
E(s)>AUT AUT(s)>E E(s)>SWE SWE(s)>E
Portion of skills inside sample of target classification
• The samples are not very aligned (AUT)
• Only 1/4 of skills overlap between them
• Transfer of information will be difficult
• SWE percentages are inconsistent due to size of the sample
27,00
9,00
39,00
44,00
0,00
10,00
20,00
30,00
40,00
50,00
60,00
70,00
AUT SWE
Number of same/conflicting mappings
# same mapping relations # conflicting mapping relations
Number of same/conflicting mappings
• Number of relations between common skills is low
• Possibly due to the misalignment of the samples mentioned above
• Higher number of conflicts could be attributed to misalignment of perception of the exact match relation
• Mapping with SWE is less aligned (more conflicts)
• possibly due to lack of descriptions
Questions?
10:45 Coffee break
Annotation process: analysis of the use cases
Final workshop: skills mapping pilot3 May 2018
STEP 4
0
5
10
15
20
25
Skills identified for each document
AUT identified SWE identified
Skills identified for each document
• Part of documents cannot be used for analysis
• Since they are not tagged by one of the MSs
• These are never the same (i.e. there is no document which would lack tagged by both MS)
• AUT on average identified more skills in each document
STEP 4USE CASE: PART 1
Example - CV_PR consultant05
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
MS PowerPoing Anwendungskenntnisse
MS PowerPoingapplication knowledge
… …
SMS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
21 Skills
7 Skills
Example - CV_PR consultant05
Starting point: SWE | Skills in sample
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
7 Skills
Example - CV_PR consultant05
Starting point: SWE | Skills in sample
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
7 Skills
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
5 Skills (2 skills lost)
Example - CV_PR consultant05
Starting point: SWE | Skills in sample
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
7 Skills
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Marknadsföring/PR Marketing / PR
MS Word MS Word
S
5 Skills (2 skills lost)
Example - CV_PR consultant05
Starting point: SWE | Skills to ESCO
5 Skills
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Marknadsföring/PR Marketing / PR
MS Word MS Word
S
Example - CV_PR consultant05
Starting point: SWE | Skills to ESCO
MS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Marknadsföring/PR Marketing / PR
MS Word MS Word
S
5 Skills
use spreadsheets software
public relations
draft press releases
plan marketing strategy
develop public relations strategies
use word processing software
…
S
24 Skills (0 skills lost)
Example - CV_PR consultant05
Starting point: SWE | Skills to AUT
use spreadsheets software
public relations
draft press releases
plan marketing strategy
develop public relations strategies
use word processing software
…
S
24 Skills
Example - CV_PR consultant05
Starting point: SWE | Skills to AUT
use spreadsheets software
public relations
draft press releases
plan marketing strategy
develop public relations strategies
use word processing software
…
S
24 Skills
Erstellung von Marketingunterlagen
Pressemitteilung
Grundkenntnisse Datenpflege
TextverarbeitungsprogrammeAnwendungskenntnisse
Online-PR
Marktstrategie
…
S
26 Skills (1 skill lost)
Example - CV_PR consultant05
Starting point: SWE | Compare to original AUT
Erstellung von Marketingunterlagen
Pressemitteilung
Grundkenntnisse Datenpflege
TextverarbeitungsprogrammeAnwendungskenntnisse
Online-PR
Marktstrategie
…
S
26 Skills
Example - CV_PR consultant05
Starting point: SWE | Compare to original AUT
Erstellung von Marketingunterlagen
Pressemitteilung
Grundkenntnisse Datenpflege
TextverarbeitungsprogrammeAnwendungskenntnisse
Online-PR
Marktstrategie
…
S
26 Skills
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
… …
S
21 Skills
Example - CV_PR consultant05
Starting point: SWE | Compare to original AUT
Erstellung von Marketingunterlagen
Pressemitteilung
Grundkenntnisse Datenpflege
TextverarbeitungsprogrammeAnwendungskenntnisse
Online-PR
Marktstrategie
…
S
26 Skills
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
… …
S
21 Skills
Example - CV_PR consultant05
Side step: Manual comparison
Example - CV_PR consultant05
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
… …
SMS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
21 Skills 7 Skills
Side step: Manual comparison
Example - CV_PR consultant05
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
… …
SMS Excel MS Excel
MS Power Point MS Power Point
Pressmeddelanden Press Releases
Krishantering Crisis management
Marknadsföring/PR Marketing / PR
MS Word MS Word
Korrekturläsning PROOFREADING
S
21 Skills 7 Skills
Side step: Manual comparison
Example - CV_PR consultant05
Side step: Manual comparison
• 6 skills can be identified
• Common according to manual comparison
• Even in optimal case the final result cannot transfer much more than 6 skill
• Only these skills are common in the starting tagged documents
Example - CV_PR consultant05
Starting point: SWE | Compare to original AUT
Erstellung von Marketingunterlagen
Pressemitteilung
Grundkenntnisse Datenpflege
TextverarbeitungsprogrammeAnwendungskenntnisse
Online-PR
Marktstrategie
…
S
26 Skills
Erstellung von Marketingunterlagen
Creation of marketing documents
Event-Management, Promotionorganisation
Event management, promotion organization
Krisenkommunikation crisis communications
Marktstrategie market strategy
Mediaplanung Media planning
MS Excel Anwendungskenntnisse
MS Excel application knowledge
… …
S
21 Skills
Example - CV_PR consultant05
Results
• Skills matched: 6
• Manually identified matches: 6
• Matching rate: 100%
• Automatically matched skills equal manually identified number
• Skill “noise”: 20 (26 – 6 matches)
• Skill “noise” rate: 77%
2,9
1,4
0
0,5
1
1,5
2
2,5
3
3,5
Average matched
Average number of correct matches
SWE > AUT AUT > SWE
77,50%
60,83%
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
90,00%
Average matching rate
Average % of success rate of the matches
SWE > AUT AUT > SWE
81,29%
91,95%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
90,00%
100,00%
Average noise ratio
Average % of noise introduced
SWE > AUT AUT > SWE
Conclusions
• Transformation provides 60/75% of matching rate between automated transformation and manual comparison
• Fairly accurate for automated process
• SWE > AUT was more accurate on all aspects compared to AUT > SWE
• Likely due to higher amount of tagged skills in the documents by AUT
• Transformation introduces high level of noise, this has to be taken into consideration when using the data for job matching
STEP 4USE CASE: PART 2
S
JV_PR consultant01
S
CV_PR consultant05
S
JV_PR consultant02
S
JV_PR consultant04
Rank 3
S
JV_PR consultant01
S
CV_PR consultant05
S
JV_PR consultant02
S
JV_PR consultant04
Rank 4*
S
S
S
*Rank 4 indicates that thereare no skill common betweenCV and JV
Analysis
• Accuracy
• Exact: 0• No documents have the same rank between the two cases
• +/- 1: 2• JV_PR consultant01 is Rank 2 in direct scenario and Rank 1 in transformed
• JV_PR consultant02 is Rank 3 in direct scenario and Rank 4 in transformed
• Both of these are off by 1 rank, therefore can be partially considered
73,08%
26,92%
65,38%
34,62%
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
+/-1 Exact
Average accuracy of ranking
AUT vs SWE>AUT SWE vs AUT>SWE
Conclusions
• Sample usable for ranking was rather small
• Results may not be representative
• If we aim for exact reproduction of the source ranking the transformed data achieve only 25/35% accuracy
• With “fuzzy” approach (+/-1 rank) the results achieve 73/65% accuracy
• Ranking using ESCO mapping provides value, but should not be considered 100% accurate
Questions?
Mapping process within the framework of the
EURES Regulation
Final workshop: skills mapping pilot3 May 2018
65
EURES
network and
portal
Cooperation for free
movement of workers
Human network of
1000 advisers
Information, recruitment, placement services
Art. 19 EURES Regulation
2016/589/EU
Better matching people to jobs
Increased IT interoperability
ESCO
Mapping framework
• Implementing acts - summer 2018• Mapping technical support: implementation manual, training
modules and webinars, helpdesk, the mapping platform -summer 2018
• " Support Package" in ESCO website• Mapping can start anytime, advisable for September
Complete framework that allows Member States to start mapping
Questions?