Artisan Profiler · Artisan Profiler In October 2017, trials were carried out using Profiler on tablet computers in a number of Power Stations across South Africa, with the

Contact: Gerald WilliamsonPhone: 011-682 1716

Email: [email protected]

Artisan Profiler

Information

Do-IT Profiler

Artisan ProfilerIn October 2017, trials were carried out using Profiler on tablet computers in a number of Power Stations across South Africa, with the aim to computerise a recruitment system for Artisans in industry. The benefits include rapid turnaround, less marking, and lower costs. It also provides standardisation and consistency across work sites and requirements with recruitment agencies.

The testing consists of the following components;

Trade theory testLiteracy testsNumeracy testsPractical tests

Trade theory testsThese were originally paper-based tests, which have now been converted to computerised versions. This includes making many of them into multiple choice (from open ended), largely carried out by STL. Most items have just a single point for each answer.

Literacy TestsThe original paper-based version had a number of different approaches including word knowledge and comprehension. These have been replaced by a series of separate tests which offer the opportunity to identify where are the specific problems. For example, it will be possible to say if the issue is literacy or second language.

Numeracy testThis test is made to evaluate numeracy skills required for the artisan jobs.

Practical testA system has been created which offers the opportunity for supervisors to score practical skill during the testing process.

Purpose of this document

What is Profiler?The Do-IT Profiler is software-as-a-web-based cloud-solution and service developed by Do-It Solutions Ltd to help identify skills, abilities, strength and weaknesses, monitor and evaluate over time, and offer support and recommendations based on data analysis at individual and institutional levels.

Key areas are:Measurement – Uses cognitive assessment and self-reporting to collect dataEvaluation – Uses complex algorithms developed with sector leadersReporting – Provides feedback and advice at individual and organisational levelsManagement – Offers extensive management capability at individual and organisation levels

Originally developed as an online service, the company has recently developed robust offline versions for intranet use (e.g. in prisons) as well as tablet standalone versions.

Profiler has Psychometric Properties

Data analytics, norming and psychometric properties

2

Test Order

Development of Phases 1 & 2 (3 & 4 in 2018)

The modules are presented in five separate “folders”, as shown below. (N.B. Numbers represent the total number of modules, and not the number of modules seen by the individual.

Priorities

To minimise time and misunderstanding, the first module only contains the two priority modules. If time is a real issue, there are the minimum requirements.

It is also possible that testing could stop at this point, since why test literacy and numeracy if they have failed these tests.

However, if they wish to re-apply, the could still do the literacy and numeracy skills modules, saving time on the next visit.

These modules have been customised to suit online

screening principles

3

Test Order



All these modules have been specially adapted and many had to be modified

in structure for computerised

testing. This took over 500 hours of

coding and development by

5 developers

4

Literacy


A series of modules have been put in place specifically aimed at the artisans recruited for Babcock. The intent is to provide guidance as to their literacy skills and potential limitations and learning needs. Currently literacy skills are not used for decision-making in the recruitment process.

A summary of the tests is provided in the panel to the right.

Action Point

It would seem reasonable to have a separate literacy criteria for each and every trade. This can only be determined once sufficient data has been collected and will be up for a review overthe next year

Modules

SpellingHelps identify English spelling skills

New word spellingWould help identify phonics skills and the ability to work with new vocabulary.

Word ChoiceProvides an indication of word knowledge.

Same / DifferentIndicate of the ability to hear words correctly, and implicitly a hearing test.

Reading comprehensionAllows the evaluation of short text passages

Which wordIdentifying the missing word in a sentence is a vocabulary and reading test.

Reading vocabularyThis test the individual at the word level, matching the written word to an image.

5

Test Order



Implemented

7 Nov 2017

6

Example: Pipe Fitters Theory Test Customisation


Pipe Fitters are the most common are of testing. Therefore this is being used as a benchmark for ensuring it is correct.

In this test, it appears an individual will be failed if they cannot answer correctly the four questions shown on the right.

To make the process quicker, the original questions 25 has be moved to question 4, and all scores recalculated.

However, a final decision needs to be made if the test should stop if they make one error in the first four questions.

7

Artisan Screener

Registration


The registration process has been simplified to speed up the process.

The System Chooses the Test Battery for the Incumbent

We have been able to programme into the system the following functionality:

When the person registers and chooses their career, the system will choose their test batteries according to their career

8

About You module


The About You module has been significantly adapted to ensure it is fit for purpose.

9

Pass / Fail Criteria


The graph below shows inconsistency in the way individuals pass or fail the qualification process. The criteria are being adjusted so that it is set at 65% for each trade. However, this can be adjust as evidence is accumulated.

A quick guide is being developed that shows if somebody has passed the theory test using the Bands option. This will mean that those who are in Band 4 (Green) will pass. However the criteria for other groups and what to do about them still need to be decided.

Suggested feedback

Band 4 (Green) – 65% or aboveResult: PASS

Band 3 (Blue) – 55%-64%Recommend further study and retry

Band 2 (Orange) – 54%-41%A lot more studying will be required before a retry is recommended.

Band 1 (Red) – Less then 40%You may wish to consider another professionThe bands for the Literacy, Numeracy and

Theory has ben agreed upon but needs to be statistically adjusted in 2018

10

Practical Testing


A new module has been demonstrated which is a way to collect data for Practical Testing when the testing is being carried out, saving the need to enter data later. In addition, several other questions have been added in careers which, within Profiler, add no more than an estimated 5 seconds per questions. This would allow additional “ready for work” questions to be added, which may aid the decision-making process. For example, one question was around punctuality and personal presentation. Scoring rubrics are being developed and refined on an on-going basis.

Action Points

Similar tests is available for all artisan professions. Where there are “must be able to do” activities, the scoring and decision-making process is mapped out clearly and consistently.

This was added to the scope Friday, 3 Nov 2017

11

Reporting


An excel download is available which, with a little calculation, can show the necessary results in terms of the theory results necessary for the decision of who passes.

Reporting currently has three elements: Literacy, Numeracy, Practical and Theoretical Test data.

Agency Registered DateDaysSin

ceDate Of

BirthAgeYear

sUserna

meFirstNa

meLastNa

meID

Number TradeLit

Spelling

Lit Spelling - Pass

Num Maths

Num Maths -

Pass Theoretical Trade Test Percent Passed

AdcorpBlu 2017-12-08 0

1965-01-01 52

flloydnero Flloyd Nero

6506165193087

Big Bore Pipe Fitter 69 Pass 97 Pass ID1208 - Pipe Fitter 76 Pass

AMT 2017-12-08 01977-01-01 40

andregolden Andre Golden

7702055204089

Quality Controller 97 Pass 100 Pass ID1203 - Eskom Site Interview 82 Pass

AMT 2017-12-08 01977-01-01 40

andregolden Andre Golden

7702055204089

Quality Controller 97 Pass 100 Pass ID1219 - Quality Controller Test - General 64 Fail

AMT 2017-12-08 01985-01-01 32

dikeledikhoba Dikeledi Khoba

8507290315080

Quality Controller 75 Pass 100 Pass ID1203 - Eskom Site Interview 82 Pass

AMT 2017-12-08 01985-01-01 32

dikeledikhoba Dikeledi Khoba

8507290315080


AMT 2017-12-08 01989-01-01 28

alwynlabuschagne Alwyn

Labuschagne

8904095072083

Quality Controller 100 Pass ID1203 - Eskom Site Interview 71 Pass

AMT 2017-12-08 01989-01-01 28

alwynlabuschagne Alwyn

Labuschagne

8904095072083

Quality Controller 100 Pass ID1219 - Quality Controller Test - General 64 Fail

AMT 2017-12-08 01988-01-01 29

mphomotlhacwi Mpho

Motlhacwi

8802060963080

Quality Controller 69 Pass 97 Pass ID1203 - Eskom Site Interview 65 Fail

AMT 2017-12-08 01988-01-01 29

mphomotlhacwi Mpho

Motlhacwi

8802060963080


Inqaba 2017-12-08 01980-01-01 37

khethang amos

Khethang Amos

8012286080087

Mechanical Fitter 9 Fail 97 Pas s ID1222 - Mechanical Fitter Test 60 Fail

Inqaba 2017-12-08 01978-01-01 39

mlindelibhukuva Mlindeli Bhukuva

7801025505086 Welders 22 Fail 94 Pass ID1214 - Welder B Class Test 75 Pass

Inqaba 2017-12-08 01987-01-01 30

mandlakayisecebekhulu

Mandlakayise

Cebekhulu

8710095918088 Tackler 12 Fail 100 Pass ID1202 - Tackler Theoretical Test 72 Pass

Inqaba 2017-12-08 01991-01-01 26

johandre due plessis

Johandre

Due plessis

9103155221081

Pipe Fitter 22 Fail 100 Pass ID1208 - Pipe Fitter 68 Pass

Inqaba 2017-12-08 01982-01-01 35

xolanikeswa Xolani Keswa

8201145459083 Welders 78 Pass ID1214 - Welder B Class Test 50 Fail

Inqaba 2017-12-08 01976-01-01 41

ramathulelaleeu

ramathulela leeu

7609225479089 Rigger 34 Fail 94 Pass ID1211 - Rigger Test 40 Fail

Inqaba 2017-12-08 01970-01-01 47

jonasletsoko Jonas Letsoko

701113537108

Mechanical Fitter 12 Fail 91 Pass ID1222 - Mechanical Fitter Test 60 Fail

Inqaba 2017-12-08 01980-01-01 37

sivuyile goodmanmafumana

Sivuyile goodman

Mafumana

8004275001083 Rigger 22 Fail 100 Pass ID1211 - Rigger Test 66 Pass

12

Artisan Profiler

Site Testing


All the tests are available online and a special App has been developed for offsite testing

No Marking – No Human ErrorFrom This: To This:

Thousands of candidates per day can be tested and scored in real-time on available devices. More testing devices means even up more scores available in real-time

13

Dashboards


Below are the activity results for Profiler.

These numbers below represent test users

14

Contact Information

15

Shaping the Learner is the service provider partner for Do-IT Solutions Profiler in South Africa, and is the support

team for all Profiler related enquiries.

Contact

Phone: 011-682 1716Email: [email protected]

Documents

Artisan Profiler · Artisan Profiler In October 2017, trials were carried out using Profiler on tablet computers in a number of Power Stations across South Africa, with the