Eindhoven University of Technology MASTER Improving the ... · Skill Training Programs at Vanderlande Eindhoven, 2015June E. (Elsa) Buijssen BSc Industrial Engineering & Innovation

Eindhoven University of Technology

MASTER

Improving the effectiveness of soft skill training progams at Vanderlande

Buijssen, E.

Award date:2015

Link to publication

DisclaimerThis document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Studenttheses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the documentas presented in the repository. The required complexity or quality of research of student theses may vary by program, and the requiredminimum study period may vary in duration.

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

https://research.tue.nl/en/studentthesis/improving-the-effectiveness-of-soft-skill-training-progams-at-vanderlande(8f08e6cc-a99f-490d-b80c-1c76b2bf08be).html

Improving the Effectiveness of Soft Skill Training Programs at Vanderlande

Eindhoven, June 2015

E. (Elsa) Buijssen

BSc Industrial Engineering & Innovation Sciences – TU/e 2012 Student identity number 0715130

In partial fulfilment of the requirements for the degree of

Master of Science In Operations Management and Logistics

Supervisors: dr. ir. P.A.M. Kleingeld, TU/e, HPM dr. S. Rispens, TU/e, HPM drs. D.J. Verheijden, Vanderlande Academy Manager

TUE: School of Industrial Engineering Series Master Thesis Operations Management and Logistics Subject headings: Competences, Soft Skills, Transfer of Training, Training Effectiveness, Training Evaluation, BOS survey

I

ABSTRACT

This master thesis project describes the result of an eight-month graduation project at Vanderlande, located in Veghel, the Netherlands. At Vanderlande, no empirical evidence is available that soft skill training programs will be effective. A format was designed for soft skill training programs at Vanderlande that contributes to the effectiveness of the intervention. Furthermore, a tool was developed to measure the effectiveness of soft skill training programs. On the basis of a detailed analysis of three soft skill training programs, best practices were established for the needs analysis; the evaluation of the quality of the soft skill training program; and for measuring its effectiveness.

II

PREFACE

This report is the result of a graduation project that has been conducted at Vanderlande in completion of the Master Operations Management and Logistics at Eindhoven University of Technology. I’m glad that I had the possibility to graduate within this company and I would like to thank all the people that made this project possible.

However, some people deserve some extra attention. Firstly, I would like to thank Ad Kleingeld for his support as my first supervisor. He is a very enthusiastic, flexible and experienced supervisor with a very kind personality. During our meetings he always provided me with a lot of feedback, which definitely strengthens my report. I’m very glad that I had the chance to be his graduate student. On top of that, he gave me the opportunity to travel six weeks around the USA at the time of my graduation, which enriched my life in a very positive way. In addition, I would like to express my gratitude to my second supervisor, Sonja Rispens, who provided a fresh perspective on my research, which was valuable during the writing of my report.

Furthermore, I would like to thank everyone at Vanderlande who contributed to a pleasant and valuable time during my graduation. In particular, I would like to thank my company supervisor, Dirk-Jan Verheijden, for his cooperation, insights, feedback and his ongoing support. Next to that, I want to thank the people from the Vanderlande Academy for the nice and cosy atmosphere. For me it became a real pleasure to graduate at Vanderlande. Furthermore, I want to express my appreciation to all the participants who were involved in the questionnaires and/or interviews. Their contribution was an important aspect for the successful completion of my master thesis project.

Above all, I would like to thank my parents, brothers and sister for their love and support in everything I do. Next to that, I would like to thank my friends for the great time I had with you on Thursday afternoons in the Villa, during dinners, and in the weekends. It provided me the necessary distraction during my graduation project. Thank you all! Elsa Buijssen Eindhoven, June 2015

III

MANAGEMENT SUMMARY

Research Background

At Vanderlande, a large variety of soft skill training programs are provided by both its own learning consultants and by external parties. Examples of soft skills are communication, negotiation, creativity and listening. However, at the moment there is no empirical evidence that soft skill training programs at Vanderlande work properly. Three phases were considered as important regarding the effectiveness of soft skill training programs: 1) needs analysis; 2) evaluating the quality of the soft skill training program; and 3) measuring the effectiveness of the soft skill training program. The Vanderlande Academy did not have sufficient insight into these phases to determine whether soft skill training programs work properly.

Research Objectives and research questions

Two research objectives were formulated for the current project. First of all, to design a format (i.e.

a checklist), including aspects that contributes to the effectiveness of soft skill training programs at

Vanderlande. Second, the design of a tool for measuring the effectiveness of soft skill training

programs at Vanderlande. This led to the following main research question: Which components

contribute to the effectiveness of soft skill training programs at Vanderlande? To provide an answer

to the research question and to maintain both research objectives, the following research sub

questions were formulated:

1. What is the best way to determine the gap between the start competence level and the

desired competence level of a Vanderlande employee?

2. What are the most important aspects for evaluating the quality of soft skill training programs

at Vanderlande?

3. How to measure the effectiveness of soft skill training programs at Vanderlande?

Theoretical Background

Soft skills are defined as the cluster of personality traits, social graces, facility with language,

personal habits, friendliness, and optimism that mark people to varying degrees. A lack of soft skills

is cited as the biggest reason for project failure (Bancino & Zevalkink, 2007).

Before conducting a soft skill training program, a solid needs analysis and organizational support are required. Training evaluation and training effectiveness are sometimes used interchangeably, but those are two separate constructs. Training evaluation is a methodological approach for measuring learning outcomes, where training effectiveness is a theoretical approach for understanding those outcomes (Alvarez, Salas, & Garofano, 2004). The suggested best practice for training evaluation is using Kirkpatrick’s hierarchy of evaluation. The model delineates four hierarchical levels of training outcomes: reaction, learning, behaviour, and results. In general, a soft skill training program may be considered effective, when its intended outcomes are achieved and when the appropriate training methods were used (Alvarez et al., 2004). With respect to Kirkpatrick’s model, a soft skill training program is considered effective when the participant experiences a positive reaction to the training program; when skills are learned during the training program; when behavioural change occurs; and when this change in behaviour results in achieving the predetermined goals and objectives.

The effectiveness of soft skill training programs will be enhanced by improving the transfer of training. Transfer of training is defined as the effective applicability and maintenance of trained knowledge, skills and attitudes in the appropriate work setting (Aguinis & Kraiger, 2009). Learning

IV

soft skills is reflected in Maslow’s four stages of learning model. The model suggested that one has to reach the final stage, showing unconsciously competent behaviour to actual learn (i.e. possess) soft skills. Moving to the final stage of learning can be achieved by improving the transfer of training. Furthermore, 70% of the lessons learned come from on-the-job experiences, reflected in the ’70:20:10’ ratio.

Finally, literature suggested to use 360-Degree Feedback to increase the reliability in measuring behavioural change. The use of 360-Degree Feedback reduces the subjectivity problem, because instead of relying on appraisals from a single source, an employer obtains evaluations from a variety of sources (e.g. supervisors, peers, and subordinates), who can observe a person’s on-the-job behaviours (DeNisi & Kluger, 2000). Method Evaluation research was determined as the research type for this master thesis project. Three soft skill training programs, all carried out at Vanderlande, were chosen as research objects. The ‘Collaboration IM/SA’ project (N=14) was about improving the collaboration between integration managers and system architects. The ‘sell valYou’ project (N=27), aimed to improve the sales skills of sales employees. The objective of the last project, ‘Professional Communication’ (N=8) was to learn communicating in a professional way. For every project, the three phases were analysed by attending and evaluating the soft skill training program and by measuring its effectiveness. Both qualitative and quantitative methods were used.

For the needs analysis, a self-assessment tool, an expert-assessment, and/or a supervisor-assessment were investigated to determine which method(s) was (were) most suitable.

The quality of the soft skill training program was evaluated by conducting interviews, using an observational checklist (developed for this project) and by analysing data obtained via the Vanderlande Academy evaluation form.

For measuring the effectiveness of the soft skill training program, a valid and reliable measurement tool was designed. This serial BOS (Behaviour Observation Scale) survey measures behavioural change, where participants had to score their behaviours on a five-point Likert scale. To measure behavioural change, a pre- and post-measurement was conducted. Next to the measurement tool, interviews with trainees and trainers were conducted to determine the effectiveness of soft skill training programs. Discussion of the results Needs analysis (sub research question 1)

A solid needs analysis is an important first step regarding the effectiveness of soft skill training programs. Before participating in a soft skill training program, one needs to know the start and desired level of the skill or competence that needs to be developed. A competence is defined as a learned ability to adequately perform a task, duty or role (Roe, 2002) At Vanderlande soft skill training programs will only be facilitated under the condition that in the current situation problems and conflicts arise or when a manager recognizes the need for improvement.

Different methods can be used to perform a needs analysis. First of all, a self-assessment, by

using the Competence Tool (known as the self-assessment tool) can be used. Next to that, a supervisor-assessment and/or an expert-assessment can be used. The obtained correlations suggest a high extent of agreement between the different methods. However, the best practice for the needs analysis is to use an expert-assessment next to the self-assessment and/or supervisor-assessment. An expert-assessment is associated with high costs and therefore the added value of an expert-assessment was investigated. The added value was expressed in the process of becoming

V

aware of the strengths and weaknesses by the participant. This process ensures that one already becomes ‘consciously incompetent’, which was required to finally possess soft skills. Quality of the soft skill training program (sub research question 2)

Evaluating the quality of the soft skill training program is important to investigate whether training objectives were achieved and whether it results in enhanced performance on the job. Based on interviews with trainees and trainers; and by participating and evaluating in three different soft skill training programs; a list of aspects was developed and used as the input for the observational checklist. The aspects that should be included within a soft skill training program to ensure the program to be successful are:

- Receiving enough feedback during the soft skill training program; - A highly motivated trainee; - Use of role-play; - Use of real-life cases (based on company context); - An enthusiastic trainer; - Interaction with other participants in order to learn from each other’s strengths and

weaknesses; - A sufficient balance between theoretical versus practical issues.

Measuring the effectiveness (sub research question 3)

To satisfy the second research objective, a reliable and validated tool was developed to measure the effectiveness of soft skill training programs. The tool, a serial BOS survey, scores behaviours on a five point Likert scale and measures the behavioural change before and after someone participates in the soft skill training program. The use of the serial BOS survey is enhanced by the finding that a relation exists between the trainees’ measured competence level and behaviour. Whether the behavioural change is statistically significant can be measured by using Wilcoxon Signed Rank Test.

Furthermore, the use of 360-Degree Feedback for conducting the BOS survey was analysed.

Based upon the obtained correlations, one can conclude that the different reviewers were not interchangeable. Therefore, 360-Degree Feedback is valuable and is considered as the best practice for measuring behavioural change in a reliable way. However, in case 360-Degree Feedback is not possible, only conducting a peer-review is considered as the best alternative. Conclusion The above results have been combined into a format containing best practices for every phase. The best practice for conducting a needs analysis is performing an expert-assessment together with a self-assessment and/or supervisor assessment, taking into account the profitability of using an expert-assessment. For evaluating the quality of a soft skill training program, the best practice is to check whether the indicated aspects are presented during the training program. Finally, for measuring its effectiveness, the best practice is to measure behavioural change with the designed tool, using the concept of 360-Degree Feedback.

Limitations and Recommendations Limitations that should be noted concern the generalizability of the study findings, due to the small sample sizes, and the reliability of the observational checklist. A recommendation to Vanderlande will be to improve transfer of training by taking into account: support from higher management, supervisors and peers; and by realising the opportunity for trainees to practice their learned skills on the job, related to the ’70:20:10’ ratio. Furthermore, a recommendation for future research is related to the tool for measuring behavioural change. To effectively apply the measurement tool for all training programs, also BOS scales need to be designed for the remaining competences.

VI

TABLE OF CONTENTS

ABSTRACT .................................................................................................................................. I

PREFACE .................................................................................................................................... II

MANAGEMENT SUMMARY ....................................................................................................... III

TABLE OF CONTENTS ................................................................................................................ VI

LIST OF FIGURES AND TABLES ................................................................................................. VIII

LIST OF DEFINITIONS................................................................................................................. IX

1. INTRODUCTION ................................................................................................................ 1

1.1. Company Description ................................................................................................................... 1

1.2. Research Background ................................................................................................................... 1

1.3. Problem statement, research objectives, research question .......................................................... 2

1.4. Report Outline ............................................................................................................................. 3

2. THEORETICAL BACKGROUND ............................................................................................ 4

2.1. Soft Skills ..................................................................................................................................... 4

2.2. The Relation between Soft Skills and Competences ....................................................................... 4

2.3. Evaluation of Training Programs ................................................................................................... 5

2.4. Factors that Contribute to the Effectiveness of a Training Program ................................................ 7

2.5. The Role of Learning and Feedback with respect to Transfer of Training ......................................... 9

2.6. Methods for Measuring the Effectiveness of Soft Skill Training Programs ..................................... 10

2.7. Summary of the theoretical background ..................................................................................... 11

3. RESEARCH APPROACH ..................................................................................................... 13

3.1. Double-Loop Learning ................................................................................................................. 13

3.2. Method ...................................................................................................................................... 14

3.3. Selection of the research object .................................................................................................. 15

3.3.1. The ‘Collaboration IM/SA’ project ......................................................................................................... 15

3.3.2. The ‘sell valYou’ project ......................................................................................................................... 17

3.3.3. The ‘Professional Communication’ project ............................................................................................ 19

3.4. Data Collection ........................................................................................................................... 19

3.4.1. Quantitative vs. Qualitative Research Methods .................................................................................... 19

3.4.2. Choice of Data Collection Methods ........................................................................................................ 20

4. METHOD: MEASURES AND DATA COLLECTION ................................................................. 22

4.1. Needs Analysis ........................................................................................................................... 23



VII


4.2. Evaluating the Quality of the Soft Skill Training Program ............................................................. 25




4.3. Measuring the Effectiveness of Soft Skill Training Programs ......................................................... 26

4.3.1. Design of the BOS survey ....................................................................................................................... 27




5. RESULTS .......................................................................................................................... 32

5.1. The ‘Collaboration IM/SA’ project (N=14) .................................................................................... 33

5.2. The ‘sell valYou’ project (N=27) ................................................................................................... 39

5.3. The ‘Professional Communication’ project (N=8) ......................................................................... 42

6. DISCUSSION ..................................................................................................................... 45

6.1. Design of the format ................................................................................................................... 45

6.1.1. Needs analysis ....................................................................................................................................... 45

6.1.2. Evaluating the quality of the soft skill training program ....................................................................... 46

6.1.3. Measuring the effectiveness of the soft skill training program ............................................................. 47

7. CONCLUSION, LIMITATIONS, AND RECOMMENDATIONS .................................................. 49

7.1. Conclusion ................................................................................................................................. 49

7.2. Limitations ................................................................................................................................. 50

7.3. Practical recommendations and future research .......................................................................... 50

7.3.1. Recommendations for Vanderlande ...................................................................................................... 51

7.3.2. Recommendations for future research .................................................................................................. 51

7.4. Concluding remarks .................................................................................................................... 52

8. REFERENCES .................................................................................................................... 53

APPENDIX I ............................................................................................................................... 56

APPENDIX II .............................................................................................................................. 57

APPENDIX III ............................................................................................................................. 58

APPENDIX IV ............................................................................................................................ 59

APPENDIX V ............................................................................................................................. 60

APPENDIX VI ............................................................................................................................ 62

VIII

LIST OF FIGURES AND TABLES

FIGURE 1: PROBLEM STATEMENT, RESEARCH OBJECTIVES, AND (SUB) RESEARCH QUESTIONS ....................................................... 2 FIGURE 2: KIRKPATRICK'S HIERARCHY OF EVALUATION (KIRKPATRICK & KIRKPATRICK, 2006)...................................................... 5 FIGURE 3: HRD EVALUATION RESEARCH AND MEASUREMENT MODEL (HOLTON, 1996) .......................................................... 6 FIGURE 4: BALDWIN AND FORD'S MODEL OF THE TRANSFER PROCESS (GROSSMAN & SALAS, 2011) ........................................... 7 FIGURE 5: LEARNING TRANSFER SYSTEM INVENTORY: CONCEPTUAL MODEL OF INSTRUMENT CONSTRUCTS (HOLTON ET AL., 2000) . 8 FIGURE 6: MASLOW'S STAGES OF LEARNING (CROSBIE, 2005) ............................................................................................. 9 FIGURE 7: THE MOST IMPORTANT FINDINGS FROM THE THEORETICAL BACKGROUND (BUIJSSEN, 2014) ...................................... 12 FIGURE 8: DOUBLE-LOOP LEARNING, BASED ON MORGAN (1997) ..................................................................................... 13 FIGURE 9: SALES PROCESS OF VANDERLANDE .................................................................................................................. 17 FIGURE 10: A CONCEPTUAL DIAGRAM OF CONTENT VALIDITY OF TRAINING PROGRAMS (GOLDSTEIN & FORD, 2002) .................... 25 FIGURE 11: PART OF THE BOS SURVEY REGARDING THE COMPETENCE 'PRESENTING' .............................................................. 30 FIGURE 12: FORMAT THAT CONTRIBUTES TO THE EFFECTIVENESS OF SOFT SKILL TRAINING PROGRAMS ........................................ 48 FIGURE 13: VANDERLANDE COMPETENCE SET WITH THE BARTRAM (2005) HIERARCHY, DERIVED FROM VAN DER HORST (2013) ... 58

TABLE 1: TYPOLOGY OF EVALUATION RESEARCH ............................................................................................................... 14 TABLE 2: OVERVIEW OF MEASURES AND DATA COLLECTION ................................................................................................ 22 TABLE 3: INTERPRETATION OF FLEISS' KAPPA .................................................................................................................. 28 TABLE 4: FLEISS' KAPPA VALUES ................................................................................................................................... 29 TABLE 5: CONCRETE BEHAVIOURS REGARDING THE COMPETENCE 'PRESENTING' ..................................................................... 29 TABLE 6: FINAL LIST OF CONCRETE BEHAVIOURS REGARDING THE COMPETENCE 'PRESENTING' ................................................... 30 TABLE 7: STRUCTURE FOR ANSWERING THE ANALYSIS QUESTIONS ........................................................................................ 32 TABLE 8: RELIABILITY OF THE BOS SURVEY FOR THE ‘COLLABORATION IM/SA’ PROJECT (N=14) .............................................. 33 TABLE 9: DESCRIPTIVE STATISTICS FOR THE 'COLLABORATION IM/SA' PROJECT (N=14) .......................................................... 34 TABLE 10: RANKING OF THE COMPETENCES (BASED ON MEAN VALUES) FOR THE 'COLLABORATION IM/SA' PROJECT ..................... 34 TABLE 11: CORRELATIONS BETWEEN SELF-REVIEW; PEER-REVIEW; SUPERVISOR-REVIEW .......................................................... 35 TABLE 12: ESTIMATION OF COMPETENCE LEVELS USING THE SELF-ASSESSMENT TOOL .............................................................. 36 TABLE 13: CORRELATION COEFFICIENTS BETWEEN BOS SURVEY AND SELF-ASSESSMENT TOOL ................................................... 37 TABLE 14: EVALUATION CRITERIA FROM EVALUATION FORM, AVAILABLE AT THE VANDERLANDE ACADEMY .................................. 38 TABLE 15: RELIABILITY OF THE BOS SCALE FOR THE 'SELL VALYOU' PROJECT (N=27) .............................................................. 39 TABLE 16: DESCRIPTIVE STATISTICS FOR THE 'SELL VALYOU' PROJECT (N=27) ........................................................................ 40 TABLE 17: CORRELATION BETWEEN SELF-REVIEW AND PEER-REVIEW BOS SURVEY FOR EACH SEPARATE COMPETENCE ................... 40 TABLE 18: CORRELATIONS BETWEEN SELF-ASSESSMENT; EXPERT-ASSESSMENT; AND SUPERVISOR-ASSESSMENT ............................ 41 TABLE 19: DISTRIBUTION OF THE NEGATIVE DEVIATIONS OF ONE LEVEL BELOW THE DESIRED LEVEL (N=29 CASES) ........................ 41 TABLE 20: RELIABILITY OF THE BOS SCALE FOR THE 'PROFESSIONAL COMMUNICATION' (N=8) ................................................. 43 TABLE 21: DESCRIPTIVE STATISTICS FOR THE 'PROFESSIONAL COMMUNICATION' PROJECT (N=8)............................................... 43 TABLE 22: CURRENT AND DESIRED COMPETENCE LEVELS FOR THE 'COLLABORATION IM/SA’ PROJECT ........................................ 56 TABLE 23: DESIRED COMPETENCE LEVELS FOR THE 'SELL VALYOU' PROJECT ............................................................................ 56

IX

LIST OF DEFINITIONS

Construct Definition

Competence A competence is a learned ability to adequately perform a task, duty or role (Roe, 2002, p. 195).

Effectiveness The extent to which the competence level developed during a training program results in the desired behaviour and outcomes on the work floor.

Hard skills The technical requirements of a job.

Soft skills The cluster of personality traits, social graces, facility with language, personal habitats, friendliness, and optimism that mark people to varying degrees. Soft skills complement hard skills (Bancino & Zevalkink, 2007).

Training effectiveness The study of the individual, training, and organisational characteristics that influence the training process before, during, and after training (Alvarez, Salas, & Garofano, 2004).

Explicate why training did or did not achieve its intended outcomes (i.e. identifying and measuring the effects of individual, organisational, and training-related factors on training outcomes such as learning and transfer of training) (Kraiger, Ford, & Salas, 1993).

Training evaluation Refers to a method for measuring whether trainees have achieved learning outcomes (e.g. measurement and design, the accomplishment of learning objectives, and the attainment of requisite knowledge and skills) (Kraiger et al. 1993).

Transfer performance The effective applicability of trained knowledge, skills and attitudes in the appropriate work setting and maintenance of these trained knowledge, skills, and attitudes (Aguinis & Kraiger, 2009).

Training program A systematic acquisition to affect KSAs (Knowledge, Skills and Attitudes) of individuals in order to improve the effectiveness of the individual, the team, and the organisation (Aguinis & Kraiger, 2009).

Training performance Training performance goes beyond learning by requiring that trainees show that they can incorporate the knowledge they have acquired into their actions (Tannenbaum, Cannon-Bowers, Salas, & Mathieu, 1993).

Training utility The usefulness of a training program in terms of benefits to the organisation (Birati & Tziner, 1999).

1

1. INTRODUCTION

Vanderlande has defined a problem that will be used as the input for this master thesis project. The problem concerns the area of soft skills. In Section 1.1, the company is briefly introduced. The reason for conducting this master thesis project, including the research assignment, is discussed in Section 1.2.

1.1. Company Description This master thesis project was conducted at Vanderlande.

The company was founded in 1949, and currently has offices in over twenty countries. More than 3.000 people are employed and the order book at the end of the financial year 2014 stood at an all-time high of more than €1 billion (Vanderlande, 2014). Vanderlande is an internationally oriented company and one of the largest and most advanced in the market, with extensive experience in many industries and is active in all major regions of the world.

The company provides automated material handling systems and services. Vanderlande is active in the markets for Baggage Handling at airports, Warehouse Automation, and sorting solutions in Parcel and Postal facilities. The company implements material handling systems of all sizes, ranging from local sorting depots, airports and distribution centres to the world’s largest facilities.

Vanderlande strives for close cooperation with the customer, extending from initial analysis of the underlying business processes through the total life cycle support (Vanderlande, 2014). The mission of Vanderlande is to support its customers in continuously improving their competitiveness by designing, realizing, managing and optimizing Automated Material Handling Processes (Vanderlande, 2014).

The Vanderlande Academy is the department of Vanderlande that is responsible for the development of employees by providing training programs. The Vanderlande Academy intends to enhance the employees’ personal development as well as their job-oriented knowledge, skills and attitudes.

1.2. Research Background The aim of the Vanderlande Academy is to support and challenge every employee in his or her personal development. The Vanderlande Academy offers a large variety of training programs for their employees, including both hard skill and soft skill training programs. The emphasis within this master thesis will be on soft skill training programs. Examples of soft skills are communication, negotiation, creativity and listening.

A training program is defined as a systematic acquisition to affect Knowledge, Skills, and Attitudes (KSAs) in the appropriate work setting and maintenance of these trained knowledge, skills, and attitudes (Aguinis & Kraiger, 2009). Offering training programs became more and more important due to the changing business world, in which the demand for a broader set of skills is increasing. Next to that, due to the huge growth of Vanderlande, more and more people need to be employed and all these new employees have a need for training.

Furthermore, the applicability of the trained knowledge, skills, and attitudes to the work setting, also known as transfer of training, is an important aspect within the effectiveness of training programs that should be taken into account.

2

At Vanderlande, soft skill training programs are provided by both its own learning consultants and by external parties. However, at the moment there is no empirical evidence that soft skill training programs at Vanderlande work properly. This is due to a number of reasons: The effectiveness of soft skill training programs is not measured; Compared to hard skills, it is hard to determine the effectiveness of soft skill training programs; It is not clear in which way soft skills are transferred to the work floor; Current literature provides few leads to the measurement of soft skills.

Before participating in a soft skill training program, one needs to know the start level and the desired level of the skill or competence that needs to be developed. A competence is defined as a learned ability to adequately perform a task, duty or role (Roe, 2002). The Vanderlande Academy already developed a competence set with in total 48 behavioural competences. Based on this set of competences, one can select the competences that need to be improved within a soft skill training program. Furthermore, for determining someone’s start competence level, the Competence Tool, developed by van der Horst (2013) can be used. The Competence Tool and the relation between competences and soft skills will be later discussed in more detail. A soft skill training program will only be facilitated under the condition that in the current situation problems and conflicts arise or when a manager recognizes the need for improvement. Therefore, the first step will be taken by the supervisor who recognizes the need for training.

This section discussed the research background and this will be the input for formulating the problem statement, the research objectives, the research question and the ensuing sub research questions.

1.3. Problem statement, research objectives, research question The problem statement for Vanderlande, the research objectives, the research question and the ensuing sub research questions are shown in Figure 1. Figure 1: Problem statement, research objectives, and (sub) research questions

Therefore, to fulfil the need for Vanderlande, a format which includes components that contributes to the effectiveness of soft skill training programs is required. The design of such a format and a tool for measuring the effectiveness of soft skill training programs are therefore determined as research objectives for this master thesis project.

3

1.4. Report Outline In the current Chapter, the research topic, problem statement, and research questions were

introduced.

Chapter 2 provides an overview of the relevant scientific literature on the topics of soft skills

and training programs. The theoretical background also covers techniques for evaluating and

measuring the effectiveness of soft skill training programs.

Chapter 3 discusses the research approach by elaborating double-loop learning, defining the

method and the possibilities for data collection.

In Chapter 4, the method is further elaborated by discussing the measures and data collection

for each of the three phases, considered relevant regarding the effectiveness of soft skill training

programs: (1) needs analysis, (2) evaluating the quality of soft skill training programs, and (3)

measuring the effectiveness of soft skill training programs.

Chapter 5 discusses the results of this research, related to the measurements discussed in the

method section.

Chapter 6 discusses the conclusions for each research sub question separately. Next to that,

the designed format including the aspects a soft skill training program should satisfy to contribute to

the effectiveness of the intervention is discussed.

The final Chapter discusses the conclusion, limitations and recommendations of the

conducted research, distinguishing between recommendations for Vanderlande and

recommendations for future research.

4

2. THEORETICAL BACKGROUND

As already discussed in the research background, the Vanderlande Academy asks for 1) a format which includes components that contribute to the effectiveness of soft skill training programs and 2) a tool for measuring the effectiveness of soft skill training programs. Therefore, this Chapter provides a theoretical background on soft skills, evaluation models, and methods for evaluating and measuring the effectiveness of soft skill training programs.

2.1. Soft Skills Soft skills are defined as the cluster of personality traits, social graces, facility with language, personal habitats, friendliness, and optimism that mark people to varying degrees (Bancino & Zevalkink, 2007, p. 20). Examples of soft skills are communication skills, listening, creativity and negotiating skills. Soft skills are the intangible and nontechnical skills that determine one’s strengths, where hard skills are known as the technical requirements of a job (Robles, 2012). Anecdotal evidence has emphasized that soft skills learned during training are significantly less likely to transfer from training to the job compared to hard skills learned during training (Laker & Powell, 2011). Furthermore, it is stated that this lack of soft skill transfer to the job results in an extremely costly waste of time, energy and money (Laker & Powell, 2011).

Laker and Powell (2011) already emphasize the importance of increasing the effectiveness of soft skill training programs. On top of that, soft skills are nowadays becoming increasingly important, also for employees in jobs in which soft skills traditionally were considered unimportant, such as technical jobs at Vanderlande. There are three driving forces behind business leaders’ increasing demand for a broader skill set from technical professionals: necessity for improvements to the bottom line; increasing competition; and globalization (Bancino & Zevalkink, 2007).

Bancino and Zevalink (2007) state that more and more technology-intensive projects tend to have high failure rates. Following a survey of more than 250 technical leaders, a lack of soft skills is cited as the biggest reason for project failure (Bancino & Zevalkink, 2007). An explanation would be that when nontechnical skills are developed to complement technical skills, personal productivity, collaboration and synergy are increased, which translates into improved project success rates, sustainable competitive advantage and increased profitability.

2.2. The Relation between Soft Skills and Competences Soft skills and competences are frequently used interchangeably, where a variety of definitions exist for both soft skills and competences. A competence is defined as a learned ability to adequately perform a task, duty or role (Roe, 2002). In contrast, a soft skill includes personality traits as discussed in the previous section. Competences can be learned, because, unlike personality traits, competences are the characteristics of individuals that are more workable and can therefore be developed or improved (Intagliata, Ulrich, & Smallwood, 2000). The model of Verheijden (2005) agrees that competences can be developed and further suggests that competences are expressed in behaviour. However, the extent to which competences can be developed and learned varies from one person to the other and partly depends on relatively stable personal factors (Verheijden, 2005).

Based upon the available definitions, it seems that competences, compared to soft skills, are more often associated with an accomplishment. However, in literature both terms are frequently used interchangeably and it became clear that a certain common ground exist between both constructs. For the purpose of this research, a soft skill is used as a more overarching concept and if a soft skill is constructed of more than one competence, the different competences need to be taken into account when measuring the effectiveness of soft skill training programs. It should be noted that also within this master thesis project both terms are used interchangeably. However, a clear

5

distinction between hard skills and soft skills is made, where this master thesis project only focusses on soft skills.

2.3. Evaluation of Training Programs Training evaluation is the measurement of a training program’s success or failure with regard to content and design, changes in learners, and organisational payoffs (Alvarez et al., 2004). Evaluating training programs is important, because it provides information that can be used to improve future training programs; one can determine whether a program should be continued, dropped or improved; and one can justify the existence of the training department and its budget (Kirkpatrick & Kirkpatrick, 2006).

“Evaluation is conducted to answer either of two questions: whether training objectives were achieved (learning issues), and whether accomplishment of those objectives results in enhanced performance on the job (transfer issues)” (Kraiger et al., 1993, p. 133). Transfer of training seems therefore an important factor within this master thesis project. However, to make transfer happen, training objectives, such as learning, need to be achieved.

Most of the models used to evaluate training programs do not consider a conceptual basis for evaluating training. Within these models, learning is measured by examining the extent to which trainees have acquired relevant principles, facts, or skills (Kraiger et al., 1993). The four well-known models for evaluating training programs are: Kirkpatrick’s four-dimensional measurement typology; an extension on Kirkpatrick’s model; Holton’s model; and Kraiger’s model (Alvarez et al., 2004). The evaluation models are related to four dimensions, suggested by Kirkpatrick’s (2006) model: (1) reactions; (2) learning; (3) behaviour; and (4) results. Kirkpatrick’s model is the most widely known model for training evaluation and is shown in Figure 2.

Kirkpatrick’s training evaluation model delineates four hierarchical levels of training outcomes: reaction, learning, behaviour, and results. Level one includes assessment of training participants’ reaction to the training program (Bates, 2004). Level two, evaluation of learning, is about quantifiable indicators of the learning and takes place during the training program (Bates, 2004). The third level, behaviour outcomes, address the extent to which knowledge and skills gained in training is applied on the job or results in exceptional job-related performance (Bates, 2004). This level is also known as the transfer of learning to the workplace, in other words, transfer of training. Paying attention to transfer of training is important with respect to the “70:20:10 ratio”, which can be applied to both hard skills and soft skills. The ratio states that 70% of the lessons learned come from on-the-job experiences. The final level is about the evaluation of results and is intended to provide some measure of the impact that training has had on broader organisational goals and objectives (Bates, 2004).

Figure 2: Kirkpatrick's hierarchy of evaluation (Kirkpatrick & Kirkpatrick, 2006)

6

In general, a soft skill training program may be considered effective, when its intended outcomes are achieved and when the appropriate training methods were used (Alvarez et al., 2004). With respect to Kirkpatrick’s model, a soft skill training program is considered effective when the participant experiences a positive reaction to the training program; when skills are learned during the training program; when behavioural change occur; and when this change in behaviour results in achieving the predetermined goals and objectives.

Furthermore, Tannenbaum expanded on Kirkpatrick’s typology by adding post-training attitudes and by dividing behaviour into two outcomes for evaluation: training performance and transfer performance (Tannenbaum & Yukl, Training and development in work organizations, 1992).

Holton, Bates, and Ruona (2000) elaborate on Kirkpatrick’s model by adding external factors that will affect the transfer of training. Reactions are excluded from Holton’s model, because they are not considered as a primary outcome of training (Alvarez et al., 2004). Holton et al. (2000) states that three crucial factors affect transfer of training (i.e. motivation to transfer; transfer climate; and transfer design). Holton’s model has been extended and is transformed into the HRD Evaluation and Research Model, shown in Figure 3. The model makes the distinction between secondary influences, motivational elements, environmental elements, outcomes, and enabling elements. Transfer of training seems to be an essential part within the HRD Evaluation and Research Model.

Figure 3: HRD Evaluation Research and Measurement Model (Holton, 1996)

A fourth evaluation model is Kraiger’s model, in which reactions are not considered as a dimension, such as within Kirkpatrick’s model. However, in this model, reactions are considered as a ‘measurement technique’ for evaluating the training program (Alvarez et al., 2004). Following this model, reactions can be used to provide insight into the quality of a training program. Kraiger states that reactions can be used for determining how effective training content and design were for the tasks to be learned (Alvarez et al., 2004). In other words, from the participant’s reactions one can evaluate the training content and design and the way participants experienced the training program.

Therefore, in order to evaluate training programs, besides the dimensions from Kirkpatrick’s model, also secondary influences (i.e. personality characteristics); motivation elements (i.e. motivation to learn and motivation to transfer); and environmental elements (i.e. transfer climate) should be taken into account.

7

2.4. Factors that Contribute to the Effectiveness of a Training Program Training evaluation is a methodological approach for measuring learning outcomes, where training effectiveness is a theoretical approach for understanding those outcomes (Alvarez et al., 2004). Alvarez et al. (2004) define training effectiveness as the study of the individual, training, and organisational characteristics that influence the training process before, during, and after training.

A training program cannot be effective unless it meets the individual, organisational, and task needs as identified by needs analysis (Alvarez et al., 2004). A needs analysis is a process that includes organisational support, organisational analysis, requirements analysis, tasks and KSA analysis and a person analysis (Goldstein & Ford, 2002). In other words, one needs to determine the knowledge, skills and attitudes that need to be developed during the training program.

Three sets of characteristics are generally known as contributions to training effectiveness (Alvarez et al., 2004): (1) Individual characteristics; (2) organisational characteristics and situational characteristics; and (3) training characteristics. Individual characteristics include the factors that trainees bring to the situation, such as personality traits, attitudes, abilities, demographics, experience, and expectations (Alvarez et al., 2004). Organisational characteristics and situational characteristics include the context in which training is implemented, such as the organisation’s climate for learning, history, policies, trainee selection technique, and trainee notification process (Alvarez et al., 2004). Support from the work environment is crucial to make transfer of training happen. Training characteristics include the aspects of the training program, such as instructional style, practice, and feedback (Cannon-Bowers, Salas, Tannenbaum, & Mathieu, 1995).

To clarify these characteristics, three leading effectiveness models will be discussed. Baldwin and Ford’s model suggests that individual and organisational characteristics are directly related to learning and transfer performance, whereas all three sets of characteristics have an indirect relationship with transfer performance through learning (Baldwin & Ford, 1988). Furthermore, transfer is facilitated when trainees are motivated to learn and transfer throughout the training process (Baldwin, Ford, & Blume, 2009). An adapted version of Baldwin and Ford’s model of the transfer process is shown in Figure 4.

Figure 4: Baldwin and Ford's model of the transfer process (Grossman & Salas, 2011)

8

The trainees’ ability to use their newly acquired competences on the job is characterized as the most important factor that contributes to training effectiveness (Salas, Wilson, Priest, & Guthrie, 2006). If a trainee cannot use the learned skills in the work environment, a positive transfer outcome will not be achieved and all the effort put in designing an effective training program became worthless. Examples of training strategies within training design that will contribute to the effectiveness of a training program include behaviour modelling, error management, and the use of a realistic training environment (Grossman & Salas, 2011). The use of real-life cases and role-plays with actors mainly seems a well strategy to contribute for effective soft skills training programs (Cole, 2008).

An extension to Baldwin & Ford’s model is known as Holton’s model, which suggests that the three sets of characteristics are directly related to learning and transfer performance where also indirect relationships exist due to the interactions between the characteristics (Alvarez et al., 2004). The integration of evaluation and effectiveness within the Holton’s model is translated into the Learning Transfer System Inventory (Holton et al., 2000), and is shown in Figure 5.

Transfer system is defined as all factors in the person, training, and organisation that influence

transfer of learning to job performance (Holton et al., 2000). Transfer can only be completely understood and predicted when all influences are examined. Examples of factors include training design, personal characteristics, opportunity to use training, and motivational influences (Holton et al., 2000).

Finally, Broad and Newstrom’s model clarifies the process nature of training effectiveness by prescribing strategies that organisations can implement before, during, and after training to enhance training outcomes (Alvarez et al., 2004). They suggest that paying attention to trainees by peers and supervisors is most important before and after the training program (Broad & Newstrom, 1992).

Figure 5: Learning Transfer System Inventory: Conceptual Model of Instrument Constructs (Holton et al., 2000)

9

Therefore, creating an environment in which learning and transfer of training is stimulated are also crucial for achieving a high effectiveness of a training program.

Baldwin and Ford (1988), Holton et al. (2000), and Broad and Newstrom (1992) suggested factors that contribute to the effectiveness of training programs. Baldwin and Ford’s model suggests that individual and organisational characteristics are directly related to learning and transfer performance, whereas all three sets of characteristics have an indirect relationship with transfer performance through learning. They mention the trainees’ ability to use their newly acquired competences on the job as the most important factor that contributes to training effectiveness.

Holton et al (2000) agreed upon Baldwin and Ford’s model, but suggested that transfer can only be completely understood and predicted when all influences (i.e. training design, personal characteristics, and motivational influences) are examined. Furthermore, Broad and Newstrom focused on strategies before, during, and after training to enhance training outcomes. They mentioned that paying attention to trainees by peers and supervisors is most important before and after the training program. Overall, all three models suggested facilitating transfer as an important factor regarding the effectiveness of a training program.

2.5. The Role of Learning and Feedback with respect to Transfer of Training As already suggested in the previous sub sections, learning and feedback might also be important in the process of transfer of training. Learning soft skills is defined as achieving unconsciously competent behaviour and seems to be more complex compared to learning in general (Crosbie, 2005). Maslow identifies four stages of learning as shown in Figure 6. .

Figure 6: Maslow's stages of learning (Crosbie, 2005)

Maslow’s contention is that learners begin unconsciously incompetent and when someone is

presented with new material, one realizes the knowledge and skills he/she does not possess (Crosbie, 2005). At this point in time, the learner is consciously incompetent and after participating in a training program one becomes consciously competent. When the participant displays the learned behaviours frequently enough, one becomes unconsciously competent and at this moment the soft skill is learned. One can retrain the brain to naturally respond in new ways, but navigating through the four stages of learning takes a lot of time and practice (Crosbie, 2005). If the participant is unable to perform the step from exerting consciously competent behaviour to exerting unconsciously competent behaviour, he/she falls back in the learning process and has to start all over again to become unconsciously competent (Crosbie, 2005). Furthermore, to learn soft skills,

10

one has to reach the final stage, because being stuck at the third stage (i.e. consciously competent) will cost a lot of effort and will become frustrating, because of the need to consciously exercise a particular behaviour all the time. From Maslow’s stages of learning model it became clear that the process after the training program seems crucial. The learning process needs to continue after the training program is finished in order to reach unconsciously competent behaviour. A feasible transfer environment is therefore crucial to reach this final stage of learning. Therefore, compared to hard skills, the learning process will lasts much longer for soft skills.

The learning process after training is also reflected in the previously mentioned ‘70:20:10’

ratio, which suggests that 70% of the lessons learned come from on-the-job experiences and emphasizes the importance of transfer of training. The other 30% in the ‘70:20:10’ ratio model is expressed by: coaching and feedback (approximately 20%); and by the actual learning during the training program (approximately 10%).

Furthermore, receiving feedback is important to be aware of the importance of training and to determine if one is on the right track. Receiving feedback affects behaviour and will positively influence the transfer of training (DeNisi & Kluger, 2000). The use of 360-Degree Feedback influences the effectiveness of feedback. Instead of relying on appraisals from a single source, an employer obtains evaluations from a variety of sources (e.g. supervisors, peers, subordinates, customers), who can observe a person’s on-the-job behaviours (DeNisi & Kluger, 2000). The use of 360-Degree Feedback reduces the subjectivity problem, which arises within the evaluation of someone’s performance. Furthermore, the use of multiple sources has a number of advantages: enhanced ability to observe and measure various job facets; greater reliability; fairness; and rate acceptance (Harris & Schaubroeck, 1988).

2.6. Methods for Measuring the Effectiveness of Soft Skill Training Programs

As already mentioned, hard skills are specific, teachable abilities that can easily be defined and measured. By contrast, soft skills are less tangible and harder to quantify (Bronson, 2007). The measurement of the effectiveness of soft skill training programs is therefore difficult and will be one of the biggest challenges within this master thesis project.

To measure soft skills, one needs to understand what contributes to the transfer of soft skills, because it is commonly cited that only around ten percent of training translates into job performance (Cheng and Ho 2001; Brinkerhoff and Gill 1994; Kurkpritz 2002). Transfer is particular low for problem solving and interpersonal skills, both examples of soft skills (Kupritz, 2002). An explanation might be that, related to soft skills, someone had to frequently exercise the desired behaviour to finally unconsciously show the desired behaviour. Therefore, the phase after participating in a soft skill training program is crucial for exercising the desired behaviour. This emphasizes the importance to further investigate the transfer of soft skills. Gulpin-Jackson and Bushe (2007) combined the most common factors to come up with five categories that contribute to the effectiveness of soft skill training programs:

1. Social support, social support seems necessary for a positive transfer of training. 2. Adoption environment, a climate is described as favourable for transfer of training if there is an

“appreciation for performance and innovation” (Burke & Baldwin, 1999, p. 229). 3. Continuity and maintenance, describes characteristics of the post-training environment that

support long-term maintenance of the learned capabilities. 4. Situational context is meant by the situation surrounding possible transfer events in the work

environment. If participants do not have the opportunity to use their skills at work, cognitive learning seems insufficient for transfer (Gilpin-Jackson & Bushe, 2007)

5. Systemic forces include several organisational system/structural aspects such as language use and cultural norms that will affect transfer of training.

11

Besides understanding the factors that contribute to the transfer of soft skills, a triangulation approach (i.e. using surveys, interviews, and observations) can be a valuable method for measuring the effectiveness of soft skill training programs. Many researchers emphasize that using a triangulation approach allows for stronger results, as the qualitative and quantitative data allows for confirmation and corroboration of the research findings (Kupritz, 2002). Furthermore, when executing a quasi-experimental research, threats like the Hawthorne effect and the extraneous variable ‘history’ seems to be important to take into account. The Hawthorne effect is defined as behavioural change as a result of awareness of being a subject in an experiment (Adair, 1984). Specific events that occur between the first and second measurement is meant by the extraneous variable ‘history’.

From the literature findings, it would appear suitable to use the Kirkpatrick’s model as starting point for assessing training, where the focus will be on the last two levels (i.e. evaluation of behaviour, and evaluation of results). Most likely, the best method for measuring the effects of soft skill training programs will be to measure behavioural change. Behaviour Anchored Rating Scales (BARS) or Behaviour Observation Scales (BOS) are widely known as reliable and validated methods for measuring behaviour. BARS are sometimes called “behavioural expectation scales”, because they occasionally ask the assessor to describe what a worker might be expected to do in a hypothetical situation (Landy & Conte, 2007). “In BARS evaluation, the rater picks a point on the rating scale that describes either how the individual has behaved in the past, or absent any direct observation of relevant behaviour by the rater, how the individual might be expected to behave (Landy & Conte, 2007, p. 231).” BOS is developed by Latham and Wexley (1994), where “the critical incidents on a BOS comprise a ‘behavioural inventory’ suitable for employee development” (Latham & Wexley, 1994, p. 86). BOS asks appraisers to report how frequently employees engage in specific behaviours. Afterwards, adding up all the ratings yields an overall measure of performance. The BOS method grew out of the idea that it would be more accurate to have assessors evaluate what someone actually did rather than what he or she might do (Landy & Conte, 2007).

BOS is preferred over BARS, because assessors find it easier to use. For example, problem occurs with BARS, when the incidents shown on the form, don’t correspond to any behaviour the assessor has observed (Jackson, Schuler, & Werner, 2009). Furthermore, BOS is highly regarded in terms of stakeholder perceptions and applicability for development appraisal (Jelley, Goffin, Powell, & Heneman, 2012). Literature made a distinction between serial BOS versus parallel BOS. A serial BOS encourages assessors to consider relative judgements of specific rate behaviours, where the parallel BOS asked assessors to rate all employees on a given behaviour, before rating all employees on the next behaviour (Jelley et al., 2012). Serial BOS seems to be a more accurate method, because with parallel BOS, the risk of comparing one’s score directly with the score of the rest of the group may occur.

2.7. Summary of the theoretical background This chapter will be concluded with a model presenting the most important findings from the theoretical background, as shown in Figure 7. Important findings include the lack of soft skills as the biggest reason for project failures within a technical oriented company. Furthermore, it became clear that a solid needs analysis, transfer of training, and Maslow’s stages of learning should be taken into account regarding the effectiveness of soft skill training programs.

12

Figure 7: The most important findings from the theoretical background (Buijssen, 2014)

13

Figure 8: Double-Loop Learning, based on Morgan (1997)

Determine start

competency level

Determine desired

competency level

Determine if gap is

correctly defined

Attending soft skill

training program

Evaluate soft skill

training program

Measuring the

effectiveness of

soft skill training

program

Design a format

for soft skill

training programs

Single-loop learning

Double-loop learning

3. RESEARCH APPROACH

The problem at Vanderlande, already mentioned as the problem statement, is formulated as ‘there is no empirical evidence that soft skill training programs at Vanderlande are effective’. The research goal for this master thesis project, which logically follows from the problem statement, is to design a format, which includes aspects that contributes to the effectiveness of soft skill training programs. Furthermore, a tool will be developed to measure the effectiveness of soft skill training programs at Vanderlande.

The theoretical background emphasizes the importance of soft skills. Therefore, recognizing the importance of soft skills by Vanderlande seems to be beneficial for the work effectiveness of all employees. This finding emphasizes the importance of this master thesis project.

From the theoretical background, three phases deemed to be relevant regarding the effectiveness of soft skill training programs: (1) needs analysis, (2) evaluating the quality of the training program, and (3) measuring the effectiveness of the soft skill training program and transfer performance. As already introduced in the research background, the research question for this project is formulated as follows:

To properly answer this research question, the following sub questions, already discussed in Section 1.2, are formulated and are based upon the findings from the theoretical background and the interviews that were conducted during the problem analysis.

1. What is the best way to determine the gap between the start competence level and desired competence level of a Vanderlande employee?

2. What are the most important aspects for evaluating the quality of soft skill training programs at Vanderlande?

3. How to measure the effectiveness of soft skill training programs at Vanderlande?

3.1. Double-Loop Learning The research background (Section 1.2) mentioned a number of reasons why there is no empirical evidence that soft skill training programs at Vanderlande work properly. The aim for this master thesis project is to come up with a solution and the approach is broadly shown in Figure 8.

RQ: Which components contribute to the effectiveness of soft skill training programs at Vanderlande?

14

From the theoretical background it became clear that the first step needs to be a solid needs analysis in order to determine the skills, knowledge, and attitudes that need to be learned during the training program. In other words, one has to determine the start competence level and the desired competence level for the competence that will be trained in the soft skill training program. After the gap between the start competence level and the desired competence level is determined, the soft skill training program will be attended and evaluated. Furthermore, the effectiveness of the soft skill training program will be measured. Finally, a format, including aspects that increases the probability that soft skill training programs will be effective, can be designed.

Following Figure 8, the approach for this master thesis project includes both single-loop learning and double-loop learning. The research question reflects the double loop, where the single loop serves as input. Morgan (1997) defines single-loop learning as generating an ability to scan the environment, determine goals, and monitor the performance of the system in relation to these goals. The type of learning emphasizes the ability of an organisation to detect and correct mistakes in relation to a given set of operating norms (Morgan, 1997). Double-loop learning occurs when errors are corrected by changing the governing values and then the actions, in other words, by questioning the relevance of operating norms (Argyris, 2002). In contrast to single-loop learning, double-loop learning involves critical reflection upon goals, beliefs, values, conceptual frameworks, and strategies (Argyris, 2002).

Within this master thesis, single-loop learning occurs from ‘measuring the effectiveness of the soft skill training program’ to ‘determine if gap is correctly defined’. This ensures feedback to the method for determining the start competence level. The double-loop learning occurs from ‘design a format for soft skill training programs’ back to the learning process.

3.2. Method An important step in starting up a study is to define the research type that will be used. The following research types can be distinguished: experimental, quasi-experimental, case-study, evaluation research, action research, and simulation research (van der Zwaan, 1990). The assignment within this master thesis project is not directly about a redesign including an intervention based on a plan of action. The aim for this master thesis project is to monitor a process and conducting measurements to the process. Therefore, evaluation research seems to be the best method, because it can be useful to know whether and to what extent a given measurement leads to the intended purpose (van der Zwaan, 1990). Furthermore, it is desirable to gather understanding about the efficacy of various interventions, measures and actions. The typology for this evaluation research is shown in Table 1.

Table 1: Typology of evaluation research

t0: needs assessment (i.e. needs analysis)

t1: intervention (i.e. reorganisation, a measure)

t2: interim (i.e. activities, processes, mechanisms)

t3: effects (i.e. intended/unintentionally ; direct/indirect; positive/negative)

Related to the typology of evaluation research, often the CIMO-logic will be used and can be applied to this research. The logic is constructed as follows: in this class of problematic Context, use this Intervention type to invoke these generative Mechanism(s) to deliver these Outcome(s) (Denyer, Tranfield, & van Aken, 2008). In the context of this master thesis project, the level of certain competences of Vanderlande employees are insufficient to perform the desired task. The improvement of those competences is known as the intervention, where the soft skill training program will be the mechanism to achieve higher competence levels to better perform the desired tasks, known as the outcome. For example, people failed meeting the deadline of a project, because they had problems with the competence ‘planning and organising’. After participating in a soft skill

15

training program aimed at improving the competence ‘planning and organising’ they might be able to better perform their task.

Within evaluation research, required comparisons can be diachronic (i.e. same object over time) and synchronic (i.e. between similar objects at the same time) (van der Zwaan, 1990). The type of evaluation that will be used for this master thesis is known as one-group-pretest-posttest design (Cook and Campbell, 1979). There is one single group under observation, with a careful measurement being done before applying the treatment and then measuring after the treatment. For evaluation research it is necessary to determine the criteria for a soft skill training program to be effective. The Vanderlande Academy deems a soft skill training program effective when a trainee possesses the desired behaviour related to the competence level three to six months after the trainee participates in the trainee program.

3.3. Selection of the research object According to the approach for this master thesis project, the research object should be soft skill training programs. For an evaluation research to be effective, the research object should be tracked from the start till the end. However, due to limited resources and the limited scope of this master thesis project, not every soft skill training program at Vanderlande can be followed from the start till the end (allowing for a pre-test and a post-test) and not every soft skill training program is suitable for evaluation. A soft skill training program is for example not suitable for evaluation if the support from higher management is lacking or when participants do not want to cooperate in the research because of possible violation of one’s privacy.

To perform the most complete research, three different soft skill training programs were chosen as research object for this master thesis project. All three soft skill training programs were carried out at Vanderlande. The first project, called ‘Collaboration IM/SA’ was primarily analysed retrospectively. This is due to the fact that this soft skill training program was already at an advanced stage of development at the start of this master thesis project. A retrospective study looks backwards and examines exposures to suspected risk or protection factors in relation to an outcome that is established at the start of the study (Cox & Hassard, 2007). The second project, called ‘sell valYou’ could be analysed from the beginning till the end. Except for the execution of a post-measurement, because the time span of this project eventually lasted longer than expected. The third project, called ‘Professional Communication’ felt within the whole time span of this master thesis project. Both projects were analysed according to a longitudinal design, meaning that a phenomenon is studied over an extended period of time. The main strength of a longitudinal research is the capacity to study change and development, which strengthens results (Saunders, Lewis, & Thornhill, 2009). It strengthens results, because the research object can be observed during the time everything actually takes place. The context of all three projects will be discussed in more detail in the next sub sections.

3.3.1. The ‘Collaboration IM/SA’ project The first project, called ‘Collaboration IM/SA’, includes a soft skill training program for Integration Managers (IM) and System Architects (SA) at Vanderlande. A system architect at Vanderlande defines the architecture of a system in order to fulfil certain requirements. The role of the system architect is being the creator or designer, with the quality of the design as the most important aspect. A system architect has to work together with an integration manager. An integration manager is someone who is responsible for the full picture, he/she had to take care of the planning and completion of a project within a given time period. Time, quality, and costs are most important factors to consider for an integration manager.

16

As both function descriptions already suggest, an area of potential tension between system architects and integration managers exists. This is due to the fact that a system architect wants to design a generic solution, in which quality is important. However, an integration manager has to complete a project within a predetermined budget and time span. And in most cases, there is no time and money available to design a generic solution, so only a standard solution can be designed. A generic solution means a system that is especially designed for a specific airport where one wants to design the system in such a way that it will be suitable for upgrades to the specific system. Due to the tension between system architects and integration managers, conflicts arise which can result in serious project failures.

Therefore, a soft skill training program was designed to improve the cooperation between system architects and integration managers. The manager of the group of system architects and integration managers determined the competences that are considered as most important to improve. It is important to notice that the soft skill training program is based upon a collective set of competences. The set of competences, derived from the competence handbook which is available at the Academy, consists of eight competences: Organisational precision and sensitivity: Recognising the influence and consequences of one’s

own activities or decisions on other parts of the organisation. Results orientation: The extent to which a person thinks, acts, and decides in a way that is

aimed at achieving the intended objectives and results. Cooperation: The extent to which a person actively contributes toward achieving collective

goals by means of making agreements, providing support and working together to fulfil tasks, without putting personal interests first.

Planning and organising: Setting objectives and priorities effectively and indicating the actions, time and resources needed in order to achieve these objectives.

Monitoring progress: Monitoring, checking and, where necessary, redirecting progress relating to employees’ processes, tasks or activities, as well as the progress of one’s own functional activities and responsibilities.

Reflection: Thoroughly understanding and recognising one’s own thought processes and behaviour, both conscious and unconscious.

Oral communication skills: Making ideas and opinions orally clear to others, using comprehensible language.

Multidisciplinary cooperation: Cooperate with people from different disciplines in a professional way, by contributing one’s own expertise.

The studied group consists of system architects and integration managers who participate in the soft skill training program to improve the eight competences. The group of system architects and integration managers consist in total of seventeen people, from which fourteen people participate in the soft skill training program. Three of them did not participate in the program, because of illness and due to a lack of willingness to participate in the program. The participants perform the following functions: Sr. System Architect (two participants) System Architect II (three participants) System Architect I (five participants)

Sr. Integration Manager (two participants) Integration Manager II (one participant) Integration Manager I (one participant)

The soft skill training program consists of different blocks, including a preparation phase with

a kick-off and personal intake; provision of general knowledge and models about cooperation; exercises with an actor to practice desired behaviours; and an integration day to summarise lessons learned and to exercise the learned behaviour with an actor.

The total length of the soft skill training program was five days. During the first block, the eight competences the soft skill training program was designed for were further elaborated and each

17

trainee was asked about his/her intentions regarding the training program. Furthermore, during the first and second block theory about providing and receiving feedback, assertiveness, and emotions was discussed. During the afternoon session of the second block, trainees participated in role plays with an actor. They practiced their behaviour in real work situations and discussed this afterwards. During the final block, attention was paid to all of the eight competences the soft skill training program was designed for. All trainees participated in role plays with an actor and afterwards they had an individual conversation with the trainer about their learning objectives for the training program.

The soft skill training program was based upon a set of given competences, predetermined by the manager of the group. It should be stressed again that the start competence level and desired competence level for each competence were based upon a collective demand for learning and no distinction was made between system architects and integration managers. Therefore, differences in start levels among individual trainees were not taken into account.

3.3.2. The ‘sell valYou’ project The second project is called ‘sell valYou’, a black belt project within Vanderlande, which means that it is a large improvement project. The project was intended to improve the sales departments of Vanderlande across the world, resulting in higher sales rates, by improving soft skills of sales employees. First of all, the project was to be conducted within the sales department in Veghel. Afterwards, it will be implemented at all customer sites of Vanderlande across the world. Objectives for the project were: improving sales knowledge; improving sales skills; and improving sales efficiency. The general sales process is shown in Figure 9. The sales process at Vanderlande is the same worldwide and includes five stages:

1. Lead: phase in which leads are identified, with or without yet identified business opportunities. 2. Pre-bid: phase in which Vanderlande further analyses and influences business opportunities. 3. Bid: phase in which Vanderlande responds to request of quotation (ROQ). 4. Closing: phase in which, either with or without negotiation, the order is being obtained, signed

and handed over to project execution. 5. Execution of the project.

One of the objectives, increasing sales skills, seems to be necessary to maximize success in

the sales process and to increase the hit rate. To improve sales skills, a soft skill training program was designed in cooperation with an external party. This research suggests that, under the most ideal circumstances, improving sales skills by education will increase sales rates.

Figure 9: Sales process of Vanderlande

18

The sales people involved in the soft skill training program were active in the sectors WPP and BHS. WPP consists of Warehouse Automation and the Parcel and Postal Market. Warehouse automation is a large and growing market consisting of many different segments with specific customer needs, where three target segments can be defined: food retail, fashion and parts & components (Vanderlande, 2015). Parcel and Postal is part of a total market for moving goods. Integrators such as DHL and UPS focus for instance on packages, time sensitive delivery and economy of scale (Vanderlande, 2015). BHS (Baggage Handling Systems) engages in airports worldwide and the corresponding services to optimize the process of transferring baggage. Vanderlande distinguish between three airport functions: (1) Regional Airport, (2) International Airport, and (3) Intercontinental Airport. To distinguish between different airport functions is necessary because of the design of the specific systems at the airports. For example, systems at regional airports are not designed to handle transfer passengers. In contrast, a system at an intercontinental or HUB market includes functionalities to handle all kind of passengers and tunnel transportation.

To achieve the objective, the soft skill training program was designed for the following six competences: Tactics: Seeing the whole picture, formulating tactical sales plans, anticipating developments

and, where necessary, adapting the sales tactic. Listening: Picking up information from verbal messages. Analysing: Recognizing important information; making connections between pieces of

information and pointing out problems. Tracking down possible (fundamental) causes of problems (diagnosis); seeking out relevant data.

Presenting: Communication ideas and facts to others in a clear way, using relevant and appropriate resources.

Creativity: Producing new or original ideas, points of view or solutions for job-related challenges/problems. Thinking up new ways of achieving a result.

Negotiating: Through consultation, reaching an agreement that serves the interests of all parties and is acceptable to all.

The desired outcome after the soft skill training program, set by the sales manager, was to achieve a more strategic sales process instead of the current purely technical sales process. Sales people of Vanderlande should use the same sales process all over the world and should be consistent in their negotiations with customers. The sales department, mainly located in Veghel, consists of three functions with associated levels: Sales Managers (levels: Sales Director; Sr. Sales Manager; Sales Manager 1; Sales Manager 2) Sales Consultant (levels: Principle Sales Consultant; Sr. Sales Consultant; Sales Consultant 1;

Sales Consultant 2) Sales Engineer (levels: Sr. Sales Engineer 1; Sr. Sales Engineer 2; Sales Engineer 1; Sales

Engineer 2).

For each function with associated level, higher management had determined the desired competence levels in terms of A, B, C, and D, where A is the lowest competence level and D is the highest competence level. The intervention included a workshop about coaching leadership for the managers and team leaders who had to coach their employees and had to facilitate transfer of training. Furthermore, a business case was carried out in order to determine the start competence level of each participant. One business case was designed and the case was applied to both WPP and BHS. During the business case everyone was assessed on the six predetermined competences. The business case, from now on called an ‘expert-assessment’, was used to determine for each participant the competence(s) that needed to be improved. The participant followed up to two or three soft skill training programs in order to improve the competence(s) in which one can gain the highest effectiveness. Before the expert-assessment took place, all managers and team leaders

19

followed a training session about coached leadership. The goal of this session was to excite and motivate the managers and team leaders to coach their employees and facilitate transfer of learning to the work situation.

The studied group consisted of sixty sales people from both WPP and BHS who first participated in the expert-assessment and afterwards in soft skill training programs to improve a certain competence. Approximately 50% of the participants were sales managers, another 25% were sales consultants and the remaining participants were sales engineers.

3.3.3. The ‘Professional Communication’ project The third project, ‘Professional Communication’, is about the soft skill training program ‘Professional Communication’. The program is facilitated by an intern trainer at Vanderlande and it is already an existing soft skill training program at Vanderlande. The soft skill training program is especially suitable for employees who will perform a managerial function in the future. The trainer of the program is a learning consultant who is active at the Vanderlande Academy. The duration of the training program is three days, where the first day is mainly about theoretical aspects, during the second day there is more space for combining theory with exercises. During the final day an actor joins the group to assist in role playing exercises. After a couple of weeks there is an evaluation day to discuss the learned skills during the training program. The objective of the training program is to communicate as a leader in a more professional way.

Eight employees of Vanderlande participated in this soft skill training program. To achieve the objective, the soft skill training program is designed upon the following eight competences: Providing Feedback: Identifying development needs and giving constructive feedback on

possibilities for improvement. Listening: Picking up information from verbal- and non-verbal messages. Oral Communication Skills: Making ideas and opinions orally clear to others, using

comprehensible language. Persuasiveness: Convincing others of a particular point of view and that aims to obtain the

agreement of others on specific plans, ideas or products (research results, etc.). Dealing with Criticism: The effect of receiving criticism on the employee’s attitude and

behaviour, and how the employee psychologically processes criticism. Results Orientation: The extent to which a person thinks, acts and decides in a way that is

aimed at achieving the intended objectives and results. Reflecting: Thoroughly understanding and recognizing one’s own thought processes and

behaviour, both conscious and unconscious. Maintaining Relationships: Maintaining a long-lasting relationship with customers and/or

colleagues that benefits the organisation.

The participants of the training program were employed within different business units within Vanderlande (i.e. parcel and postal, warehousing operations, and baggage handling). The participants performed different functions and varied in age from 24 to 50 years.

3.4. Data Collection The information that needs to be collected in order to effectively answer the research question could be gathered from various sources. In this section the data collection methods are discussed.

3.4.1. Quantitative vs. Qualitative Research Methods Quantitative methods, qualitative methods, or a combination of the two, can be used as a data collection method. The goal of a quantitative research method is to measure and analyse causal relationships between variables within a value-free framework (Denzin & Lincoln, 1994). This

20

method of data collection is useful when the researcher has clarity about the topic, when variables can be identified and when relationships can be measured. The techniques to ensure quantitative research include randomization, blinding, highly structured protocols, and written or orally administered questionnaires with a limited range of predetermined responses (Sale, Lohfeld, & Brazil, 2002).

However, a disadvantage of quantitative methods such as closed-format surveys and experiments is that they limit the possible way in which a respondent can react to and express social behaviour (van Aken, Berends, & van der Bij, 2007). It is therefore a reflection of the researchers’ assumptions. In contrast, qualitative methods such as open-ended questionnaires, semi-structured interviews and observations are useful at the individual level to investigate in depth what people think or feel and can be used when variables are complex, interwoven and difficult to measure (Smith, 1983). However, the main disadvantage of qualitative methods is the difficulty of analysing and interpreting the data. A combination of both quantitative methods and qualitative methods is known as the mixed-method. For this master thesis project, it seems obvious to use a mixed-method, because the aim of the project is to gain clarity about the change in behaviour one experiences.

3.4.2. Choice of Data Collection Methods Conducting interviews was the first step to understand the problem and to gather the necessary information in a timely manner. Interviews can be structured, semi-structured, and unstructured. For this project, semi-structured interviews seems to be most valuable, because this type includes mostly open-ended questions and leaves the opportunity for the respondent to take multiple paths and explore different thoughts, feelings and experiences. However, the researcher is still in control of the interview and can guide the interview back to the subject under discussion in order to jointly discover the real causes of a problem. Semi-structured interviews were conducted with the managers related to all three projects, and at a later stage with the trainers of all three soft skill training programs. Interviews were conducted to investigate which factors contributed to the effectiveness of soft skill training programs and how the transfer of soft skills could be optimized. However, semi-structured interviews may inadvertently guide respondents to give answers expected by the interviewer. Therefore, before conducting semi-structured interviews, research was conducted to investigate what is the best method for performing semi-structured interviews to be prepared for the pitfalls.

Besides interviews, informed observations and surveys were used. Informed observers are defined as individuals who have worked long enough with the trainees (both before and after the trainee went through the soft skill training program) (Buijssen, 2014). A manager or supervisor can be asked about their experience of the training program and about the behaviour of the participant during the training. With this information the quality of a soft skill training program can be evaluated. A valuable point about observation is that one can collect the original data at the time they occur (Blumberg, Cooper, & Schindler, 2008). On top of that, observational checklists can be used to measure the quality of the soft skill training program (i.e. the quality of the program in terms of: use of appropriate training methods; feedback during training; and enthusiasm of the trainer).

A closed-format survey and the available Competence Tool at Vanderlande can be used as quantitative methods to gather the necessary data for answering the research question. As already mentioned in the theoretical background (Section 2.6), BOS and BARS can be used to measure behaviour and both methods are valuable tools for a closed-format survey. BOS is preferred over BARS because of the ease of use and the applicability as a rating instrument. Furthermore, literature made a distinction between serial BOS and parallel BOS. For this research, serial BOS is better suited,

21

because with parallel BOS, the risk of comparing one’s score directly with the score of the rest of the group may occur.

To summarise, the following, both qualitative and quantitative methods, will be used as data collection methods: interviews; (informed) observations; observational checklists; closed-format surveys; and assessment methods, such as the available Competence Tool, from now on called self-assessment tool.

22

4. METHOD: MEASURES AND DATA COLLECTION

This section discusses the three main phases deemed to be relevant: (1) needs analysis, (2) evaluating the quality of the soft skill training program, and (3) measuring the effectiveness of the soft skill training program and transfer performance. First of all, a total overview is shown in Table 2, which shows for every project, the chosen data collection method for every phase. Afterwards, for all three projects, three phases are discussed in detail. Table 2: Overview of measures and data collection

Phase Project

(1) Needs analysis

(2) Evaluate the quality of soft skill training programs

(3) Measuring the effectiveness of soft skill training programs

Main characteristics of needs analysis within the project

Methods for determining the effectiveness of the needs analysis

Collaboration IM/SA

Collective start and desired competence levels established by manager.

Comparison of levels among: * Collective results self-assessment tool (pre-and post-measurement) * Individual results self-assessment tool (post-measurement)

Observational checklist Interviews with trainer and participants Evaluation form from the Academy

BOS post-measurement (e.g. self-report; peer-review; supervisor-review) Interview with trainer

sell valYou Start competence levels established by: * Self-assessment tool * Supervisor-assessment * Expert-assessment Desired competence levels per function determined by management

Comparison of levels among: * Self-assessment tool * Supervisor-assessment * Expert-assessment

Observational checklist Interviews with trainer and participants Evaluation form from the Academy

BOS pre-measurement (e.g. self-report and peer-review) Interviews with trainees

Professional Communication

Individual start and desired competence levels established by the manager or supervisor

Check whether the start competence level of the participant fits with the level of the soft skill training program

Observational checklist Interviews with trainer and participants Evaluation for from the academy

BOS pre- and post- measurement (e.g. self-report)

23

4.1. Needs Analysis From the theoretical background it became clear that performing a solid needs analysis is an important first step in contributing to the effectiveness of a soft skill training program. During the needs analysis one determines the knowledge, skills, and attitudes that need to be developed and/or improved during the training program. In other words, the current competence levels and the desired competence levels have to be determined to fit the soft skill training program with the determined gap. If the soft skill training program does not fit with the predetermined gap, the effectiveness of attending the soft skill training program will be doubtful from the outset. At the Vanderlande Academy, the self-assessment tool, developed by Van der Horst (2013), is available to determine one’s start competence level.

The self-assessment tool is based upon the Vignette Matching Procedure (VMP) method of Gonsalvez (2012). “A vignette is a piece of text around 150 words created by an expert as an indication of behaviour at a certain competence level. The idea of the VMP is that the different levels/vignettes of a competence are calibrated, such that they are statistically different from one another” (van der Horst, 2013, p. 15). The purpose of the self-assessment tool is to identify an employee’s start competence level, where his/her competence level will be categorized as one of the four levels (i.e. A, B, C, and D, where A is the lowest level and D is the highest level). One’s level for a selected competence is determined from the set of statements one will choose.

The next subsection provides for each of the three projects the main characteristics of the needs analysis and afterwards the methods for determining the effectiveness of the needs analysis are discussed.

4.1.1. The ‘Collaboration IM/SA’ project The soft skill training program for Integration Managers and System Architects was aimed at improving the cooperation between both groups. The start competence level and desired competence level for each competence were based upon a collective demand for learning and no distinction was made between system architects and integration managers. Therefore, it can be the case that someone’s individual gap differs from the collective gap. This mismatch could have an impact on the effectiveness of the soft skill training program. However, for analysing this mismatch a quasi-experimental design was not possible; because the individual start level of a certain competence was unknown (i.e. only the results of the post-measurement of the self-assessment were available). The self-assessment tool was filled in by the manager, who takes the opinions of system architects and integration managers into account to determine the collective start competence level and desired competence level for each of the eight competences. For the purpose of this research, the self-assessment tool was filled in by the group as a whole and individually after the soft skill training program had ended.

Afterwards, the effectiveness of the needs analysis was determined by comparing among the

collective results of the self-assessment tool and the individual results of the self-assessment tool. The start and desired competence level for each competence is shown in Appendix I, Table 22.

4.1.2. The ‘sell valYou’ project The soft skill program for the ‘sell valYou’ project aimed at improving the sales process of Vanderlande all over the world. At the start of the ‘sell valYou’ project, the project manager together with managers from both WPP and BHS determined the desired competence level for each competence per function. These levels are shown in Appendix I, Table 23.

24

As already mentioned, both WPP and Baggage participated in the soft skill training program. However, due to the lack of management support, only employees from the Baggage department were involved in the analysis phase.

Besides the use of an expert-assessment and a supervisor-assessment, for the purpose of this

research also the self-assessment tool was completed to independently determine the gap between trainees’ current and desired competence level. Based upon the expert-assessment and the supervisor-assessment, the need for training was determined and each individual attended at least one, and up to three soft skill training programs to improve one or more of the competences, out of the set of six competences, which require the most training.

The expert-assessment consisted of a business game, developed by the training company. The business game, which is also called a sales game, focuses on the sales process of Vanderlande and had been developed by the training company in cooperation with sales managers of Vanderlande. The sales game is called the ‘flight case’ and is about the situation in 2030 when oil and gas reserves are getting more and more scares and extensive urbanisation frustrates the grows of airports. The sales employees had to think about a way to achieve an optimization in the number of movements of goods from one location to another. A solution, proposed by a fictive company, was to optimize the transport of luggage, mail and packages by building new logistic centres in a number of strategic airports where these goods are jointly handled and finally transported in the same aircrafts. Vanderlande could be the company to design and build such combined logistics centres and the sales employees had to convince the fictive company to choose for Vanderlande. The sales game took place at the Philips Stadium in Eindhoven and was intended to make the Vanderlande employees aware of their weaknesses. Furthermore, an environment was created in which learning was stimulated which is, as discussed in the theoretical background, crucial for achieving a high effectiveness of a subsequent training program

The goal of the game was to gain insight into the shareholders acting on the six focus competences when following the general sales process of Vanderlande. The game was played in teams of a sales manager, a sales consultant and one or two sales engineers. Furthermore, the customers within the case were played by employees of Vanderlande who played a predetermined role associated with a certain attitude. During the game, everyone was observed by trainers of the training company and employees of Vanderlande.

The current competence level of each individual was determined by using the self-assessment tool, the expert-assessment and the supervisor-assessment. For the purpose of this research, the effectiveness of the needs analysis was determined by comparing the results of the self-assessment tool, the expert-assessment, and the supervisor-assessment. The next step was to investigate whether correlations exist between the different methods used during the needs analysis. If no strong correlation exists between the score of the self-assessment tool and the score generated by using the expert-assessment, one should wonder if it is useful to make use of an expert-assessment instead of the self-assessment tool. However, using an expert-assessment has the advantage that it already represents some kind of training and a supervisor can use displayed behaviours as input to provide feedback. A drawback of using an expert-assessment is the associated high costs.

4.1.3. The ‘Professional Communication’ project The goal of the soft skill training program ‘Professional Communication’ is to learn to communicate in a professional way. The supervisor or manager of each participant decides whether one should participate in the soft skill training program. The needs analysis was based upon the manager’s vision and one’s own opinion. The effectiveness of the needs analysis was checked to determine to

25

what extent the start competence levels of the participants fit with the level of the soft skill training program.

4.2. Evaluating the Quality of the Soft Skill Training Program The second step in contributing to the effectiveness of the soft skill training program is a well executed evaluation of the training program. The theoretical background already suggests that aspects such as providing feedback during the training program; use of real-life cases; peer support; use of role plays; and trainer support; positively influence the transfer of training and contributes to an effective soft skill training program. Therefore, an observational checklist was designed in which an observer can list whether and to which extent certain factors are present during the soft skill training program. The observational checklist is shown in Appendix II. Besides, the observational checklists also informed observations and interviews were conducted to evaluate the quality of the soft skill training program.

The observational checklist was based upon the theoretical background, findings from Elbers (2010), and interviews with trainers and participants. Sixteen aspects were chosen as deemed to be important for evaluating the quality of a soft skill training program. Examples of these aspects are: amount of received feedback during training program (Elbers, 2010); motivation of the trainee (Holton E. , 1996); use of role play (Cole, 2008); level of interaction with other participants (Elbers, 2010); and the opportunity for own input (Grossman & Salas, 2011). Furthermore, the content validity of the soft skill training program was checked by a conceptual diagram (Goldstein & Ford, 2002), shown in Figure 10.

The horizontal axis across the top of Figure 10 represents the dimension of importance, or criticality, of the KSAs as determined by a training needs assessment. The vertical axis represents the degree of emphasis for the KSAs in training. Both boxes A and D provide support for the content validity of the training program. Conceptually, to the degree to which KSAs fall into categories A and D, one can think about the training program as being content-valid (Goldstein & Ford, 2002).

Box B represents a potential error and could affect the degree to which a program is judged content-valid. Box C represents KSAs that are emphasized in training but are judged as less important for the job. This is sometimes a criticism of training programs (Goldstein & Ford, 2002). For every soft skill training program, the competences the training program was designed for were classified in this model to judge the content validity. After observing the soft skill training program, every competence was classified in the model by box A, B, C, or D.

Figure 10: A conceptual diagram of content validity of training programs (Goldstein & Ford, 2002)

The observational checklist was only filled in by myself, the master student. A drawback from only filling in the checklist by only one person is the risk of subjectivity. However, all aspects within the checklist are based upon literature and interviews. Furthermore, the checklist is

26

quantified by adding a scale ranging from (1) poor – (2) barely acceptable – (3) acceptable – (4) good – to (5) excellent; representing the frequency a certain aspect was addressed. Evaluation was carried out by first observing and then interpreting. A first step was observing the soft skill training program and listing everything what happened and the moment it happened. Afterwards, the frequency of the happenings was determined to subsequently judge it on the given scale. For example, from literature it is known that the level of interaction is an important factor that contributes to the effectiveness of the soft skill training program. During the training program, the moments interaction between participants took place were listed. Afterwards the aspect was scored on the scale, while keeping in mind what is known as ‘acceptable’ from literature. The checklist can, to a certain extent, be filled in an objective way and is therefore assumed to be a valid evaluation tool.

4.2.1. The ‘Collaboration IM/SA’ project The soft skill training program for the ‘Collaboration IM/SA’ project was already at an advanced stage at the start of this master thesis project. Therefore, only the final part of the soft skill training program was evaluated. The evaluation was performed by conducting interviews with participants and the trainer. The observational checklist was used to evaluate the final part of the soft skill training program. Furthermore, also the general evaluation form, available at the Vanderlande Academy, was completed by the trainees and the results were used to evaluate the quality of the soft skill training program.

4.2.2. The ‘sell valYou’ project Prior to the soft skill training program a workshop for all managers and team leaders took place. This workshop, called ‘coached leadership’, was about enthusing and motivating the managers and team leaders to coach their employees and facilitate transfer of learning to the work situation. Afterwards, some of the managers and team leaders were interviewed to gain insight into their experiences about the workshop ‘coached leadership’.

For all six competences of the ‘sell valYou’ project, soft skill training programs had been developed, where the competences ‘analysing’ and ‘listening’ were combined to one soft skill training program. The programs were evaluated by using the observational checklist to list whether and to which extent certain factors were addressed during the training program. Furthermore, interviews with participants and the trainers were conducted to evaluate the soft skill training program.

4.2.3. The ‘Professional Communication’ project The observational checklist was used during the training sessions and afterwards interviews were conducted with both the participants and the trainer about their experiences with the soft skill training program. Furthermore, the general evaluation form, available at the Vanderlande Academy and filled in by the trainees, was also used to evaluate the soft skill training program. 4.3. Measuring the Effectiveness of Soft Skill Training Programs As already mentioned in Section 3.2, the Vanderlande Academy deems a soft skill training program effective when a trainee possess the desired competence levels three to six months after the trainee participated in a soft skill training program. The effectiveness of the soft skill training program was measured by using a BOS survey. For the purpose of this research, an improvement exists if a significant difference (i.e. α = .05) between the pre- and post-measurement is found and/or when a reasonable effect size of the difference between the pre- and post-measurement exists. However, for small sample sizes, it is difficult to find significant effects. For this reason, the focus will be on measuring effect sizes. Arthur Jr., Bennet Jr., Edens and Bell (2003) indicated an effect size between r = .30 and r = .60 as reasonable for soft skill training programs. Furthermore, Guzzo, Jette, and Katzell (1985) reported a mean effect size of r = .44 for all psychologically based interventions.

27

Therefore, for the purpose of this master thesis, an effect size between r = .30 and r = .60 is determined as an acceptable effect size, indicating an improvement.

The next sub section will explain how the BOS survey was designed following the required steps for developing BOS surveys. Then, the way in which the BOS survey is used for each of the three projects is explained.

4.3.1. Design of the BOS survey A BOS survey was developed to measure the effectiveness of soft skill training programs. The starting-pint in the design process was the specific steps required for developing BOS surveys (Latham & Wexley, 1981):

1. Critical incidents that are similar, if not identical, in context are grouped together to form one behavioural item.

2. Behavioural items that are similar are grouped together by job incumbents and/or analysts to form one BOS criterion.

3. Inter-judge agreement is assessed to determine whether another individual or group of individuals would have developed the same behavioural criteria from the critical incidents obtained in the job analysis.

4. The BOS criteria are examined regarding their relevance or content validity. 5. The appraisal instrument is developed by attaching a five-point Likert scale to each

behavioural item. 6. Many items on the BOS, although critical in terms of defining highly effective or ineffective

performance occur either so frequently or infrequently that they do not differentiate good from poor job incumbents. Therefore, these types of items are eliminated by conducting an item analysis, which involves correlating the scores on each behavioural item with the sum of all items so that each section on the appraisal instrument is unambiguous to the appraiser.

7. If there are approximately three to five times as many individuals, to be rated as there are behavioural items, a factor analysis can be conducted.

Interviews were conducted with managers of all three projects to ask for critical incidents

and/or desired behaviours that are specific for a certain competence. The three projects together cover eighteen separate competences for which concrete behaviours needed to be designed. In addition to the specific competences the soft skill training was designed for, the BOS survey also includes two unrelated competences. These unrelated competences were measured in order to control for validity of the measurement tool and to control for the absence of a control group. The results on the two unrelated competences should remain the same for both the pre- and post-measurement, because the specific soft skill training program was not designed for these competences. Including these unrelated competences should control for the absence of a control group. The competence set of Vanderlande integrated within Bartram’s (2005) hierarchy was used to establish which competence could be included to serve as an unrelated competence. For the ‘Collaboration IM/SA’ project, the competences ‘Creativity’ and ‘Planning and Organising’ were selected as unrelated competences. The second project, ‘sell valYou’ included the unrelated competences ‘Delegating’ and ‘Planning and Organising’. For the project ‘Professional Communication’, the competences ‘Creativity’ and ‘Planning and Organising’ were chosen as unrelated competences. Competences were selected as unrelated competences, because following Bartram’s (2005) hierarchy, the selected competences did not relate to one of the specific competences the soft skill training program was designed for. The competence set of Vanderlande with the Bartram hierarchy can be found in Appendix III, Figure 13,

28

After adding these unrelated competences, a total set of 128 behaviours was developed by conducting interviews, asking for critical incidents, and by translating descriptions from the competence handbook to concrete behaviours. Purposely a large number of behaviours for each competence were used to take into account that establishing a suitable and validated number of behaviours for each competence would probably entail deletion of some behaviours. The list of behaviours was randomized and three volunteers from the Vanderlande Academy were asked to allocate the behaviours to the most related competence. The clustering was necessary for assessing the inter-rater reliability for a fixed number of assessors. Employing multiple assessors was necessary to control for the reliability and validity of the clustering.

Fleiss’ Kappa was used as a statistical measurement for inter-rater reliability. The measurement calculates the degree of agreement in classification over that which would be expected by chance (Shoukri, 2011). It expresses the extent to which the observed amount of agreement among assessors exceeds what would be expected if all assessors made their ratings completely random (Fleiss, 1971).

The derivation of Fleiss’ kappa can be found in Appendix III. Fleiss’ Kappa can be calculated with the following formula:

Following Landis and Koch (1977), the values for Fleiss’ Kappa can be interpreted following Table 3. However, one should note that the magnitude of Kappa’s value is influenced by factors such as the weighting applied and the number of categories in the measurement scale. The larger the number of scale categories, the greater the potential for disagreement, with the result that Kappa will be lower when there are many categories and higher when there are fewer categories (Sim & Wright, 2005). Table 3: Interpretation of Fleiss' Kappa

Interpretation

< 0 Poor agreement

0.01 – 0.20 Slight agreement

0.21 – 0.40 Fair agreement

0.41 – 0.60 Moderate agreement

0.61 – 0.80 Substantial agreement

0.81 – 1.00 Almost perfect agreement

The general approach for calculating the values for Kappa is as follows: 1) calculating the overall value for Kappa, based upon the total set of competences; 2) calculating the value for Kappa for every competence separately; 3) removing items if the value for Kappa is insufficient and return to step 2. The lower limit to determine a set of behaviours as reliable and valid is assumed to be at least 0.61. For Kappa values beneath this score, items will be removed in order to reach a more acceptable value. However, if the content of the competence after removing certain behaviours is no longer covered, based upon the competence descriptions from the competence handbook, a lower value for Fleiss’ Kappa will be accepted.

The overall value for Fleiss’ Kappa was calculated and equalled 0.65, which can be

interpreted as a substantial agreement. Therefore, the clustering seemed reliable and the set of behaviours could be used for designing the BOS surveys. Furthermore, Fleiss’ Kappa was calculated for every competence separately. For a few competences, the Fleiss’ Kappa proved to be too low to be reliable. For clarity, the final values for Fleiss’ Kappa, after removing items, are listed in Table 4.

e

e

P

PP

1

29

Table 4: Fleiss' Kappa values

Competence #items (initial)

Fleiss’ Kappa (initial value)

#items (removed)

Fleiss’ Kappa (final value)

#items (remaining)

Organisational Precision and Sensitivity

7 ≈ 0.22 3 ≈ 0.72 4

Results Orientation 6 = 0.25 2 = 1 4

Cooperation 10 ≈ 0.38 4 ≈ 0.68 6

Planning and Organising 10 ≈ 0.13 7 = 1 3

Monitoring Progress 10 ≈ 0.09 4 =1 6

Reflection 5 ≈ 0.42 1 = 1 4

Multidisciplinary cooperation 5 ≈0.42 0 ≈ 0.42 5

Delegating 5 ≈ 0.38 1 = 0.4 4

Creativity 8 ≈ 0.91 3 = 1 5

Providing Feedback 6 ≈ 0.76 2 = 1 4

Listening 8 ≈0.74 3 = 1 5

Oral Communication Skills 5 ≈ 0.48 1 ≈ 0.67 4

Persuasiveness 8 ≈ 0.48 5 = 1 3

Dealing with Criticism 4 = 1 0 = 1 4

Maintaining Relationships 5 ≈ 0.17 1 = 1 4

Analysing 8 ≈ 0.26 0 ≈ 0.26 8

Tactic 5 ≈ 0.08 2 ≈ 0.61 3

Presenting 6 ≈ 0.44 1 ≈ 1 5

Negotiating 7 ≈ 0.52 0 ≈ 0.52 7

To illustrate the design of the BOS survey, the process for designing the BOS survey will be

explained step by step for the competence ‘Presenting’.

1. Translating descriptions for the competence ‘Presenting’ to concrete behaviours. Interviews were conducted with managers to ask for critical incidents about presenting in practice. Furthermore, the descriptions from the competence handbook were translated to concrete behaviours. The formulations of the concrete behaviours were translated to English and checked by a supervisor, because the BOS survey for the project ‘sell valYou’ was conducted in English. Table 5 shows the concrete behaviours regarding the competence ‘Presenting’.

Table 5: Concrete behaviours regarding the competence 'Presenting'

Competence Concrete behaviours

Presenting 1. Conveys ideas and facts in a clear manner to others. 2. Makes proper use of resources during a presentation. 3. Dares to speak in public. 4. Uses variations in intonation and non-verbal behaviour while speaking. 5. Behaves in a relaxed way while presenting. 6. Keeps the attention of the audience by method of presentation.

2. Clustering behaviours. As already mentioned, volunteers were asked to cluster the concrete

behaviours to the best suited competence. Afterwards, a reliability analysis was conducted and

Fleiss’ Kappa was calculated. The calculated Fleiss’ Kappa for the competence ‘Presenting’ was

equal to ≈ 0.44. The lower limit to accept a set of behaviours was set to at least 0.61.

Therefore, the next step was to remove items to obtain a higher value for Fleiss’ Kappa.

30

3. Removing items to improve Fleiss’ Kappa. Items were removed in order to obtain a higher value for Fleiss’ Kappa. However, items were only removed if the content of the competence is still enough covered. The final list of concrete behaviours is shown in Table 6. Fleiss’ Kappa was calculated again and the new value for Fleiss’ Kappa was equal to = 1, indicating a perfect agreement between the assessors.

Table 6: Final list of concrete behaviours regarding the competence 'Presenting'

Competence Concrete behaviours

Presenting 1. Makes proper use of resources during a presentation. 2. Dares to speak in public. 3. Uses variations in intonation and non-verbal behaviour while speaking. 4. Behaves in a relaxed way while presenting. 5. Keeps the attention of the audience by method of presentation.

4. Designing the survey. At the moment the final list of concrete behaviours was available and

examined regarding their relevance and content validity, the lay-out for the BOS survey was designed. A five-point Likert scale was used for the BOS survey and ranged from (1) totally not applicable – (2) a little bit applicable – (3) somewhat applicable – (4) mostly applicable – to (5) totally applicable.

5. Furthermore, strategies were undertaken to control for the halo-error and the leniency- and

severity error. The halo-error is a phenomenon when the assessors’ overall impression or evaluation is thought to strongly influence ratings of specific attributes (Robert, Jako, & Anhalt, 1993). When an assessors’ tendency is to rate all employees at the positive end of the scale or at the low end of the scale is known as the leniency error and the severity error, respectively (Latham & Wexley, 1981). To control for the mentioned errors, negative formulations of the statements were used and no heading text was used prior to the statements.

A clear introduction to the survey was made to create trust from the participant to fill in the survey as honestly as possible. The participant was asked to indicate for each statement to what extent the behaviour applies to him/her. Also negative formulations were used to control for the halo- error. Figure 11 shows a part of the BOS survey related to the competence ‘Presenting’.

totally not applicable

a little bit applicable

somewhat applicable

mostly applicable

totally applicable

22. Makes proper use of resources during a presentation.

23. Has problems with speaking in public.

24. Uses variations in intonation and non-verbal behaviour while speaking.

25. Behaves in a relaxed way while presenting.

26. Keeps the attention of the audience by method of presentation.

Figure 11: Part of the BOS survey regarding the competence 'Presenting'

Three BOS surveys were designed, because every project includes a soft skill training program

that was designed for a different set of competences. As already mentioned in the theoretical

31

background (Section 2.5), to reduce the subjectively problem, the concept of 360-Degree Feedback was used. In order to do so, the BOS survey was filled in by several people (i.e. a supervisor-review; a peer-review), instead of completing the survey only by the participant him/herself (i.e. a self-report). From literature is known that in general it is not likely to find high positive and significant correlations between groups in case 360-Degree-Feedback is used. This finding is quite logical, because otherwise 360-Degree Feedback would be unnecessary. For analysing the data, Spearman correlations will be calculated, because one cannot assume linearity and because the data possess an ordinal scale. Research from Fletcher, Baldry and Cunningham (1998) and Furnham and Stringfield (1994) indicate a Spearman correlation coefficient of at least .30 as an acceptable value to assume a relationship between reviewers within 360-Degree Feedback. Furthermore, Harris and Schaubroek (1988) investigated the correlations between peer-supervisor; self-supervisor; and self-peer. The results, based on a meta-analysis, indicated a relatively high correlation between peer and supervisor ratings (ρ = .62), but only a moderate correlation between self-supervisor (ρ = .35) and self-peer ratings (ρ = .36) (Harris & Schaubroeck, 1988). Literature offered a number of explanations why low correlations exist between different assessors. The most common is known as egocentric bias, wherein a self-rater is inclined to inflate his/her rating in order to enhance the evaluation, resulting in higher correlations between others (i.e. peers and supervisor) (Holzbach, 1978).

4.3.2. The ‘Collaboration IM/SA’ project Conducting both a pre- and post-measurement was not possible for measuring behavioural change, because the ‘Collaboration IM/SA’ project was already in progress at the start of this master thesis project. However, a BOS survey was designed for a post-measurement and to check for correlations between a self-report, a peer-review, and a supervisor-review. Furthermore, all participants were asked to fill in the competence tool at the end of the soft skill training program, and a collective competence level was determined by the group as a whole as well. The outcomes from the self-assessment tool were compared with earlier results. Next to that, the manager of the system architects and integration managers was interviewed to gather information about improvements of the employees who participated in the soft skill training program.

4.3.3. The ‘sell valYou’ project To measure the effectiveness of the soft skill training program of the ‘sell valYou’ project it was desired to use the BOS survey and perform both a pre- and a post-measurement. However, the soft skill training program ended one month before the end of this master thesis. Therefore, it was not possible to carry out the post-measurement at least three to six months after the participants completed the soft skill training program. It has been considered to still conduct a post-measurement only one month after the soft skill training program has ended. However, literature mentioned that it won’t be likely to find improvements after such a short time. Furthermore, the number of people who finished the soft skill training program one month ahead of the end of the master thesis is too small to draw worthwhile conclusions. An acceptable alternative to measure to some extent the effectiveness of the soft skill training program, is by conducting structured interviews with seven people who finished the soft skill training program at least a month ago and were involved in the pre-measurement.

4.3.4. The ‘Professional Communication’ project The effectiveness of the soft skill training program for the project ‘Professional Communication’ was measured by using a pre- and post-measurement with the BOS survey. The post-measurement was conducted three months after the end of the soft skill training program. In contrast to the other two projects, only a self-report was conducted, because the participants perform a wide variety of functions. Therefore, assigning a peer-reviewer was too difficult. Furthermore, the support from the managers to complete a supervisor-review was lacking.

32

5. RESULTS

For this master thesis project, three research sub questions were already formulated. However, ten analysis questions were formulated in order to gather an answer to the sub research questions following the method section, as discussed in the previous Chapter.

Research sub questions: 1. What is the best way to determine the gap between the start competence level and the

desired competence level of a Vanderlande employee? 2. What are the most important aspects for evaluating the quality of soft skill training

programs at Vanderlande? 3. How to measure the effectiveness of soft skill training programs at Vanderlande?

Analysis questions:

a. What is the extent of agreement between different methods that were used for the needs analysis?

b. What is the added value of using an expert assessment instead of only a self-assessment and or supervisor-assessment to determine someone’s need for training?

c. Is it beneficial to use the self-assessment tool for determining the collective demand for learning?

d. What is the extent of agreement between a collective measurement and an individual measurement to determine someone’s competence level?

e. How is the quality of the three soft skill training programs evaluated in terms of the evaluation form of the Academy; the observational checklist; and interviews with the trainer and trainees?

f. What is the extent of agreement between different reviewers when measuring behaviour, or in other words, what is the added value of 360-Degree Feedback?

g. Is the BOS survey a reliable tool for measuring changes in behaviour? h. Is the difference between the pre-and post-measurement of behavioural change when

attending a soft skill training program significant? i. Is there a relation between someone’s competence level and someone’s behaviour?

Some of the analysis questions can only be answered using a particular project, because not

all three projects were performed in the same way and/or with the same measurement techniques. For clarity, Table 7 shows the structure for answering the analysis questions. For example, analysis question ‘a’ is answered to provide an answer to sub research question ‘1’, by analysing the data of the ‘sell valYou’ project. The following sub sections provide an answer to the analysis questions regarding the particular project. Table 7: Structure for answering the analysis questions

Analysis question Project ‘Collaboration IM/SA’

Project ‘sell valYou’ Project ‘Professional Communication’

a. sub-RQ 1

b. sub-RQ 1

c. sub-RQ 1

d. sub-RQ 1

e. sub-RQ 2 sub-RQ 2 sub-RQ 2

f. sub-RQ 3 sub-RQ 3

g. sub-RQ 3 sub-RQ 3 sub-RQ 3

h. sub-RQ 3

i. sub-RQ 3

33

5.1. The ‘Collaboration IM/SA’ project (N=14) Fourteen participants were actively involved in the project and the data file was based upon the response of these people. Determining the reliability of the BOS survey

In order to draw conclusions from the calculations, first the reliability of the BOS survey was determined, by calculating Cronbach’s Alpha. For explorative research, a value of at least 0.60 is acceptable (Field, 2009). The values for Cronbach Alpha were listed in Table 8. Table 8: Reliability of the BOS survey for the ‘Collaboration IM/SA’ project (N=14)

Competence Value for Cronbach alpha

Organisational Sensitivity and Precision α = 0.648

Results Orientation α = 0.788

Cooperation α = 0.736

Planning and Organising α = 0.690

Monitoring Progress α = 0.817

Reflection α = 0.419

Oral Communication Skills α = 0.708

Multidisciplinary Cooperation α = 0.358

Delegating α = 0.218

Creativity α = 0.759

Overall, for most of the competences the BOS scale seems reliable. The results suggest that

the BOS scale for the competences ‘Reflection’, ‘Multidisciplinary Cooperation’, and ‘Delegating’ are not reliable (i.e. unreliable values for Cronbach Alpha are shown in bold type). Due to the small sample size it is hard to find high values for Cronbach Alpha. However, the complete BOS survey will be used for the analysis, but caution is required in interpreting findings for these competences. Descriptive statistics for the ‘Collaboration IM/SA’ project

Table 9 illustrates the descriptive statistics for the ‘Collaboration IM/SA’ project. For every competence, the mean, standard deviation and rank position were calculated. Overall, the mean scores are highest for the supervisor-review and lowest for the peer-review. The mean scores for the self-report are closest to the mean scores for the peer-review. The high mean scores for the supervisor-review may explained by the fact that the supervisor (i.e. the manager of the system architects and integration managers) perhaps wants to indicate that the behaviour of his employees has significantly improved compared to the start of the soft skill training program. The standard deviation is the actual difference in behaviour between the employees and by the differences in norms of the different assessors. The supervisor-review was performed by one person (i.e. the manager) and the peer-review was performed by fourteen different assessors. This may explain why the standard deviation for the supervisor is lowest and the standard deviation for the peer-review is highest.

Furthermore, the ranking was calculated in order to determine whether a correlation exists between the rank positions by the assigned different reviewers. Table 10 shows the correlation matrix, where the correlations are based on Spearman rank correlation test. The correlation coefficients are significant and high enough (i.e. r > .3) to conclude that a positive relation exists between the rankings, performed by the self-report; peer-review; and supervisor-review. Furthermore, this suggests that the competences that were scored highest are the same across different reviewers. In other words, the extent to which competences are ranked relative to another competence is the same across different reviewers. Especially the ranking performed by peers and the self-report is highly correlated.

34

Table 9: Descriptive statistics for the 'Collaboration IM/SA' project (N=14)

Competence Self-report Peer-review Supervisor-review Total of three sources

Mean SD Rank Mean SD Rank Mean SD Rank Mean SD Rank

Organisational Precision and Sensitivity 4.07 0.43 3 4.07 0.47 1 4.03 0.35 4 4.06 0.27 2

Results Orientation 4.13 0.40 1 4.05 0.57 2 3.89 0.46 7 4.02 0.35 4

Cooperation 4.06 0.40 4 3.95 0.67 4 4.18 0.22 2 4.06 0.29 2

Planning and Organising 3.95 0.57 5 3.81 0.64 6 3.93 0.47 5 3.90 0.43 6

Monitoring Progress 3.81 0.58 7 3.80 0.44 7 3.88 0.38 8 3.83 0.36 7

Reflection 3.73 0.32 8 3.48 0.55 10 3.80 0.28 9 3.67 0.22 10

Oral Communication Skills 3.73 0.42 8 3.75 0.65 8 3.93 0.32 5 3.80 0.32 8

Multidisciplinary Cooperation 4.09 0.49 2 4.00 0.47 3 4.20 0.22 1 4.10 0.20 1

Delegating 3.95 0.38 5 3.84 0.56 5 4.04 0.29 3 3.94 0.21 5

Creativity 3.71 0.39 10 3.70 0.65 9 3.75 0.34 10 3.72 0.35 9

Table 10: Ranking of the competences (based on mean values) for the 'Collaboration IM/SA' project

self <-> peer self <-> supervisor peer <-> supervisor

.939** .642* .663*

** .Correlation is significant at the.01 level * .Correlation is significant at the .05 level Extent of agreement between different reviewers

To establish the extent of agreement between different reviewers, or in other words, the added value of 360-Degree Feedback, for each competence the correlations were calculated. The correlation coefficients between the self-review; peer-review; and supervisor-review are listed in Table 11. As already mentioned in the method section, a positive Spearman correlation coefficient of at least .30 was indicated as an acceptable value to assume a relationship between reviewers within 360-Degree Feedback.

35

Table 11: Correlations between self-review; peer-review; supervisor-review

Competence self <-> peer self <-> supervisor peer <-> supervisor

Organisational Precision and Sensitivity .202 .499 .005

Results Orientation .214 .558* .311

Cooperation .138 .122 .140

Planning and Organising .211 .558* .388

Monitoring Progress .348 .578* .135

Reflection -.163 -.305 .239

Oral Communication Skills .374 -.120 .107

Multidisciplinary Cooperation -.069 -.294 -.263

Delegating -.179 .036 -.327

Creativity .070 .089 .248

*p < .05 and N = 14 For the competences ‘Organisational Precision and Sensitivity’, ‘Results Orientation’,

‘Planning and Organising’, and 'Monitoring Progress’ acceptable values for the correlations were found (indicated in bold type). Therefore, a relationship between the self, peer and supervisor is assumed. Therefore, the mean values of these competences were used for comparing the scores with the scores of the self-assessment tool. However, the values for the Spearman correlation coefficients for the other competences are very low or even negative. Especially for the competences ‘Reflection’, ‘Multidisciplinary Cooperation’, ‘Delegating’ and ‘Creativity’, the Spearman correlation coefficients are not acceptable. The competences ‘Delegating’ and ‘Creativity’ were for this project known as the unrelated competences and might be wrongly interpreted by the different assessors. Furthermore, the competences with low values for the Spearman correlation coefficient are the ones with an unreliable scale, based on the calculation of Cronbach’s Alpha (Table 8). The unreliable scales may be the reason for the low or even negative correlations that were found within these competences. Next to that, the competence ‘Reflection’ might be a competence that is hard to interpret, which could be the reason for the negative correlation coefficients. Another reason for the low correlation coefficients might be that the peers are not trained in observing behaviour. Furthermore, for the supervisor it might be hard to assess the competence ‘Reflection’, because the supervisor (i.e. the manager) is not cooperating with all his employees in practice, while the peers are selected upon their cooperation with each other during projects.

To summarise, the obtained correlations indicate some extent of agreement between

different reviewers when measuring behaviour. However, when high and significant correlation coefficients were found, 360-Degree Feedback would be superfluous. The results indicate that the groups (i.e. self; peer; and supervisor) are not interchangeable and 360-Degree Feedback is therefore valuable. Nevertheless, the combination of a low reliability and a low correlation is problematic, which is the case for the competences ‘Reflection’, ‘Multidisciplinary Cooperation’, and ‘Delegating’. Therefore, it is doubtful to generalize the findings regarding these competences. Self-assessment tool for determining the collective demand for learning

This sub section discusses whether it would be beneficial to use the self-assessment tool for determining the collective demand for learning. Furthermore, the degree of agreement between a collective measurement and an individual measurement when using the self-assessment tool will be discussed. Finally, it will be investigated whether a relation between someone’s assessed competence level and someone’s observed behaviour exist.

During the evaluation day of the soft skill training program, the collective post-measurement

by using the self-assessment tool was conducted by the manager together with the group of participants of the soft skill training program during the evaluation day of the soft skill training

36

program. So, the manager, system architects, and integration managers discussed until they reached a consensus about the attained level for each of the eight competences the soft skill training program had been designed for. Table 12 shows the collective pre- and post-measurement of the estimation of the competence levels using the self-assessment tool. The post-measurement was conducted without knowledge of the desired competence level. Furthermore, the extent of the improvement is shown in brackets, where the collective post-measurement is compared with the collective pre-measurement.

Table 12: Estimation of competence levels using the self-assessment tool

Competence Pre-measurement (collective)

Post-measurement (collective)

Desired post-measurement (manager)

Goal achieved?

Organisational Precision and Sensitivity

B C D No (+)

Results Orientation B C C Yes (+)

Collaboration B C D No (+)

Planning and Organising B B C No (=)

Monitoring Progress B B C No (=)

Reflecting A B C No (=)

Oral Communication Skills A B C No (+)

Multidisciplinary Cooperation B B D No (=)

‘+’ means an improvement ‘-‘ means an impairment ‘=’ means remained unchanged

Following Table 12, the goal in terms of the desired competence level was not achieved for most of the competences, but for most of the competences still an improvement was obtained. Reasons for not achieving the desired result can be that the manager set too ambitious goals. Furthermore, the post-measurement was conducted right after the soft skill training program had ended. However, from literature it is known that the learning process for soft skills lasts for a long period and most of the learning actually takes place at the workplace after someone participated in the soft skill training program, related to the ’70:20:10’ ratio. Based upon the results and the available scientific literature, determining the demand for learning is essential prior to participating in a soft skill training program. The self-assessment tool was originally intended for individual use in order to determine someone’s current competence levels (i.e. the demand for learning). However, the self-assessment tool can be used to assist the process for determining the collective demand for learning, by using an open discussion in order to reach consensus about the demand for learning. Extent of agreement between a collective- and an individual measurement

For each competence, the collective pre-measurement, the collective post-measurement, and the individual post-measurement were compared for all the fourteen participants separately. In this way, one can determine whether the average individual competence level is similar to the estimation of the collective post-measurement level. The comparison of the values can be found in Appendix V. Overall, the estimation of the own competence level by the individual seems to be higher compared to the estimation of the competence level by the group as a whole; the average of the individual scores is higher compared to the group as a whole. Only for the competences ‘Organisational Precision and Sensitivity’ and ‘Results Orientation’ the estimation of the competence level during the post-measurement is lower for the individual compared to the estimated level by the group as a whole. For the competence ‘Cooperation’ the estimation of the competence level is the same. A reason for the higher estimation of the competence level by the individual compared to the estimation of the competence by the group as a whole could be that the individual

37

overestimates him/herself, also known as overestimation in self-reports (Mayer, et al., 2007). Overestimation with respect to the self-assessment tool will probably be the case during post-measurements, because after participating in the soft skill training program one is already aware of the competences and the regarding levels. Therefore, it will be easy to consciously or unconsciously answer the desired competence level. Furthermore, Mabe III and West (1982) mentioned that self-evaluations are subject to a great deal of error resulting from self-enhancement desires; and people cannot analyse themselves objectively or reliably enough to give accurate information.

Based on these issues with self-reports, in order to determine an improvement with regard

to one’s competence level, a discussion with the group as a whole to reach a consensus is preferred over the individual post-measurement, because of the likelihood of self-overestimation. However, the collective post-measurement is only preferable in case of an open discussion with all the participants and without knowledge of the desired competence level. Relation between competence level and behaviour

To determine whether a relation exists between someone’s assessed competence level and someone’s observed behaviour, correlations were calculated. The Spearman correlation coefficients were calculated between the scores of the BOS survey compared with the individual scores of the self-assessment tool, measured after the soft skill training program has ended. The Spearman correlation coefficients are shown in Table 13, where a correlation coefficient of at least .30 is indicated in bold type. As already mentioned, in case high correlations were found between the BOS scores of the self-review, peer-review, and supervisor-review, the average score was added for this analysis. A positive relation between the assessed competence level and observed behaviour is critical for the design of a valid measurement tool. Therefore, correlations between the self-assessment tool (i.e. measures someone’s competence level) and the BOS survey (i.e. measures someone’s behaviour) need to be found. As already mentioned within the method section, a positive correlation of at least .30 is indicated as acceptable. If, besides a positive correlation between the BOS self-review and the self-assessment tool, also a positive correlation exists between the average, peer-review, or supervisor-review and the self-assessment tool, it can be concluded that the competence level can be estimated using the peer-review and or the supervisor-review. In other words, if a positive correlation of at least .30 exists between the self-assessment tool and for example the BOS peer-review, both the self-assessment tool (which was based on competence descriptions) and the BOS tool (which was based on behaviours) measured the same. This suggested that the estimation of someone’s competence is behaviour related, which enhances the validity and reliability of the BOS tool.

Table 13: Correlation coefficients between BOS survey and self-assessment tool

Competence self <-> tool peer <-> tool supervisor <-> tool average <-> tool

Organisational Precision

and Sensitivity

.407 .204 -.131 .216

Results Orientation .061 .263 .361 .238

Cooperation .118 .404 .000

Planning and Organising .284 .457 .189 .444

Monitoring Progress .446 .425 .439 .618*

Reflecting -.213 .403 .442

Oral Communication Skills .407 -.095 -.340

Multidisciplinary

Cooperation

-.218 .341 .000

* .Correlation is significant at the .05 level (2-tailed)

38

A value of at least .30 for the Spearman correlation coefficient is assumed as acceptable, because of the use of different assessors (i.e. 360-Degree Feedback). Overall, for most of the competences a positive relation exists between the BOS survey and the self-assessment tool. This finding indicates that a relation exists between someone’s competence level and someone’s behaviour. However, some correlation coefficients were too small or even negative. For these cases, the reviewers already mutually disagree about the scores for the competence levels in the analysis about the extent of agreement between different reviewers. This could be the reason for the small or even negative correlation coefficients. In general, correlations would not be high when the variation within the data is low. Overall, Appendix V shows a low variety for the scores of the self-assessment tool. Due to this lack of variation within the estimation of someone’s own competence level, the obtained correlations with someone’s observed behaviour were for some competences low. Nevertheless, overall the obtained correlations were moderately high enhancing the existence of a relation between someone’s assessed competence level and someone’s observed behaviour.

Furthermore, based on Table 13, the highest positive correlations exist between the peer-

review and the self-assessment tool. Therefore, when 360-Degree Feedback is not possible in order to measuring the effectiveness of a soft skill training program (i.e. behavioural change), performing only a peer-review for the BOS survey seems the best option. Evaluating the quality of the soft skill training program

The quality of the soft skill training program was evaluated by using the observational checklist; the evaluation form, available at the Academy; and interviews with both trainees and the trainer. Table 14 shows the most important evaluation criteria from the evaluation form, available at the Academy. Every participant was asked to rate the soft skill training program in terms of those evaluation criteria. The average grades for all three projects are listed in Table 14 for comparing the projects. Table 14: Evaluation criteria from evaluation form, available at the Vanderlande Academy

Evaluation criteria

Average grade

‘Collaboration IM/SA’ project (N=14)

‘sell valYou’ project (N=27)

‘Professional Communication’ project (N=8)

Amount of practical exercises during training program

8.2 8.3 7.8

Enthusiasm and expertise of the trainer 9.0 8.8 8.4

Level of interaction and opportunity for own input

9.0 8.8 8.6

Use of an actor 8.7 N/A 9.0

Training program meets trainee expectations

8.5 8.2 7.9

The soft skill training program was very well evaluated and subjects that were reported as

important were: the role-plays; determination of learning objectives; the reflection and feedback received from the trainer. In addition to the evaluation form, the observational checklist was used to evaluate the quality of the soft skill training program. The complete observational checklist is listed in Appendix VI. Positive aspects, which were observed during the soft skill training program, are the use of role-play, the use of an actor and the use of real-life cases. These aspects made the soft skill training program very valuable. Furthermore, the enthusiasm of the trainer and the motivation of the trainee were observed as ‘good’, which was mentioned in the theoretical background as a very important

39

factor to contribute to the effectiveness of a training program. The aspect ‘correct start level for training program (based on needs analysis)’ was rated as 3, ‘acceptable’, because of the different start competence levels between the participants. The differences were very obvious during the soft skill training program, shown in the proficiency during exercises and role-plays. From the evaluation form it already became clear that the trainees were very positive about the soft skill training program. Furthermore, interviews with trainees emphasized this finding. Most of the trainees reported that the soft skill training program definitely met their expectations; they considered the training program very valuable; and the use of role-play seems to most of the trainees the most valuable aspect of the soft skill training program. Another important aspect, mentioned by the trainees, is to learn from other participants and the feedback they received. The trainer of the soft skill training program was very positive about the motivation of the trainees and the input one provided. Furthermore, he stated that the support from the manager was very important for the success of the soft skill training program. Measuring the effectiveness of the soft skill training program

Due to the fact that a pre-measurement with the BOS survey was lacking, it was impossible to measure the behavioural change and therefore the effectiveness of the training program with a BOS survey. However, the self-assessment tool was completed before and after the soft skill training program. Based upon the results from the self-assessment tool, an improvement is indicated. Based upon these results, the BOS post-measurement, and the interviews with the trainees and the manager, it is concluded that the soft skill training program is effective for the following reasons. First, the manager recognizes improvements regarding the behaviour of his employees he observed during projects and in practice. Secondly, comparing the scores of the BOS post-measurement and the pre-measurement with the self-assessment tool, one recognizes an improvement. Nevertheless, related to the ’70:20:10’ ratio, most of the learning takes place after the soft skill training program and paying attention to the transfer phase might be important.

5.2. The ‘sell valYou’ project (N=27) Twenty-seven participants were included in the analysis of the ‘sell valYou’ project. Determining the reliability of the BOS survey

In order to draw conclusions from the dataset, first the reliability of the BOS survey was determined, by calculating Cronbach’s Alpha. The results were listed in Table 15. Seven of the eight competences are measured in a sufficiently reliable way, because for these competences the value for Cronbach’s Alpha is at least 0.60. Only for the competence ‘Delegating’, the value is below 0.60 (indicated in bold type). The competence ‘Delegating’ is one of the unrelated competences for this project and for the participants this competence might be hard to interpret, which could be the reason for the low value for Cronbach’s Alpha. Table 15: Reliability of the BOS scale for the 'sell valYou' project (N=27)

Competence Value for Cronbach’s Alpha

Listening α = .742

Analysing α = .906

Creativity α = .786

Tactic α = .703

Presenting α = .813

Negotiating α = .722

Delegating α = .455

Planning and Organising α = .658

40

Descriptive statistics for the ‘sell valYou’ project Table 16 illustrates the descriptive statistics for the data set. The competences ‘Delegating’

and ‘Planning and Organising’ are the unrelated competences for this project. For every competence, the mean, standard deviation and rank position were calculated. Furthermore, the ranking was calculated to determine whether a correlation exists between the assigned rank positions by the different reviewers. The mean scores for the peer-review are lower for all the competences compared to the mean scores for the self-report. The cause might be overestimation in self-reports, which could occur consciously and unconsciously. Next to that, it is believable that others are likely to rate their colleagues lower than they would rate themselves. Furthermore, a strong and significant correlation between ranks of the self-report and peer-review BOS ratings was found (Spearman’s ρ = .85, p < .01). This finding suggests that the extent to which competences are ranked relative to another competence is the same across the different reviewers. Table 16: Descriptive statistics for the 'sell valYou' project (N=27)

Competence Self-report Peer-review Average of both sources

Mean SD Rank Mean SD Rank Mean SD Rank

Listening 3.87 0.48 4 3.61 0.60 4 3.74 0.46 4

Analysing 3.88 0.53 2 3.71 0.66 2 3.79 0.55 3

Creativity 3.67 0.64 6 3.34 0.70 7 3.51 0.53 7

Tactic 3.68 0.71 5 3.46 0.69 6 3.57 0.59 5

Presenting 3.44 0.75 8 3.28 0.68 8 3.36 0.58 8

Negotiating 3.59 0.54 7 3.48 0.51 5 3.54 0.42 6

Delegating 3.88 0.53 2 3.80 0.41 1 3.84 0.35 2

Planning and Organising 3.99 0.65 1 3.70 0.61 3 3.85 0.52 1

Extent of agreement between different reviewers

To measure the extent of agreement between different reviewers, or in other words, the added value of 360-Degree Feedback, the ‘Collaboration IM/SA’ project already suggests that overall some extent of agreement exists between the reviewers due to the positive correlations that were found. However, to verify this finding, also for the ‘sell valYou’ project, correlations between the self-review and peer-review were calculated. The Spearman correlation coefficients between the self-review and peer-review are listed in Table 17. Following Table 17, for most of the competences, acceptable values for the correlation coefficients (i.e. ρ > .30) between the self-review and peer-review were found. This suggests a relation between the BOS scores for a particular competence assigned by the self-report and by the peer. Only the correlation coefficient for the competence ‘Delegating’ is low and this may be due to the low value for Cronbach Alpha. If the reliability for this competence was already low, questions belonging to the same competence were differently interpreted, resulting in a low correlation coefficient. Table 17: Correlation between self-review and peer-review BOS survey for each separate competence

Competence self <-> peer

Listening .334

Analysing .621**

Creativity .228

Tactic .405*

Presenting .318

Negotiating .333

Delegating .111

Planning and Organising .333

* .Correlation is significant at the .05 level (2-tailed)

41

** .Correlation is significant at the .01 level (2-tailed)

Extent of agreement between different methods for the needs analysis Different methods (i.e. self-assessment; assessment centre; supervisor assessment) were used

during the needs analysis according to the ‘sell valYou’ project. For six participants, the way of allocation with regard to the six competences was analysed in more detail. This results in a dataset with 36 cases, because for only six participants the complete information was available. For these 36 cases, the correlations between the scores in terms of A, B, C, and D, for the self-assessment, expert-assessment, and supervisor-assessment were calculated. The Spearman correlation matrix is listed in Table 18.

Table 18: Correlations between self-assessment; expert-assessment; and supervisor-assessment

Self-assessment Assessment centre Supervisor-assessment

Self-assessment 1.00 .497** .671**

Expert-assessment 1.00 .619**

Supervisor-assessment 1.00

** .Correlation is significant at the 0.01 level (2-tailed)

One can conclude that the estimation of the self-assessment, expert-assessment, and supervisor-assessment are significant and highly correlated with each other. Therefore, there is high agreement between the different methods used for the needs analysis.

The added value of an expert-assessment

Furthermore, the added value of using an expert-assessment instead of only a self-assessment and/or supervisor-assessment for determining someone’s need for training was investigated. For the six participants, the allocation process was analysed in more detail. The result (i.e. the score in terms of A, B, C, or D) of the self-assessment, the expert-assessment, and the supervisor-assessment for the relevant six competences was known, together with the particular soft skill training program the participant were allocated to. As already mentioned, each participant will attend at least one, and up to three soft skill training programs (i.e. ‘Listening and Analysing’; ‘Tactic’; ‘Presenting’; ‘Negotiating’, and/or ‘Creativity’). First of all, the number of negative deviations was calculated, by comparing the assigned score (in terms of A, B, C, or D) with the desired competence level, for respectively the self-assessment, expert-assessment, and supervisor-assessment. For every participant, three methods were used to determine someone’s score in terms of A, B, C, or D for each of the six competences (i.e. 6 participants * (3 methods * 6 competences) = 108 possible deviations).

In more than a quarter of the 108 cases, a negative deviation existed between the

competence levels, assigned by the self-assessment tool, expert-assessment, and supervisor-assessment, compared to the desired levels. Most of the negative deviations were one level below the desired level. The distribution of the negative deviations of one level below the desired level is listed in Table 19.

Table 19: Distribution of the negative deviations of one level below the desired level (N=29 cases)

Method for needs analysis Magnitude (#cases) of the negative deviation

Self-assessment tool 9

Expert-assessment 10

Supervisor-assessment 8

42

Based on Table 19, the added value of using an expert-assessment instead of only using the self-assessment tool and/or supervisor-assessment to determine someone’s competence level was quite small. However, based on the observations of the expert-assessment (i.e. the sales game) and interviews with participants and trainees, it was very valuable to perform an expert-assessment. Besides the function of determining someone’s need for training, the expert-assessment also ensured some kind of learning and the sales game was therefore already part of the intervention. The sales game made people aware of their strengths and weaknesses, which was experienced as very valuable. Participants already entered the stage of ‘consciously incompetent’, referring to Maslow’s second stage of learning. Evaluating the quality of the soft skill training program From Table 14 it became clear that the soft skill training program was positively rated by the participants. Most of the participants considered the interaction with the trainers and with other participants as very valuable. Furthermore, most of them reported the exercises and the use of real-life cases and the expertise of the trainers as useful aspects. Negative comments about the soft skill training program include speaking too much in abstract terms; and no clear relation to the competences the soft skill training program was essentially designed for. This point of criticism might be explained by the used training method. The trainers used a more abstract training method and used of a lot of anecdotes, which was different compared to more standard training methods. A more standard training method included the explanation of theory in a bookish way and afterwards some role-plays. For some participants this ‘different’ method of training (i.e. lacking the bookish training method) could cause some confusion. The observational checklist further rates the level of interaction with other participants and the use of real-life cases (based on company context) as excellent. Aspects that might be improved include ‘clear objective of the training program’ and ‘use of role play’. The trainers were sometimes too abstract in their way of working and for some topics it seems better to exercise with an actor. Measuring the effectiveness of the soft skill training program As already mentioned, conducting a post-measurement for the ‘sell valYou’ project was not possible, because the end of the soft skill training program was too close to the end of this master thesis project. However, structured interviews were conducted with seven participants who finished the soft skill training program and were involved in the pre-measurement. Most of the interviewees reported that the time after the end of the soft skill training program was too short to recognise already significant improvements. However, everyone reported being already consciously competent about their behaviour, indicating that participants right now behave according to the third stage of Maslow’s four stages of learning model (Figure 6). Furthermore, most of the interviewees reported that the process of becoming consciously competent’ was also noticed for peers who participated in the soft skill training program. Therefore, based on this analysis and for the purpose of this research, it is assumed that the soft skill training program related to the ‘sell valYou’ project was effective.

5.3. The ‘Professional Communication’ project (N=8) Eight participants were involved in the soft skill training program and this section provides the results according to the ‘Professional Communication’ project. Determining the reliability of the BOS survey

In order to test the reliability of the BOS survey, values for Cronbach’s Alpha were calculated and the result is shown in Table 20. Overall, the BOS scale seems reliable and therefore the BOS survey can be used for the estimation of competence levels. Only for the competence ‘Listening’, the value for Cronbach’s Alpha is negative, indicating a negative value for the mean of the inter-item

43

correlation (indicated in bold type in Table 20. However, the value for Cronbach’s Alpha is based on a very small sample and this could be the reason for the negative value for Cronbach’s Alpha. The competence ‘Listening’ is also included in the ‘sell valYou’ project and within this project the value for Cronbach’s Alpha is equal to .742. Therefore, for the purpose of this research, the BOS survey related to the ‘Professional Communicating’ project is assumed to be a reliable tool. Table 20: Reliability of the BOS scale for the 'Professional Communication' (N=8)

Competence Value for Cronbach’s Alpha

Providing Feedback α = .634

Listening α = -.115

Oral Communication skills α = .627

Persuasiveness α = .267

Dealing with Criticism α = .735

Results Orientation α = .688

Reflecting α = .453

Maintaining Relationships α = .845

Creativity α = .809

Planning and Organising α = .706

Descriptive statistics ‘Professional Communication’ project

For every competence the mean and standard deviation were calculated for the pre-measurement and the post-measurement, listed in Table 21. The mean scores for every competence were improved comparing the pre-measurement with the post-measurement. Even the mean scores for the competences ‘Creativity’ and ‘Planning and Organising’ were increased, while these two competences were selected as the unrelated competences based on Bartram’s (2005) hierarchy. Table 21: Descriptive statistics for the 'Professional Communication' project (N=8)

Competence Pre-measurement Post-measurement Wilcoxon Signed Rank Test

Mean SD Mean SD

Providing Feedback 4.16 0.50 4.31 0.29 .416

Listening 4.03 0.25 4.08 0.43 .864

Oral Communication Skills 3.75 0.73 3.79 0.43 .854

Persuasiveness 3.63 0.52 3.96 0.33 .052

Dealing with Criticism 3.63 0.55 3.88 0.38 .084

Results Orientation 4.25 0.38 4.41 0.46 .197

Reflecting 3.59 0.42 3.84 0.55 .071

Maintaining Relationships 3.66 0.76 4.13 0.57 .054

Creativity 3.30 0.57 3.40 0.49 .234

Planning and Organising 3.96 0.49 4.25 0.43 .034*

*The significance level is .05 level (2-tailed) Evaluating the quality of the soft skill training program The findings from Table 14 show that the soft skill training program was very well evaluated. Especially the use of an actor and the level of interaction with other participants and the opportunity for own input was highly scored by the participants. The results from the observational checklist agreed upon the findings from the evaluation form, where the support from the trainer and level of enthusiasm of the trainer was very well scored. The aspects ‘pace/motion’ and ‘clear training program material’ score compared to the other aspects the lowest. Some participants seemed during the first day a bit disinterested and bored, because for them the focus was too long on one particular theory, which they already understood.

44

However, overall the quality of the soft skill training program was very well evaluated and also the trainer was very enthusiastic about the motivation of the participants. Measuring the effectiveness of the soft skill training program After attending a soft skill training program, it is assumed that someone experiences behavioural change. The BOS survey was indicated as a reliable tool for measuring changes in behaviour. A non-parametric test is required to determine whether the difference in scores of the BOS survey between the pre- and post-measurement is significant. A non-parametric test is required, because of the following reasons: the data follows a non-normal distribution; the sample size is very small; and the data set includes ordinal data, ranked data and outliers that cannot be removed. Therefore, the best suitable non-parametric test for this research is the Wilcoxon Signed Rank Test. This test compares two related samples on a single sample to assess whether their population mean ranks differ (Field, 2009). The Wilcoxon Signed Rank Test needs to be significant to detect an improvement for a particular competence. The results of the Wilcoxon Signed Rank Test are the last column of Table 21.

Following Table 21, none of the related competences show a significant difference (p<.05).

Only the difference for the unrelated competence ‘Planning and Organising’ is significant, which is an unexpected result, because the soft skill training program was not supposed to focus on this competence. However, most of the participants reported that their planning and organising skills were indeed improved due to the soft skill training program. An explanation might be that someone who works more results-oriented unconsciously enhanced his/her planning and organising skills.

The differences for the other competences were not significant at the significance level of .05. However, the differences for ‘Persuasiveness’, ‘Dealing with Criticism’, ‘Maintaining Relationships’, and ‘Reflecting’ approached significance (p<.10). The almost significant differences are promising results, taking into account the small sample size (N=8). Furthermore, the absolute values of the mean scores for all the competences were increased. Therefore, with caution, an improvement for the competences was assumed. Conclusion Overall, the obtained results tend to the desired effectiveness of the soft skill training programs for all three projects. However, also some weaknesses of the research were visible within the results section and these will be further discussed in the next Chapters. Furthermore, the next Chapter suggests best practices for every phase regarding the effectiveness of soft skill training programs.

45

6. DISCUSSION

This section presents the conclusive elaboration of the research sub questions, leading to the opportunity to elaborate the earlier stated research objectives in Section 1.3. All statements and conclusions follow from the literature, models, and measurements discussed in Chapters 1 to 5.

The research objectives for this master thesis project were formulated as 1) a format that contributes to the effectiveness of soft skill training programs at Vanderlande; and 2) a tool for measuring the effectiveness of soft skill training programs at Vanderlande. The tool, the BOS survey, was already discussed in the method (section 4.3.1). The format provides an answer to the research question of this master thesis project: “Which components contribute to the effectiveness of soft skill training programs at Vanderlande”. The research sub questions are answered in line with the design of the format, because the research sub questions were considered as input for the format.

6.1. Design of the format For the design of the format the three sub research questions were used as input and those questions will be answered in this section. The sub research questions were formulated as: 1. What is the best way to determine the gap between the start competence level and the desired competence level of a Vanderlande employee? 2. What are the most important aspects for evaluating the quality of soft skill training programs at Vanderlande? 3. How to measure the effectiveness of soft skill training programs at Vanderlande?

From the theoretical background, three phases (i.e. needs analysis; evaluating the quality of the soft skill training program; and measuring the effectiveness of the soft skill training program) are deemed to be relevant regarding the effectiveness of soft skill training programs. The format focuses on these three phases in particular. For every phase the best practices will be discussed, which are logically the answers for the research sub questions. The designed format is shown in Figure 12.

6.1.1. Needs analysis A solid needs analysis is required for the success of soft skill training programs. Without knowing someone’s exact need for training, the training will not be effective, because one does not know the knowledge, skills and attitudes that need to be developed during the training program. During the needs analysis one determines the gap between someone’s start competence level and desired competence level. Different methods can be used to perform such a needs analysis. First of all, a self-assessment, by using the Competence Tool (known as the self-assessment tool) can be used. Next to that, a supervisor-assessment and/or an expert-assessment can be used. A high extent of agreement was found between the different methods. Therefore, all three methods can be conducted to perform the needs analysis. However, the best practice for the needs analysis is to use an expert-assessment next to the self-assessment and/or supervisor-assessment.

The added value for using an expert assessment is expressed in the process of becoming aware of the strengths and weaknesses by the participant. From this research it can be concluded that such a process of becoming aware is required for the learning of soft skills, because it ensures that one becomes ‘consciously incompetent’. So, the expert-assessment had more value as initial training intervention than as a way of establishing training needs. However, the drawback of using an expert-assessment is the associated high costs. Therefore, for every particular case a cost-benefit

46

analysis should be considered to determine whether it will be profitable to use an expert-assessment next to the self-assessment and/or supervisor-assessment.

Furthermore, using the self-assessment tool for determining the collective demand for learning is not recommended, because the self-assessment tool was originally intended and designed for individual use. However, in case only a collective measurement is possible, an open discussion with all the participants and without knowledge of the desired competence level is the best practice.

6.1.2. Evaluating the quality of the soft skill training program The next phase is known as evaluating the quality of the soft skill training program. As already mentioned within the theoretical background, training evaluation is the measurement of a training program’s success or failure with regard to content and design, changes in learners, and organisational payoffs. Evaluation is important to investigate whether training objectives were achieved and whether it results in enhanced performance on the job. If, for the design of a soft skill training program the best practices are known, one can use this list of aspects for the negotiation process with external training parties. Based on interviews with trainees and trainers; and by participating and evaluating in three different soft skill training programs; a list of aspects was developed and used as the input for the observational checklist. The aspects that should be included within a soft skill training program to ensure the program to be successful are:

- Receiving enough feedback during the soft skill training program; - A highly motivated trainee; - Use of role-play; - Use of real-life cases (based on company context); - An enthusiastic trainer; - Interaction with other participants in order to learn from each other’s strengths and

weaknesses; - A sufficient balance between theoretical versus practical issues.

Receiving enough feedback during the soft skill training program was reported as very valuable by almost every participant. The received feedback is considered as valuable, because it allows someone to verify that he/she is on the right track. Furthermore, people were aware of the need for training. Furthermore, following Holton’s evaluation model (1996), a high motivated trainee is important to reach the outcome ‘learning’. Trainers of the different soft skill training programs expressed agreement with this scientific finding. Role plays and the use of real-life cases were considered as especially important within soft skill training programs. Soft skill training programs are about learning behaviour. Therefore, the way participants behave during role plays appears valuable for learning the appreciate behaviour in particular situations. On top of that, using real-life cases, which are based on the company context, was reported as most appreciated, because one can behave within his/her normal habitat. An enthusiastic trainer is reported as a success factor within a soft skill training program, because it ensures the motivation of the trainee, which is important to achieve learning. Another interesting aspect, which was reported by most of the participants, is the interaction with other participants during the soft skill training program. The interaction results in another way of learning, in which one can learn from each other’s strengths and weaknesses and the problems someone is struggling with. The last aspect is about a sufficient balance of theoretical versus practical issues during the soft skill training program. From the literature it became clear that both practical and theoretical issues should be addressed during the soft skill training program. Participants reported that the amount of practical and theoretical issues during a soft skill training program should be balanced, to avoid that trainees become bored and unfocused.

47

6.1.3. Measuring the effectiveness of the soft skill training program The final phase includes the measurement of the effectiveness of soft skill training programs. To measure the effectiveness, one needs to measure behavioural change before and after someone participates in the soft skill training program. The best practice is known as the reliable and validated serial BOS survey, a tool which scores behaviours on a five-point Likert scale. The use of the serial BOS survey is enhanced by the finding that a relation exists between the trainees’ measured competence level and behaviour. Both a pre- and post-measurement should be conducted to measure behavioural change, and consequently the effectiveness of the soft skill training program. Based on Maslow’s four stages of learning, one has to practice the learned behaviour to arrive at the final stage of learning, ‘unconsciously competent’. However, it takes time to reach this final stage of learning. Therefore, the post-measurement should be conducted at least three to six months after participation in a soft skill training program. Whether the behavioural change is statistically significant can be measured using Wilcoxon Signed Rank Test.

Furthermore, the use of 360-Degree Feedback for conducting the BOS survey was analysed. 360-Degree Feedback is very valuable within measuring behaviour, which reduces the subjectively problem. This research investigated the extent of agreement between the different reviewers. One can conclude that the groups were not interchangeable and 360-Degree Feedback is valuable and is considered as the best practice. However, in case 360-Degree Feedback is not possible, only conducting a peer-review is considered as the best alternative. Last but not least, this master thesis project recommended to take two factors into account with respect to the effectiveness of soft skill training programs. First of all, support from higher management, from his/her supervisor, and from peers. Secondly, the opportunity for trainees to practice their learned skills on the job, referring to the ’70:20:10’ ratio, which suggests that for soft skills, 70% of the lessons learned come from on-the-job experiences.

48

Figure 12: Format that contributes to the effectiveness of soft skill training programs

GAP

Methods for needs analysis:* self-assessment tool (for individual use)* supervisor-assessment* expert-assessment

Best practice: self-assessment and/or supervisor-assessment together with an expert-assessment (to become consciously competent)

desired competence level

1. Needs Analysis

2. Evaluating the quality of the soft skill training program

Desired aspects for the content of soft skill training programs:* Receiving enough feedback during the soft skill training program;* A high motivated trainee;* Use of role-play;* Use of real-life cases (based on company context)* Interaction with other participants in order to learn from each other’s strengths and weaknesses;* A sufficient balance between theoretical versus practical issues.

Best practice: Check whether the indicated aspects are presented during the soft skill training program.

3. Measuring the effectiveness of soft skill training programs

Measurement tool à serial BOS survey* Pre-measurement (before soft skill training program)* Post-measurement (three to six months after soft skill training program)

Best practice: The serial BOS survey, using 360-Degree Feedback (self-review; peer-review; supervisor-review) for measuring someone’s behavioural change. If 360-Degree Feedback is not possible, conducting only a peer-review is the best alternative.

FORMAT (that contributes to the effectiveness of soft skill training programs at Vanderlande)

Recommendations to Vanderlande to realise effective soft skill training programs

* Support from higher management; supervisors; peers.* Opportunity for trainees to practice their learned skills on the job.

start competence level

49

7. CONCLUSION, LIMITATIONS, AND RECOMMENDATIONS

The final Chapter discusses the overall conclusion, limitations, and recommendations of the conducted research. 7.1. Conclusion Before conducting this master thesis project, no empirical evidence was available that soft skill training programs at Vanderlande work properly. This led to the undesirable situation that a lot of money was spent without knowing whether the soft skill training program was effective. Furthermore, employees at Vanderlande lack to some extent soft skills in order to perform the desired behaviour according to their job. Therefore, the objectives for this master thesis project were the design of a format that contributes to the effectiveness of soft skill training programs at Vanderlande; and a tool for measuring the effectiveness of soft skill training programs at Vanderlande. The designed format emphasizes the importance of the three phases deemed to be relevant according to the effectiveness of soft skill training programs. A solid needs analysis was considered as an important first step related to the effectiveness of soft skill training programs. During a needs analysis one determines the knowledge, skills, and attitudes that need to be developed during the training program. Furthermore, Goldstein and Ford (2002) mentioned that organisational support should be included in the needs analysis, because the organisation and/or supervisor should motivate the trainee, emphasise the importance of improving his/her skills and should set the desired competence level. Without knowing the gap between someone’s start and desired competence level, attending a random soft skill training program would not be effective beforehand. The needs analysis is necessary to fit a soft skill training program to the gap between someone’s start and desired competence level. Therefore, investigating the best method for assessing someone’s need for learning is necessary, where literature suggested to discuss the need for learning with someone’s supervisor. Furthermore, this master thesis project investigated, besides using a supervisor-assessment, also the use of a self-assessment and an expert-assessment for conducting the needs analysis. The suggested best practice for performing a needs analysis is by performing an expert-assessment together with a self-assessment and/or supervisor-assessment. By noting that one should determine beforehand whether conducting an expert-assessment is profitable for the particular group size.

Next to that, best practices for evaluating the quality of the soft skill training program were investigated. Based upon the theoretical background, interviews with trainers and trainees, and research from Elbers (2010), an observational checklist was developed including aspects and training methods that contribute to the effectiveness of soft skill training programs. An important aspect was to exercise the desired behaviour by using role-plays and real-life cases which were based on company context. Furthermore, the interaction with other participants and receiving a sufficient amount of feedback by the trainee is required. On top of that, motivation of both the trainer and the trainees plays an important role. Additional factors that are crucial regarding the effectiveness of the soft skill training program include the support from higher management, supervisors, and peers and the opportunity for trainees to practice their learned skills on the job. The second objective of this master thesis project relates to measuring the effectiveness of soft skill training programs. A validated and reliable BOS survey was developed to measure behavioural change. Following this research, performing 360-Degre Feedback for the BOS survey seems to be the best practice. Furthermore, in case 360-Degree Feedback is not possible, regarding the obtained correlations and theoretical findings, conducting a peer-review was suggested as the best alternative.

50

7.2. Limitations There are several limitations concerning this master thesis project that should be discussed. First of all, the population of all three projects is small in order to generalize the findings. Furthermore, participants of the soft skill training program ‘Professional Communication’ only filled in the self-report resulting in a potential social desirability bias. However, the clear introduction prior to the survey would reduce the potential social desirability bias by informing the participant that the results would not be shared within the company (Nederhof, 1985). Furthermore, the introduction ensured the participants that the outcomes of the survey would not be used for assessment purposes. Secondly, both the ‘Collaboration IM/SA’ project and the ‘sell valYou’ project were not suitable for measuring behaviour change, because only a pre- or post-measurement was conducted. However, both projects were valuable for validating the measurement tools.

Another limitation for the ‘Collaboration IM/SA’ project is the absence of an individual start and desired competence level for each competence prior to the soft skill training program. During the needs analysis the start and desired level were based upon a collective demand for learning. Therefore, it could be the case that someone’s individual gap differs from the collective gap. This mismatch could have an impact on the effectiveness of the soft skill training program, but this cannot be investigated in more detail. The consequence might be a difference in start levels between the participants. However, this difference can actually be an advantage, because participants can learn from each other.

Furthermore, the person who performed the peer-review often was also a participant of the soft skill training program, which possibly gives a distorted result. However, it is expected that this influence is rather small, because of the clear introduction of the survey, the guaranteed anonymity of the survey and the assurance that the survey would not be used for assessment purposes such as a Mid-Year Review (MYR), which was emphasized by the introduction to the survey.

Another limitation of this research has to do with the design of the BOS survey. After clustering the list of behaviours, Fleiss’ Kappa was calculated to control for the inter-rater reliability. However, removing items in order to achieve a higher and sufficient value for Fleiss’ Kappa possibly results in the disappearance of a facet of the competence. To take the content of the Competence Handbook and the self-assessment tool into account, items were therefore only removed when this did not result in a loss of coverage. However, for some competences this result in Fleiss’ Kappa values lower than the acceptable value of 0.61, indicating that the inter-rater reliability cannot be guaranteed. Therefore, caution is required in interpreting findings for these competences.

Furthermore, the training observations, using the observational checklist, were individually performed and therefore, the reliability was not checked. And although criteria within the observational checklist were obtained from the literature, they were not explicitly operationalized and no explicit grading anchors were defined. Nevertheless, the observational checklist was used for observing the quality of the soft skill training program. And the observational checklist was filled in by one person, because there were not enough resources available to get the observational checklist filled in by multiple assessors.

7.3. Practical recommendations and future research In this section the recommendations for Vanderlande and recommendations for future research are discussed.

51

7.3.1. Recommendations for Vanderlande Based on the analysis and overview of results, several recommendations to Vanderlande are derived to improve the effectiveness of soft skill training programs.

Five years ago Elbers (2010) already investigated the factors contributing to transfer of training at Vanderlande. Elbers (2010) mentioned the following factors were most likely to enhance transfer of training: peer support; openness to change work environment; and performance-based coaching of trainees. It was suggested that without transfer of training, soft skill training programs would never be effective. Investigating in depth the transfer of training felt outside the scope of this master thesis project. However, this research suggested that transfer of training could be improved by: support from higher management, supervisors and peers; and the opportunity for trainees to practice their learned skills on the job. Despite two independent studies (i.e. Elbers (2010) and the current master thesis project), the same factors for improving the transfer of training were suggested. Therefore, a recommendation to Vanderlande will be to seriously take those factors into account, for example by organising coaching workshops (e.g. the workshop ‘coached leadership’ related to the ‘sell valYou’ project). Secondly, a solid needs analysis is an important factor that contributes to an effective soft skill training program. Therefore, I advise the Vanderlande Academy to require a solid needs analysis of everyone who will participate in a soft skill training program. A suggestion for the needs analysis is conducting an expert-assessment in case a whole department or large team wants to participate in a soft skill training program. For smaller groups, due to the financial aspect, an expert-assessment may not be profitable and a needs analysis using the self-assessment tool and/or supervisor-review are good alternatives. Besides the function of a needs analysis, an expert-assessment also ensures the participants to become already aware of their strengths or weaknesses. Or in in other words, an expert-assessment ensures people to already move to the second stage, referring to Maslow’s stages of learning, which was mentioned as very valuable. Trainee motivation was determined as an important aspect for the success of a soft skill training program. Therefore, I advise the Vanderlande Academy to use stricter rules for participants who do not show up for a soft skill training program without a valid reason. Sometimes it happened that only half of the participants show up for a soft skill training program, while soft skill training programs are very costly. Lastly, the pre-measurement of the BOS survey should be conducted before the expert-assessment takes place, because of the unintended learning effect during the expert-assessment. This master thesis project investigated the use of an expert-assessment as a method for needs analysis. It became clear that performing an expert-assessment was already part of the training intervention, because participants became already aware of their strengths and weaknesses, referring to Maslow’s second stage of learning (i.e. ‘become consciously incompetent’). Therefore, the pre-measurement of the BOS survey should be conducted before the expert-assessment takes place, because of the unintended learning effect, which results in a measurement bias.

7.3.2. Recommendations for future research Next to the practical recommendations for Vanderlande, a recommendation for future research can be presented. The previous section already suggested that improving the transfer phase is an important aspect for soft skills training. Coaching and support by supervisors and higher management (e.g. organising a ‘coaching leadership’ workshop regarding the ‘sell valYou’ project) is mentioned as an example for improving the transfer of training (Elbers, 2010) (Gilpin-Jackson & Bushe, 2007). However, future research should investigate in more depth the effectiveness and best

52

practices for performing such ‘coaching leadership’ workshops and other possible solutions to learn supervisors and higher management with their supporting role during the transfer phase. Furthermore, the BOS survey was designed and validated for nineteen competences, where sixteen of those scales were sufficiently reliable. To improve the unreliable scales, it is necessary to develop new formulations of behaviour and subsequently cluster these formulations in order to determine the inter-rater reliability. The method for the design of the BOS survey is discussed in Section 4.3.1. To effectively apply the measurement tool for all training programs, BOS scales also need to be designed for the remaining competences. The design of the BOS scales can be performed using the described method in Section 4.3.1.

7.4. Concluding remarks To conclude this master thesis project, the most important findings will be briefly mentioned. Both a format and a valid and reliable measurement tool were designed to satisfy the research objectives. On top of that, an observational checklist was designed including aspects a soft skill training program should meet in order to guarantee its quality. According to three soft skill training programs, for every phase deemed to be relevant regarding the effectiveness of soft skill training programs, best practices were developed. For conducting a solid needs analysis, a self-assessment and/or supervisor-assessment should be performed together with an expert-assessment. The expert-assessment makes participants aware of their strengths and weakness, which was mentioned as very valuable. Secondly, the quality of the soft skill training program should be properly evaluated by checking whether the indicated aspects are presented during the soft skill training program. Examples of those aspects include: receiving enough feedback during the soft skill training program; the use of real-life cases; and interaction with other participants. Finally, for measuring the effectiveness of soft skill training programs the designed measurement tool can be used to conduct both a pre- and post-measurement, while performing 360-Degree Feedback.

53

8. REFERENCES

Adair, J. (1984). The Hawthorne Effect: A Reconsideration of the Methodological Artifact. Journal of Applied Psychology, 69(2), 334-345.

Aguinis, H., & Kraiger, K. (2009). Benefits of Training and Development for Individuals, and Teams, Organizations and Society. The annual review of psychology, vol. 60, p. 451-474.

Alvarez, K., Salas, E., & Garofano, C. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, 3(4), 385-408.

Argyris, C. (2002). Double-loop learning, teaching, and research. Academy of Management Learning & Education, 206-219.

Arthur Jr., W., Bennet Jr., W., Edens, P., & Bell, S. (2003). Effectiveness of Training in Organizations: A Meta-Analysis of Design and Evaluation Features. Journal of Applied Psychology, 88(2), 234-245.

Baldwin, T., Ford, K., & Blume, B. (2009). Transfer of training 1988-2008: an updated review and agenda for future research. International Review of Industrial and Organizational Psychology, 24, 41-70.

Bancino, R., & Zevalkink, C. (2007). Technical professions increasingly require a broader skill set, but career and technical educators can help their students add soft skills to their hard-core technical skills. Techniques, 20-22.

Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal of applied Psychology, 90(6), 1185-1203.

Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347.

Birati, A., & Tziner, A. (1999). Economic utility of training programs. Journal of Business and Psychology, 14(1), 155-164.

Blumberg, B., Cooper, D., & Schindler, P. (2008). Chapter 9: Observations, content analysis, action and ethnographic research. In B. Blumberg, D. Cooper, & P. Schindler, Business Research Methods (Second European Edition) (pp. 343-373). Berkshire: McGraw-Hill.

Broad, M., & Newstrom, J. (1992). Transfer of Training: Action-Packed Strategies to Ensure High Payoff from Training Investment. Reading, MA: Addison-Wesley.

Bronson, E. (2007). Career and technical education is ideally suited to teaching the soft skills needed to succeed in the 21st century workplace. Techniques: Connecting Education & Careers, 82(7), 30-31.

Buijssen, E. (2014). Literature Review (Master Thesis Project). Eindhoven: Eindhoven University of Technology.

Burke, L., & Baldwin, `. (1999). Workforce training transfer: a study of the effect of relapse prevention training and transfer climate. Human resource management, 227-242.

Cannon-Bowers, J., Salas, E., Tannenbaum, S., & Mathieu, J. (1995). Toward theoretically-based principles of training effectiveness: A model and initial empirical investigation. Military Psychology, 7(3), 141-164.

Cole, N. (2008). How long should a training program be? A field study of "rules-of-thumb". Jounal of Workplace Learning, 54-70.

Cox, J., & Hassard, J. (2007). Ties to the past in organization research: a comparative analysis of retrospective methods. Organization, 14(4), 475-497.

Crosbie, R. (2005). Learning the soft skills of leadership. Industrial and Commercial Training, 37(1), 45-51.

DeNisi, A., & Kluger, A. (2000). Feedback effectiveness: Can 360-degree appraisals be improved? Academy of Management Executive, 14(1), 129-139.

Denyer, D., Tranfield, D., & van Aken, J. (2008). Developing Design Propositions through Research Synthesis. Organization Studies, 393-413.

54

Denzin, N., & Lincoln, Y. (1994). Introduction: Entering the field of qualitative research. In N. Denzin, & Y. Lincoln, Handbook of Qualitative Research (pp. 1-17). Thousand Oaks: CA: Sage.

Elbers, W. (2010). Improving transfer of training. Eindhoven: Eindhoven University of Technology. Field, A. (2009). Discovering Statistics Using SPSS, third edition. Thousand Oaks: SAGE Publications

Ltd. Fleiss, J. (1971). Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin,

76(5), 378-382 . Fletcher, C., Baldry, C., & Cunningham-Snell, N. (1998). The psycholometric properties of 360 degree

feedback: An empirical study and cautionary tale. International Journal of Selection and Assessment, 19-34.

Furnham, A., & Stringfield, P. (1994). Congruence of self and subordinate ratings of managerial practices as a correlate of supervisor evaluation. Journal of Occupational and Organizational Psychology, 57-67.

Gilpin-Jackson, Y., & Bushe, G. (2007). Leadership Development Training Transfer: A Case Study Of Post-Training Determinants. Journal of Management Development, 26(10), 980-1004.

Goldstein, I., & Ford, K. (2002). Chapter 3: The Needs Assessment Phase. In Training in Organizations: Need Assessment, Development, and Evaluation (4th ed.) (pp. 34-84). Belmont USA: Wadsworth Thomson Learning.

Gonsalvez, G. (2012). The vignette-matching procedure: An innovative approach to assess competencies in psychology and clinical practicum. Sydney: Australian Learning and Teaching Council.

Grossman, R., & Salas, E. (2011). The transfer of training: what really matters. International Journal of Training and Development, 15(2), 103-120.

Guzzo, R., Jette, R., & Katzell, R. (1985). The effects of psychologically based intervention programs on worker productivity: A meta analysis. Personnel Psychology, 275-291.

Harris, M., & Schaubroeck, J. (1988). A Meta-Analysis of Self-Supervisor, Self-Peer, and Peer-Supervisor ratings. Personnel Psychology, 41(1), 43-62.

Holton, E. (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7(1), 5-21.

Holton, E. I., Bates, R., & Ruona, W. (2000). Development and validation of a generalized learning transfer system inventory. Human Resource Development Quarterly, 11(4), 333-360.

Holzbach, R. (1978). Rater bias in performance ratings: Superior-, self-, and peer ratings. Jounal of Applied Psychology, 579-588.

Intagliata, J., Ulrich, D., & Smallwood, N. (2000). Leveraging Leadership Competencies to Produce Leadership Brand: Creating Distinctiveness by Focusing on Strategy and Results. Human Resources Planning, 12-23.

Jackson, S., Schuler, R., & Werner, S. (2009). Managing Human Resources, 10th edition. Mason: South-Western Cengage Learning.

Jelley, R., Goffin, R., Powell, D., & Heneman, R. (2012). Incentives and Alternative Rating Approaches, roads to greater accuracy in job performance assessment. Journal of Personnel Psychology, 11(4), 159-168.

Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs. San Francisco: Berrett-Koehler Publishers, Inc.

Kraiger, K., Ford, J., & Salas, E. (1993). Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation. Journal of Applied Psychology Monograph, 78(2), 311-328.

Kupritz, V. (2002). The relative impact of workplace design on training transfer. Human resource development quarterly, 427-447.

Laker, D., & Powell, J. (2011). The Differences Between Hard and Soft Skills and Their Relative Impact on Training Transfer. Human Resource Development Quarterly, 111-120.

55

Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 159-174.

Landy, F., & Conte, J. (2007). Work in the 21st century: an introduction to industrial and organizational psychology (second edition). Oxford: Wiley-Blackwell.

Latham, G., & Wexley, K. (1981). The Development and Validation of Appraisal Systems. In G. Latham, & K. Wexley, Increasing Productivity Through Performance Appraisal (pp. 37-77). Reading: Addison-Wesley Publishing Company.

Latham, G., & Wexley, K. (1994). Increasing productivity through performance appraisal (2nd ed.). Reading: MA: Addison-Wesley.

Mabe III, P., & West, S. (1982). Validity of Self-Evaluation of Ability: A Review and Meta-Analysis. Journal of Applied Psychology, 67(3), 280-296.

Mayer, R., Stull, A., Campbell, J., Almeroth, K., Bimber, B., & Chun, D. (2007). Overestimation Bias in Self-reported SAT Scores. Eductational Psychological Review, 443-454.

Morgan, G. (1997). Images of Organization. London: Sage Publications. Nederhof, A. (1985). Methods for coping with social desirability bias: a review. European Journal of

Social Psychology, 263-280. Robert, K., Jako, A., & Anhalt, R. (1993). Nature and Consequences of Halo Error: A Critical Analysis.

Journal of Applied Psychology, 78(2), 218-225. Robles, M. (2012). Executive Perceptions of the Top 10 Soft Skills Needed in Today's Workplace.

Business Communication Quarterly, 75(4), 453-465. Roe, R. (2002). Competencies- Een sleutel tot integratie in theorie en praktijk van de A&O-

psychologie. Gedrag en Organisatie, 15(4), 203-222. Salas, E., Wilson, K., Priest, H., & Guthrie, J. (2006). Chapter 18: Design, Delivery, and Evaluation of

Training Systems. In E. Salas, K. Wilson, H. Priest, & J. Guthrie, Handbook of Human Factors and Ergonomics (pp. 472-512). Hoboken, NJ: John Wiley & Sons.

Sale, J., Lohfeld, L., & Brazil, K. (2002). Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research. Quality and Quantity, 43-53.

Saunders, M., Lewis, P., & Thornhill, A. (2009). Research methods for business students (fifth edition). Harlow: Pearson Education Limited.

Shoukri, M. (2011). Chapter 6: Coefficients of Agreement for Multiple Rates and Multiple Categories. In M. Shoukri, Measures of Interobserver Agreement and Reliability, 2nd Edition (pp. 213-249). Boca Raton: CRC Press, Taylor & Francis Group.

Sim, J., & Wright, C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85(3), 257-268.

Smith, J. (1983). Quantitative versus qualitative research: An attempt to clarify the issue. Educational Researcher, 12: 6-13.

Tannenbaum, S., & Yukl, G. (1992). Training and development in work organizations. Annual Review of Psychology, 43, 399-441.

Tannenbaum, S., Cannon-Bowers, J., Salas, E., & Mathieu, J. (1993). Factors that influence training effectiveness: A conceptual model and longitudinal analysis. Technical Rep. No. 93-011, Orlando, FL: Naval Training Systems Center.

van Aken, J., Berends, H., & van der Bij, H. (2007). Problem Solving in Organizations - A Methodological Handbook for Business Students. Cambridge: Cambridge University Press.

van der Horst, S. (2013). Competency Management: A self-assessment tool for competency needs analysis. Eindhoven: TU/e.

van der Zwaan, A. (1990). Organisatie-onderzoek: leerboek voor de praktijk: het ontwerp van onderzoek in organisaties. Assen: Van Gorcum.

Vanderlande. (2014). Year in Review 2014, Condensed Annual Report Financial Year 2014. Vanderlande Corporate.

Vanderlande. (2015). Vikipedia. Verheijden, D. (2005). Competentie Management.

56

APPENDIX I

Table 22: Current and Desired competence levels for the 'Collaboration IM/SA’ project

Competence Level A B C D

Organisational Precision and Sensitivity Current Level

Desired Level

Results Orientation Current Level

Desired Level

Cooperation Current Level

Desired Level

Planning and Organising Current Level

Desired Level

Monitoring Progress Current Level

Desired Level

Reflection Current Level

Desired Level

Oral Communication Skills Current Level

Desired Level

Multidisciplinary Cooperation Current Level

Desired Level

Current level

Desired level

Table 23: Desired competence levels for the 'sell valYou' project

57

APPENDIX II

Date: Group: Name of the training program:

(1=poor; 2=barely acceptable; 3=acceptable; 4=good; 5=excellent)

Amount of received feedback during training program. 1 2 3 4 5 NVT

Support from trainer during training program. 1 2 3 4 5 NVT

Motivation of the trainee. 1 2 3 4 5 NVT

Use of role play. 1 2 3 4 5 NVT

Use of an actor. 1 2 3 4 5 NVT

Use of real-life cases (based on company context). 1 2 3 4 5 NVT

Level of enthusiasm of the trainer. 1 2 3 4 5 NVT

Level of interaction with other participants. 1 2 3 4 5 NVT

Opportunity for own input. 1 2 3 4 5 NVT

Clear training program material. 1 2 3 4 5 NVT

Correct start level for training program (based on needs analysis). 1 2 3 4 5 NVT

Training program meets trainee expectations. 1 2 3 4 5 NVT

Clear objective of the training program. 1 2 3 4 5 NVT

Group size. 1 2 3 4 5 NVT

Pace/motion of the training program. 1 2 3 4 5 NVT

Theoretical versus practical issues. 1 2 3 4 5 NVT

Content validity of the soft skill training program (measures if the training program is designed to reflect the KSA’s as developed during needs assessment). Each competence the soft skill training was designed for can be placed in the following matrix: Comments about the soft skill training program:

58

APPENDIX III

The Great 8 18 Competency Dimensions 48 Behavioural Competencies

1. Providing Leadership & Decision-making

Need for power and control

1.1 Deciding & Taking action

Decisiveness

Making Choices

Taking the Initiative

Self-confidence/Independence

Organisational Sensitivity

1.2 Providing leadership & Supervising

Delegating

Group-oriented Leadership

Individual-oriented Leadership

2. Supporting & Cooperating Friendliness

2.1 Teamwork & Supporting

Providing Feedback

Listening

Cooperation

2.2 Helping Customers & Clients

Customer-orientation/Service Provision

Advising

Maintaining Relationships

3. Interacting & Presenting Extraversion

3.1 Creating Relationships & Networking

Sociability

Sensitivity

3.2 Persuading

Persuasiveness

Negotiating

3.3 Communicating & Presenting

Presenting

Oral Communication Skills

Appropriate Conduct

4. Analysing & Interpreting General Reasoning Ability (‘IQ’)

4.1 Application of Experience & Technology

Knowledge of Environmental Factors

Corporate Social Responsibility

4.2 Problem-solving

Evaluating/Judgment

System Thinking

Analysing

5. Creating & Conceptualizing Being open to Experience

5.1 Learning & Researching Learning Ability

Learning to Learn

5.2 Creating & Innovating

Creativity

Innovating

Designing

5.3 Forming Strategies & Concepts Vision

Defining and Developing Strategies

6. Organising & Implementing Conscientiousness

6.1 Planning & Organising Planning and Organising

Monitoring Progress

6.2 Following Instructions and Procedures

Discipline

Accuracy

Safety Awareness

7. Adapting & Processing

Emotional Stability

7.1 Adapting & Reacting to Change Flexibility/Adaptability

7.2 Dealing with Pressure & Setbacks

Stress Resistance

Perseverance

Dealing with Criticism

8. Enterprise & Performance Drive to Achieve

8.1 Achieving Results & Career Development

Ambition

Results Orientation

Energy/Enthusiasm for Work

Reflecting (Self-reflection)

8.2 Enterprise & Commercial Thinking Daring

Entrepreneurship

Figure 13: Vanderlande Competence set with the Bartram (2005) Hierarchy, derived from van der Horst (2013)

59

APPENDIX IV

As already mentioned, Fleiss’ Kappa was used to check the inter-rater reliability. This section discusses the derivation of Fleiss’ Kappa formula.

The notation for Fleiss’ Kappa is as follows. Let N represent the total number of subjects, n the number of ratings per subject, and k the number of categories into which assignments are made. Let the subscript i, where i = 1, N, represent the subjects, and the subscript j, where j = 1, k, represent the categories of the scale. Define nij to be the number of assessors who assigned the ith subject to the jth category, and define (Fleiss, 1971):

N

i

ijj nNn

p1

1

Some extent of agreement is to be expected solely on the basis of chance. If the assessors made their assignments purely at random, one would expect the mean proportion of agreement to be (Fleiss, 1971):

k

jje pP

1

2

The overall extent of agreement can be measured by (Fleiss, 1971):

N

i

iPN

P1

1 , where

k

jiji nn

nnP

1

)()1(

12

The quantity eP1 measures the degree of agreement attainable over and above what

would be predicted by chance. The degree of agreement actually attained in excess of chance is

ePP , so that a normalized measure of overall agreement, corrected for the amount expected by

chance, known as Fleiss’ Kappa (Fleiss, 1971):

e

e

P

PP

1

60

APPENDIX V

Individual level below collective level

Individual level above collective level

Note: the last column shows the translation of the A, B, C, or D’s in absolute values (i.e. A=1; B=2; C=3; D=4) for comparing the overall scores. Competence: Organisational Precision and Sensitivity

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Pre-measurement (collective)

B B B B B B B B B B B B B B


C C C C C C C C C C C C C C 42

Post-measurement (individual)

C B C C B C C B C C C D C C 40

à Result: on average, the estimation of the competence level during the post-measurement is lower for the individual compared to the estimated level by the group as a whole. Competence: Results Orientation

1 2 3 4 5 6 7 8 9 10 11 12 13 14






C C C B C B B B C C C C C C 38

à Result: on average, the estimation of the competence level during the post-measurement is lower for the individual compared to the estimated level by the group as a whole. Competence: Cooperation

1 2 3 4 5 6 7 8 9 10 11 12 13 14






C C C C B C C C C C C D C C 42

à Result: on average the estimation of the competence level during the post-measurement remains the same by comparing the individual and the group as a whole. Competence: Planning and Organising

1 2 3 4 5 6 7 8 9 10 11 12 13 14




B B B B B B B B B B B B B B 28


C B C C C B C B C C B C C B 37

à Result: on average, the estimation of the competence level during the post-measurement is higher for the individual compared to the estimated level by the group as a whole.

61

Competence: Monitoring Progress

1 2 3 4 5 6 7 8 9 10 11 12 13 14






D B D C C B B C C C C D C D 43

à Result: on average, the estimation of the competence level during the post-measurement is higher for the individual compared to the estimated level by the group as a whole. Competence: Reflecting

1 2 3 4 5 6 7 8 9 10 11 12 13 14


A A A A A A A A A A A A A A




B C B B D B D B C C C C C D 40

à Result: on average, the estimation of the competence level during the post-measurement is higher for the individual compared to the estimated level by the group as a whole. Competence: Oral Communication Skills

1 2 3 4 5 6 7 8 9 10 11 12 13 14


A A A A A A A A A A A A A A




C C C C C D B B D C C C B C 41

à Result: on average, the estimation of the competence level during the post-measurement is higher for the individual compared to the estimated level by the group as a whole. Competence: Multidisciplinary Cooperation

1 2 3 4 5 6 7 8 9 10 11 12 13 14






C B C C B B C B C C C C C C 38

à Result: on average, the estimation of the competence level during the post-measurement is higher for the individual compared to the estimated level by the group as a whole.

62

APPENDIX VI

Observational checklist for the ‘Collaboration IM/SA’ project

Date: 03-09-2015 (DAY 3) Group: Collaboration IM/SA, 14 participants Name of the training program: Samenwerken (Collaboration IM/SA)



Support from trainer during training program. 1 2 3 4 5 NVT









Correct start level for training program (based on needs analysis).

1 2 3 4 5 NVT


Clear objective of the training program. 1 2 3 4 5 NVT


Pace/motion of the training program. 1 2 3 4 5 NVT


(1 = Organisational Sensitivity and Precision; 2= Results Orientation; 3 = Cooperation; 4 = Planning and Organising; 5 = Monitoring Progress; 6 = Reflecting; 7 = Oral Communication Skills; 8 = Multidisciplinary Cooperation) Comments about the soft skill training program: The trainer is very professional and the role play with an actor was seen as very valuable. Furthermore, the participants thought that not all competences were discussed during the training program. The manager of the system architects and integration manager was very enthusiastic and this has a positive effect on the motivation of the participants.

1,8 5

7 2,3,4,6

63

Observational checklist for the ‘sell valYou’ project

Date: 18 – 05 -2015 Group: Bagage & WPP, 11 participants Name of the training program: Listening & Analyzing (sell valYou)



Support from trainer during soft skill training program. 1 2 3 4 5 NVT









Correct start level for soft skill training program. 1 2 3 4 5 NVT


Clear objective of the soft skill training program. 1 2 3 4 5 NVT


Pace/motion of the soft skill training program. 1 2 3 4 5 NVT


Note: the attended soft skill training program only focuses on the competences ‘Listening’ and ‘Analysing’. Therefore, the conceptual diagram to determine content validity is missing. Comments about the soft skill training program: A lot of interaction and feedback takes place between the trainer and between the participants. Combination of the two trainers works very well. The attendance of someone from the management board was very beneficial for creating support and transfer of training for the WPP participants.

Date: 19 - 05 -2015 Group: Bagage & WPP, 11 participants Name of the training program: Listening & Analyzing (sell valYou)











64



Soft skill training program meets trainee expectations. 1 2 3 4 5 NVT





Note: the attended soft skill training program only focuses on the competences ‘listening’ and ‘analysing’. Therefore, the conceptual diagram to determine content validity is missing. Comments about the soft skill training program: A lot of interaction takes place. Many anecdotes that makes the participant very enthusiasm about the training program. The feedback that was given with regard to the sales game was very helpful. There exists a very good collaboration between the two trainers. There was a bit more emphasis on Analysing compared to Listening, but this seems a good balance for most of the participants. Observational checklist for the ‘Professional Communicating’ project

Date: 12-01-2015 (DAY 1) Group: Com-5-53, 8 participants Name of the training program: Professioneel Communiceren

(1=poor; 2= barely acceptable; 3=acceptable; 4=good; 5=excellent)

















65

(1 = Providing Feedback; 2 = Listening; 3 = Oral Communication Skills; 4 = Persuasiveness; 5 = Dealing with Criticism; 6 = Results Orientation; 7 = Reflecting; 8 = Maintaining Relationships) Comments about the soft skill training program: The trainer is very enthusiastic and is a beginner within performing this training program. In the beginning, the participants are a bit restrained. After a while they became also enthusiastic and want to learn new things. During exercises they all participate in a good way and after a while they became honest about their weaknesses instead of only telling their strengths.

Date: 26-01-2015 (DAY 2) Group: Com-5-53, 8 participants Name of the training program: Professioneel Communiceren

(1=poor; 2= barely acceptable; 3=acceptable; 4=good; 5=excellent)

















1,2

3,5 6

4,7,8

66

(1 = Providing Feedback; 2 = Listening; 3 = Oral Communication Skills; 4 = Persuasiveness; 5 = Dealing with Criticism; 6 = Results Orientation; 7 = Reflecting; 8 = Maintaining Relationships)

Comments about the soft skill training program: The level of interaction is still increasing during the time. They want to learn a lot and the cooperation with the actor works very well. The trainer also is very good at summarising and asking for improvements from the participants.

4,6,7

8 3,5

1,2

Documents

Eindhoven University of Technology MASTER Improving the ... · Skill Training Programs at Vanderlande Eindhoven, 2015June E. (Elsa) Buijssen BSc Industrial Engineering & Innovation