© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 1
Evaluation, test and development ofsoft skills
Evaluation of the test: results from students´projectsWhat do test measure?Testing soft skills
Methods of measuring and improving soft skillsespecially planning games
Prof. Dr. Ulrich Holzbaur AAUAS - Aalen University of Applied Science
faculty of economics, dept. for engineering management STZ Applied Management, Aalen
Steinbeis Tranfer Centre for Applied Management, Aalen
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 2
Own experiences/ backgroundProf. Dr. U. D. Holzbaur
Involved in education, teaching, training , leadership since 1974 Actively involved in training/teaching with Army, University Studies for secondary education Head, airborne radar systems group, EADS
Professional experience in Flight Trainer Design with EADS Prof. at AAUAS since 1990, students´projects in:
Computer Science and Software Development Environmental Management and Project Management First grade students´research projects 100 master thesis at Ulm University and AAUAS
LARS project „Planning Games“ Grant from the Studienkommission Hochschuldidaktik Baden-Württemberg
(university education commission) 1997 to 2002 workshops on planning games 1997 to 2002 Development of planning games for sustainability, management, skills
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 3
Evaluation of the tests by students´ teams
Situation: within their 1st and 7th semester, students work on a project in teams of 4 to 5 1st term: methods of project management and research 7th term: quality- and environmental management Two joint projects:
Public transportation system for students´ requirements, students´ticket Evaluation of the test
Participants: Pascal Cromm, Antje Discher: tests 4 + 5 students´ teams as object (participants) 1 + 1 students´ teams doing the evaluation
Students´ project run from march 2003 to July 2003 (final report)
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 4
Example of the results from tests
-30
-20
-10
0
10
20
30
No
rma
lbe
reic
h li
eg
t b
ei -
48
bis
+ 4
8
Gruppen
Interkulturelle Kommunikation
Soft Skills
ÖPNV
Event
Grüner Gockel
"W"
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 5
Evaluation or measurement of skills
With respect to any property PX of a person X we have several values which we may try to measure:
PX "how am I" is the "real" value of X however that may be defined. I(PX) "index" is an indicator for X, e.g. the classical IQ defined by an IQ test. S(PX) "how do I see me" is the self evaluation of the individual. F(PX,Y) "how do others see me?" is the external evaluation by another person or
group of persons Y, this is the image or aura of a person. S(F(PX,Y)) "how do I assume to be seen by others?" "do I know my own image?" F(S(PX),Y) "how do others assess my self-estimation?" F(F(PX,Y),Z) "how do others assess my image?"
In addition, the property PX leads to some performance P in a real world situation R or to some result P in a test situation T.
With regard to testing, we have I(PX) = P(PX,T). The 180°-feedback measures P(F(PX,Y=peer-group), T=180°-feedback)
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 6
What do tests measure?
The tests measure P(F(PX,Y),T) in comparison to P(S(PX),T) by using the same test.
The real blind spot ist the difference between F(PX,Y) and S(F(PX,Y))which could be assessed by testingP(F(PX,Y),T) in comparison to P(S(F(PX,Y)),T).
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 7
Test principle and structure
skills and behaviour
Self-assessment
Perception by others
180°- feedback
test
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 8
What can tests measure? e.g. intercultural competence
Ethical Competence: dealing with different sets of values To decide in an ethical conflict
Personal Competence and awareness: perception of different values, behavior and scripts in different cultures
To see when and whether a intercultural difference occurs Social Competence/ Management competence
To be able to integrate different cultures, to be able to lead a team with members from different cultures
Problem solving competence To solve problems embedded in or arising from
an intercultural difference Knowledge
To know the parameters that determine a cultural behavior Facts
To know the dos and donts of specific cultures
tests
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 9
Results from the students´questionnairs
Project: questionnaire for self-assessment questionnaire to assess the team members
questionnaire developed by students´s team to assess the test
self
180°-feedback
evaluation of the test
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 10
Evaluation Typical distribution of students´ answers: fine/ good 20%, poor: bad: 20% too complex (> 50 %) took too much time ( > 50 %)
typical responses (free test) tests are too long and too complex many questions that are similar problems to see the meaning students get tired of filling the questionnaires not adapted to / adequate for students´situation
suggestions (free text) test should be anonymous role play would make more sense same method within all test
Evaluation and typical response
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 11
Overall evaluation by the students:do the test make sense?
Fandest Du diese Tests sinnvoll?
0
2
4
6
8
10
12
14
16
10=s
ehr g
ut 9 8 7 6 5 4 3 2 1
0=se
hr sc
hlech
t
Bewertung
Hä
ufi
gk
eit
Innovation/Kreativität & Risikobereitschaft
Team und Projektmanagement
Interkulturelle Kompetenz
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 12
Problems with translation and meaning
e.g.: „Ich verhalte mich unvermutet“ -> unexpected -> spontaneous behaviour positive ? creativity ? within a management/ leadership context, this is a rather negative property:
unberechenbar = unpredictable, unfair.
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 13
Overall evaluation of the quality of the tests:points (max = 10)
innovation/ risk taking skills: 5.5 team & project management skills: 5.5 intercultural competences: 4.2
Suggestions: Test should be shorter, more precise and systematic Try to compare
self assessment to „real“ skills perception by others to self assessment of perception
There should be some method / tool to measure the skills Test should be in a setting, preceded or based on some common experience
e.g. an extensive project experience.
Summary
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 14
Planning Game:teaching and experiment
planning game:management game,simulation gamewargame
Model of the real world Simulation Controlled experiment Method of teaching Method of learning
Next, let us lower the sales figures and measure his stress symptoms
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 15
Planning Games
Learning by playing Didactics Situations
Planning Game: Initial Conditions = Starting Position Decisions to be made Dynamics = Rule Criteria
Conceptual Basis: Dynamics Decision Goal Didactics
Planning game is more than hard + soft - ware
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 16
Planning games
Role-Based Computer-Based
Facts
SelfProblem solving
Your role is ... You are ....You are responsible for ...You will be evaluated ...You may ...
Your role is ... You are ....You are responsible for ...You will be evaluated ...You may ...
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 17
Simulation games
Wargames and strategic games The Settlers of Catan, …..
Economic Simulation Games with graphic visualization Beer, Zoo Tycoon, …
Number-Based Economic Simulation Games TOPSIM, FondManager, ….
Management Simulations Project management,
Internet Based Games MARGA, TOPSIM, StockMarketTraining
Paper and Board Simulation Factory,
Startup Computer Simulation (typically based on MS-Excel) UGS-SIM, UGS-Game
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 18
Computer based planning gamese.g.: TOPSIM Startup
strategic start-up simulation Phases
Idea and business plan Foundation - constitutive decisions Market entry and competition Evaluation
business areas Production (high-tech sports equipment) Commerce (eCommerce, books/CDs) Services High-Tech (bio-technology)
Problem areas/ challenges Business / finance Technical Social
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 19
Other examples of planning games for management skills (paper based)
Number factory Players “produce” numbers Each player has different
Information Role and responsibility Behaviour
Planning takes place on different levels: Production (to calculate the mumbers) Organization (mass production, push-pull, R&D, ) Communication (rules, culture)
Albuchmühle Players plan the future for a mill Core aspects:
Marketing Environmental and risk management
Players have hidden agenas
Your role is ...
You are ....
You are responsible for ...
You will be evaluated ...
You may ...
You want to ...
You are planning to ...
Your role is ...
You are ....
You are responsible for ...
You will be evaluated ...
You may ...
You want to ...
You are planning to ...
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 20
Öko² - oeko2 Planning Game for Sustainable Development
Evaluation of the player according to the relevant variables
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 21
How to use planning games for skills development
Education (university, school): Special course within the curriculum Embedded method within courses on special subjects
management, quality/ project/ risk/ environmental management accounting, marketing, controlling, enterpreneuring any other subject
Startup for newcomers: Facts Skills Motivation
Lifelong learning: Test & Evaluate … Gain & Improve & Enlarge … Evaluation of skills
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 22
Planning games match entrepreneurship skills
Planning games provide an optimal match for the competences needed by entrepreneurs:
Give the potential entrepreneur the Facts and Knowledge Problem solving competence Social Competence and Intercultural Competence Personal Competence and self-esteem Ethical Competence
which are needed to Plan: start a company and make the right decisions Communicate: get the money, personnell and customers Manage: run the company Control the operations and establish the management systems Decide optimally in routine and critical situations
Business Simulations allow to identify misplanning and risks
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 23
How to use planning games for skills assessment
Start with planning games Dedicated games for
Skills Aspects Properties
© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 24
Summary
Test have been evaluated by students´ teams too complex too much questions with little differentiation inhomogeneous answering and evaluation system what do the tests measure?
Planning Games are a valuable tool to teach, experiment, learn evaluate soft and hard skills Create a basis for skill evaluation
Planning games can support soft skills Facts, knowledge and understanding: I KNOW Motivation and self-esteem: I CAN Action: I WILL