Measuring Employability Gain approaches from different ... · PDF fileeducationally purposeful activities that contribute directly to ... experiences Some items are ... developed Often

1 Qualitätssicherung und -entwicklung an systemakkreditierten Hochschulen Expertengespräch Systemakkreditierung, Berlin, 13. November 2017

Measuring Employability Gain – approaches from different countries

Methods for assessing competencies in higher education in Germany – and its reflection

PD Dr. Edith Braun HIS-HE Higher Education Governance


In the past: higher education was aiming for academic education

Now Over 50% of a cohort starts higher education programmes

– graduates enter labour market


Definition

• "competences“

goes beyond cognitive skills – Mastering complex situations, adaptation to novel settings and situations, “effective citizenship”

• "employment-related competences”

cognitive (discipline-specific) and non-cognitive aspects of competences: personal and social skills, leadership and communication skills, life long learning

• Definition of Masten and Coatsworth (1998)

‘a pattern of effective adaptation to the environment, […] broadly defined in terms of reasonable success with major developmental tasks expected for a person of a given age and gender in the context of his or her culture, society, and time […] This definition takes into account the dual meaning of competences and considers track record of achievements in the past (competent performance) as well as individual’s capability to perform well in future.’

Edith Braun


Assessment of Competences

Indirect measurements:

1. Self-report of competences

2. Job requirement approach

3. Student engagement approach

Direct measurements.

4. Achievement tests

5. Performance based testing: Role plays

Five established approaches:

Edith Braun


CIRP (Sax, Astin, Korn, & Mahoney, 2000):

“rate your current level of competence in comparison to an average person of your age“

‘‘Highest 10%’’, ‘‘Above average’’, ‘‘Average’’, ‘‘Below average’’ and ‘‘Lowest 10%’’

Questionnaire:

1) „Leadership ability“

2) „Public speaking ability“

3) „Self-confidence (social)“

Based on self-report of competences rely on students’ self-perceptions and assessment of their competences


Edith Braun



Advantages

Help to assess how students perceive themselves

Meaningful correlations between self-reports and behavior

External validity of self-reports

Systematic correlations with external criteria (independent observer ratings, course grades, learning styles of students)

Assessment of communication skills and personal/social development through items on the questionnaires

Items are found to be internally consistent with each other (high Cronbach’s alpha)

Simple and cost-efficient

Disadvantages

Not completely free of biases

Questionable quality of self-report assessment using in case of vague formulation of items

Prone to cultural and context-specific influences and interpretations

Based on self-report of competences rely on students’ self-perceptions and assessment of their competences

Edith Braun



Basic assumptions:

1. activities that individuals engage in at their workplace reflect the demands and requirements of their work

2. the best way to get information about job-related activities and requirements is to ask the employees directly

3. generally, employees are able to share information about their duties and requirements at work in a less biased manner and also indicate how often they engage in what type of activities

4. compared to self-report JRA allows for using job-related activities and requirements as a potentially less biased proxy for measuring job-related competences

Approach for measuring vocational tasks and activities (Felstead et al. 2007; Klaukien et al. 2013)

Edith Braun


• Based on self-descriptions and cannot differentiate between various levels of competences

We suggest a cautious interpretation of activities as a proxy for competences

• Helpful in identifying activities that higher education graduates engage in at work

• Identification of amount of graduates who are on an (in-)adequately position

• Addresses many of the weakness of self-ratings

• Take into account the actual activities and behaviors that a person is involved with

• Questions are asked within the context of the work environment

Advantages: Disadvantages:


Edith Braun



This approach has been employed in surveys for assessing students’ learning environment and activities they engage in at higher education institutions (Kuh, 2003) “the quality of effort students themselves devote to educationally purposeful activities that contribute directly to desired outcomes.” (Hu & Kuh, 2002) Four most prominent subscales (‘Benchmarks of Effective Educational Practices’) (McCormick, A. C. , J. Kinzie & R. M. Gonyea; 2013):

1. academic challenge

2. active and collaborative learning

3. student-faculty interaction

4. supportive campus environment

Edith Braun


• Unclear definition of ‘educationally purposeful activities’

need for more theoretically and empirically driven research

• Empirical correlations between the surveyed activities and more objectives measurements of learning outcomes (GPA) are quite weak

• Helps to identify aspects of teaching and learning which need improvement

• Students provide relevant information regarding activities they have been involved with in higher education

• Adoption of the NSSE survey in itself is an intervention in promoting students’ engagement

• Opportunity to voice the opinion on the overall learning process and environment which could in turn lead to better engagement in higher education

• Main advantage is its underlying emphasis on learning opportunity

Advantages: Disadvantages:


Edith Braun



Direct measurement of the level of competences of students and graduates

• Measuring cognitive, mostly discipline-specific competences or knowledge

A lack of tests available for objectively assessing students’ competences in higher education

IQ tests are sometimes used

Important distinction between competences and intelligence

Edith Braun


AHELO (Assessment of Higher Education Learning Outcomes)

Conduct a international and inter-institutional comparison of competences in higher education system

between 2009 and 2012 implemented on a pilot basis

17 countries participated

• Areas of competences: ‘generic skills strand’, ‘economics strand’, and ‘engineering strand’

• Not proceeded

KoKoHs (Competency models and instruments of competence assessment in higher education)

German national wide program

Create a important propulsion and to close gaps in the area of competence assessment in higher education

• First funding period (2011-2015)

24 projects were initiated

• Second periode (2015-2019)

15 projects have been initiated

Edith Braun


• scientific and methodological

challenges Students are highly

diverse in their profiles and educational experiences

Some items are hard to translate into other languages

• Students are often not motivated to take achievement tests

• Do not take into account the dimensions and complexity of competences in a broad understanding

• Allow a direct measurement of the level of competences of students and graduates

Advantages Disadvantages:


Edith Braun



Modeling competence as a continuum (Blömeke et al. 2015)

Edith Braun


Put students in specific scenarios where they take on different roles and responsibilities

Advantages over other approaches:

• Socially accepted method to activate competences that are of interest for training and assessment needs

a sufficient method to assess competences in the way we have defined it above

• Students are highly motivated to participate in role plays • Instructions and standardised observer forms


Edith Braun


Conclusion

Edith Braun


Concluding remarks Comparison of five approaches for assessing competences

Self-reports

Job requirements

Student engagement

Achievement tests

Role plays

Definition of competence

Mostly not existing

Self-rated performance of job-related activities

learning opportunities and activities that students are engaged in

Cognitive skills and/or knowledge

Performance in complex and authentic situations

Indirect/ direct assessment

Indirect assessment

Indirect assessment

Indirect assessment

Direct assessment

Direct assessment

Added value (changes in competence over time)

No Less quantifiable

Less quantifiable

Quantifiable on level of competences

Quantifiable on level of competences

Effort of development

Required effort often under-estimated

Often well developed




Effort of applying

Low Low Low High High

Students` Motivation to participate

Medium Medium Medium Low High

Edith Braun


It is essential to either use these approaches in combination or to develop innovative methods which are equally suitable for measuring discipline-related as well as more generic competences such as leadership ability, social well-being, life long learning

Edith Braun


Based on

Braun, E., & Mishra, S. (2016). Methods for Assessing Competences in Higher Education: A Comparative Review. In Theory and Method in Higher Education Research (pp. 47-68). Emerald Group Publishing Limited.

Braun, E., Woodley, A., Richardson, J.T., Leidner (2012). Self-rated competences questionnaires from a design perspective. Educational Research Review, 7, 1–18.

Braun, E. M., & Brachem, J. C. (2015). Requirements Higher Education Graduates Meet on the Labor Market. Peabody Journal of Education, 90(4), 574-595.

Edith Braun, Ulrike Schwabe, Georgios Athanassiou, Daniel Klein (under review). Performance-based tests: Using role plays to assess communication skills. International Handbook of Vocational Educational Training. Springer.

Edith Braun


Edith Braun

HIS-HE Hochschulmanagement Hannover

[email protected]

Feedback and Discussion

Remarks?

Questions?

Edith Braun


Higher education and its complexity

Braun, Weiß & Seidel (2014)

Edith Braun


Result of changes

• key concepts in quality assurance and accreditation of degree programs and higher education institutions

• Focus on the knowledge society

stronger push towards equipping higher education graduates with the right skills and competences for labor market requirements as well as for social well-being

• Responsibility of higher education institutions

help their graduates acquire and develop broad sets of skills and competences that match with the changing requirements of the employers and labour market

E.g. non-cognitive competences & skills as important outcomes of higher education in the context of a dynamic labour market

Edith Braun


References • Alkin, M. C. & Christie, C. A. (2002). The Use of Role-Play in Teaching Evaluation. American Journal of

Evaluation, 23(2), 209–218. • Allen, J., Ramaekers, G., Velden, R. van der., (2005) Measuring Competencies of Higher Education Graduates.

In D. J. Weerts & J. Vidal (Eds.), Enhancing Alumni Research: European and American Perspectives, New Directions for Institutional Research. Number 126 (pp. 49-59). Jossey-Bass, San Francisco.

• Astin, A. (1999). Student involvement: A developmental theory for higher education. Journal of College Student Development, 40(5), 518-529.

• Beard, R. L., Salas, E., & Prince, C. (1995). Enhancing transfer of training: using role-play to foster teamwork in the cockpit. The International journal of aviation psychology, 5(2), 131–143.

• Blömeke, S., Gustafsson, J. E., & Shavelson, R. J. (2015). Beyond Dichotomies. Zeitschrift für Psychologie, 223 (1), 3-13.

• Blömeke, S., Zlatkin-Troitschanskaia, O., Kuhn, C., & Fege, J. (2013). Modeling and measuring competencies in higher education: Tasks and challenges. In Blömeke, S., Zlatkin-Troitschanskaia, O., Kuhn, C., & Fege, J. (eds.) Modeling and measuring competencies in higher education: Tasks and challenges, Rotterdam, The Netherlands: Sense Publishers.

• BMBF (Bundesministeriums für Bildung und Forschung), Koordination DZHW (2013). Datenportal. http://www.datenportal.bmbf.de/portal/de/K25.gus.

• Bologna Declaration. (1999). The Bologna declaration of June 1999. Joint declaration of the European Ministers of Education. Retrieved from http://www.ond.vlaanderen.be/hogeronderwijs/bologna/documents/ MDC/BOLOGNA_DECLARATION1.pdf.

• Boni, A., & Lozano, J.F. (2007). The generic competences: An opportunity for ethical learning in the European convergence in higher education. Journal of Higher Education, 54, 819-883.

• Braun, E. M., & Brachem, J. C. (2015). Requirements Higher Education Graduates Meet on the Labor Market. Peabody Journal of Education, 90(4), 574-595.

• Braun, E., & Leidner, B. (2009). Academic course evaluation: Theoretical and empirical distinctions between self-rated gain in competences and satisfaction with teaching behavior. European Psychologist, 14(4), 297-306.

• Braun, E., Athanassiou, G., Pollerhof, K. & Schwabe, U. (submitted). Wie lassen sich kommunikative Kompetenzen messen? Konzeption einer kompetenzorientierten Prüfung kommunikativer Fähigkeiten.

• Braun, Edith, Alan Woodley, John TE Richardson, and Bernhard Leidner. "Self-rated competences questionnaires from a design perspective." Educational Research Review 7, no. 1 (2012): 1-18.

• Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages*. Research in higher education, 47(1), 1-32.

• Centra, J. (1998). The development of the Student Instructional Report II. Princeton, NJ: Educational Testing Service

• Chan, R.Y., Brown, G.T.L., & Ludlow, L.H. (2014). What is the purpose of higher education?: A comparison of institutional and student perspectives on the goals and purposes of completing a bachelor’s degree in the 21st century. Paper to be presented at the annual American Education Research Association (AERA) conference. Philadelphia, PA: April 5, 2014. Retrieved from https://www.dal.ca/content/dam/dalhousie/pdf/clt/Events/Chan_Brown_Ludlow%282014%29.pdf


• Chapman, J.D. and Aspin, D.N. (2013). A problem-solving approach to addressing current global challenges in education. British Journal of Educational Studies, 61(1), 49-62.

• Coates, H., & Mahat, M. (2014). Assessing student engagement and outcomes: Modelling insights from Australia and around the world. International Journal of Chinese Education, 2(2), 241-264.

• Dublin Descriptors. (2004). Shared Dublin Descriptors for Bachelor’s, Master’s and Doctoral awards. Retrieved from http://www.tempus.ac.rs/here/tl_files/Dokumenti/Dublinski%20deskriptori.pdf.

• Enders, J., De Boer, H., File, J., Jongbloed, B., & Westerheijden, D. (2011). Reform of higher education in Europe (pp. 1-10). SensePublishers.

• European Commission. (2011). Supporting growth and jobs- an agenda for the modernisation of Europe’s higher education systems. Brussels: European Commission.

• Felstead, A. et al., 2007. Skills At Work, 1986 to 2006, Oxford: ESRC Centre on Skills, Knowledge and Organisational Performance.

• Gonyea, R. M., Kish, K. A., Kuh, G. D., Muthiah, R. N., & Thomas, A. D. (2003). College Student Experiences Questionnaire. Norms for the fourth edition. Bloomington, IN: Indiana University Center for Postsecondary Research, Policy and Planning.

• Gordon, J., Ludlum, J., & Hoey, J. J. (2008). Validating NSSE against student outcomes: Are they related?. Research in Higher Education, 49(1), 19-39.

• Guliker, J. T. M., Bastiaens, T. J., Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67-86.

• Habermas, J. (1984). The theory of communicative action, Volume I. Boston: Beacon. • Harden, R. M., & Gleeson, F. A. (1979). Assessment of clinical competence using an objective structured

clinical examination (OSCE). Medical education, 13(1), 39-54. • Hartig, J., Klieme, E., & Leutner, D. (Eds.). (2008). Assessment of competencies in educational contexts.

Hogrefe Publishing. • Hovdhaugen, E. (2013). AHELO: Norway’s Experience of Participation in the Feasibility Study. In Braun, E.,

Donk, A., & Bülow-Schramm, M. (Eds.). AHELO goes Germany. Hannover: HIS. 7-12. • Klaukien, A., Ackermann, D., Helmschrott, S., Rammstedt, B., Solga, H. & Wößmann, L. (2013). Grundlegende

Kompetenzen auf dem Arbeitsmarkt. In B. Rammstedt (Ed.), Grundlegende Kompetenzen Erwachsener im internationalen Vergleich. Ergebnisse von PIAAC 2012 (p. 127-166). Münster: Waxmann.

• Koucky, Jan, Christoph Meng, and Rolf Van Der Velden. "Reflex country study." Maastricht: Research Centre for Education and the Labour Market/REFLEX (2007).

• Kuh, G. D. (2003). What we're learning about student engagement from NSSE: Benchmarks for effective educational practices. Change: The Magazine of Higher Learning, 35(2), 24-32.

• Kuh, G. D. (2009). The National Survey of Student Engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 141, 5–20.

• Kunina, O., Wilhelm, O., Formazin, M., Jonkmann, K., & Schroeders, U. (2007). Extended criteria and predictors in college admission: Exploring the structure of study success and investigating the validity of domain knowledge. Psychology Science, 49(2), 88.

• Kyllonen, P. (2005). The case for noncognitive assessments. R & D Connections, 1-7. • Lawless, C., & Richardson, J. T. E. (2004). Monitoring the experiences of graduates in distance education.

Studies in Higher Education, 29, 353–374.


• Lisbon Strategy. (2000). The Lisbon Strategy in short. Retrieved from https://portal.cor.europa.eu/europe2020/Profiles/Pages/TheLisbonStrategyinshort.aspx.

• Lozano, J. F., Boni, A., Peris, J., & Hueso, A. (2012). Competencies in higher education: A critical analysis from the capabilities approach. Journal of Philosophy of Education, 46(1), 132-147.

• Lucas, R., & Baird, B. (2006). Global self assessment. In M. Eid & E. Diener (Eds.), Handbook of psychological measurement: A multimethod perspective (pp. 29–42). Washington, DC: American Psychological Association.

• Masten, A.S. & Coatsworth, J.D. (1998). The development of competence in favorable and unfavorable environments. American Psychologist 53 (2), 205-220.

• Mulder, M., Gulikers, J., Wesselink, R., & Biemans, H. (2008). The new competence concept in higher education: error or enrichment. Paper presented to AERA New York. Retrieved from http://www.mmulder.nl/PDF%20files/2008-03-31%20Paper%20Mulder%20AERA%202008.pdf.

• OECD (2013). OECD Skills Outlook 2013. First results from the survey of adult skills. Paris: OECD. • Pascarella, E. T., Seifert, T. A., & Blaich, C. (2010). How effective are the NSSE benchmarks in predicting

important educational outcomes?. Change: The Magazine of Higher Learning, 42(1), 16-22. • Peterson, N. G., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K. Y., Campion, M. A., Mayfield, M. S.,

Morgeson, F. P., Pearlman, K., Gowing, M. K., Silver, M. B. & Dye, D. M. (2001). Understanding work using the Occupational Information Network (O*NET): Implications for practice and research. Personnel Psychology, 54(2), 451-492.

• Sax, L. J., Astin, A. W., Korn, W. S., & Mahoney, K. M. (1999). The American Freshman: National Norms for Fall 1999. Higher Education Research Institute, UCLA Graduate School of Education and Information Studies, 3005 Moore Hall, Mailbox 951521, Los Angeles, CA 90095-1521.

• Spinath, B., Spinath, F. M., Harlaar, N., & Plomin, R. (2006). Predicting school achievement from general cognitive ability, self-perceived ability, and intrinsic value. Intelligence, 34(4), 363-374.

• Umbach, P. D., & Wawrzynski, M. R. (2005). Faculty do matter: The role of college faculty in student learning and engagement. Research in Higher Education, 46(2), 153-184.

• Van Ments, M. (1999). The effective use of role-play: Practical techniques for improving learning (2nd ed.). London: Kogan Page.

• Walker, M. (2003). Framing social justice in education: What does the capabilities approach offer? British journal of educational studies, 51, 168-187.

• Westerheijden, D. F., Stensaker, B., & Rosa, M. J. (Eds.). (2007). Quality assurance in higher education: Trends in regulation, translation and transformation. Dordrecht, NL: Springer.

• Wilson, K., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the Course Experience Questionnaire. Studies in Higher Education, 22, 33–53.

• Winteler, A. & P. Forster (2008). Lern-Engagement der Studierenden: Indikator für die Qualität und Effektivität von Lehre und Studium. Das Hochschulwesen, 56, 162-170.

• Zlatkin-Troitschanskaia, O., Pant, H. A., Kuhn, C., Toepper, M. & Lautenbach, C. (2016). Messung akademisch vermittelter Kompetenzen von Studierenden und Hochschulabsolventen. Ein Überblick zum nationalen und internationalen Forschungsstand. Wiesbaden: Springer.

• Zlatkin-Troitschanskaia, O., Shavelson, R.J., & Kuhn, C. (2015). The international state of research on measurement of competency in higher education. Studies in higher education, 40(3), 393-411.

https://portal.cor.europa.eu/europe2020/Profiles/Pages/TheLisbonStrategyinshort.aspx

http://www.mmulder.nl/PDF files/2008-03-31 Paper Mulder AERA 2008.pdf





Documents

Measuring Employability Gain approaches from different ... · PDF fileeducationally purposeful activities that contribute directly to ... experiences Some items are ... developed Often