Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:
McNamara, Judith & Field, Rachael(2007)Designing Reflective Assessment for Workplace Learning in Legal Educa-tion.In Southwell, D (Ed.) ATN Evaluation and Assessment Conference.Department of Teaching and Learning Support Services, QUT, Australia,pp. 87-93.
This file was downloaded from: https://eprints.qut.edu.au/12576/
c© Copyright 2007 QUT Teaching and Learning Support Services andJudith McNamara and Rachael Field
This work is copyright. Apart from any use permitted under the CopyrightAct 1968, no part may be reproduced by any process without written per-mission from the copyright holders.
Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.
https://eprints.qut.edu.au/view/person/McNamara,_Judith.htmlhttps://eprints.qut.edu.au/view/person/Field,_Rachael.htmlhttps://eprints.qut.edu.au/12576/
2007
ASSESSMENT AND EVALUATION FOR REAL WORLD LEARNING
A conference for university teachers 29–30 November 2007
ATN Evaluation and Assessment Conference: assessment and evaluation for real world learning ISBN: 1 74107 135 6 Published by the Department of Teaching and Learning Support Services Queensland University of Technology Brisbane, Queensland Australia 4059 © Copyright 2007 QUT Teaching and Learning Support Services and individual authors. This work is copyright. Apart from any use permitted under the Copyright Act 1968, no part may be reproduced by any process without written permission from the copyright holders.
ATN EAC2007 Steering Committee Ms Deborah Southwell, Director, TALSS, Queensland University of Technology Ms Ros McCulloch, Associate Director, TALSS, Queensland University of Technology Dr Deborah Peach, Learning and Teaching Consultant, TALSS, Queensland University of Technology Ms Carol‐joy Patrick, Program Manager, Industrial Affiliates Program, Griffith University Ms Helen Dwyer, Program and Events Coordinator, TALSS, Queensland University of Technology Ms Jodie Lugton, Program Support Officer, TALSS, Queensland University of Technology Ms Bree Williams, Program Support Officer, TALSS, Queensland University of Technology Mrs Bergita Shannon, Program Support Officer, TALSS, Queensland University of Technology Mrs Rebecca Somerville, Program Support Officer, TALSS, Queensland University of Technology Ms Kellie O’Sullivan, Administration Officer, TALSS, Queensland University of Technology
Conference support
Administrative support staff, TALSS Department of Marketing and Communication, TILS Media Development staff, TALSS QUT Printing Services QUT Publications CPE (Continuing Professional Education)
Principal sponsor
Queensland University of Technology
Additional sponsor
The Australian Technology Network of Universities
Referee panel Ms Lyn Alderman Ms Frances McGlone Ms Allison Brown Dr Deborah Peach Ms Caroline Cottman Professor Peter Taylor Dr Robert Ellis Professor Stephen Towers Ms Wendy Harper Professor Sue Savage Associate Professor Margaret Hicks Ms Jude Smith Associate Professor Dineli Mather Ms Karen Whelan Dr Jo McKenzie
Welcome
On behalf of the ATN Evaluation and Assessment Conference 2007 organising committee, I extend you a warm welcome to this event.
The conference theme of ‘Assessment and evaluation for real world learning’ provides an occasion for reflection upon the ways in which we might take advantage of the opportunities offered to teaching and learning when these are situated in real world contexts. How can we work together to enhance learning and provide a quality, active, engaged and connected educational experience for our students?
I am sure you will find the range of topics covered by the three keynote speakers and presented in the collection of peer‐reviewed papers to be both stimulating and thought provoking. There will also be opportunity for hands‐on engagement in the variety of workshops on offer.
I hope you will use this opportunity to build collaborations with colleagues who have similar interests in assessment and evaluation for real world learning to progress our knowledge and practice in the teaching and learning field.
Deborah Southwell Director Teaching and Learning Support Services (TALSS) Queensland University of Technology
Table of Contents
page
Pathways to the profession: a study in integrating journalism degree programs with the world of work 1 Leo Bowman and Ann Lund
Teaching and assessment for workforce capacity building 9 Sarojni Choy
Levelling expectations across different years in an undergraduate degree to support work-related learning: evidence for a multi-pronged approach 17 Moira Cordiner, Deborah Stenzel and Louise Hafner
Development of a cross-faculty model for the enhancement of academic standards in assessment of work-integrated learning programs 25 Linda Corrin and Martin Smith
Managing consulting: delivering work-ready students through experiential learning 31 Ian R. Douglas
Course sustainability = relevance + quality + financial viability 37 Sonia Ferns, Beverley Oliver, Sue Jones and Rosemary Kerr
Benchmarking capstone projects in UTS Faculty of Engineering 45 Rob Jarman and Keith Willey
Exploring PBL as a strategy for improving student learning outcomes 59 Carmen Joham and Marilyn Clarke
ICT student projects: assessing students engaged in the community 67 Kathleen Keogh and Anne Venables
From optional to mandatory to assessed: updating models of university student ePortfolio use 75 Sarah Lambert and Linda Corrin
Evaluation and assessment of problem-based and case-based learning in the Mining Education Australia (MEA) collaborative initiative 81 Gavin Lind and Trish Andrews
Designing reflective assessment for workplace learning in legal education 87 Judith McNamara and Rachael Field
Towards an improved understanding of methods for obtaining information relevant for evaluating work integrated learning programs 95 Bruce Moulton
Mapping curricula: ensuring work-ready graduates by mapping course learning outcomes and higher order thinking skills 103 Beverley Oliver, Sue Jones, Sonia Ferns and Beatrice Tucker
Are our students work-ready? Graduate and employer feedback for comprehensive course review 111 Beverley Oliver, Beatrice Tucker, Sue Jones and Sonia Ferns
Pharmacy assessment and building skills within placements towards professional competencies 117 Susanne Owen and Ieva Stupans
(ii)
A reflective account of WIL from the students’ perspective 127 Sharon Schembri
Assisting academics to use self and peer assessment processes to integrate graduate attribute development with discipline content delivery 133 Keith Willey and Anne Gardner
* * * * *
Bowman and Lund 1
Pathways to the profession: a study in integrating journalism degree programs with the world of work
Dr Leo Bowman Creative Industries Faculty
Queensland University of Technology [email protected]
Ms Ann Lund Creative Industries Faculty
Queensland University of Technology
Abstract
At QUT, the journalism program has developed a successful work integrated learning approach that integrates learning, teaching and assessment with the goal of providing student pathways into industry. This paper will outline the structure and approach of the program to clearly identify the reasons for its success, and also to outline the particular benefits that this program has for universities that run professional/vocational degree programs.
In particular, it shows how it is possible to inculcate a ‘professional stance’ in programs dedicated to professional vocational outcomes.
Introduction
In her 1989 paper, ‘Learning from action: a conceptual framework’, Miram Hutton says that, pedagogically, her focus is ‘on knowledge for use and learning from doing’ (Hutton, 1989, p. 50). Such an understanding permeates the journalism course at QUT, where the staff seek to develop an intellectual approach that leads to a disinterested examination of the ‘full range of opinions’ in line with various liberal theories of the media (Schultz, 1998; Allan, 1999).They are aware that they need to situate such philosophies and the associated practices of inquiry (Sedorkin & McGregor, 2002; Adkins, 1989; 1992) within the particular constraints that govern the ‘manufacture’ of news for a range of media outlets. Further, they are fully aware that in the dynamic field of journalism (as in many other arenas) the answer to any particular question is highly context dependent, and that a range of different answers are available to any particular question, depending on the time and place of its posing (Bowman & Mcilwaine, 2004, pp. 126–127). The challenge, then — as it is in a range of professional vocational programs — is to adopt teaching approaches that develop in students the particular professional stances that meet industry needs and expectations.
Setting the scene
Such understandings of the nature of journalism practice meant that the journalism curriculum had, firstly, to establish that students needed to be clear that there was no ‘single best’ solution; rather, that a sound professional approach or professional stance was based not on the ‘rightness’ of individual approaches to the coverage of an issue, but depended on the development of a mature judgment, apprised of the full set of circumstances that affect — and constrain — journalistic practice in the modern media. The dilemma that Hutton (1989) identifies in her paper is indeed the dilemma of the self‐conscious professional journalist at the point of filing a story and, hence, a pre‐occupation of the QUT professional vocational approach to curriculum development. Indeed, this paper argues that the situation applies in a range of professional vocational settings where judgments are subject to temporal pressures.
How do we make a decision that implies closure, and yet which is flexible enough to allow us to adapt to ongoing changes?
The solution for courses and programs that want to develop such skills is that the student work needs to take place within a stable process rather than as part of a focus on a fixed outcome.
2 ATN EAC2007: Assessment and evaluation for real world learning
Once a stable process is in place, Hutton talks of a situation that epitomises the state of mind needed for the practice of journalism and that applies equally to a range disciplines that operate in a dynamic environment. Hutton stated: ‘… we are not concerned about the predictable outcome. Rather we begin where we are and learn as we act’ (1989, p. 54).
A stable process, of course, requires some things that are predictable and to which journalists (and other professionals who are required to make many decisions ‘on the run’) can adapt their practices. Hutton (1989) suggests that the particular circumstances about which knowledge can be fixed are the conditions in which one works, and the particular stance that one brings to the work that they do, and to the particular workplace. ‘Conditions are those facts which may be known, such as physical characteristics, history, culture, resources available, structure, relationships and so on. Stance includes attitudes, values, beliefs expectations and assumptions’ (Hutton, 1989, p. 53). In the particular case under examination (that is, the journalism program at QUT), in order to inculcate a professional stance we needed to establish the particular process through which news is made (the conditions). We then needed to develop within this process the particular way of operating or the particular stance that distinguishes the journalistic way of operating from the stance exhibited by other professionals.
To do so at QUT, we developed a range of media outlets (print, online, radio and television) where students could learn to adapt a particular stance in the face of the conditions in which they practice their journalism. In this ‘laboratory without walls’, QUT journalism students take on the stories of the day, side by side with industry professionals. They interview the main newsmakers in the same settings, and under the same constraints, as the working professionals. The products that they produce are presented within industry deadlines and are published, or broadcast, in the same fashion, to a real audience. To confine the scope of this paper, we have limited the examination of this dynamic process to the mediums of radio and television.
The background
Students usually started studying their radio and television units in the second semester of their second year, so they already had grounding and understanding of what makes news and how to write it, in a broader context. Then they were introduced to the processes of radio and television news through a series of lectures and specialised tutorials. The tutorials initially concentrated on the technical aspects of the process: the software involved (audio editor in radio, a news production system in television), studios, recording equipment, etc.
Such an approach was seen as a necessary precursor to the selection, analysis and implementation of stories and story ideas, as students cannot concentrate on this higher priority until the processes needed to craft such ideas into a media product are second nature. To bridge the gap between these factors, the students were encouraged in a sort of role‐play environment to use the software and equipment to write stories.
How it works: radio
QUT Radio News uses an audio‐editing system called NewsBoss. It is an industry recognised system used by many radio networks (for example, Austereo) in Australia. There are many different audio‐editing systems around the world, and the understanding was that the students would be able to transfer these NewsBoss skills onto other applications fairly easily (once they were familiar with the processes of recording and editing interviews using this application), gaining valuable industry skills and presenting — to themselves and to others — that they were ‘on top of things’ in the particular industry environment. They were encouraged to interview each other, edit grabs and write mock stories within a controlled environment where an intensive learning session suddenly incorporated an element of fun.
In the next tutorials the students had to present a ‘real’ topic or subject matter, interview people and write a radio story incorporating interview grabs. This session also concentrated heavily on their writing style, because one of the most difficult things for journalism students to achieve is that graduation from writing for print to writing for broadcast.
Bowman and Lund 3
As highlighted already, the repetition of the processes through the sessions and making the students comfortable with the equipment and radio writing style was primarily to try to take the element of surprise out of their assessed practical radio sessions, known as praxis, which is the process of putting the theoretical knowledge into practice.
QUT Radio News runs for up to nine weeks of each semester. It broadcasts for half an hour from 1 pm, Monday to Thursday, on ethnic community radio station 4EB in Brisbane; the students roster themselves on for a four‐day week to fit around their studies and work commitments. Each week the radio team will consist of ten to fourteen students who are the radio reporters and presenters, two Assistant Producers (usually double‐degree students who have completed all their journalism units but still want to keep their skills up to date), and an Executive Producer (EP). This person is either a member of QUT’s academic staff or a sessional member of staff, all of whom have an extensive background working in radio journalism. Each day begins with an editorial meeting at 0800 taken by the EP of the day. The students are required to come to the meeting with real story ideas that they want to pursue for that day’s program. They are expected to have ideas for talent they can contact for interview, and to be able to demonstrate why that story is news, and why it is relevant to a lunchtime community radio station audience.
The story ideas are then ‘workshopped’ by the students and the EP, which encourages the students to think and work as a team — part of a newsroom — rather than as individuals completing a piece of assessment. A criticism often levelled at journalism students is that they lack the ‘hunger’ or enthusiasm to dig up stories and be proactive in the workplace. This is not just in the realm of broadcast journalism; it’s industry wide and has become an important focus, particularly of the radio praxis where students are expected to come in with story ideas thought through.
The radio newsroom is open plan with eight workstations in the centre of the room; four other workstations to the side double up as interviews stations where students can conduct telephone interviews. There are also two soundproof studios: one is the main broadcast studio where the bulletin is presented from, but both studios are used for quality phone or face‐to‐face interviews. In addition, the students have access to mini‐disc recorders so they can conduct outside interviews and vox pops.
Each day the students are expected to produce a radio package that consists of a solid newsreader introduction (lead) and their script, incorporating grabs (sound bites) from their interviewees (talent). The package is expected to run for least one‐and‐a‐half minutes. The students are faced with a fairly demanding deadline. Scripts should be ready for subbing at 1100 to allow time for everyone to record their voices and edit their packages ready for broadcast. At 1300, two of the students present the bulletin live to air, panelled by one of the Assistant Producers. The whole process is designed to be as close to a real radio newsroom as possible. While there are not hourly news bulletins or deadlines, the process of producing a radio news package each day incorporates almost every aspect of radio news that a journalist would realistically be expected to do: attending an editorial meeting, presenting relevant story ideas, conducting interviews, selecting grabs, writing a script, recording their voice, editing and presenting.
How it works: TV
In line with the induction processes implemented in radio, the television tutorials were also initially focused on the software employed. QUT TV News used ENPS (Electronic News Production System), which was originally written for the BBC in the 1990s. It is an industry recognised system that is used in Australia by the Nine Network and Channel 10. Once the students gained familiarity with the writing processes on ENPS they were introduced to the technical processes (previewing raw footage, time‐coding grabs, script writing, etc.) Because television employs so many other people — camera crews, editors, etc. — it was not possible to give the students the full experience of working in TV in the tutorial sessions. To ensure a better understanding of the process, technical staff were encouraged to produce a simple video presentation that was delivered in a lecture to better explain the detailed processes. The students were then taken on short familiarisation tours where they were shown edit suites, the control room and the studio. QUT works closely with the Teaching and Learning Support Services (TALSS) department in this endeavour: TALSS supplies all the technical staff (camera crews, editors) and equipment (studios, edit suites) outside the newsroom.
4 ATN EAC2007: Assessment and evaluation for real world learning
QUT TV News operates for eight weeks each semester. It broadcasts from Monday to Friday at 6.30 pm on community television on Channel 31; the students roster themselves on for either a three or five day rotation, depending on their experience. Each week the TV newsroom consists of nine to twelve students who are the reporters, and an Assistant Producer (usually a senior student wanting to gain additional experience). In addition, each day there is a Chief of Staff (COS), a Program Producer and an International Producer (these roles are filled by full time academic staff or sessional staff, all of whom have extensive television news experience). The camera crews and editors employed by TALSS are professionals, and are either still working in the TV news industry or have extensive experience within it. On any one day QUT TV News usually has four or five camera crews operating (not as many as commercial news organisations, but certainly more than the ABC).
Each day the Chief of Staff arrives early (0600) and starts to prepare the stories to be shot locally. Students arrive by 0800 to be briefed on the stories they might be covering. The students are divided into three groups: those covering local stories, those covering national stories and those covering international stories. This rotation is then swapped each day so the students get to experience each area. Those students on local stories are briefed by the Chief of Staff and go with their crews to film their stories. All students are required to deliver a piece‐to‐camera in the field, helped and encouraged by the crews. On return to the newsroom the students preview their footage, time‐code their grabs and write their stories.
The national story footage is mainly supplied by Channel 7; a strong working relationship with that organisation over the years has afforded QUT a direct link to the station and access to raw material that would otherwise be unavailable to us. QUT also has a formal agreement with the Reuters News Agency, which supplies raw international footage to news organisations throughout the world. The footage is recorded directly into QUT’s computing system so that the students can preview the footage, time‐code their grabs and begin the writing process. This access to national and international footage is invaluable to students and expands their understanding of news substantially.
Once the students have written their scripts they are subbed by one of the TV journalism staff members, then they record their scripts and sit with an editor who professionally cuts the piece into a news package ready for broadcast. While students are expected to write at least one news package each day, they may actually write more than one, and will often be required to write short reader voice overs (RVOs). In addition, each semester students are able to audition to be a TV newsreader — those who are successful are rostered on for a week and rotate between the lead and second newsreader and weather presenter. More than twenty students get this opportunity each semester.
The working relationship with other news organisations, particularly in our television news, has been clearly demonstrated over the past few years. As other news chiefs of staff recognise that those teaching television journalism are ‘real’ industry people too, new lines of communication have opened up between us. QUT’s Chief of Staff now liaises with them on a daily basis, enabling a pooling of resources.
Assessment
We measure the journalism students’ ability to adopt a professional stance in two ways. In the first instance, we monitor their activity in our own newsrooms and, secondly, we examine data from employers, supervisors and journalism interns from the industry placements that many students attend on completion of their practical sessions in our newsrooms.
For both radio and television, our students are assessed across a number of criteria, including: • editorial (news sense and knowledge) • production (writing ability, voice delivery, and use of grabs) • technical ability (use of software and equipment) • teamwork.
Bowman and Lund 5
In order to integrate our learning and assessment program with the industry interns program we provided formal feedback sessions to give students the opportunity to adapt their stance in preparation for the potential uptake of industry internships. Here are some of the responses from students following our first feedback session:
The feedback session was great! I found it to be a great help and the individual time (Which is often rare at Uni) was valuable.
The session was very helpful and you gave constructive feedback that I can use to improve.
The feedback session has given me a little more confidence in myself.
The feedback was very helpful and provided me with some good advice in the things I really need to improve on. I believe these sessions should be compulsory as it’s very encouraging.
Real-world experience
Such feedback was encouraging, but to ensure that it was re‐enforcing an industry‐ready professional stance, we needed to test our approaches in the field through monitoring the student transition from our own newsrooms to those of industry.
The dual benefits were seen as establishing whether our approaches were a good ‘fit’ with contemporary industry practice, and also the extra confidence engendered through a student discovering that their particular stance was a ‘realistic’ approach to life as a professional journalist. To achieve such a goal, our internship program had to be wide and varied, incorporating as many different aspects of the broadcast media as possible: commercial radio and television, public broadcasting, news and programming.
We now offer upward of sixty broadcast placements every year. News organisations involved include: • Public broadcaster: ABC Radio and Television news, ABC online/broadband, 612 ABC Brisbane. • Commercial radio: 4BC, B105, Triple M, Hot Tomato. • Commercial TV: Channels, 7, 9 and 10, WIN TV.
The data that we have collected from both industry participants in the internship program and from students who have undertaken these internships indicates that our interns are work‐ready and that their particular professional stance fits industry requirements. The internship program with 4BC used to work as a one‐week placement, but after a number of recent successes News Director Chris Adams was keen to extend the placements to two weeks:
I am happy to have a constant flow of interns from QUT – in fact I would like to extend the on‐going work experience programme which seems to have worked well
(Chris Adams, 05/11/06)
In 2007 we started a formal internship with the Austereo network, which operates two radio stations in Brisbane: B105 and Triple M. After the first two students completed their internship this was the feedback from News Director Melissa Martin:
They have been absolutely fantastic. It was such a pleasure to have them both with us. They really jumped in and got involved. They did general newswriting, conducted phone interviews and cut audio…It was so great to have students who were willing to do whatever was asked of them and who fitted in so well. They’re both a credit to QUT.
(Melissa Martin, 07/06/07)
We were gratified, but not entirely surprised, by such feedback, which indicated to us that our efforts to provide suitable conditions and an industry replicable stance were working. In purely practical terms, it shows that our students are used to working in a ‘fast paced’ newsroom at university and have been able to apply that experience in a real newsroom and in the field. Indeed, this year, a number of students have been offered ongoing work experience, paid freelance work or fulltime employment.
Thanks very much for setting me up with the ABC Internship last semester. Itʹs paid off. Iʹd told you previously that they wanted to extend my internship to have me come in on weekends to help out on the sports desk. Now, they have offered to pay me for the trouble... making me a casual. So Iʹm on the books!
(Student, 05/09/07)
6 ATN EAC2007: Assessment and evaluation for real world learning
Such comments are not exceptional, as News Directors are regularly commenting that they enjoy the fact that students can quickly fit into the newsroom environment and actively start working almost straight away.
Shona Mackenzie is the Unit Manager and Personal Assistant to the Director of News at Channel 7:
I think Tom will actually be doing some ongoing work experience in our sports department. (Shona Mackenzie, 16/03/07)
Just wanted to drop you a line and tell you how wonderful Emily has been in her first week. Weʹre looking forward to having her back next week. From feedback around the newsroom, everybody has been impressed with Emilyʹs attitude, work ethic and abilities and I’m sure she will be welcome in the newsroom as an ongoing member of the team.
(Shona Mackenzie, 10/08/07)
The fact that our program, our students and their abilities are becoming such a ‘known entity’ has further strengthened our industry links and the rapport that we are developing, and has sharpened the feedback that we are receiving. The following internship arrangement serves as a case in point.
The new News Director at WIN News on the Sunshine Coast is Brendan Hockings: a former QUT Journalism student. We began a formal internship program with WIN this year and, with Brendan taking over recently, have recently been able to extend this relationship. Brendan already has at least three former QUT Journalism students working full time in his newsroom. He knows firsthand what we teach, the equipment we use (ENPS) and what he can expect from the students. He regularly provides feedback on how interns are going:
Katie was great – I’m going to get her back in when we can – at different times I had her working on two stories in a day – and I let her voice stories for air. She’s a chance of freelance work down the track if she can get up here for more work exp.
(Brendan Hockings, 30/08/07)
Similar cases are also observable in radio, where another QUT graduate is guiding the progress of students at the ABC .Just days after the students completed their internships with 612 ABC Brisbane this year, Kellie Riordan, Acting Program Director, emailed this:
… great to have in the office. Candice is working for us (paid) tonight as a field reporter, plus as a producer a few days in the next two weeks. Michelle is working for us today (paid) doing some online posting of The Conversation Hour. My feeling is Candice is a natural. Good ideas, quick‐thinker. Sheʹll go places.
(Kellie Riordan, 05/09/07)
Many of student testimonials speak for themselves, including this one from a student who graduated at the end of 2006:
Just thought I would let you know that Iʹve secured full time employment with Prime News in Wagga Wagga. Naturally very excited. I feel very fortunate to have gained employment so quickly after finishing uni...especially in the field I really wanted to get into: TV. Thank you all for your wisdom, kind words, and advice during my degree. Your passion has certainly worn off on me.
(19/12/06)
Workplace learning
In addition to the internship program, we actively encourage students to use their internships for credit under a unit called Workplace Learning. Again, the feedback from students (who are required to keep a reflective journal as part of their assessment) is tremendous:
I was not afraid to ask questions, rattle off a piece to camera with enthusiasm and confidence and speak to key players and other journalists. This came mainly from QUT news experience.
The journalists at 10 use ENPS, which is the same software as QUT, making my initiation a little easier.
Felt I understood everything that went on out in the field and in the newsroom, as QUT news prac had me well‐prepared.
Bowman and Lund 7
As for Channel 10, I think itʹs a good testament to QUT how similar most elements of the news room and production are the same.
Comparing things to the TV work we do at QUT it was very similar. The newsroom system seems to work in a very similar way and the ingest of tapes, selection of footage using time‐codes and editing is all done in pretty much exactly the same way. So having all my QUT experiences I think is very helpful, because none of what they do is all that new to me and I already have a good understanding of how to do things.
The following comment is clear testament to the fact that we teach real skills and provide a realistic environment in which to learn, as well as a clear pathway for students into the industry. But importantly, it also demonstrates that this has to be coupled with real industry experience for students to feel the full benefit of integrated teaching and learning:
I feel like Iʹve had a real insight into the realities of the industry and itʹs really opened my eyes, to some of the exciting opportunities and stories that happen, and also some of the less glamorous parts of being a TV journalist. QUT News has been great in teaching us about TV journalism, but I feel like being up there has taught me things that I honestly couldnʹt have learnt without going.
Conclusion
The QUT experience is that the creation of conditions of work that can buttress the appropriate professional stance that professional/vocational programs aspire to has produced first‐rate outcomes. The integration of the program with the ‘real world’ outlets connects us with organisations where our students aspire to work, plus serves as a check on the validity of the program and a valuable pathway for our students into the world of work.
We plan to continue this reflexive conversation with industry in the hope of further improvements to our program. We believe that our efforts have succeeded in providing a professional stance appropriate to the journalism field. We also believe the particular organisational dynamic that we have created, coupled with its integration with industry, is a model suitable for use in a range of professional areas with similarities to the journalism field.
References
Adkins, B. (1989). Targetting the issues on time: An analysis of topic development and its constraints in radio current affairs interviews. Honours thesis, University of Queensland, Brisbane.
Adkins, B. (1992). Arguing the point: The management and context of disputations and challenges in radio current affairs interviews. Australian Journalism Review, 14(2), 37–48.
Allan, S. (1999). News culture. Buckingham, UK: Open University Press.
Bowman, L., & Mcilwaine, S. (2004). A structured return to the “Fourth Estate”: Interviews and inquiry that challenge entrenched authorities “knowledge”. Australian Journalism Review, 26(1), 121–133.
Hutton, M. (1989). Learning from action: A conceptual framework. In. S. Weil & I. McGill (Eds.), Making sense of experiential learning: Diversity in theory and practice. Milton Keynes, UK: Society for Research into Higher Education & Open University Press.
Schultz, J. (1998). Reviving the fourth estate: Democracy accountability and the media. Melbourne: Cambridge University Press.
Sedorkin, G., & McGregor, J. (2002). Interviewing: A guide for journalists and writers. Sydney: Allen & Unwin.
Copyright statement Copyright © 2007 Bowman and Lund.
The author(s) assign to QUT and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to QUT to publish this document in full on the World Wide Web (prime sites and mirrors), publication to CD-ROM and in printed form within the ATN EAC2007 conference proceedings. Any other usage is prohibited without the express permission of the author(s).
Choy 9
Teaching and assessment for workforce capacity building
Dr Sarojni Choy Faculty of Education
Queensland University of Technology [email protected]
Abstract
Organisations that sponsor staff for academic studies in order to build their workforce capacity expect their learning experiences and outcomes to be aligned with their business. There are challenges not only for facilitation of such learning, but also its assessment. Despite recent research on learning in the workplace the development of rigorous pedagogy to underpin work‐based learning and its assessment is still in its infancy (Brodie & Irving, 2007).
This paper describes the teaching and assessment approaches negotiated for a cohort of ‘worker‐learners’ to develop their leadership capacity through a Graduate Certificate in Education (Executive Leadership) course. The course was delivered using an organisation centred curriculum framework that aligned the learning and assessment tasks to the functional roles of the learners and the strategic goals of their organisation. The framework transported the learning curriculum into the realm of the workplace and its context, exemplifying the integration of academic learning into the workplace.
Keywords
work integrated learning, workplace learning, workplace assessment, organisational centred curriculum, workforce capacity building
Introduction
A booming ‘knowledge economy’ relies on highly competent ‘knowledge’ workers; hence, universities are increasingly under pressure to align their curriculum to real work tasks in order to adequately develop such workers (Symes & McIntyre, 2000; Tennant, 2000). Consequently, there is renewed interest in work integrated learning (WIL) to provide learning experiences and outcomes that are more in tune with the real world. For this to happen, new partnership arrangements need to be forged between universities and organisations to actively diversify the study repertoire into the workplace and simultaneously improve outcomes for individuals and their workplaces.
This paper describes the author’s approach to aligning teaching and assessment for real work outcomes using an organisation‐centred curriculum (OCC) framework. Four units of study were brokered for a Graduate Certificate in Education (Executive Leadership) course to develop the leadership capacity of a learning cohort based in the Queensland Health and Community Services Workforce Council (Workforce Council). The paper begins with an overview of the OCC framework; the teaching and learning strategies, workplace support strategies, and assessment approaches embedded in the OCC framework are then briefly described. The paper goes on to summarise the preliminary outcomes of learning while developing the leadership capacity of the cohort.
A framework for work integrated learning
Work integrated learning (WIL) and work based learning are often conceptualised in similar ways (see Boud & Solomon, 2001 for some examples). In this paper, work integrated learning is conceptualised as a moderately tailored version of an existing course delivered to a cohort of learners in a particular work site. Strategically, its design focuses on outcomes that lead to the achievement of the organisation’s strategic goals, yet maintains the academic standard and quality. Various arguments (for example, by Billett, 2001; Brown, 1998; Symes & McIntyre, 2000; Billett & Boud, 2001; Bryson, Pajo, Ward, & Mallon, 2006) in favour of the workplace as an effective learning environment provided justification for embedding the leadership development program within an OCC framework.
10 ATN EAC2007: Assessment and evaluation for real world learning
A partnership between the university and the Workforce Council was forged to develop a boutique curriculum. Four units packaged for the course were Leadership for Change; Politics of Diversity and Identity; Managing Knowledge in Organisations; and Changing Agendas in Leadership.
The OCC framework was founded on the organisation’s strategic goal, philosophies and culture to simultaneously meet the needs of the individuals and their organisation. Accordingly, the strategic goal of the organisation was positioned at the core of the framework, surrounded by its philosophies and culture. Organisational culture is usually described as the taken‐for‐granted values, underlying assumptions, expectations, collective memories and definitions present in an organisation (Cameron & Quinn 1999, p. 14). These elements of the organisational culture impact on how new ideas are viewed in an organisation and also influence their interpretation for practice. At the next level, the university learning units surrounded these structures (organisational goal, philosophies and culture). The teaching and assessment strategies that emerged from the unit objectives were designed to influence changes that traverse through the organisation and its environmental cultures, while at the same time respecting its philosophies and values.
The learning cohort of twelve staff formed a critical mass that could sense, construe, analyse and respond to changes that would help the Workforce Council achieve its strategic goal. The worker–learners contextualised what they learned to initiate and also facilitate a gradual change process. A partnership with the academics and a coalition with staff across the organisation allowed them to collaborate and negotiate the bridge between theory and practice, and create new knowledge that added value to their organisational services and strategic goal.
The success of work integrated learning in this instance hinged on interactions between the disciplinary area, professional practice and the workplace goals. The implementation of the OCC framework required special attention to design appropriate pedagogies. It was premised on the notion that learning would be influenced by (and, in turn, influence) the strategic direction of the organisation. Therefore, the content needed to be relevant to the organisation — that is, fulfil the requirements of both the intended and emergent strategic plans — and at the same time be relevant to the learners, with obvious application to their functional roles. Furthermore, the learning strategies needed to suit the learning preferences of the cohort. The designers interpreted learning strategies as the combination of the learning processes and activities, as well as the assessment tasks to demonstrate that learning had occurred. The arrangements for an OCC framework were deliberated against the university policies, systems and structures.
Teaching and assessment strategies
The learning strategies and the assessment tasks had to be appropriate to achieve learning objectives and this configuration is termed ‘constructive alignment’ (Biggs, 1999). As a basis for constructive alignment, Biggs’ (1999) ‘Structure of observed learning outcome’ (SOLO) model and Delahaye’s (2005) ‘Hierarchy of learning outcomes’ (HLO) were employed. The SOLO model proposes aligning learning strategies and learning outcomes and describes how a learner’s performance grows in complexity when mastering a topic. The HLO provides the designer with two important indicators: the type of content that the learners need to cover and, secondly, the most appropriate type of learning strategy that should be used. The design of the learning activities was therefore founded on three governing values:
1. The andragogical principles of: ― Work centred problems and issues. ― Learning that acknowledged and utilised the work experience of adult learners. ― Acknowledgement that self‐evaluation should be combined with evaluation by others. (Rogers
(1983) suggested that self‐evaluation is more important than evaluation by others.)
2. Overtly situated learning within the strategic directions and cultural context of the host organisation.
3. Individual learning embedded within the normal workplace tasks, processes and goals to concurrently achieve learning outcomes and organisational goals, as well as develop sustainable knowledge reservoirs.
Choy 11
Facilitation of learning
In acknowledging the work experience of the adult learners, the course was facilitated using a blended approach comprising internet‐based delivery and face‐to‐face on‐site consultation. The worker–learners were well acquainted with electronic communication and utilised a well‐designed information and computer technology system available at the work site. The learning materials were posted on the university’s course website, allowing the cohort to access and interact with the materials from their work or home computers and at a time of their own choosing. The cohort determined the content and structure of the face‐to‐face consultations. Individuals were required to send requests for specific information or topics/issues to be discussed at least two working days before the session to allow for adequate preparations.
The objectives of the first unit (Leadership for Change) and those of the individuals (based on their work projects) were aligned to the organisational objectives interpreted from its strategic goal. As a result of this alignment, the Leadership for Change unit offered development of expertise in three areas:
• Investigating and analysing organisational and environmental cultures (themes = the changing nature of organisations, organisational culture, organisational values).
• Accountability (themes = ethics, relationship building, accountability and organisational improvement).
• Change management (themes = communication, the change process, leading the change process).
According to the ‘Hierarchy of learning outcomes’ (Delahaye, 2005), such learning outcomes need more self‐directed learning strategies, experiential learning and problem‐based learning, thus reinforcing the need for blended learning based on questions initiated by the learners.
Individuals self‐selected themselves into one of three teams, each focusing on one of the areas. The use of teams was reinforced by the writings of Knowles (1998), Revans (1988) and Wenger (1998), who emphasised that adult learning is a social process. Each team shared three features: a common domain of knowledge to be developed; a community that cares and takes responsibility for the domain knowledge in the organisation; and shared practice to effectively interpret and apply their domain knowledge. In this way the communities or teams became nodes for exchange and interpretation of their domain knowledge. These features are characteristic of communities of practice (Wenger, 1998). The nodes transpired into a useful knowledge management tool that supported and stimulated learning within the organisation.
The intention of the teams was to develop in‐depth knowledge and understanding of their area of study and to explore the diversity of applications across various aspects of their functions. In other words, the learners were expected to operate at the relational level of the SOLO model (Biggs, 2003) so that they would be able to ‘compare/contrast, explain causes, analyse, relate and apply’ (p. 48). They researched and updated their knowledge base on developments both internally (within the organisation) and externally (nationally, internationally and beyond their industry). Members of the groups read, summarised and critically analysed the assigned readings and additional articles they sourced. The assessment tasks (described later in this paper) were designed to generate more learning and required them to state how the ideas could be applied to the functions of the Workforce Council. Regular staff meetings and professional development workshops offered them opportunities to share, reflect on and discuss concepts and ideas with others in the workplace. In this way they retained their knowledge domains in ‘living’ ways.
At the beginning of the semester, each learner selected a work project to apply and integrate knowledge from the three areas: investigating and analysing organisational and environmental cultures; accountability; and change management. They set their personal objectives, the performance measures and indicators to measure progress and outcomes.
The strategies described above are common practice in normal university teaching; however, they have particular significance in the context here and highlight the alignment of university learning to real work outcomes. Traditionally, when such activities are conducted in lecture rooms the authenticity of alignment and application of knowledge is left to the individual learners. The validity and feasibility is often unverified, mainly because that is an assumed role of the worker–learners. The alignment and application discussed in this paper reflects real and productive outcomes for the cohort and the Workforce Council.
12 ATN EAC2007: Assessment and evaluation for real world learning
Learning support in the workplace
The learning process and experiences of the cohort were supported by features of expansive learning environment, as identified by Fuller and Unwin (2004). In an expansive learning environment, learners participated in multiple communities of practice including those external to the organisation (its regional staff, networks and stakeholders). They engaged in diverse tasks, knowledge bases and experiences to encourage learning. Their learning was acknowledged, supported and formally valued as organisational capability. Workplace pedagogies, described by Billett (2001), were made available and utilised by the cohort. The pedagogies included questioning and getting explanations, observation and listening, interacting with others, accessing documents in the workplace and learning from daily work activities through practice. These pedagogies were supported by workplace affordances that included fortnightly team meetings; internal communication systems (oral, written and electronic); time allocation for group learning; opportunities to solve problems; systemic knowledge of the workplace; shared responsibility for learning and achieving organisational goals; and timely access to assistance from others, including from the organisational liaison person and the academics. Each team met regularly to plan, review and complete their activities. Some of these meetings were held in the workplace during working hours, while others were informal meetings after work, often outside the work site.
Assessment
There were two major assessment tasks for this unit, each equivalent to 50%. Each of these tasks needed to demonstrate constructive alignment (Biggs, 1999) and, again, the SOLO model (Biggs, 2003) and the HLO (Delahaye, 2005) proved useful here.
The first task was team based and involved two deliverables (25 per cent each): oral seminars and a resource folder. Each group presented a seminar on their area of study. The presentations needed to demonstrate deep analysis and application of theoretical concepts to the work tasks — the relational level of SOLO (Biggs, 2003) and the deeper levels of the ‘Task’, ‘Relationship’ and ‘Critical thinking’ categories of the ‘Hierarchy of learning outcomes’ (Delahaye, 2005). After the seminar, each group collated the summaries of the readings and added these to the resource folder. The summaries and analysis from each group were presented in a format that allowed the remaining two groups to gain a basic understanding of the themes within the area of study, and also to apply the theories and learning into the context of their work. The resource folders also included tools, websites and other usable materials for other staff in the organisation, their stakeholders and networks. The tools included definitions of key terms, guidelines, theoretical and conceptual frameworks, evaluation items, surveys, checklists, to‐do lists, relevant legislations, regulatory frameworks, case study examples and benchmarks. A single set of these was deposited in the central office of the Workforce Council.
The second assessment task was completed individually and was based on personal learning objectives. It included two aspects: oral seminar and written essay. Each learner presented a case example that demonstrated the application of their understanding gained from the study unit content. During the seminar there was much discussion around the diversity in the application of the new knowledge in the context of their work. Interestingly, they began linking the concepts and ideas beyond their projects, to other areas of work in the organisation. This linking took the learners beyond the relational level into the extended abstract level of the SOLO model (Biggs, 2003). The presentations were assessed by peers and an academic panel. Engagement of peers in the assessment and evaluation process facilitated the validation process. This was useful because while the academic panel could assess the applications from a theoretical perspective, they were not fully conversant with the socio‐cultural contexts of the Workforce Council to validate the feasibility of those applications. After the seminar each learner wrote a reflective essay (2000 words) on their presentation.
Although these assessment strategies sound familiar and are commonly practised in university settings, their contribution to organisational outcomes was substantial. The preliminary outcomes are summarised as follows.
Choy 13
Preliminary outcomes of the organisation-centred curriculum framework
The research team of academics involved in the development of the OCC framework received a small grant from the Queensland University of Technology to evaluate the framework. The evaluation was conducted at the completion of the first study unit on Leadership for Change. Data was collected from interviews with ten learners who volunteered to participate in the study, and a focus group of six interviewees. Transcripts of each interviewee were coded as WFC 001 to WFC 010.
The findings on the evaluation of the framework itself will become a forthcoming publication. In this paper, preliminary outcomes of the first unit are summarised to draw attention to the results for individuals and their workplaces when teaching and assessment adequately integrate university units with real work, and learning is situated in the workplace. The teaching and assessment approaches for an OCC framework contributed to individuals and the Workforce Council in four main areas: development of knowledge capital, development of skills, cultural change in the workplace, and improvements to organisational systems. The most immediate and greatest change was evident in the new learning culture.
Development of knowledge capital
The learning cohort acquired new information and knowledge about the theories on leadership, change management and workforce development. One of the first activities in the assessment task required the learners to conduct an environmental scan of their workplace, stakeholders, networks and industry. During this exercise they learnt about the socio‐cultural elements and their significance for the Workforce Council’s functions and services. The cohort also critically reviewed their internal organisational culture. Their analysis alluded to several enhancing and limiting socio‐cultural aspects that impacted on the leadership of the Workforce Council. Their search and review of literature in these topics identified benchmarks and case examples to appraise their current policies and practices, as well as plan for new directions in order to improve their organisational functions. They shared these during formal and informal interactions with members of the cohort and other staff. The Executive Director of the Workforce Council described the place of new knowledge capital, saying ‘… it’s not a physical product, but it is actually an intellectual product that’s now built into, in lots of ways, the conversation and the way decisions are made’. The new knowledge capital of the individuals and collective teams became a powerful tool for initiating change and motivating more learning.
Development of skills and confidence
As well as enhancing their learning skills in web and database searches, use of computer softwares, setting objectives and performance measures and indicators, and in academic writing, they also refined their critical and reflective thinking skills. The cohort gained an appreciation of critical and reflective thinking and their significance for everyday work practices. For instance, in determining the relevance of the new knowledge and understanding of theories and case examples, they engaged with colleagues across the organisation. This type of active learning made them critically analyse, challenge and contest their own belief systems, practices and cultures. As a result they initiated improvements to their overall services. The cohort also developed skills in oral and written presentation of well researched proposals for change, justifying their positions with relevant theories and literature. Their new skills boosted their confidence in leading change for the reforms that the Workforce Council was responsible for. Statements from two interviewees reflect common views about enhanced confidence amongst the cohort members.
I think the intrinsic value of what we’ve done is very high. People’s sense of self worth, their sense of an impact, they have something quite significant to contribute to the organisation. (WFC 001)
I think what itʹs done is given me more confidence in my communication with others generally and that also means the stakeholders. The fact that I am more au fait with change management and the issues in change management has helped me deal with issues as they’ve come up in a better way. (WFC 006)
The knowledge capital, skills and confidence enhanced the learning culture in the Workforce Council.
14 ATN EAC2007: Assessment and evaluation for real world learning
Cultural change
As a community of learners, the cohort gradually refined their motives and intentions of learning making the subjective relevance of the unit content more explicit. Consensual validation during informal discourses, group seminars and the symposium (formal discourses) resulted in common interpretations and validation of the application of new knowledge and understandings. These discourses were facilitated largely by the nature of relationships that the members of the cohort developed over the period of the course. The open nature of group dynamics during the seminars and symposium diminished scepticism among their colleagues, and they valued the rigour of academic research and standards. A team approach had an infectious effect and promoted systems thinking. They implemented internal systems and processes to share and maintain the knowledge and skills base, as well as sustain a learning culture through regular staff development activities. One interviewee described her perspective on the cultural change as follows:
I would have to say the big thing at an organisational level is probably the connectedness among colleagues, and the sense that you have been empowered from the process, that you can actually step forward and have the ability to critically analyse certain things and always question work processes. So it’s giving us the capacity to work in a continuous improvement manner. (WFC 006)
The nature of discussions at various forums about what they learnt, and frequent references to the content of the course units indicated a high level of interest and energy. Such a level of energy in a cohort is ideal for organisational change. However, more planning and resources are needed to accommodate other creative outputs of the cohort.
Enhancement of organisational systems
The resources and tools developed as part of the learning and assessment activities were used to refine internal policies and processes. For instance, the current communication protocols and tools developed for engaging and interacting with networks from the various diversity groups were revised in light of new understandings about learning styles of people from the different diversity groups and cultural sensitivities. A change in approach to intentional invitation is expected to make engagement with networks more inclusive and genuine. The management of diversity issues at the Workforce Council is now an integral part of all aspects of the Workforce Council’s everyday business.
When describing the approach under an OCC framework, the Executive Director said:
When people undertook that study, there was a practical application of what they were learning immediately, not just in terms of the assessments that they might undertake, but immediately, how they can better work with groups, to lead and support groups and to develop people within groups, to lead and support groups.
His comments — and the preliminary outcomes — illustrate that the learning and assessment activities were well aligned to the functional roles of the individual learners and the Workforce Council’s strategic objective of providing high quality services to the health and community services industry.
Acknowledgement
Associate Professor Brian Delahaye, Dr Lisa Ehrich, Dr Tricia Fox and Dr Ian James of the Queensland University of Technology provided support in the development of the OCC framework and the compilation of the Graduate Certificate program for the cohort. Staff from the Faculty of Education enrolment section and QUT Library provided support to the cohort. The Executive Director, liaison person and members of the cohort from the Workforce Council contributed to the evaluation data.
Choy 15
References
Biggs, J. B. (1999). Teaching for quality at university: What the student does. Buckingham, UK: Open University Press.
Biggs, J. B. (2003). Teaching for quality learning at university (2nd ed.). Maidenhead, UK: Open University Press.
Billett, S. (2001). Learning in the workplace: Strategies for effective practice. Sydney: Allen & Unwin.
Billett, S. & Boud, D. (2001, July 26–28). Participation in and guided engagement at work: Workplace pedagogic practices. Paper presented at Researching work and learning: Second international conference on learning and work, Calgary, Alberta.
Boud, D. & Solomon. (2001). Work‐based learning: A new higher education? Buckingham, UK: Society for Research into Higher Education & Open University Press.
Brodie, P. & Irving, K. (2007). Assessment in work‐based learning: Investigating a pedagogical approach to enhance student learning. Assessment & Evaluation in Higher Education, 32(1), 11–19.
Brown, B. L. (1998). Applying constructivism in vocational and career education. Information Series No. 378 (Eric Document Reproduction Service No. ED428298). Columbus, OH: ERIC Clearinghouse on Adult, Career, and Vocational Education. Available from http://cete.org/acve/majorpubs.asp.
Bryson, J., Pajo, K., Ward, R., & Mallon, M. (2006). Learning at work: Organisational affordances and individual engagement. Journal of Workplace Learning, 18(5), 279–297.
Cameron, K. S., & Quinn, R. E. (1999). Diagnosing and changing organizational culture. Reading, MA: Addison‐Wesley.
Delahaye, B. L. (2005). Human resource development: Adult learning and knowledge management (2nd ed.). Brisbane: John Wiley & Sons Australia.
Fuller, A., & Unwin, L. (2004). Expansive learning environments: Integrating organisational and personal development. In H. Rainbird, A. Fuller, & A. Munro, (Eds.), Workplace learning in context (pp. 126–44). London: Routledge.
Knowles, M. S. (1998). The adult learner: A neglected species (5th ed.). Houston: Gulf Publishing.
Revans, R. W. (1988). The ABC of action learning. Kent, UK: Chartwell‐Bratt.
Rogers, C. R. (1983). Freedom to learn for the 80s. Columbus, OH: Charles E. Merrill Publishing.
Symes, C., & McIntyre, J. (2000). Working knowledge: An introduction to the new business of learning. In C. Symes & J. McIntyre (Eds.), Working knowledge: The new vocationalism and higher education (pp. 1–14). Buckingham, UK: Society for Research into Higher Education & Open University Press.
Tennant, M. (2000). Learning to work, working to learning: Theories of situational education. In C. Symes & J. McIntyre, (Eds.). Working knowledge: The new vocationalism and higher education. Buckingham, UK: Society for Research into Higher Education & Open University Press.
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press.
Copyright statement Copyright © 2007 Choy.
The author(s) assign to QUT and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to QUT to publish this document in full on the World Wide Web (prime sites and mirrors), publication to CD-ROM and in printed form within the ATN EAC2007 conference proceedings. Any other usage is prohibited without the express permission of the author(s).
Cordiner, Stenzel and Hafner 17
Levelling expectations across different years in an undergraduate degree to support work-related learning: evidence for a multi-pronged approach
Ms Moira Cordiner Teaching and Learning Support Services Queensland University of Technology
Dr Deborah Stenzel Faculty of Science
Queensland University of Technology
Associate Professor Louise Hafner Faculty of Science
Queensland University of Technology
Abstract
One of the ways in which university departments and faculties can enhance the quality of learning and assessment is to develop a ‘well thought out criterion‐referenced assessment system’ (Biggs, 2003, p. 271). In designing undergraduate degrees (courses) this entails making decisions about the levelling of expectations across different years through devising objectives and their corresponding criteria and standards: a process of alignment analogous to what happens in unit (subject) design. These decisions about levelling have important repercussions in terms of supporting students’ work‐related learning, especially in relation to their ability to cope with the increasing cognitive and skill demands made on them as they progress through their studies. They also affect the accountability of teacher judgments of students’ responses to assessment tasks, achievement of unit objectives and, ultimately, whether students are awarded their degrees and are sufficiently prepared for the world of work.
Research reveals that this decision‐making process is rarely underpinned by an explicit educational rationale (Morgan et al, 2002). The decision to implement criterion referenced assessment in an undergraduate microbiology degree was the impetus for developing such a rationale because of the implications for alignment, and therefore ‘levelling’ of expectations across different years of the degree. This paper provides supporting evidence for a multi‐pronged approach to levelling, through backward mapping of two revised units (foundation and exit year). This approach adheres to the principles of alignment while combining a work‐related approach (via industry input) with the blended disciplinary and learner‐centred approaches proposed by Morgan et al. (2002). It is suggested that this multi‐pronged approach has the potential for making expectations, especially work‐related ones across different year levels of degrees, more explicit to students and future employers.
Keywords
levelling, work‐related, alignment, multi‐pronged approach
Introduction
The Faculty of Science at the Queensland University of Technology is in the early stages of implementing a criterion‐referenced assessment (CRA) system. This is to enhance the quality of learning and assessment and to make the work‐related connections in degrees more explicit for students. (Note that at QUT, ‘unit’ refers to a component of a ‘degree’ or ‘award’. At other universities, ‘unit’ may be termed ‘subject’, ‘course’, ‘module’ or ‘program’, while ‘degree’ may be termed ‘course’ or ‘program’.) To facilitate the implementation process we set up a team of three: two of us (Stenzel and Hafner) are senior academics in the Microbiology disciplines of Parasitology and Mycology respectively and the third (Cordiner) is an academic staff developer with expertise in assessment.
18 ATN EAC2007: Assessment and evaluation for real world learning
The aims were not only to revise two units in a highly regarded undergraduate Microbiology degree (foundation and exit year) so that they reflected the principles of CRA, but also to make more explicit to students the level of expectations of all aspects of the course across these years; that is, to carry out a process of ‘levelling’. The multi‐pronged approach to levelling that evolved from our collaboration was not based on any particular model or on research — it evolved logically from what we knew ‘worked’ and was valued by industry. The units that comprise undergraduate degrees are assigned to different year levels, usually based on the difficulty or complexity of content and skills (which is only one facet of levelling). Effective levelling is important because it ‘helps students recognise their involvement in a cohesive curriculum (Allen, 2004), affects the ‘good management of student learning and progression through their programs’, underpins valid ‘credit and transfer’ systems (Morgan, Watson, McKenzie, Roberts, & Cochrane, 2002) and ‘captures the dynamic nature of students’ working futures in the profession’ (Scott & Gribble, 2005). Ultimately, decisions about levelling affect the accountability of teacher judgments about whether or not students are awarded degrees and are sufficiently prepared for the world of work. This paper begins by examining the few previously published approaches to levelling. It then argues and provides preliminary evidence for a multi‐pronged approach to levelling, and concludes with suggestions about the potential for this approach across the university sector.
Previous approaches to levelling
An Australian national survey (Morgan et al., 2002) revealed that universities have no policies stating explicitly what year levels imply in undergraduate degrees, nor are expectations at each level clearly linked to educational theory. Our worldwide literature search also indicated there has been little research on how universities carry out the levelling process. The disciplinary approach appears the most common way of levelling, where content‐sequencing is the primary determinant of the ‘level’ (Morgan et al., 2002). An alternative approach is learner‐centred levelling, in which taxonomies of educational objectives or learning outcomes, for example, Bloom et al. (1956), Biggs and Collis (1982), Marzano and Kendell (2006), are used to ‘structure levels of understanding’ at both course and unit level. Biggs (2003, p. 38) states that adapting his taxonomy for these purposes is ‘straightforward’. However, in a national survey (Morgan et al., 2002), only two universities mentioned using a taxonomy for levelling and its use was not described. The approach taken at the University of Sydney was a combined discipline and learner‐centred approach. Their discipline approach was based on an action research methodology that views discipline knowledge as being framed by concepts and ideas that form models or theories of reality. These models or theories then are applied and tested in contexts (simulated or real world situations) with tools that include relevant methods, techniques and instruments. The level of discipline knowledge in a unit is determined by the scope and depth of the interrelationships between the models, tools and context that are offered (Morgan et al., 2002). Once this is decided, a learner‐centred approach (based partially on Bloom’s taxonomy) is used to define expectations of how — and to what degree — students engage with the knowledge. Short generic descriptions of expectations at three levels of an undergraduate degree evolved from mapping discipline expectations against learner expectations (Morgan et al., 2002).
More recently described approaches to levelling involve using the principles of alignment to level expectations across degrees. For example, Allen (2004) refers to Diamond’s commonsense approach to designing degrees: introducing important learning objectives early, then reinforcing and developing further throughout the course. Allen describes this approach by an unnamed faculty at California State University, where unit objectives were put into levels by classifying them as ‘introduced’, ‘practiced’, or ‘demonstrated’ and within these categories, further classified as ‘basic’, ‘intermediate’ or ‘advanced’. While a generic example was provided by Allen (2004) of the resulting mapping or matrix, it would have been more useful if definitions of the categories and of the process used for levelling the objectives had been included. This would have enabled us to extrapolate to our disciplines and context and to determine whether or not the definitions could be mapped from one discipline to another.
Scott and Gribble (2005) also described a mapping process as a way of levelling, but this was a much more complex form involving industry input by an external industry consultant. Engineering academic staff used this approach to review and reform a traditional course at Curtin University of Technology.
Cordiner, Stenzel and Hafner 19
Their six‐stage approach included mapping each component of the degree to two sets of generic attributes: those prescribed by Engineers Australia, and those developed by the university. The result, at the end of the fifth stage, was several matrices for each year level showing what each unit contributed to the development and demonstration of the attributes. This was mostly based on discipline content and skills expressed in terms of percentage contribution to each unit and to credit points, as well as broad statements of evidence. The final (and yet to be completed stage of this aforementioned project) is for academic staff to work out how this information is to be expressed in unit objectives in the different year levels of the program (degree) and how these unit objectives subsequently affect the nature of assessment tasks for the unit.
A multi-pronged approach: aligning disciplinary, learner and industry expectations
Unaware of these previous approaches to levelling
