Embed Size (px)
Citation preview
DLHE Good Practice
17 September 2009
London
Objectives
• Introduce good practice within DLHE record
• Demonstrate tools designed to aid good practice
• Provide institutional perspectives on completion of survey and data collection
• Highlight data quality issues and checking
Target response rates
Target response rates
• 80% for FT UK domiciled students (including Research Council students)
• 70% for PT UK domiciled students
• 50% all other EU
Difficulties in meeting targets
• Increasing student mobility reduces the reliability of contact information
• Public scepticism about what personal data is held against them at a governmental level
• Survey saturation
• High targets set
Current response rates
• 78.5% for FT UK domiciled (-0.4% when compared with 2006/07)
• 70.7% for PT UK domiciled (-0.4% when compared with 2006/07)
• 54.7% for all other EU (+0.7% when compared with 2006/07)
Preparing for DLHE 2008/09
April target list system
• Uses a student file from that year
• Reduced validation
• Generates an accurate population file to survey against
• System currently underused
Population
• Important to check the POPDLHE file generated from the Student Record submission…
• …make sure you are not missing students…
• …or that no one is going to be surveyed that shouldn’t be
• The Match Report will show you how successful you were at this
Know your graduates
• Up-to-date contact information
• Some graduates will like to complete forms online, others will not, so offer as many methods of completing the survey as possible
• Call at the right times
• Identify your part-time and ITT graduates
Sections D and E
• Ensure part-time and/or ITT students complete the sections relevant to them
• Failure to do so will result in validation warnings
• Be aware of students who changed to a PT mode of study at the end of their course but studied the majority as a full-time student
Telephone interview
• Ask the respondent the core questions first if graduate is in a hurry…
• …without them answered you do not have a valid response
• Clarify contradictions or information that would appear incorrect
Location information
• Location of employment information is very important to statutory users of data
• Allows analysis of student’s geographical movement from entry to HE to the labour market
• Try to code to full postcode, however at least outward postcode level or town postcode should be reported
Audit
• The data should be kept only as long as is necessary
• The completed questionnaires should be kept (as either a hardcopy or electronic version) for each individual leaver for one year after the data is returned to HESA
• If the purpose is research then data may be kept for a longer period of time
• There is no need to anonymise the data
Aardvark system changes
• From now on, in addition to the Access Codes, users will also need a PIN code to create new accounts and/or add permissions to existing accounts. The PIN codes will be distributed by letter a few days in advance of the Access Codes
• Both the Access Codes email and the PIN letters are sent to the nominated record contact at institutions
• The system will now send an email to the transaction owner when a transaction has completed processing
DLHE online service
Why use it?
• Help to improve response rates by offering an alternative to paper or phone based questionnaires
• Students can complete it at their own accord and at their own pace
• Intuitive and free to register and use
• Built in validation
System Usage
Percentage of HEIs using system
31
43
53
0
10
20
30
40
50
60
S05018J S06018J S07018J
Online DLHE Questionnairefor both January and April census
Graduates complete form at www.dlhe.ac.ukor www.dlhe.ac.uk?INST=nnnn
to bypass institution selection
Institutions download data from submit.hesa.ac.uk
DLHE Online Statistics 1
DLHE online completed surveys
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
S05018J S06018J S07018J
Completed Surveys
DLHE Online Statistics 2
% Response rates of HEIs using S07018 for first time
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
A B C D E F G H I J
C06018
C07018
DLHE Online Statistics 3
Percentage response rates
78.478.6
76.2
76.7
75.0
75.5
76.0
76.5
77.0
77.5
78.0
78.5
79.0
C06018 C07018
Institutions usingDLHE online
Institutions notusing DLHE online
Uses of online system
• Existing institutions have used the system in different ways…
• …either as an additional tool to the mailings and telephone follow-ups…
• …or as part of a full integration policy
• Many institutions also use the system as a data inputting tool for staff
Registering for system
• Ahead of each DLHE survey you must pre-register for use of the system by the date we publish…
• …and as such requires incorporation into planning
The Aardvark system
• Register to use DLHE online on Aardvark
• Control panel allows you to specify your requirements for system e.g. inclusion of institution's student number or including a PIN
• Download the data of completed surveys from Aardvark
CASCOT
CASCOT (IER)
• CASCOT is designed to assign a code to a piece of text. e.g. a SOC code to a job title from the DLHE survey
• When CASCOT assigns a code to a piece of text it also calculates a score from 1 to 100 which represents the degree of certainty that the given code is correct
• Available to download from: https://submit.hesa.ac.uk/aardvark/CASCOT.ASP
Benefits of using CASCOT
• Ensures consistency within institution and also across sector
• Reduces difficulty of interpreting text
• Improves efficiency and reduces burden of data inputting
Potential problems
• The quality of coding depends on the quality of the input text:
"advertising & marketing"
- These are the area of work. Are they an advertising executive, an advert designer or an advertising salesman?
"answering phone enquiries“
- This is a job description, not a title.
Working with batches
• CASCOT can be used to code on a record-by-record basis or can work in batches
• Input files allow user to code as many records as they require
• Setting up an output file will then store the results in a spreadsheet for you
Interpreting results
• Always interpret the information accurately…
• …however you do not have to use the recommendation made by CASCOT (regardless of the score it is given)
• Use your judgement to assess whether it is the most accurate code and reflects the true nature of the job
For example
• Student employed as an architectural model maker for which CASCOT gives the following code…
• ‘Modeller (Architectural) 5315’
• …however that is classified as a non-graduate job…
• …whereas ‘Modeller, Artistic 3411’ is a graduate job
DLHE good practice
The School of Pharmacy
17th September 2009
Mark Gittoes
Head of Quantitative Analysis for Policy, HEFCE
Areas of use
• Institutional/subject area reporting
• Research
• Understanding of provision
• Overview of HE
• Other bodies
Institutional/subject area reporting
• HEI Performance indicators
• Teaching quality information and UniStats
• Regional profiles
HEI Performance indicators (1)
• Proportion of graduates who are working or studying (or both) by institution
• Currently published for full-time first degree graduates
• Extending to other cohorts
• Interest in the development of other indicators
HEI Performance indicators (2)
Unistats (1)
• “Employment prospects” broken down by institution and subject area
• Aimed at potential students
• Reports:
– Employment status
– Top 10 profession types
– Job category (graduate, non-graduate)
Unistats (2)
Research
• Student ethnicity
• Job types
Student ethnicity (1)
• Relationship between a student’s ethnicity and experience in HE:
– Entrant profiling
– Progression through HE
– Attainment in HE
– Satisfaction
– Graduate destinations
Student ethnicity (2)
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
White Black Pakistani orBangladeshi
Indian, Chineseor Other Asian
Mixed or Other
Per
cen
tag
e em
plo
yed
or
stu
dyi
ng Young Mature
Research: Job types (1)
• Examination of Standard Occupational Classification (SOC)
• Study of relationship with student perception of criteria needed for job?
• Makes use of responses to DLHE question:
– “Would you have been able to get the job you were
doing on 14 April 2008/12 January 2009 without the
qualification you recently obtained (the actual
qualification, not the subject of study)?”
Research: Job types (2)
Understanding of provision
• Strategically important and vulnerable subjects
• Foundation degrees
• Erasmus and placement courses
• Further Education Initial Teaching Training
SIVS: Early careers of graduates
“To address the lack of information on the supply and demand of STEM skills, HEFCE should … publish an annual report describing: undergraduate subject trends; recent graduate jobs and salaries; and the subjects where employers and government departments believe that there are, or are likely shortly to be, shortages of graduates with key skills.”
SIVS: Early careers of graduates
Destination No. %
Studying (not employed) 1,850 47%
Studying and employed 825 21%
Employed 1,015 26%
Total employed 1,840 47%
Unemployed 105 3%
Other 110 3%
Total DLHE respondents
3,900 100%
Note: Full-time foundation degree qualifiers registered at English HEIs, 2005-06
Foundation degrees: destinations
Erasmus and placement courses:Salary six months after qualifying
£-
£5,000
£10,000
£15,000
£20,000
£25,000
Erasmus Placement year Other 3 yeardegree courses
Other 4 yeardegree courses
Type of degree undertaken
Mea
n s
alar
y o
f g
rad
uat
es
Other bodies• Office of National Statistics
– UK migration mapping
• Department for Business, Innovation and Skills
– Parliamentary questions
– Research
• Research Councils UK
• Higher Education Policy Institute
– Male and female participation in HE
• Other UK HE funding bodies
Quotes
• “It could be that males are more likely to be rejected for the jobs that they apply for, …, and females are experiencing discrimination in the salaries that they are paid.” Male and female participation and progression in HE, HEPI June 2009
• “Graduate employment data makes grim reading” The Times Higher, July 2009
• “More graduates finding work” The Times Higher, August 2008
The future?
• Move from DIUS to DBIS
• Increased focus of employability
• Development of additional indicators
• HE framework and capturing institutional strength
DLHE Good Practice Seminar: 17th September 2009
Nathalie ReesSenior Careers Information
Officer
Background & current role
• Survey experience
• Review of First Implementation& good practice document
• Survey co-ordinator/manager at the University of Chester
• Departmental contact for data
Who makes DLHE work @ Chester?
But also:
• MIS colleague
• Temporary telephone team
• Career Link Tutors
• Departmental Administrators
Procedures & training• Preparation: procedures checklist
• Training for new staff members: coding and SITS
• Regular updates at team meetings; keeping the team informed of progress
Mailing & processing questionnaires
• Preparation for first mailing & online survey
• First mailing
• Processing incoming questionnaires
• Coding
Telephone follow-up
• Recruitment of phoners
• Training
• Documentation/resources
• Quality checking
Liaison with departments
• Contact details
• Obtaining information
• Recruitment of students for telephone follow-up
• Raising awareness of DLHE survey
Inputting & data quality control
• Tools used
• Coding checking, inputting & validation
• Spelling & postcodes
• Excel spreadsheets – MIS partnership
Challenges
• Up-to-date contact details
• Staffing/administration of the survey
• Survey overload
• Meeting targets for specific cohorts etc…
Room for improvement
• Updating procedures/documentation
• Increasing online replies
• Using DLHE data
• Survey awareness raising with final years & academic departments
Internal use of DLHE data
• Academic departments
• Careers Advisers
• Publications & website
Any questions
DLHE Good PracticeUniversity College Falmouth
Contents of Presentation
• Profile of University College Falmouth• DLHE outcomes at Falmouth over the last 10 years• Outline of the DLHE process at Falmouth• The role of Alumni• Lessons to be learnt• Tricks of the trade• Wish list
Profile of University College Falmouth
• Small HEI – just under 3000 students• Specialist – Art, Design, Media and now Performance• Practice based – most courses are studio based with
students spending 80% of their studies doing hands-on practical work & 20% of their time on written work.
• No exams – continual assessment• High levels of dyslexia and mental health issues
DLHE at Falmouth over the last decade
• Admissions staff at Falmouth have responsibility for HESES, HESA, DLHE, Enrolment, Assessment Processing, Student Transcripts, Open Days etc.
• Average response rate is just above 50%, Falmouth’s employability rate is normally between 91%-94%.
• The worst employability rate came the year we
employed an ex-BBC researcher to work on the DLHE for 6 months. Response rate went up to 97%. We came bottom of the employment tables with 43% of graduates obtaining jobs. Luton University came top with a mere 42% response rate.
Outline of the DLHE process at Falmouth
Inclusion of a letter in their graduation pack
Letter & form to their home address
Letter & form again to their home address
Using ‘text tools’ to send text reminders to mobile ‘phones
A postcard ‘where are you’ to their parents asking them to forward on any envelopes bearing the College logo.
Emailed reminders
Phoning graduates during the day
Phoning graduates between 6 & 7:30 pm
Utilising tutors private Alumni
Use other contacts and other methods
The Role of Alumni
Tutors tend to operate a private Alumni, whilst often effective for DLHE purposes, transgress the boundaries of the Data Protection Act.
Falmouth have established an official Alumni in the last 18 months which it is hoped will improve communication with graduates by maintaining up-to-date contact details, networking, course cameraderie and another conduit for the DLHE.
Lessons we have learnt
• Visual thinkers such as art and design students do not engage well with the DLHE official questionnaire
• 31 questions are 27 questions too many for our graduates• Students are suffering from questionnaire fatigue• Postcards rather than letters have a better response rate• Mothers, house-mates, student ambassadors are invaluable in
nudging graduates and/or providing new contact details• Graduates with dyslexia or mental health issues do not engage
well with the DLHE• Cheap tricks like ipods or book tokens don’t work• Don’t try too hard to contact 100% of your graduate cohort• Graduates tell lies or bend the truth.
Tricks of the trade (1)
• Use student ambassadors, they know who knows who and they talk the same language (doh..)
• Utilise technology such as social networking sites e.g. face book, friendsreunited, twitter.
• Parents and mature graduates prefer printed post e.g. reminder postcards and stamped addressed envelopes.
• Graduates prefer electronic communication – invest in ‘text tools’ to perform mass SMS texting, download spreadsheets of private email addresses to upload into Outlook and b.c.c DLHE messages to graduates
• Ask course teams to feed back on references for graduates.• Set up an Alumni; use case studies in Alumni newsletters.
Tricks of the trade (2)
• In ‘phone and text chase-ups, there are 4 key questions to ask• Have you got a job, where, who with, doing what.• You can use the web to find out post codes (Q10), what the
organisation does (Q9), size of organisation (Q11), the type of school (Q26, Q27) etc.
• The kind of work the graduate is in, can often provide missing answers – e.g. you don’t need a degree in Graphic Design to work as a shelf stacker in Asda (Q12, Q13).
• Don’t offer gifts, extend the use of workshops, studios, library to Alumni instead.
• Don’t use careers to give DLHE talks, invite successful Alumni to give DLHE talks to 3rd years.
Our wish list
• Severely cut the number of DLHE questions• Create a visual mind map version of the DLHE• Produce a summarised parent’s version of the special
issue ‘statistics focus’ viewable via http://www.direct.gov.uk
• A DLHE user group on JISCMAIL
79
University of LondonGood Practice
Oliver Gardham &
Julie Cheung-Inhin
80
Introduction
• Who we are• Where we are• How many?• The Careers Group
81
Who We Are
• A ‘family’ of universities and institutes affiliated with or bound to The University of London
• Some colleges constitute one HESA return, others are combined
82
Who We Actually AreRoyal Holloway,
UoLKings College
LondonUCL
Goldsmiths Courtauld IoA Queen Mary, UoL
SOAS School of Pharmacy LSHTM
Cancer Research Heythrop Institute of Education
Birkbeck Royal Veterinary Coll.
St George’s HMS
Others…. UoL Institutes (below)
Warburg Institute UoL in Paris Inst. Advanced Legal Studies
Inst. Classical Studies Inst. Commonwealth Studies
Inst. English Studies
Inst. Germanic & Romance Stud
Inst. Historical Research Inst. Musical Research
Inst. Philosophy Inst. Study of the Americas Millport Marine Biological Station
83
Where We Are
Royal Holloway
84University Marine
Biological Station,Millport
85
Millport
86
University of London Institute in Paris
87
Senate House andStewart House
88
How Many?
• Between 45,000 and 50,000 total graduates
• Around 30,000 of which are U of L• Most colleges request internationals,
due to diversity of London graduates• It increases every year
89
The Careers Group• Central Careers Administration• Products and services designed to help
individuals achieve and maintain job satisfaction throughout their working lives
• Customers include academic institutions, students, graduates, graduate recruiters, and other careers services providers
• Products and services include careers advice, graduate fairs, online services, courses, professional training
• Graduate Research
90
Graduate Research
• Full-time DLHE team of 4 all year round, one of which is the Call Centre Manager
• On-site call centre• Department with a collective 20
years’ experience, including manager with 10 years of experience
91
DLHE Problems
• Spotting trends and problems in time– Bad numbers– Unemployment
• Quality control• Language barriers• Promoting/engaging with the data
92
Bad Numbers
Near-instant reporting on:
• Bad numbers• Unemployment rates• Response rates
93
Quality Control
• Invest in a good capture system• Call afternoons and evenings, week
and weekends if necessary)• Experienced callers• Employee retention is valuable• International codes and time zones• Language diversity
94
Colleges with Language Schools
• UCL (Dutch, French, German, Greek, Hebrew, Italian, Spanish, the BA Modern Languages course allows over 250 combinations of modern languages)
• SOAS (over 50 languages, including Arabic, Amharic, Bengali, Chinese, Hausa, Hebrew, Hindi, Japanese, Korean, Nepali, Swahili, Thai, Tibetan, Urdu, Vietnamese and Zulu)
• King’s (Arabic, Bengali, Catalan, Dari, Farsi, French, German, Greek, Gujarati, Hebrew, Hindi, Italian, Japanese, Mandarin, Punjabi, Polish, Portuguese, Russian, Sanskrit, Spanish, Swedish, Turkish, Urdu)
• Royal Holloway (French, German, Hispanic Studies, Italian)• Birkbeck (French, German, Japanese, Spanish, Portuguese)• Goldsmiths (French, German, Spanish, Portuguese)• Queen Mary (French, German, Hispanic, Russian)• Institute of Germanic and Romance Studies (French, German,
Hispanic, Italian)• University of London Institute in Paris (French)
95
Engaging with the Data
• Familiarise yourself with both survey and registry data. Over 100 fields of information are available for each graduate allowing simple or complex cross-referencing
• Prepare a menu of available reports for academics or other interest groups who may not be aware of DLHE’s potential, for example…– Graduate level work and/or post-graduate study– Destination summaries– Work sectors– Geographical reporting– Top ten employers/jobs
• Allow for various breakdowns/versions for each report, for example– PT/FT, UG/PG, UK/EU, Department, Course, Gender, Ethnicity,
Class
96
SIC/SOC & SectorsSIC CODE SIC Description The Careers Group Sector
6010 Radio broadcasting Motion Picture Production/Sound Recording/Broadcasting
6020 Television programming and broadcasting activities Motion Picture Production/Sound Recording/Broadcasting
6100 Telecommunications Telecommunications
6110 Wired telecommunications activities Telecommunications
6120 Wireless telecommunications activities Telecommunications
6130 Satellite telecommunications activities Telecommunications
6190 Other telecommunications activities Telecommunications
6200Computer programming, consultancy and related
activities Computer Programming, Consultancy & Related
6201 Computer programming activities Computer Programming, Consultancy & Related
6202 Computer consultancy activities Computer Programming, Consultancy & Related
6203 Computer facilities management activities Computer Programming, Consultancy & Related
97
A Few of Our Sectors
The Careers Group Sector
Architectural/Engineering Activities
Arts, Entertainment & Recreation
Building & Landscape Services
Computer Programming, Consultancy & Related
Construction
Motion Picture Production/Sound Recording/Broadcasting
Telecommunications
Transportation & Storage
98
“Helping Graduates in Work” Website
• Mayor of London website• http://
www.london.gov.uk/graduates/Default.aspx
• “What can I do with my XXXX degree?”
• Jobs, salaries, employers• “What does a XXXX do?”
99
Destinations Website
• Online access to wide range of reports for all DLHE years with different levels of access for Head of Careers Services, Careers Advisers, academics
• Destinations Breakdown, Top Work Locations (using Google Maps), Average Salary, Employment Index, Graduate-Level Employers, and Graduate-Level Jobs
• Institutional breakdowns at UoL, college, department and course level
• Different levels of log-in
100
Summary
• Monitor and report collection data asap
• Retain staff where possible• Be prepared for international calling• Make the data interesting and
useable for academics/CAs/principles• Give people reporting options
101
Thank you
102
Q & A?
DLHE Data Quality
Exception checks
• Designed to pick up peculiarities within the data
• Can be errors or warnings
• Feedback to user given at point of ‘COMMIT’ transaction
• Significant exceptions will be raised by Data Quality Assurance at HESA
Common issues
• Students working and studying in full-time modes
• Incorrect salaries being reported – salary information should be reported as FTE
• Don’t return ‘0’s for salary if information not collected - use Xs instead
• Part-time and/or ITT students not completing the relevant sections
• Incomplete postcodes – unknown regions
Teaching Quality Information
• Teaching Quality Information (TQI) is displayed in three tables:
- Table 5: Destinations information
- Table 6: Job categories information
- Table 7: Common job types information
• Ensure you check each table as the information is widely used…
UNISTATS
• In 2008 – 565,250 unique visits
• In 2009 so far – 409,157 unique visits (comparable time in 2008 there had been 299,982 unique visits)
• International readership
• Significant advertising campaign placing links to UNISTATS on popular websties
Table 6: Job categories information
Check documentation…
Future developments
Catherine Benfield
Review of Early DLHE
• Implementation 2011/12 (Apr 2012/Jan 2013)
• Review will take place from Autumn 2009 and be completed by February 2011
• Usual HESA review procedures including review group with institutional representation
The review will include…
• Revisions to the questionnaire and telephone script
• Review of supporting documentation
• Development of online systems and support
• Technical update to use XML
• Other aspects of the survey
Review items – questionnaire and telephone script
• As well as the usual DLHE review items which include…– Reassess core and non-core
questions– Ensure routing works (issues with
sections D and E)– Any design improvements– Explore any problems that have
been reported
Review items – questionnaire and telephone script (cont.)• …there are some specific items for
review/consideration this time round:– Remove NHS question (Q8)– Update method categories to reflect the
current options– Capture of free text information– Q1 and Q2 options – Use of JACS for Q21 (subject area of
study or training)– Section E (part-time leavers)– How the survey works for self-employed
leavers
Review items – supporting documentation
• Review letter and email texts
• Review the methodology document
• Possible SOC 2010 implementation
SOC10 implementation
• SOC 2010– Will be finalised by ONS soon– There are significant changes planned,
particularly to Major Group 1 (Managers)– Some of the disaggregation currently done
at 5-digit level will be available at 4-digit (e.g. separate codes for barristers and solicitors)
– No decision yet on the need for a 5th digit– “Graduate jobs” will need to be re-specified
Review items – technical update to XML
• eXtensible Markup Language
• The HESA Student Record has been translated to XML
• Data standards - Information Standards Board
• Hierarchical data structures
Review items – other issues
• Coverage– BIS is commissioning research to
investigate feasibility of including international students in DLHE. This will feed into DLHE development
– There is ongoing discussion about the exclusion from DLHE of those who receive their awards from dormant status
– Multiple qualifications for a student - HIN
Review items – development of online systems and support
• Investigate the development of an input-only version of the online questionnaire
• Remove the HTML provision in order to focus on improving the DLHE Survey System
• Assess the use of the April target List System and whether or not it can be removed
• Look at how the DLHE Survey System is hosted
The future for the Longitudinal Survey?
• Results of second survey published earlier this month
• The funders of the survey (Funding Councils, DELNI, TDA and RCUK) will take a decision on the future of the survey in the next few weeks
• Quite separately, RCUK intend to undertake further follow-up with the 2004/05 cohort of PGR graduates
Operational Training
• DLHE Introductory seminar
• 9 February 2010 in Cheltenham
• www.hesa.ac.uk/seminars
