Institutional ResearchWhat problems are we
trying to solve?
Mantz Yorke & Bernard Longden
Sorting out types of problems
“Reports that say that something hasn't happened are always interesting to me, because as we know,
there are known knowns; there are things we know we know.
We also know there are known unknowns; that is to say we know there are some things we do not know.
But there are also unknown unknowns the ones we don't know we don't know.”
Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002
The challenge facing any University is to
Be smarter and better manage the known knowns;
Inquire into university life to seek out the known unknownsand Reduce the risk of unknown unknowns as they relate to:
Demographic changes
National policy drivers
International opportunities
Types of problem :Types of institutional research
What’s going on type of research
Why is it going on type of research
ExplanatoryDescriptive
Without theory, research is impossibly narrow. Without research, theory is mere armchair contemplation. (Silverman, 2001: 294)
Two broad types of research
Institutional research - four faces
Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR.
Research forimprovement
Information
Educationalresearch
Audience
Role
Administrative,Institutional
Formative & internalfor improvement
Summative & externalAccountability
Academic,Professional
Presentation
Terenzini’s organisational intelligence
Contextual intelligence
Issues Intelligence
Technical / Analytical Intelligence
Hierarchy of
complexity
Factual knowledgeanalytical skills
Major issues or decision areas facing the university
Culture – knowledge of what a college is and where it has come from
What problems are we trying to solve?The view from inside - BL
Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR.
Information
Audience
RoleAdministrative,Institutional
Formative & internalfor improvement
Summative & externalAccountability
Presentation
Administrative/Formative-known knowns
IR as Information Authority
Administrative – formative and internal
Emerging within institutions as:
Information officeManagement informationPlanning and developmentInstitutional Research
Primary functions:
Clarifying the known knowns
Describing the university in data terms
Educating the university community about itself
What problems are we trying to solve?
Information authority
‘Base data’ about the institution – profile and fact bookStatutory returnsValidating data – is this really us?Size and shape of the institution to assist in setting targets and plans
Terenzini’s organisational intelligence level – Technical/analytical intelligence
In the UK these are well developed possibly over load the university with data
The challenge is to reduce the plethora of tables and cross tabs – to identify Key Questions that are crucial to the organisation for success
What problems are we trying to solve?
Presentation facility
Marketing the best aspects – advocate roleAudit and external accountability Performance measureLeague tablesTerenzini’s organisational intelligence level – Technical analytical intelligence
Spin and managing data
Political pressure to maximise impact of data
Denial about the message data gives
Risks and concerns
Primary functions:
Conform data to agreed descriptors Performance Indicators
Satisfy public funding legislative requirements
Student Data flow
Student data
Students
Aggregate early return to HEFCE
Data informs funding- audit
IndividualSummative activitydata return to HESA
Data passed to HEFCEwho recreate Decemberaggregate return
But same data also used to inform public through:Publications, commodification of data Rankings
Data sources to assist with the questions
UKHigher Education Statistics Agency
Published data sets HEDI
UCASUnistat
HEFCE (?)National Student Survey
US National Centre for Education StatisticsIPEDS – universe institutional census NPSAS – National Post Secondary
Student aid StudyBPS – Beginning Post Secondary -
Longitudinal StudyB&B – Baccalaureate & Beyond -
Longitudinal Study
National Science Foundation
Some propositions that we might need to address
Should the IR questions be articulated more clearly and data collected in response, or should we continue to collect data and then use what we have for policy decision?
Do universities make good use of the data within their own university?
Does IR constitute value for money?
Is data critical to teachers, administrators, decision makers?
Is the emphasis between data collection and analysis appropriate?
What lessons can be learned from the research community’s use of data sets in the US?’
Comparison – sometimes interesting
• UK HESA and US IPEDS – similar objectives and operation
• US interest in scientific advancement not reflected in the UK – Possibly RAE provides nearest comparison
• US positively encourages Institutional Research– Association for Institutional Research (US) in conjunction
with National Centre for Education Statistics provides funded summer school to encourage analyses of public data
• UK implicitly discourages Institutional Research – HESA charge for the use of our data!
What problems are we trying to solve?The view from outside - MY
Research forimprovement
Educationalresearch
Audience
Role
Formative for improvement
Summative & externalAccountability
Academic,Professional
Academic/Formative: known knowns and unknowns
IR as Research for Improvement
Academic – formative and internal
Analysing (afresh?) existing data, or investigating institutionally-focused research questions, with a view to influencing practice.
Problem: such activity is often scattered within an institution, and unco-ordinated.
Research for improvement
Examples:
Student feedback, including NSS
Outcomes of quality assurance
Assessment results, cross-tabulated with demographics, subject disciplines, etc.
Incidence of plagiarism
Research assessment exercise
What problems are we trying to solve?
Academic/Summative: known unknowns
IR as Educational Research
Academic – summative and external, though with some formative and internal relevance
Investigation of issues which are of wider relevance than to a single institution, but which may be triggered by an internal institutional need.
Evidence needs to be particularly robust.
Potential for dissemination of findings beyond the individual institution, in which case criteria for peer review may well apply.
Educational Research
Examples:
Pedagogical innovations of various kinds
Assessment of achievement in workplace settings
Variation in honours degree classifications across subject disciplines and institutional types
Relation of PT employment to achievement on FT programmes
Students’ experiences of HE (FYE; PT; HE in FE, etc)
Non-completion
What problems are we trying to solve?
In the end what answer can we give to Gnothi Sauton – ‘Know thyself’?