20
Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Embed Size (px)

Citation preview

Page 1: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Institutional ResearchWhat problems are we

trying to solve?

Mantz Yorke & Bernard Longden

Page 2: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Sorting out types of problems

“Reports that say that something hasn't happened are always interesting to me, because as we know,

there are known knowns; there are things we know we know.

We also know there are known unknowns; that is to say we know there are some things we do not know.

But there are also unknown unknowns the ones we don't know we don't know.”

Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002

Page 3: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

The challenge facing any University is to

Be smarter and better manage the known knowns;

Inquire into university life to seek out the known unknownsand Reduce the risk of unknown unknowns as they relate to:

Demographic changes

National policy drivers

International opportunities

Page 4: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Types of problem :Types of institutional research

What’s going on type of research

Why is it going on type of research

ExplanatoryDescriptive

Without theory, research is impossibly narrow. Without research, theory is mere armchair contemplation. (Silverman, 2001: 294)

Two broad types of research

Page 5: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Institutional research - four faces

Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR.

Research forimprovement

Information

Educationalresearch

Audience

Role

Administrative,Institutional

Formative & internalfor improvement

Summative & externalAccountability

Academic,Professional

Presentation

Page 6: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Terenzini’s organisational intelligence

Contextual intelligence

Issues Intelligence

Technical / Analytical Intelligence

Hierarchy of

complexity

Factual knowledgeanalytical skills

Major issues or decision areas facing the university

Culture – knowledge of what a college is and where it has come from

Page 7: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

What problems are we trying to solve?The view from inside - BL

Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR.

Information

Audience

RoleAdministrative,Institutional

Formative & internalfor improvement

Summative & externalAccountability

Presentation

Page 8: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Administrative/Formative-known knowns

IR as Information Authority

Administrative – formative and internal

Emerging within institutions as:

Information officeManagement informationPlanning and developmentInstitutional Research

Primary functions:

Clarifying the known knowns

Describing the university in data terms

Educating the university community about itself

Page 9: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

What problems are we trying to solve?

Information authority

‘Base data’ about the institution – profile and fact bookStatutory returnsValidating data – is this really us?Size and shape of the institution to assist in setting targets and plans

Terenzini’s organisational intelligence level – Technical/analytical intelligence

In the UK these are well developed possibly over load the university with data

The challenge is to reduce the plethora of tables and cross tabs – to identify Key Questions that are crucial to the organisation for success

Page 10: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

What problems are we trying to solve?

Presentation facility

Marketing the best aspects – advocate roleAudit and external accountability Performance measureLeague tablesTerenzini’s organisational intelligence level – Technical analytical intelligence

Spin and managing data

Political pressure to maximise impact of data

Denial about the message data gives

Risks and concerns

Primary functions:

Conform data to agreed descriptors Performance Indicators

Satisfy public funding legislative requirements

Page 11: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Student Data flow

Student data

Students

Aggregate early return to HEFCE

Data informs funding- audit

IndividualSummative activitydata return to HESA

Data passed to HEFCEwho recreate Decemberaggregate return

But same data also used to inform public through:Publications, commodification of data Rankings

Page 12: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Data sources to assist with the questions

UKHigher Education Statistics Agency

Published data sets HEDI

UCASUnistat

HEFCE (?)National Student Survey

US National Centre for Education StatisticsIPEDS – universe institutional census NPSAS – National Post Secondary

Student aid StudyBPS – Beginning Post Secondary -

Longitudinal StudyB&B – Baccalaureate & Beyond -

Longitudinal Study

National Science Foundation

Page 13: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Some propositions that we might need to address

Should the IR questions be articulated more clearly and data collected in response, or should we continue to collect data and then use what we have for policy decision?

Do universities make good use of the data within their own university?

Does IR constitute value for money?

Is data critical to teachers, administrators, decision makers?

Is the emphasis between data collection and analysis appropriate?

What lessons can be learned from the research community’s use of data sets in the US?’

Page 14: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Comparison – sometimes interesting

• UK HESA and US IPEDS – similar objectives and operation

• US interest in scientific advancement not reflected in the UK – Possibly RAE provides nearest comparison

• US positively encourages Institutional Research– Association for Institutional Research (US) in conjunction

with National Centre for Education Statistics provides funded summer school to encourage analyses of public data

• UK implicitly discourages Institutional Research – HESA charge for the use of our data!

Page 15: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

What problems are we trying to solve?The view from outside - MY

Research forimprovement

Educationalresearch

Audience

Role

Formative for improvement

Summative & externalAccountability

Academic,Professional

Page 16: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Academic/Formative: known knowns and unknowns

IR as Research for Improvement

Academic – formative and internal

Analysing (afresh?) existing data, or investigating institutionally-focused research questions, with a view to influencing practice.

Problem: such activity is often scattered within an institution, and unco-ordinated.

Page 17: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Research for improvement

Examples:

Student feedback, including NSS

Outcomes of quality assurance

Assessment results, cross-tabulated with demographics, subject disciplines, etc.

Incidence of plagiarism

Research assessment exercise

What problems are we trying to solve?

Page 18: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Academic/Summative: known unknowns

IR as Educational Research

Academic – summative and external, though with some formative and internal relevance

Investigation of issues which are of wider relevance than to a single institution, but which may be triggered by an internal institutional need.

Evidence needs to be particularly robust.

Potential for dissemination of findings beyond the individual institution, in which case criteria for peer review may well apply.

Page 19: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Educational Research

Examples:

Pedagogical innovations of various kinds

Assessment of achievement in workplace settings

Variation in honours degree classifications across subject disciplines and institutional types

Relation of PT employment to achievement on FT programmes

Students’ experiences of HE (FYE; PT; HE in FE, etc)

Non-completion

What problems are we trying to solve?

Page 20: Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

In the end what answer can we give to Gnothi Sauton – ‘Know thyself’?