Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden Slide 2 Sorting out types of problems Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns the ones we don't know we don't know. Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002 Slide 3 The challenge facing any University is to Be smarter and better manage the known knowns; Inquire into university life to seek out the known unknowns and Reduce the risk of unknown unknowns as they relate to: Demographic changes National policy drivers International opportunities Slide 4 Types of problem : Types of institutional research Whats going on type of research Why is it going on type of research ExplanatoryDescriptive Without theory, research is impossibly narrow. Without research, theory is mere armchair contemplation. (Silverman, 2001: 294) Two broad types of research Slide 5 Institutional research - four faces Volkwein (1999) has provided a useful analysis of the aims of IR The four Faces of IR. Research for improvement Information Educational research Audience Role Administrative, Institutional Formative & internal for improvement Summative & external Accountability Academic, Professional Presentation Slide 6 Terenzinis organisational intelligence Contextual intelligence Issues Intelligence Technical / Analytical Intelligence Hierarchy of complexity Factual knowledge analytical skills Major issues or decision areas facing the university Culture knowledge of what a college is and where it has come from Slide 7 What problems are we trying to solve? The view from inside - BL Volkwein (1999) has provided a useful analysis of the aims of IR The four Faces of IR. Information Audience Role Administrative, Institutional Formative & internal for improvement Summative & external Accountability Presentation Slide 8 Administrative/Formative-known knowns IR as Information Authority Administrative formative and internal Emerging within institutions as: Information office Management information Planning and development Institutional Research Primary functions: Clarifying the known knowns Describing the university in data terms Educating the university community about itself Slide 9 What problems are we trying to solve? Information authority Base data about the institution profile and fact book Statutory returns Validating data is this really us? Size and shape of the institution to assist in setting targets and plans Terenzinis organisational intelligence level Technical/analytical intelligence In the UK these are well developed possibly over load the university with data The challenge is to reduce the plethora of tables and cross tabs to identify Key Questions that are crucial to the organisation for success Slide 10 What problems are we trying to solve? Presentation facility Marketing the best aspects advocate role Audit and external accountability Performance measure League tables Terenzinis organisational intelligence level Technical analytical intelligence Spin and managing data Political pressure to maximise impact of data Denial about the message data gives Risks and concerns Primary functions: Conform data to agreed descriptors Performance Indicators Satisfy public funding legislative requirements Slide 11 Student Data flow Student data Students Aggregate early return to HEFCE Data informs funding - audit Individual Summative activity data return to HESA Data passed to HEFCE who recreate December aggregate return But same data also used to inform public through: Publications, commodification of data Rankings Slide 12 Data sources to assist with the questions UK Higher Education Statistics Agency Published data sets HEDI UCAS Unistat HEFCE (?) National Student Survey US National Centre for Education Statistics IPEDS universe institutional census NPSAS National Post Secondary Student aid Study BPS Beginning Post Secondary - Longitudinal Study B&B Baccalaureate & Beyond - Longitudinal Study National Science Foundation Slide 13 Some propositions that we might need to address Should the IR questions be articulated more clearly and data collected in response, or should we continue to collect data and then use what we have for policy decision? Do universities make good use of the data within their own university? Does IR constitute value for money? Is data critical to teachers, administrators, decision makers? Is the emphasis between data collection and analysis appropriate? What lessons can be learned from the research communitys use of data sets in the US? Slide 14 Comparison sometimes interesting UK HESA and US IPEDS similar objectives and operation US interest in scientific advancement not reflected in the UK Possibly RAE provides nearest comparison US positively encourages Institutional Research Association for Institutional Research (US) in conjunction with National Centre for Education Statistics provides funded summer school to encourage analyses of public data UK implicitly discourages Institutional Research HESA charge for the use of our data! Slide 15 What problems are we trying to solve? The view from outside - MY Research for improvement Educational research Audience Role Formative for improvement Summative & external Accountability Academic, Professional Slide 16 Academic/Formative: known knowns and unknowns IR as Research for Improvement Academic formative and internal Analysing (afresh?) existing data, or investigating institutionally- focused research questions, with a view to influencing practice. Problem: such activity is often scattered within an institution, and unco-ordinated. Slide 17 Research for improvement Examples: Student feedback, including NSS Outcomes of quality assurance Assessment results, cross-tabulated with demographics, subject disciplines, etc. Incidence of plagiarism Research assessment exercise What problems are we trying to solve? Slide 18 Academic/Summative: known unknowns IR as Educational Research Academic summative and external, though with some formative and internal relevance Investigation of issues which are of wider relevance than to a single institution, but which may be triggered by an internal institutional need. Evidence needs to be particularly robust. Potential for dissemination of findings beyond the individual institution, in which case criteria for peer review may well apply. Slide 19 Educational Research Examples: Pedagogical innovations of various kinds Assessment of achievement in workplace settings Variation in honours degree classifications across subject disciplines and institutional types Relation of PT employment to achievement on FT programmes Students experiences of HE (FYE; PT; HE in FE, etc) Non-completion What problems are we trying to solve? Slide 20 In the end what answer can we give to Gnothi Sauton Know thyself?