The Knowledge Pirates Guide to Institutional Researchaair.org.au/app/webroot/media/pdf/JIR/PirateGuide.pdf · The Knowledge Pirates Guide to Institutional Research by A G Nolan OAM

The Knowledge Pirates Guide to Institutional Research by A G Nolan OAM JP ISBN 0-9751338-0-2

G’day,

My name is Tony, and I would like to welcome you to my Knowledge Pirates Guide to Institutional Research.

This E-Book is not your usual statistical or research guide to conducting institutional research. It is more like a Digest of helpful tips, hints, reflections and tricks on Institutional Research. I hope you find it useful !!!

This resource is provided free for you to use.

I love the practice of Knowledge Philanthropy, and I would like to ask you a favour. If you get some use out of this E-Book, you could practise some Knowledge Philanthropy yourself. Pass on some piece of knowledge that you have to others, with a similar request to them aswell. Then we can all benefit from each other, and help the world become a better place through the sharing of knowledge.

The Knowledge Pirates Guide to Institutional ResearchPage No How to use this E-Book Contents Page

Book TitleTo go back to the previous page

The page numberTo go forward to the next page

Chapter Title Go to the 1st Contents Page

Welcome to the guide on how to read this E-Book.

Each page has a navigation table at the top. As you can see at the top of this page, there are a series of hyperlinks to help you navigate through this E-Book.

The Book Title, Chapter Title and Page Number are displayed for information, and are not designed to be interactive.

Both previous page and next page take you forward and backwards through the E-Book.

The 1st Contents Page is a hyperlink which will take you straight to the 1st page of the Contents Page. Each entry to the Contents Page is a hyperlink to the specific entry in the chapter. By clicking on the desired entry, you can jump directly to that page and area of interest.

What is Institutional Research ? Information – Intelligence – Knowledge – Wisdom Getting to Know the Problem Data Warehouses Data Marts Data Library Storage & Retrieval Population Sample Non-Probability Sampling Accidental Samples Convenience Sampling Quota Sampling Expert Sampling Probability Sampling Simple Random Samples Stratified Random Samples Clustered SamplesSampling With or Without Replacement The Data: Pieces of the Puzzle Categorical Ordinal Interval

Ratio The Statistics Frequency Distribution Measures of Central Tendency The Arithmetic Mean The Median The Mode Measures of Dispersion The Range The Standard Deviation Variance Normal Distribution & Normal Probability Curve Skewed Distributions Kurtosis Standard Error of the Mean Z Scores T-Scores Percentiles Correlations Non Parametric – For Ordinal and Categorical Scales Contingency Tables Chi Squares One Tailed Tests and Two Tailed Tests

The Knowledge Pirates Guide to Institutional ResearchPage No Contents Contents Page

678999910101010101011111111111112131313

1314141414141415151515151516161616161617171717

Experimental Design Reliability Validity Internal Validity External Validity Constructs Variables Pilot Tests Questions Questions: Tick and Scale Responses Questionnaire ScalesQuestions - Again and Again and AgainSimple Questions and One at a TimeAreas of Concern for Research Exp. and Designs ?? Maturation History Instrumentation Mortality Group Maturation Laboratory and Fields Experiments Time Frames Survey Replies Input Research Design Randomised 2 Group Design Randomised 2 Group Design: Before and After

Solomon 4 Group Design Factorial Design – 4 Groups Static Group Comparison Design Pretest: Posttest Nonequivalent Control Group Design Regression: Discontinuity Design One Group Pretest-Posttest Design Interrupted Time-Scale Design Replicated Interrupted Time-Scale Design A Hypothesis Design Qualitative and Quantitative Survey or Interview Gap Analysis Software Modelling Data Mining Data Modelling Simple Modelling Complexity Modelling Boolean Logic Fuzzy Logic Reporting Operational Reports Analytical Reports Summary Reports Working Reports / Interim


18181818181818191919191919202020202020202021222222

22222222232323232425252627282828282828282929293030

Annual Reports Fact Sheets Web Pages Graphs / Graphics Location on the Page Colours Titles & LabelsNumber of VariablesLegend & ScalesGet a 2nd Opinion Professional Standards & Ethics Level of Service Access to Information Censorship / Propaganda / Marketing Privacy Copyright and Intellectual Property Training and Up Skilling Hints for Graphs and Tables Hints for Graphs and Tables


30303031313131313232333333333434343536

Institutional Research is primarily what it suggests; a series of research style functions and investigations that examine and report on the how, who, where and why of an institution.

The following areas examine the way an institution functions; Marketing, Logistics, Budgets, Products and Services, Human Relations, Sales, and Management.

At all times a company or organisation needs to be well informed, and to try and follow an Action Philosophy Model. There needs to be a cycle of Action, Reflection and Decision, for a company or organisation to function effectively and to have the ability to grow and expand. An Institutional Research Perspective is essential and perhaps paramount for its survival. As information gatekeepers, knowledge key-holders and protectors of the Facts, it is important for the IR Professionals to be able to practise their trade in a fitting manner.

Traditionally IR was developed and focused primarily for the educational sector. It became an essential part of their reporting, logistical operations and strategic planning activities. IR has been a part of other disciplines, and has a lot of usage within the business and scientific sectors. Indeed, many of the primary skills and tools cut across a wide range of disciplines and activities.

With this in mind, interest in IR has spread out to the other disciplines, and IR now plays a role in many business and research activities. As can be seen as you read this E-Book, especially as you review the contents page, there are many different facets, skills and functions to the IR process and output. Whether it be statistical, reporting, data management or experimental design, there will always be the need to examine, reflect and benchmark institutions. The more rigorous the examinations, and the more professional the research and reporting activities, the better the decision making of the institution and the decision makers.

The Knowledge Pirates Guide to Institutional ResearchPage No 6 What is Institutional Research ? Contents Page

We know that data gets converted into information due to the brain’s ability to recognise and classify. Information then converts to intelligence (value added information). How? Intelligence becomes knowledge when there is a context, and a subjective cognitive interpretation has occurred. Thus - all knowledge is subjective, as is the same for truth! Hence the saying - the only absolute truth in this world is that there isn’t any.

When we produce knowledge to output into the world, what we give out as knowledge to ‘us’, is only information to others. The receivers have to go through their unique process to turn it into knowledge.

In transferring our knowledge for learning to occur, there needs to be thought put into the content of the information and the context of the information. Reinforcement of the information must occur from a variety of sources. The dream cycle is for when the newly constructed knowledge, or the reinforcement of already constructed knowledge, is to be integrated into their knowledge base.

The process is often that complex, you have to wonder how communication and learning ever occur.

As will be seen, this model looks at how we experience events, and how that builds into information, knowledge and finally wisdom. It is important to understand what we are receiving, so as to understand on what basis we are making our decisions.

The Knowledge Pirates Guide to Institutional ResearchPage No 7 Information – Intelligence – Knowledge – Wisdom Contents Page

Knowledge: the cognitive state beyond awareness. Knowledge implies an active involvement and understanding, and the ability to extend the level of understanding to meet life’s contingencies. Knowledge can also refer to the organised record of human experience given physical representation. (Books, Reports, etc.)

Wisdom: Implies the application of knowledge as contained in human judgement around certain values that are generally accepted by culture or society.

Information: the cognitive state of awareness (as in being informed) given representation in physical form (data). This physical representation facilitates the process of knowing.

Cognitive Driven Segment- - - - - - - - - - - - - - - - - - - -

Data Driven Segment

Data: Letters, numbers, lines, graphs and symbols. Used to represent events and their state, organised according to formal rules and conventions.

Rules & Information: Rules are used to organise such representations, and also in generating data.

Symbols: Representation of the event.

Events: The event is an occurrence, some condition or change in the state of the world.

It is important to recognise the problem you are reporting on. A problem may not always be seen as a problem. The problem may just be as simple as supplying some numbers to a colleague at your institution, or the problem may be a physical problem to solve. The main point is to know what you are about to undertake. If you are answering a question or solving a problem for someone else, then you need to make sure that you know what their problem is.

Too many times, people seek information without fully understanding what they are seeking. Generally, it then becomes the information provider’s fault for providing the wrong information. Instead, what is required is a better understanding of what the appropriate information actually is, before the problem at hand can truely be solved.

A good rule of thumb is to always know what data you have. Also important, is to seek out and explore the questions being asked of you. Feedback along the way is also very good. The saying ‘a stitch in time saves nine’ is very appropriate in information and intelligence work. So, if there is an element of being unsure as to whether you’re on the right path, it never hurts to ask.

Some good questions to ask are:

● What is the problem?● What causes the problem?● Why is it a problem?● What is the history of the problem?● Do I know what they want?● Do they know what they want?● Do I know what I don’t know?

The Knowledge Pirates Guide to Institutional ResearchPage No 8 Getting to Know the Problem Contents Page

Data WarehousesThis is where you keep your entire store of data housed in electronic or paper format. It is usually kept in a relational database, has a series of data tables, reference tables, and some sort of search and query. These types of databases are usually more a data storage and backup facility than an everyday working structure. It is a good practice to backup these databases, and to also have a copy to work off.

Data MartsThese are miniature subsets of the Data Warehouses, which are the real working backbone of the IR data sourcing. These are usually made up of various tables and extracts that are used as a working set. They may have been combined from various data sources, have been washed or cleaned up, have reference values inserted for raw terms or abbreviations, have constructed measurements, and so forth. These also need to be backed up. These should be time stamped to remind you that they are a snapshot of the data warehouse. Always remember that data aging is a real threat. If data marts are useful, then data updates and new data dumps from the main data warehouse are advisable. Data LibraryA Data Library or a Report Library, are where you keep copies of previous reports or extracts, usually with a data and time code in the name, together with some form of brief description. This is great as a fast paced data reference tool. It also helps to have a history of previous IR activities and results of the intitutions research and reports.

Storage & RetrievalA directory structure on ‘where and how’ data is kept, can be on a hard disk, as well as floppy backups, CD, and off site storage. If you have a distribution network of data or reports to various other offices or locations, it is best to have a labelling section of the various data / reports that are available. You should have some sort of classification and notation system, which includes the topic area, data types and date / time stamps of the data dumps or reports.

The way a data storage directory system is designed can make the retrieval process a lot easier and incredibly functional. By the way, when you construct the main and sub directories, you create a knowledge structure that tells an institutional researcher a story about the data structure.

The Knowledge Pirates Guide to Institutional ResearchPage No 9 Data Management Contents Page

PopulationIs the total volume of the data including all variables and information for the entire database. Population can also be used as a statistical term to represent the total number of persons in a subset of the main population. However, the population must be the total possible number of values to be examined.

SampleA sample is a subset of the population that is usually extracted though various methods. It is generally considered to be a true representation of the population for testing and comparison, and is often used in ‘what if’ scenarios. It is not always easy to know the correct sample size to use. Some times a random sample of as little as 10% of is used.

There are 2 main types of sampling - Non-Probability and Probability.

Non-Probability Sampling

Accidental SamplesThis is where the researcher just picks values out of the population until the desired number of values has been achieved. It is often used in generalisations, but may not be very reliable, as there is little structure in the data selections.

Convenience SamplingThis is where a sample is taken to represent the population because of the ease of accessing the sample. This is often used in survey sampling.

Quota SamplingThis is sometimes known as representative sampling, occuring where we pick a few variables and ensure that there are at least a certain number of representative values included in the sample. An example would be if you were conducting a study of different geographical areas in which 20 values would be located from each area.


Expert SamplingThis is where the sample to represent the population is drawn by a person or process that uses an expert with some special knowledge that is outside of the sample or population itself.

Probability Sampling

Simple Random SamplesIs where all values or representations in the population have an equal chance to be selected. There is a very good chance that most unique combinations will end up being represented if the sample size is big enough.

Stratified Random SamplesThis is exactly the same as simple random sampling. However, the population is broken up into a group of sub-populations with a simple random sample taken from each sub-population and combined into the sample to be examined and tested.

Clustered SamplesThis occurs when the population is divided into clusters, usually by a geographic categorisation. The group is then stratified, and a random sample taken with checks to see that all clusters are included. If not, intervention may occur to ensure that a cluster is represented.

Sampling with or without replacementWhen a sample is picked out of the population, the population size is reduced by one. This means that the chances of the next sample being chosen has increased by one. For example when you have a bag of ten marbles, each marble has one chance in ten of being picked. However the next time each marble has one chance in nice, and so on. Some researchers wish to make the chance of an sample being picked equal for all picks. To achieve this, the replace the drawn sample with a different item, or they return the same item to the population, with the opportunity to be chosen again.


One real problem with data is the way data isn’t organised, and the lack of respect for what it has to tell us. The other problem with data is that it can be manipulated to tell almost any story the producer wishes it to suggest. So there is always a moral and ethical question. If you have any questions on the ethics of data, I would suggest having a look at your local library association, which usually has suggestions on moral and ethical issues. Many universities have a Research Ethics Committee, which may need to be consulted regarding research on people.

A good method of practise is to have a Data Dictionary, which gives you a list of the variables and their characteristics, like the type of data, length of data, range of data, source of data, any reference or cross references to other data sources.

An example would be:

Name Type Length Range Source Restricted Data X- Ref Ref Date Created

Sex Cat 2 M/F Stud. R No Yes No 4/6/1980

You need to:

● Identify the topic in question or problem● Ask what are the questions we need to ask● Ask what data do we need to collect● Ask what limits do we have in dealing with data● Establish sources of data● Determine privacy & copyright issues● Comprehend Information & Intelligence - how to tell the story

The Knowledge Pirates Guide to Institutional ResearchPage No 12 The Data - Pieces of the Puzzle Contents Page

CategoricalData is represented in categories, usually in words and with no sequence or order to them. For example - Male / Female or Red / Blue / Green.

OrdinalData that has some sort of order or rank to it, can be in either words or numbers. An example would be like Low / Medium / High or 1 / 2 / 3. It is a more subjective measurement, and is mainly measured in whole units.

Interval Data that is recorded in parts of whole units - that is a true measurement (like on a ruler). All interval data is based on an equal unit of measurement. Interval data does not have a true zero, and cannot be used for ratios. In the measurement of temperature there is an artificial zero in Celsius and Fahrenheit degrees. When there is an artificial zero, true ratios do not exist. For example, 20 Celsius is not twice as hot as 10 Celsius. In fact, Kelvin is the ratio measurement scale for temperature, so 0.0 Kelvin is at -273 C degrees and 20 Celsius is really 293 Kelvin. When the temperature increase appears to have doubled in Celsius, the increase is really only 3.6%.

RatioRatio data is basically the same as interval data, with one major difference. Ratio data has a true zero, so 20 is always 1⁄2 of the 40 measurement. So, when you have walked 20 kilometres rather than 10 kilometres, then you have really walked twice as far.

The Knowledge Pirates Guide to Institutional ResearchPage No 13 Data Types Contents Page

Statistics can be the Institutional Researcher’s best friend, yet also their worst enemy at the same time. Statistics help us to analyse data, to tell a story, construct a picture and perhaps give us an indication of the future.

They can be so complex and specific on where you can use different statistical tools, so I have put together a simple selection of the various statistical information as a rough guide.

Frequency DistributionA frequency distribution is a basic list of all values of a variable, and the count of all such values. They help the Institutional Researcher to determine what other statistical tests may be of use, as well as providing good visual means to check for data errors or corrections.

Measures of Central TendencyThe Mean, Median and Mode are a set of measures that examine the average values of a data set. They give an indication of the shape and distribution of the data set, and also help to describe a sample or population for further analysis. They are only used for interval and ratio measurement scales.

The Arithmetic MeanThis is also known as an ‘Average’. Divide the sum of all values by the numbers of all values by the number of values recorded. The MedianThe Median is the middle score. For odd numbers of scores, make a list of the scores in descending order and find the exact middle one. For even numbers of scores, find the two middle scores, add them together and divide by 2. The Median can be used on ordinal, interval and ratio measurement scales.

The Knowledge Pirates Guide to Institutional ResearchPage No 14 The Statistics Contents Page

The ModeThe Mode is simply the value with the highest number of scores. The Mode can be used on all measurement scales.Measures of Dispersion

The RangeThe range is the lowest value to the highest value.

The Standard DeviationThis standard deviation describes the spread of scores within a variable. In conjunction with the mean, it shows you the shape of the bell curve. The standard deviation is essential for the majority of parametric statistical tests. The great number of observations used to create the standard deviation makes statistical tests more reliable. This is only used on interval and ratio data.

VarianceThe variance is the square of the standard deviation.

Normal Distribution & Normal Probability CurveA series of scores that are used from a sample or a population while using interval or ratio data. A normal distribution looks like a bell curve, with the right-hand side being a mirror image of the left-hand side. Because the bell curve is symmetrical it can be used for probability calculations and estimates. It also allows for comparisons between samples, populations and data sets.

Skewed DistributionsThese types of distributions have a significantly higher number of scores at one end of the range of data. A positively skewed distribution has the greater number of scores in the lower half of the distribution. A negatively skewed distribution has the greater number of scores in the upper half of the distribution. If the skew value lies outside the range of -0.5 to 0.5, then a skewed distribution is said to occur, which means that statistical tests such as Z-scores and T-scores cannot be used.


KurtosisThis measure is often partnered with the skew of a distribution. It measures whether the distribution is normal (mesokurtic) with a value of 0. If the distribution rises to a sharp point (leptokurtic) it has a positive kurtosis. If the distribution falls from a normal curve to have a more flattened curve (platykurtic) it has a negative kurtosis value.

Standard Error of the MeanThis is a measurement that tells you the possible error from the mean. When we measure a group of scores we have a range of values which will all have an effect on moving the mean forwards or backwards. The standard error of the mean tells you the possible variance the actual mean may be within the standard error from the statistical mean.

Z ScoresThis is a score that represents the percentage location of a score under the normal bell curve. It is expressed in standard deviation units and gives us the confidence to compare the differences between samples to see if there is commonality or not between the samples. Z Scores tend to be used with larger numbers of observations in the sample

T-ScoresWhen you are using very small sample sizes, specifically samples under 30 observations, you need to use a T-score. Then the distribution needs to be artificially adjusted.

PercentilesA percentile states that a particular score resides within a percentile. Thus a when a score is in the 69th percentile, that translates as 69% of the population achieving scores below it and 31% achieved scores above it.

CorrelationsPearson’s (Interval Scales) and Spearman’s (Ordinal Scales) correlations are measures of how alike 2 data sets are. The higher the value the more alike they are.


Non Parametric – For Ordinal and Categorical Measurement Scales

Contingency TablesThese tables are usually applied when you have only 1 or 2 variables, and you record the frequencies to give an example of the range of counts. These would be used for sex, pass / fail rates, etc.

Chi SquaresThis is where you can test your expected frequencies against your actual frequencies, to see if the sample matches the population. This test cannot be used with less than 5 observations recorded in any cell. A Chi Square is used to show if the variables are independent or dependent.

One Tailed Tests and Two Tailed TestsResearchers need to understand the different types of statistical tests, and the populations / samples they are exploring, to choose either a One Tailed Test or Two Tailed Test.


Experimental DesignSometimes people in IR will want to run an experimental design to test a hypothesis. For example, a teacher may want to examine a series of experiments to confirm if a hypothesis is the same as in the previous year. An experiment will then be run on different students to establish if the hypothesis holds true. Preferably this experiment will utilise the same test. Below are some considerations to be determined when dealing with experiments.

Reliability This is when the observations are consistent, and as free as possible from random error. In surveys and experiments, this would mean that the answers are consistent throughout the whole survey or experiment.

ValidityThis is where the questions are designed to record what the researcher wants, and not what the questions may ask. The wording is very important in ensuring that the desired variety of responses is consistent with the questions.

Internal ValidityThis is where the research needs to consider if the results of an experiment or investigation can be carried and applied to the outside world, or whether they are limited to the laboratory setting.

External ValidityThis is where a researcher considers whether the results of an external study can be applied to an internal setting. This would be of interest to people who wish to implement other studies conducted elsewhere into their own environment.

ConstructsThese are abstractions that researchers wish to measure, and will often combine different variables to analyse a construct.

VariablesThese are the questions and the measurements used to explore both population and data warehouses to test constructs.


Pilot TestsWhere possible, researchers should test a survey on a small group of people first. It is then wise to process and complete a full analysis. If possible, do try and run different pilots, and then make any necessary adjustments from the results.

QuestionsSometimes questions can be obtuse. Always try to make sure that the survey questions are only asking a single question.

Questions: Tick and Scale ResponsesWhen using a series of tick responses, try to be consistent and to make all the scales in the same direction, ie lowest to highest, etc. The best format to use is lowest to highest or negative to positive, left to right. This fits into the general perceptions of society. Other directions can be stressful, and cause the respondents to make errors.

Questionnaire ScalesLikert scales are the usual 1 – 5 or very low to very high. All 4 measurement scales can be used. ‘Yes and No’ or ‘True / False’ can only be used with Categorical tests. Writing a number response between 0 – 10 are interval / ratio tests.

Questions – Again and Again and AgainResearchers are a curious bunch, and sometimes are not always a trusting bunch. Hence, they may like to ask questions a few times over and over again, in different parts of the survey, to try and trick their respondents. If you are one of these people and you have matched questions, you will need to make sure that the questions contribute to the same constructs and have the same reliability and validity. However, they will need to have different wording to be sure as to not cause other type errors in the measurements.

Simple Questions and One at a TimeOften researchers want to know complex issues, and have very long or involved questions. However, this may just confuse the respondents. It is often better to have simple questions, and to make sure that the questions are only investigating one issue at a time. Double barrel questions tend to confuse people and increase errors.


Some areas of concern for research experiments and designs:

MaturationThis is where you may have other factors affecting your results such as, boredom, sickness, growth, or stress / tiredness.

HistoryThat people being tested may have already had a similar test, or have remembered the same survey when retesting occurs.

InstrumentationSometimes researchers reflect on possible changes, and mid-way through the study, change the questions in some way. This can change the validity of the results. When this happens, it negates the changed questions because the respondents will take the question a different way. You thus increase the chance of error in the study.

MortalityWhen you attempt a longitudinal study, extra numbers in the groups are needed because people may drop out of the process along the way. Obviously this affects the completion of the experiment.

Group MaturationWhen you conduct experiments, you may find that the groups under the study may have a specific influence that the other groups may not have. This influence may have some affect in that group. For example, If a person or a group of people in one group have some extra information or experience, they can advance or retard that group’s results.

Laboratory and Field ExperimentsLaboratory experiments have high reliability, because all the factors involved can be controlled to a high degree. Field experiments occur in their natural settings, and subjects may not know they are being studied.

Time framesTry to always make sure that when you start an experiment that you have a time frame and a suitable ending in mind.

The Knowledge Pirates Guide to Institutional ResearchPage No 20 Experimental Design Contents Page

Survey replies inputGood researchers always knows their tools, but they must also know their data. Entering in some, if not all, of the data, gives the researcher a feel for their respondents, the survey instrument and the sample and populations. It is just good practise to do this, and often gives other insights that may have been missed.

The Knowledge Pirates Guide to Institutional ResearchPage No 21 Experimental Design Contents Page

When conducting a group of experiments, there is a selection of designs you can use. Some use control groups, others are just pure observation. Some use only one survey, others use a before and after approach. Some take a longitudinal approach, and attempt to survey the same group of people over weeks, months and perhaps years.

Below are a series of experimental designs that can be used. (Experiment means to conduct a survey or process data / observe means to collect data and or publish findings.)

Randomised 2 Group DesignRandomly assign two groups of people, and then experiment and observe.

Randomised 2 Group Design – Before and AfterRandomly assign two groups of people, observe and then experiment and observe.

Solomon 4 Group DesignThis is a combination of Group Design and Group Design before and after. A researcher will run a design where there is a before and after, on only a few groups. The group design only occurs on the majority of the groups.

Factorial Design – 4 GroupsThis is where there are 2 types of testing for various interventions. Group 1 receives only Type A intervention, but receives it twice. Group 2 receives both Types A & B. Group 3 receives both types, but in reverse order, hence Type B & A. Group 4 only receives Type B, but receives it twice. An observation on all groups then occurs. Groups are randomly assigned.

Static Group Comparison DesignThis is where you have 2 or more groups, but usually where the groups are paired. Only one group receives an intervention, and then both groups are measured. Groups are randomly assigned.

Pretest – Posttest Nonequivalent Control Group DesignThis is where you have 2 or more groups (usually in pairs), where both groups are tested both before and after the intervention. But only one group receives the intervention.

The Knowledge Pirates Guide to Institutional ResearchPage No 22 Research Design Contents Page

Regression – Discontinuity DesignThis is where the researcher artificially assigns members to the groups, and no random allotment occurs. This is where you have 2 or more groups (usually in pairs), where both groups are tested both before and after the intervention. But only one group receives the intervention.

One Group Pretest-Posttest DesignThis is used where there is only 1 test to be conducted. However, there is a pretest and a posttest, with only one intervention.

Interrupted Time-Scale DesignThis is where there is a series of observations, then an intervention, and another series of observations. This is to see, over time, what the effects of the intervention are.

Replicated Interrupted Time-Scale DesignThis is where there is a series of observations, then an intervention, and another series of observations. This is to see, over time, what the effects of the intervention are. This is repeated in other groups, but their time frame of intervention is different to each other group.

The Knowledge Pirates Guide to Institutional ResearchPage No 23 Research Design Contents Page

Put forward a Null Hypothesis.Girls score higher grades than boys in mathematics.

Put forward an alternative hypothesis.Boys score higher grades than girls in mathematics.

Fix a set value for the Alpha Level.Lets use the 90% confidence interval.

Choose a research design.Use a two tailed test.

Choose the desired statistical test.As we are using 70 observations, we will use a z-test.

Calculate the Statistics.Calculate a z-test, and then compare previous results with current results.

Check to see if the results fit into the desired confidence level.Do the calculations, and check the results.

Choose the correct hypothesis and publish your findings.Accept or reject the correct hypothesis.

The Knowledge Pirates Guide to Institutional ResearchPage No 24 A Hypothesis Design Contents Page

Qualitative and Quantitative When conducting qualitative research, there are 2 different methodologies for using a framework for analysis. One is to pre-develop the framework from the constructs, and then fit the questions to the framework for analysis. The other way is called grounded theory. This is where you design the questions from the variables, and then construct the framework for the analysis from the very words written on the page or the transcripts.

I always find it better to use the grounded approach, but I do try to frame it within a bounded rationality. It is then possible to get some very good results, but still be able to contain them within a semi rigid framework, and thus get the best of both worlds. The section on Gap Analysis is a good example of how qualitative and quantitative can be used together to get a very strong research based impression on customer expectations and experiences.

Survey or Interview

There are many different ways to conduct qualitative research. Some people use interviews, some people use recordings, and others use surveys. Some people use only qualitative methods, and others use a combination.

Factors that often contribute to which option you choose are most often based on cost, resources and timeframe to be completed, and reported. There is always the possibility that results can be tainted (as mentioned in the area on reporting). If you are happy to take an action learning / action research approach, then using a more interactive questioning system allows you to gather better data though exploring issues that arise in the process. For instance, a focus group or a ‘one on one’ survey, allows you to follow up on interesting aspects. However, there is an increased chance of contamination occuring, and you may need to undertake some extra statistical compensation to accommodate any unexpected effects.

If you use a more static device such as a survey / questionnaire, with carefully designed questions, there is less of a chance of contamination, however, you will not be able to explore any extended developments.

The Knowledge Pirates Guide to Institutional ResearchPage No 25 A Hypothesis Design Contents Page

Gap analysis is a tool for quantifying the difference between a customer’s expectations and experiences with regard to a product or service. It also demonstrates to what extent the product / service is important to customer. This is a survey technique, where you use 3 questions to every question. The first 2 questions are for the liket or quantitative questions. These questions are used for the analysis of customer expectations and satisfaction. The 3rd question is used for qualitative information. You then make a matrix, and you can get an idea of the group think of various people, with regard to satisfaction and expectation.

The Knowledge Pirates Guide to Institutional ResearchPage No 26 Gap Analysis Contents Page

V H Extreme S High S Medium S Low S EqualH High S Medium S Low S Equal Low DM Medium S Low S Equal Low D Medium DL Low S Equal Low D Medium D High DVL Equal Low D Medium D High D Extreme D

VL L M H VH

Satisfaction DissatisfactionEqual This is where the customer’s satisfaction & dissatisfaction are balanced. This is to say that the customers are contented with the product / service. Low Low satisfaction is where the customer thinks that the product / service is

just above where they expected it to be.

Their opinion may be easily be shattered, and drop down in their satisfaction level, if they find a change they didn’t expect and disagree with.

Low dissatisfaction is where the customer thinks that the product / service is just below where they expected it to be.

However, their slightly negative perception may well be because of a misguided or poor understanding of the product / service, and it may be possible to move these people up a few levels, with little resource impacts.

Medium Medium satisfaction is where the customer thinks that the product / service is above where they expected it to be.

These customers are more comfortable with the product / service, and are more likely to ride out any discomfort that may arise.

Medium dissatisfaction is where the customer thinks that the product / service is below where they expected it to be.

These customers have a greater discomfort with the product / service, and would need more resources to move them up the satisfaction scale.

High High satisfaction is where the customer thinks that the product / service is well above where they expected it to be.

These customers are more comfortable with the product / service, and are increasingly more likely to ride out any discomfort that may arise.

High dissatisfaction occurs where the customer thinks that the product / service is well below where they expected it to be.

These customers have a great discomfort with the product / service, and would need extra resources to move them up the satisfaction scale.

Extreme These customers are well and truly over satisfied with your product / service, and regard it in the highest esteem.

It would be near impossible to lose them as customers.

These customers are so dissatisfied with your product or service, they might actually hate it.

It isn’t worth the resources to try and move them on purpose.

The combination of software is very important for the success of any project. Through combining the various, differing software outputs including integration between the different software applications, it becomes possible to have a fast and accurate picture. This gives a speed advantage, and increases the researchers ability to answer adhoc queries. This integration also gives researchers the opportunity to be data explorers, to freely follow patterns of data trails and to refine data searches. More advanced software (such as COGNOS) also allows us exploration of relevancy in some investigations. It also allows the researcher to check some of the variables in the data sources for inclusion or exclusion in the analysis.

The following applications are considered to be important to the investigation.

MSWord: A word processor used for the creation of reports.MsAccess: A database program used for data warehousing, data manipulation and creation of files for analysis.MsExcel: A spreadsheet program used for compiling results, graph creation and some basic analysis.WordPerfect: A word processor for reporting functions.QuattoPro: A spread sheet program used for data analysis and graphics.SPSS: A statistical package, which is used to clean data, provide statistic analysis, produce graphs, performance reporting activities, clustering activities and some basic prediction.Impromptu: A data warehouse interrogation program, which is used by other COGNOS products to produce analysis and reports. It is used for the creation of data marts and snapshots of data for analysis.Powerplay: An investigation and reporting software package, which makes the statistical outs more flexible. Also used to create various reports, and to answer adhoc questions.Transformer: A data transforming program to create powerplay cubes for analysis and reporting.Scenario: A data mining computer package that sorts through the bulk of the data, points towards impactive areas and explores the value of various subsets of the data for further investigations.4Thought: A data mining and data exploration software, that creates models of best fit, trend analysis, what if scenarios, strengths and weakness, grouping linking and interactions, degrees of significance, simulation and modelling.NUDIST: A qualitative data package used in survey analysis. It is mostly used for further marketing data activities, such as surveys and focus groups.

The Knowledge Pirates Guide to Institutional ResearchPage No 27 Software Contents Page

Data MiningThis is where people and software explore data populations to extract statistics, samples and combinations to enable a description to be given to the greater population. These applications and methodologies tend to look for connections, relationships, and cause & effects between variables.

Data ModellingThis is where the most basic data analysis is used to combine and cluster different variables by other variables, to develop data sets, samples, and subsets of various pieces of information. This then allows development of statistical tests to see if constructs are valid.

Simple ModellingSimple modelling is where no more than 3 variables are examined for their dynamic effect on each other. This allows for the researcher to examine different cause and effects, and some very basic ‘what if’ scenarios about some simple constructs.

Complexity ModellingThis is perhaps the most challenging piece of modelling, because it examines the between group and within group sensitivities of various dynamic effects of data sets. Here people become desktop detectives and data explorers, trying to find new ways to understand the data and the complex relationships it suggests.

Boolean LogicThis is the type of logic that builds up a picture through ‘Yes and No’. It is very static, and the outputs give a very basic picture, with no colour to the outputs.

Fuzzy LogicThis is where ‘shades of grey’, come into effect, and there is quite often no straight ‘yes or no’ answer, rather a series of variables contribute to the answer.

The Knowledge Pirates Guide to Institutional ResearchPage No 28 Modelling Contents Page

Reports are often the only product that people see, and people rarely consider all the work and competences that are used in making a report. Reports are the end of the Institutional Researchers main work role. Reports produced can sometimes make or break an organisation, and one can never be so sure if it will sit on a shelf or be used everyday for weeks on end.

Reports should be short, to the point, relevant, have dates and ownership details, a good clear title and ending. Other helpful inclusions are a title page, contents page, introduction, index and a glossary. If you have data that needs to be shown, that affects the smoothness of the report, place them in an appendix. You may also find that the use of Headers and Footnotes, may aid in the understanding of more difficult constructs.

It is also helpful to remember who your audience will be, to use the appropriate style, level of language, frequency of jargon, and to have a good balance of graphs and images in the report. Sometimes a simple and short report will be more effective than a complex and long report.

It is also good to remember that there are different ways to deliver reports. Such mediums may be Verbal, Physical or Electronic. Each medium has its own special nature, and you can not always just pick up a report that was designed for one medium and automatically have it fit another medium.

Operational ReportsThis type of report is usually an internal report which has the logistical information to run an organisation. It is mainly focused on a specific section or function, and is usually short in length.

Analytical ReportsThis type of report contains some form of statistical analysis, and compares and contrasts different items of data. It also offers some form of commentary. It usually has a range of tables and apendices that contain raw data. The report also takes a scientific and structural format in both analysis and format.

The Knowledge Pirates Guide to Institutional ResearchPage No 29 Reporting Contents Page

Summary ReportsThis type of report may contain some basic data processing, but its main function is simply to collate and reproduce like variables and values into small summaries. It offers slight commentary and descriptions, but delivers no in-depth analytical statements.

Working Reports / InterimThese reports are more like reports ‘in progress’, or ‘internal section’ reports. They don’t have the final ‘polished’ touch. They mainly answer a really simple question, and are usually used for very short-term projects.

Annual ReportsThese reports are usually produced every 12 months, and have the appearance of a company report. They usually contain several years’ worth of data, and include wide ranging information from Products & Services, Financial & Budget and Marketing.

Fact SheetsThese are very basic reports, usually just a single page and never more than 2 pages. Fact sheets include basic descriptions, but mostly contain facts and figures, with very little commentary or graphics.

Web PagesPeople these days often put up web pages as a report. They can be very interactive, dynamic, colourful and have movement to catch the reader’s attention. But remember that they are also a different size and medium. They need to be updated often; the audience is less controlled and a simpler format may be needed. They also need to have lots of space on them; it is a lot harder to read amounts of text off a computer screen. You will also need to remember that the screen size is a lot different to a physical page size to print off a screen report onto A4 paper - it looks tardy and is a waste of space. It just makes a bad impression. When using a screen at A4 paper size, there is a lot of scrolling which makes work for the reader. This may turn them off gaining valuable insights from information. Where possible have a web report on the web, with a downloadable copy that is designed for printing.



Graphs / GraphicsWhen you go and open up a report and start to read, it is always nice to see a graph or icon. The saying that ‘a picture paints a 1000 words’ has a ring of truth to it. However, not every one gets a benefit from a graphic, nor does every one get the same message. Sometimes a graphic can cause people to stop reading, or even mislead or complicate the message you are trying to get out. There are a few tips and tricks to keep in mind when using graphs and graphics.

Location on the PageWhen you read your eyes follow across the page from left to right, and your eyes also move down the page. This creates a line called Readers Gravity. It’s the point where your eye is naturally drawn to on the page. Research has shown that anything located on this line that is big or bulky, acts as an obstruction to the natural flow of eye movement. It will cause a reader to loose momentum. Sixty per cent of readers loose enough interest at this point to stop reading. It is a good idea to not locate objects on this line, but to place them in other areas of the page.

ColoursColours are a great way to brighten up a page, and to attract people’s attention to a specific point. They are also great for diversifying and highlighting different aspects and attributes of variances of variables and data. But too many colours confuse, and may cause information overload. There is always an extra complication for people who may be colour-blind. There is no hard and fast guide of how many colours to use, but common sense should give you an indication.

Titles & LabelsWhen you look at a graph or even a table, you need to check and see if it is clearly titled and labelled. A title needs to be distinct and succinct in context and words. All areas need to be labelled clearly, although the audience needs to be considered for level and accuracy, it is a good idea to consider aiming for the lowest level of pre-knowledge. Thus people unfamiliar with the data information knowledge will be able to gain an understanding from the report.

Number of VariablesIt is very easy to overcrowd a graph by trying to cram too much information on it. This creates information overload and thus confusion. Graphs are used to make things simple yet people try and make them so complex. It’s a good idea to try and keep

The Knowledge Pirates Guide to Institutional ResearchPage No 32 What is Institutional Research ? Contents Page

graphs as simplistic as you can.

Legend & ScalesIt is important to always have the legend clear and understandable. It should always give an indication to what the sections of the graphs / data represent. The use of Scales is very important to the graph, and may need to be adjusted when a wide range of values is represented.

Get a 2nd OpinionWhen you have finished the graph, go and try it out on a person. See what they can tell you about the graph, without any prompting.

Professional Standards are important to the Institutional Researcher because it helps to shape the industry, and gives confidence to the clients in the product of researchers work. It also helps to develop a working respect of the skills and competences of the IR professional with regards to their behaviour, interactions / communications and diagnostic and reporting abilities.

When dealing with sensitive information and with the growing realisation that knowledge can be power, information / knowledge is fast becoming the world currency. People trade in information all their lives. Because people can get so emotive on the issues of privacy and legitimacy of information, ethics has become an important issue in the workplace. As Institutional Researchers are often information gatekeepers, and therefore responsible for the distribution of knowledge, there are some considerations and good sense practices to be examined. Another consideration is that most Universities and Government departments have ethics committees, which deal with privacy and ethical issues in surveys.

The following list is a combination of items drawn from different professional organisations, such as Institutional Research Committees and also Library & Information Science professional bodies.

Level of ServiceIt is best to aim for the highest level of service. It’s a good idea to treat everyone as a ‘best customer’, and to always maintain a high level of quality, focus and production.

Access to Information This incorporates, with due regard to the sensitivity and privacy of information, that an open system of public style information should be maintained. It also maintains that, wherever public information is produced, all specialised terms and jargon are fully explained.

Censorship / Propaganda / MarketingWhere information is freely available as information, its design, content, appearance and structure should be neutral in its presentation, and clearly portrayed as purely information. Where information is used for propaganda or marketing purposes, the information should not be distorted, used out of context, or give the wrong or misleading impression.

The Knowledge Pirates Guide to Institutional ResearchPage No 33 Professional Standards & Ethics Contents Page

Training and Up SkillingIt is another good idea to take the time out to complete some job / career research and become a generalist in the area of IR, even if some of these functions aren’t in your current duties. With the world of Knowledge Management, Intelligence Work, Institutional Research and Management Information changing so fast, multi-skilling may turn out to be very handy. So maintaining professional knowledge and different methodologies for doing things, could prove to be very profitable.

AAIR has a very good webpage on ethics on their website http://www.aair.org.au/Pages/AAIREthics.htm

The Knowledge Pirates Guide to Institutional ResearchPage No 34 Professional Standards & Ethics Contents Page

Where possible, it is a good idea to also try and remove personal feelings from the information being provided, and to produce it in a straight, factual and non-emotional way. It is not easy to always maintain a balance between feelings and some information or analysis being considered or produced. As professionals, there may be times when there is a contrast, and at these times a quiet reflection and some distance out of the heat of the moment may shed light on the issue.

PrivacyThat all care, sensitivity and responsibility are undertaken when dealing with information that is considered private. It is also a wise move to regularly visit the Australian Privacy Commission’s website, as well as your own institution’s policy and procedures manual. Another consideration is the privacy issue for the people requesting the information when such information may be sensitive to the institution’s strategic plans.

Copyright and Intellectual PropertyOften in the production of reports a document, or a series of tables or graphs may be created. On the other hand, a researcher may reproduce a document, table or graph. It is a good practice to acknowledge where the item has come from, or place your ownership details on what you have produced. This allows for credit to be given to the rightful persons. It also allows for a better follow-up of the details, for either furthering information or for the correction of details.

• Try and make sure that coloured graphs can be photocopied well.• Make sure the left hand scales are labelled with the type of measurement• That the bottom positions are always on the same scales – eg bar graphs.• Always label the dimensions / variables.• When comparing two graphs, make sure they are to the same scale, and the values are in the same order on both

graphs.• Use sensitivity in naming labels• You don’t always need a zero value• Area charts don’t always show a value like a bar chart• 2D is often less confusing than 3D. The 3D effect often confuses.• Don’t have the sections of the graphs the same colour as the background.• Always add in a title.• When using graphs and tables together, make sure they are simply linked, the more complex or busy the page the more

confusion it creates.• When using a single object to describe a value, the size doesn’t matter. When using separate images to describe different

values, don’t make all the images the same size. Make them proportional to their values.• Try not to use two line legends for describing the variable names.• When using percentages, remember to use the percentage sign.• Try not to use abbreviations in titles.• Make sure the graph isn’t too small on the page, that it is unreadable.• When making Pie charts, try to include values or percentages.• Try not to have too many elements in pie charts.• Try and make sure that the graph actually adds to the value of the text it is accompanying.• Try and make the topic flow off the tongue, and not be over long or too complex.• Use table / graph header and footnotes to add in extra details.• When using charts and tables together, always put the graph on top.• Use readable fonts and don’t try to use more than 3 or 4 fonts.

The Knowledge Pirates Guide to Institutional ResearchPage No 35 Hints for Graphs and Tables Contents Page

The Knowledge Pirates Guide to Institutional ResearchPage No 36 Hints for Graphs and Tables Contents Page

• When choosing fonts, try and make sure that they contrast.• Use dividing lines to make sections stand up, also shading can help to highlight sections.• If you use a watermark, make sure it is very soft in the background.• The more variety the more distractions from the main message.• When using tables and graphs displaying the same data, group them as close as you can.

Documents

The Knowledge Pirates Guide to Institutional Researchaair.org.au/app/webroot/media/pdf/JIR/PirateGuide.pdf · The Knowledge Pirates Guide to Institutional Research by A G Nolan OAM