Introduction to Quantitative Research Methods

Quantitative Research Methods

• Put simply – quantitative research methodsbasically means using numbers to study things

• Numbers can be used to study patterns and phenomena that cannot be directly observed

• In the Social Sciences, statistics is the backbone of quantitative research methods

• There are many things to consider when conducting quantitative research

What is a Research Question?

• Research attempts to explain phenomena, usually driven by a problem (not necessarily tangible)

• Developing a strong research question is essential. It will guide you all the way through your research and writing process.

• Research is focused on relevant, useful, and important questions. Without a question, research has no focus, drive, or purpose

• Usually driven by a research problem• Your research question should concisely summarise the

focus of your research

Data• Data can be anything, from angles of observed stars in the sky to a

conversation between two people• Some things are more easily enumerated than others• Most quantitative social science research uses pre-existing scales (i.e.

a series a questions or tasks) or observations which are then quantified

• There is a whole branch of statistics used in the development of scales

• Both scales and observations raise many issues that require consideration which are detailed later

• Different scales/observations can produce different numerical outputs. It is important to understand which format your data will take

Levels of Measurement

There are several types of number scales which determine different statistical analyses:• Nominal• Categorical• Ordinal• Interval/Ratio

Interval/ratio format provides more information, generally better to use interval/ratio variables as there are more statistical tests available to explore results.

Methods of Data Collection

• Methods of data collection are as varied as types of data, and are influenced by the technology and resources available to the researchers

• Most common in education would be assessment data (e.g. aptitude tests), surveys and observations

• Surveys can now be delivered online (to the right audience or sample), which can increase efficiency of data entry

Sample• Who is your sample? What characteristics do they have?• How representative are they of the broader population of interest?

E.g. SES, ESL, refugee, etc.• How many will there be, generally quan is not interested in less than

15-20 people• If using group statistics, who will you be comparing them to? Are

they comparable/homogenous? • How accessible? Given voluntary consent is a must, assume that

response rates will not be much higher than 30%, but can be higher in educational settings

• If students are not the sample, what recruitment methods will you use to recruit, say, teachers or parents

• Sampling bias : can’t be removed, but can be explored

Hypothesis Testing

• Hypothesis testing is the tradition from which most statistical tests derive

• Involves specific predictions, followed by experiments in order to test outcomes and generate theories on the basis of outcomes

• Traditionally the domain of hard sciences, e.g. physics, chemistry, biology, but has spilled across into many social sciences

• Independent and Dependent Variables, suitable to a laboratory, not as suitable to society

• Statistical tests determine whether hypothesis is accepted or rejected

Descriptive statistics

Descriptive statistics are the first step of statistical analysis. Descriptive statistics describe but do not infer relationships or differences• Central tendency:

– Mean, median, mode– Percentages, rank ordering– Plot data on graphs

• Dispersion:– Standard deviation

• Report descriptives before moving to inferential statistics, not least of which for assumption-checking (more later)

Inferential statistics• Inferential statistics go beyond describing data, and attempt to look

for patterns in data that suggest differences, relationships or predictions

• Interested in testing whether results from a sample can be generalised to a broader population, as it is almost impossible to sample an entire population

• Many different statistical tests generated by statisticians for this purpose

• Generally, inferential tests will look for either group differences or relationships/predictions; except for advanced techniques which can do a degree of both

• Attempt to measure the likelihood beyond chance of finding a difference or relationship/prediction

p or significance or‘sig’

• Almost all inferential statistical tests produce a p figure. Also called significance or ‘sig’

• This figure uses probability theory to test whether there is a significant difference or relationship present vs arriving at that finding by chance

• Up to the researcher to set the significance level they accept, usually p<.05, .05 equates to a 5% chance that the result is due to chance

• Takes into account standard deviation and sample size, larger samples more likely to reach significance, even when differences may be very small

Assumption checking• All statistical tests require assumptions to be checked before they can

be used. • These assumptions are rules or conditions under which this test can be

used – IMPERATIVE to check that your data meets the assumptions required, or results can be meaningless

• Different tests have different assumption checks that must be completed – there is a subfield of study devoted to it

• Most are done by SPSS (Statistical Package for the Social Sciences) either before the test is run or during

• You just have to check before interpreting the results• If assumptions are violated, sometimes you must use a different test• Find a list of instructions for the particular test you choose, and consult

VU’s statistical advice service

Are you normal?Parametric or non-parametric

• Most statistical tests that use an interval/ratio scale rely on datasets being ‘normal’, or parametric. Also called Gaussian

• Complex mathematical underpinning principles• Bell-curve: the perfect normal distribution

Differences across groups• Very commonly used across many research fields, including medical

treatments, psychological experiments, educational interventions, etc.

• Chi-squared for non-parametric:– group differences – goodness of fit

• T-tests: – single sample– paired samples– Independent samples

• Analysis of Variance – more than 2 groups• “The differences between the two groups were not found be

significant, t = 1.41, df = 18, p=0.175”

Some interesting facts

• The t-test was invented by William Sealey Gosset, who at the time of invention was a worker at the Guinness Brewery. As Guiness wouldn’t allow him to publish under his own name and he used the pseudonym ‘student’, therefore it is sometimes called students t-test

• He invented the test to cheaply monitor the quality of ‘stout’

• Shown to be more robust to data that doesn’t conform perfectly to the normal distribution than other similar tests for difference

Relationships and Predictions• The famous correlation, Pearson’s product-moment for parametric and

Spearman’s Rho for non-parametric• Looks at the relationship between two variables at a single time• Works on linear relationships, plot data first, some relationships can be

curvilinear (e.g. motivation and challenge) and not suitable• Requires normality checking as well• Not causation! This requires underlying mechanism and triangulation

of findings• Positive and negative relationships, r is the correlation coefficient, runs

between 0-1, r squared gives percentage, also has a p• Example Statement: r=.15, N=20, p=0.53• R2, .15 x .15 = .0225, so 2% of variance shared – SPSS will usually

output the r2 figure

Scatterplot examplePositive relationship

GLM, SEM• General linear modeling (GLM), multiple regression, structural

equation modeling (SEM), more advanced• Regression uses correlation like mathematics to test how well

several variables predict an outcome variable• Different methods: enter, stepwise, hierarchical• SEM can also test a pre-existing model of how several variables

should interact• In the social sciences, the advantage of statistical modeling is to

account for variables that cannot be manipulated by the researcher, and if time is a factor to demonstrate directionality and chronology

• Certain types of variance can be removed/controlled in the model

Growth scores

• Especially important in quantitative education research is growth scores, usually calculated by subtracting the earlier assessment (usually lower) score from the later assessment (usually higher) score of a test

• Takes out the beginning and end-point, focuses on growth, removes some of the distortions presented by students who are starting at a higher point, e.g. high SES

• Some limitations, growth may not be linear, growth curve may not be available

• Try to choose scales that ideally have detailed growth points over time, e.g. VALS, .25 over 6 months

Choosing scales• Psychometrics: scales measure latent constructs• Validity of scales – are they measuring what we think they’re

measuring? • Types of validity:

– Face and content validity– Factor structure– Chronbach’s alpha– Convergent and discriminant validity– Predictive validity

• Reliability – stability over time– Test-retest– Testing for differences under different conditions– Split-half

Other scale considerations• What is the output of the scale? Subscales?• What are the conditions under which the scale is delivered? Can they

realistically be maintained? Are these consistent with the validity?• How long does the scale take to complete? Are there shorter versions

that are equally valid?• Copyright and accessibility? • Translations? • Report the past validity and reliability in your paper to make your

statistical findings compelling. Also calculate validity figures yourself to test for differences

• Your method section should provide sufficient detail for the study to be re-run in future

Observations• Sometimes observations are quantified and used as data• What will you be looking for? Will you devise a scale or use an existing

one?• Low inference will be a count of behaviours, high inference require more

complex judgment• Commonly, multiple observers are used in order to overcome bias and

because of the time required in observational studies to collect sufficient data

• Observations may be intrusive and may influence the observed to improve performance or alter behaviour from normal

• Inter-rater reliability is a measure of observer consistency in how similarly they rate observations. It is the measure of difference between how the same case is rated across observers

• Training is required in order to increase consistency

Meta-Analyses

• Meta-analyses are an amalgamation of many studies• Many samples that are similar, statistical findings are

all added together into one big study• What is the purpose of this?• When amalgamated, many smaller samples become

one bigger sample, and can be used to test how applicable the findings are to a broader population

• Also can be used to test for how consistent the findings are across many conditions

Extraneous Variables

• Attempt to control/account for as many extraneous variables as possible

• Past research will help to determine what these may be, e.g. SES in education impacts on performance

• How to control for the impact of these variables – inclusion/exclusion criteria, include a brief measure then remove outliers post data-collection

• More advanced statistics are able to factor out variance

Choosing your method

• Be realistic in how much time you have• Independent groups (comparing growth scores across

two like groups), or single sample (comparing a group’s growth score to a norm over that time period) t-tests are likely feasible and are not statistically overwhelming

• To calculate growth scores you must have data from two time points for the same people

• Ensure to discuss your statistical design with your supervisor or the VU statistical consultancy service before collecting data

Tenets of Positivism

The positive bias towards quan methods in research has led to certain assumptions:• There is an objective and consistent reality accessible via research• Physically observable, either directly or via tools that serve as our

eyes• Quantitative data is superior to qualitative, more objective and

consistent• Researcher is neutral and ‘outside’ of the research process• Experimental method, IVs and DVs, hypothesis testing• Theories develop to match the evidence that emerges, theory

Darwinism

The Bad

• Highly suitable to studying some aspects of the physical world, arguably a hangover from Newtonian physics (i.e. Einstein and quantum contradicts principles)

• Suitable to a laboratory setting, where almost all variables that may influence an outcome can be manipulated and controlled

• Problematic when applied to studying human thought and behaviour. Multiple variables all impacting upon one-another simultaneously, unable to be untangled or controlled, researcher as part of the process, reductivist

The Good

• Throw out empiricism with the bathwater?• Numbers are still a highly useful way of charting

phenomena that we are unable to observe directly, e.g. attitudes, aptitudes, personality, etc.

• Aimed at mapping latent constructs• Efficiently deployed, can test generalisability of findings• Prediction rather than proof• Do not have to accept all tenets of positivism in quan

research• Mixed methods, can be complementary

Mixed methods

• Combining qualitative and quantitative methodologies is quite common now, and can be complementary

• They can combine in several ways, the most common being a quantitative study to give an overall picture, then drilling down using qualitative methods to investigate a stand-out group;

• Another common methods begins with a qualitative study into a particular phenomenon, followed by a larger quantitative study that attempts to determine whether findings are generalisable

To End

• Quantitative research is broad• Often looks for differences or relationships• Not without limitations• Seeks to efficiently investigate larger

populations

Group Exercise

• Formulate the design of a quantitative research project on one of the following topics:– Class size and student performance in literacy– Matching student and teacher gender and student performance

in literacy• Discuss the following:

– What is your research question?– What are the variables, and how will they be measured?– Will you use scales or observations? How will you ensure they

are valid?– What will be the methodology?– What are the strengths and weaknesses of the study?

References

• Mujis, D. (2004). Doing Quantitative Research in Education with SPSS. – this book is available in full online via VU library

• Brace, N. Kemp, R. & Snelgar, R. (2009). SPSS for Psychologists 4th edition. Palgrave Macmillan: Basingstoke. – Useful simple steps to conducting statistical analyses in

SPSS, available in hard copy via VU library

VU Statistical Consultancy Service

To seek advice on your statistical design, contact the VU Coordinator of Statistics Support A/Prof. Neil Diamond via either:• Email: Neil.Diamond@vu.edu.au; or• Phone: 9919 9302

Example of the steps required to conduct an independent samples t-test

• Ensure your samples are independent • Assumes dataset fits a normal distribution, therefore checks must be made to

ensure it does• These checks are simple steps that you undertake• Skewness (where the mean is located on the distribution) and kurtosis (the

steepness or flatness of the distribution) should be less than 2x their standard error (they can have negative values which is fine) – this is a simple check of figures in the output

• Levene’s test of homogeneity (should be above p=.05 if variances are comparable, i.e. not different – if a significant difference exists SPSS outputs different figures)

• One tailed/directional or two tailed/non-directional hypotheses, if hypothesis is one tailed, SPSS automatically uses two-tailed and so can divide the p value by 2 (giving an even lower p value)

• If the variables are not normally distributed within these constraints, which is unlikely, you may have to use another statistical test – consult a statistician

Example SPSS output

Note Skewness and Kurtosis are less than 2x their SD (or standard error) – this is enough to confirm the normality of the distribution so that parametric tests are acceptable. Difference statistical tests have other assumptions to be checked, however. It is best to find a guide for your specific test.

Examples SPSS output

Levene’s test for equality of variances confirms the distributions are similar enough for comparison, the test indicates they are NOT significantly different (i.e. p value is not less than .05).

Example Summary Statement

• The results of this analysis may be reported as “The differences between the two groups were not found be significant, t = 1.41, df = 18, p=0.175 [2 tailed]”

• The p figure is above .05, therefore we conclude that any differences are due to chance. Therefore no significant difference exists

• Note if it were 1 tailed, the two tailed p figure that SPSS outputs could divided by two, bringing the figure closer to .05, but still not close enough to meet the cut-off, when divided p=.087

• How meaningful is this finding?