Dr Lisa Wise 20/09/2002 Designing Web Surveys Dr Lisa Wise

Dr Lisa Wise 20/09/2002

Designing Web Surveys Dr Lisa Wise


Overview

• web surveys• overview of survey design • when to use surveys• how to design survey questions• data analysis and reporting

– data be collection, storage and analysis

– ethical issues to do with handling sensitive or personal data


Web-based Surveys

• can do from anywhere with web access – but samples are biased against those with no

internet access (> 50% of Aust households are connected - AC Nielson Mar 2002)

• data can be “validated” before submission• note that validation is in terms of correct data

type rather than meaningfulness• issue of multiple submissions vs anonymity

• quick data collection phase for researcher


Web Survey

web server

databasesurvey form processing scripts

emailSurvey data might go to adatabase or might be storedin a file in the file systemor could be emailed to the researcher

filesystem


Web survey design

• use standard web design principles • especially ensure that

– survey is not too long (or can be saved)

– information travels correctly between screens

– error messages, if they occur, are meaningful

– all fields are validated and can handle appropriate input and reject inappropriate input

– check field length, data type, illegal characters


Making a web survey

• put the questions into an HTML form• get the data formatted for analysis using

the scripting tools of your choice• set up access controls if required• test across a range of browsers with a

range of good and bad input– ensure security / integrity of your data, but also

of the system collecting your data


Data collection and storage

• use cgi-mailer or FormMail to email formatted form variables to researcher– http://www.its.monash.edu.au/web/resources/cgi-mailer.html

– http://www.its.monash.edu.au/web/resources/formmail.html

• use a script (eg Perl, PHP) to write to a file on the filesystem

• use a script to put data into a database– don’t put into a database if you are then going

to take it out into SPSS / excel to analyse it !!


Privacy

• “Privacy laws impose quite specific requirements regarding data storage and security, access and correction rights, ensuring data is accurate before use, used only for the purpose for which it was collected or for which consent has been given, and disclosure only in limited circumstances.”

• “Legal requirements may be more detailed than ethical requirements …”

• http://www.monash.edu.au/resgrant/human-ethics/privacy/index.html


Handling data

• shared responsibility of technical staff and researchers to ensure that privacy and confidentiality requirements are met– “information is only de-identified only if all

identifying information has been irreversibly removed from the record”

– “retention of codes which allow recovery of identifiable information means record is not de-identified”


Cost effectiveness

• even a simple web survey takes a couple of days to implement and test

• cost-effectiveness of solution needs to include real costs– eg email vs database solution might involve 3

days of coding for programmer versus 1 day of data entry (cutting and pasting from email to analysis program) for researcher

– cost saving or cost shifting?


Making a survey ...

• technical “design” requirements covered in previous slides but what about designing the actual survey questions ?

• there is a whole discipline area focused on designing surveys and questionnaires and analysing data collected using these tools

• it is not a trivial exercise and committees / managers are not ideal survey designers


Overview of survey design

• survey design requires that you have clear research questions

• survey questions need to be focused on answering your research questions

• survey design includes generating the questions and planning the data analysis

• planning the data analysis happens before the survey is released !!


When to use surveys

• surveys allow you collect lots of data relatively quickly and cheaply

• lots of poor quality data is never better than smaller amounts of high quality data

• lack of time and money are not excuses for collecting poor quality data - if you can’t afford to collect reasonable quality data, don’t do the research.


Personal data

• most surveys request demographic information about respondents

• usually ask for opinions about something• collecting either type of information has

ethical and privacy implications• survey designers should be familiar with:

– http://www.monash.edu.au/resgrant/human-ethics/

– above page has link to Use of Personal Information


… from Privacy statement on web

“On-line SurveysAll research surveys conducted on-line by

University staff and /or students which involve the collection of personal information, will have received approval from the University's Committee for Human Ethics in Research.

A survey might ask visitors for unique identifiers (such as login information).”


Sensitive information

• “Do the records or information you are collecting or using include any sensitive information (such as political opinion or memberships, religious beliefs or affiliation, philosophical beliefs … )”– note that peoples’ opinions are considered to

be personal information

– be aware of perceived power relationships and potential access to confidential information


Classroom and workplace surveys

• work / class surveys have ethical issues related to perceived power relationships between respondents and researchers

• even if survey does not require specific ethics committee approval, the ethical and privacy principles should be considered

• go through all the ethics forms whether or not they need to be submitted


SCERH ethics forms

• based on National Statement on Ethical Conduct in Research Involving Humans (NH&MRC, 1999)

• apply to “anyone who is gathering information about human beings and organisations through interviewing, surveying, administering questionnaires, observing human behaviour, taking human tissue / fluids …”

• “ … there are no exceptions, exclusions or blanket permissions”


Sampling

• Sampling design and survey design must be tightly coupled

• Who or what are you planning to draw conclusions about ?

• Are they a homogenous group or are there sub-groupings in the population ?– do you need a stratified sample?

– do you want to compare between groups?


Sampling

• Surveys usually use convenience samples• Demographic information is collected to

allow comparisons between target groups • If targeting specific groups is critical to

your research, consider interviewing participants, or delivering and collecting surveys from selected participants

• convenience sample =/= random sample


Sampling

• for statistical techniques, calculate the sample size required for valid conclusions

• consider whether you want lots of general data or whether you are actually interested in very specific focussed data

• a large survey doesn’t give more objective data than eg a focus group unless it follow rigorous methodology


Types of questions

• Open ended– more difficult to answer and to code or to

analyse objectively

• Closed questions– forced choice (one of two mutually exclusive)

– multiple choice (one of several)

– checklist (one or more of several)

– partially closed (alternatives including “Other”)


Questions

• avoid double-barrelled questions• avoid leading questions• avoid motherhood statements• avoid undefined terms• ensure that your questions lead to

responses that interest you and conversely that responses that interest you are elicited by your questions


Order effects

• funnel questions from general to specific• can use general filter questions to

determine whether respondent should be asked detailed questions

• can have some practice questions• counterbalance order of presentation• prevent response sets• can use alternate forms of questions


Content analysis

• talk to expert about content analysis• consider whether you are happier with the

assumptions underlying content analysis or your research team’s ability to interpret and code responses

• if you code responses, take measures of inter- and intra-rater reliability – do raters make similar / consistent judgements


Response scales

• response scales should allow people to communicate what they want– eg there are differences between neutral,

undecided, don’t know, don’t care, n.a.

• anchor points should be bipolar– need to test this on pilot sample

boring interesting

boring fun


Measurement scales

• Ratio scales (true zero - eg age, height)• Equal Interval scales

– Thurstone scales, Likert scales, Guttman scales, semantic differentials

• Ordered scale (eg <18, 18-30, 31-50, >50)• Categories (ITS, Med, Sci, Arts …)• Different types of data are amenable to

different analyses: consult a statistician!

Albrecht et al, Social Psychology, pp 190-198


Rating scales

stronglydisagree

stronglyagree

undecided

Likert scales should have 5 marked values not 7

The following multiple choice format is still an interval scale1. Strongly disagree2. Disagree3. Undecided4. Agree5. Strongly agree

disagree agree


Rating scales• constructing rating scales to have specific

psychometric properties is labour-intensive and requires expertise– multiple forms or minimal number of questions?

– group similar questions or intermix?

– alter format to avoid response bias or keep consistent to avoid response errors?

– normalise data or trust participants responses?

• consult a statistician or social scientist


Measurement

• rating scales should be equal-interval scales to use parametric statistics

• the fact that you have numerical data does not mean it is accurate or reliable– male (1) - female (0) … obvious that 0 and 1

are codes not numbers

– rarely (1) - sometimes (2) - often (3) … things that can be ranked are not necessarily equal interval scales - can you take the mean?


Statistics

• Descriptive– describe a sample distribution in terms of

shape, central tendency and variance

• Inferential– draw inferences about population parameters

based on sample statistics

– hypothesis testing - test whether a statement is true or false based on sample statistics


Reliability and validity

• Reliability– test - retest

– surveys are only a snapshot at particular time

• Validity– face validity

– criterion-based validity

– construct validity

– internal / external validity


Hypothesis testing

• null hypothesis H0 (no effect of experimental manipulation)

• alternative hypothesis H1 (experimental manipulation has an effect on this test statistic)

• type 1 error (accept H1 when H0 true)

• type 2 error (accept H0 when H1 true)


Inferential statistics

• use statistics to make inferences about populations from your samples

• need to be aware of assumptions underlying statistical tests– eg t-tests and anovas assume continuous

underlying variable, normal distribution and homogeneity of variance

– what happens when assumptions are violated?


Statistics

• most survey data is not ratio and may not be interval data (depending on your perspective on this)

• non-parametric statistical tests don’t make assumptions about underlying distribution

• don’t use statistics to show things you can’t see by eye - statistics help decide if what you can see is “significant”


Correlational studies

• surveys usually correlate opinions with demographic information

• correlations show relationships between variables but don’t address causality

• correlation coefficients indicate strength of relationship (between 0 and 1)– r of .8 explains 64% of variance, or the degree

to which knowing X can predict the value of Y


But I’m not doing real research ...

• I just want to run a quick survey• This stuff doesn’t apply to me cos I’m not

doing real research

• . . . so what are you doing?– “anyone who is gathering information about human

beings and organisations through interviewing, surveying, administering questionnaires, observing human behaviour, … no exceptions ...”


What constitutes research?

– If you are going to summarise responses on your survey and you are going to act on them in any way, you are doing research

– for that research to be meaningful, you need be aware of proper research methodology and the limitations of what you are doing

– if you can’t do it properly, don’t do it at all !!

– informed professional opinion can far more valuable than poor quality survey data


When to use web surveys

• small amount of non-confidential data from wide range of people– not good for sensitive or confidential data

– not good where loss of data would be a major problem

– not good for long surveys (usability issues)

• can use web to download a survey which is printed out and submitted

Documents

Dr Lisa Wise 20/09/2002 Designing Web Surveys Dr Lisa Wise