Quantitative Analysis of Survey Data & Related Design Issues...Results and Implications "Expert, lay "Presentation "Analysis 24-Nov-16 Kornbrot Quantitative Methods for Surveys . Project

Quantitative Analysis of Survey Data &

Related Design Issues

Diana E Kornbrot University of Hertfordshire [email protected]

Plan ! Project Overview ! Design and Measurement ! Workshop Participant Surveys & Issues ! Survey Design Issues

" Respondent perspective " Investigator perspective

! Methods ! Results and Implications

" Expert, lay " Presentation " Analysis

24-Nov-16 Kornbrot Quantitative Methods for Surveys

Project Overview ! Plain English Formulation

"  Primary Goal, Secondary Goals

! Measures "  Primary Outcome

e.g. satisfaction (ordinal), weight (metric), success (binary) "  Factors: groups, treatments, metric predictors

! Methods: what was Planned "  N/group: power based on evidence or availability " Measures, analyses, procedures

! Results "  Summary: Graphic, Tables, Analysis: 2 or multi-variable. Models

!  Implications and Dissemination: "  lay, expert

! Need statistics or design advice? Get it Early!!!


Investigation Design ! Ethics ! Qualitative, Quantitative

" Qualitative: categories, narrative " Quantitative: ordered, or metric numbers

! Subjective v Objective " Objective available to all " Subjective

Type 1: available only to participants, maybe quantitative: pain Type 2: classified by investigators, often qualitative: transcript

! Experiment, observation 24-Nov-16 Kornbrot Quantitative Methods for Surveys

Measurement

24-Nov-16

Type Measure Distribution Display Compare ModelNominal, categorical frequency multinomial pie, histogram chi-square cart treeeg: favourite proportion group histogram forest treeOrdinal frequency histogrameg: Likert, strength binary proportion binomial group histogram regression,logistic,categorical regression

cumulative proportion ordinal cumulative proability regression,ordinal,categorical multiple regressionMetric mean normal means t-test regressioneg weight, biological variance, range confidence levels ANOVA regression, multiple

MIXED:unequal variancecounts mean Poisson means MIXED regressioneg number clicks variance, range confidence levels multiple regressionscale instrument mean J_shape as ordinal, binary as ordinal, binary as ordinal, binarye.g. Beck depression variance, range as normal as normal as normal

Kornbrot Quantitative Methods for Surveys

Method: Planned & Actual ! Ethics

" Sufficient power " Anonymity " Distress – use known instruments, if able

! How many subjects? " Scientific, implication " Statistical: power, effect size " Pragmatic: time, availability, custom

! Analyses " Description: summaries, graphs " Hypotheses and multiple testing

Bonferoni, Benjamini-Hochberg

! Trial Registration


Survey Design ! Participant perspective

" Short, clear " Options: include neutral, include none " Open ended as list: e.g. Up to 5 “good”

! Investigator Perspective " Easy to summarize, analyse " Number categories, as needed " Validated instruments if available


Results Presentation ! Description

" Means, proportions, regression " By group[s] " With confidence levels " Graphs and/or Tables " Data Reduction, Model diagram

! Inference " Table " Effect size: Cohen’s d, % variance " Exact significance, p(null) " Power if not significant


Useful SPSS procedures

24-Nov-16

Procedure Sub-procedure Inferential Graph Descriptive cross-tabs chi-square group bar graph

explore box plot q-q plot test normality q-q plot

frequency histogram Compare means one-way ANOVA unequal variance t-test Welch General Linear Model univariate compare factors means

assume equal var interaction interaction means repeated wide compare factors means

assume equal variance interaction interaction means

Generalized Linear Model Generalized Linear Model distribution based

Generalized Estimating Equation

distribution based long format

Regression Linear: normal various Automatic Linear Modeling multivariate good selection various

Logit: binary probabilities

Probit :v binary probabilities

Ordinal: ordered Non-parametric All avoid Dimension reduction Factor on metric or transform % variance, exploratory factor scores

Classify Tree Cluster


Example ! National Student Survey: HEI, subject, sex, disability,+


Measures ! 5 Category Likert Response

" Single or cumulative proportions, p 5 category cumulative binary: e.g., 4,5=good V rest; 5=excel V rest;1,2=poor V

rest " Logit or probit of p

! Descriptive many category: HEIs " Crocodile: individual quartiles, CLs

! Descriptive few category: domain, goal " Cumulative probabilities: Interpretable " Logit, probit: Generalized Linear Models


Group Comparison ! Domain, goal, cohort, HEI type

" Goal, HEI type for each domain Domain interacts with everything Cohort effect negligible

! Analyses " Ordinal logit " Binary logit: excellent, good, poor


Compare Measures by Subject Group: Line

24-Nov-16

All subjects Difference from group means as a function of question and measure

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

Q01 Q02 Q03 Q04 Q05 Q06 Q07 Q08 Q09 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22

excellent good poor


Compare Measures by Measure, Subject Group: Colour

24-Nov-16

Subject Group explain interest enthuse s.mulate criteria fair prompt detail clarify advice contact choice .metable no.fy smooth library IT labs confident comms novelty overall

Grand Mean

Good: Grand Mean Good = 80.2% BioAcademic 12.4% 5.2% 8.6% 9.2% -5.8% -5.0% -18.0% -17.3% -21.0% -2.6% 7.0% -5.0% 3.0% 0.2% 0.3% 5.9% 5.8% 3.1% -0.3% 3.2% 0.2% 7.8% 80.2% Bioprofessional 10.5% 5.6% 8.7% 8.0% -5.1% -7.0% -16.3% -15.9% -19.7% -1.2% 5.5% -3.0% -3.2% -8.1% -9.1% 7.1% 8.7% 4.1% 5.4% 9.9% 6.3% 6.1% 80.2% PhysicalAcademic 11.6% 2.9% 8.4% 9.2% -5.2% 0.0% -13.0% -14.0% -14.7% 0.2% 9.1% -2.4% 2.1% 2.4% 2.4% 7.5% 7.2% 3.8% -2.0% 0.3% 2.0% 8.3% 80.2% Physicalprofessional 6.0% -4.5% 1.4% 2.2% -5.7% -3.5% -19.8% -19.2% -19.1% -3.0% 5.7% -5.2% -0.1% -2.2% -5.2% 7.0% 8.5% 1.6% -1.9% 1.2% 1.1% 2.4% 80.2% PhysicalConstruction 6.8% 1.7% 5.8% 3.3% -11.0% -12.0% -16.1% -13.6% -16.8% -3.3% 4.5% -5.1% -1.7% -4.7% -9.5% 5.9% 4.9% -2.7% -0.5% 4.9% 1.6% 3.2% 80.2% SocialAcademic 11.8% 5.8% 8.8% 7.2% -5.0% -3.5% -14.3% -7.5% -17.1% -3.8% 5.9% -7.6% 4.1% 0.2% 1.4% -1.4% 1.0% -5.6% -2.2% 1.4% -1.9% 6.5% 80.2% SocialProfessional 9.1% -3.0% 3.2% 0.9% -4.4% -6.0% -17.0% -17.1% -22.4% -6.2% 4.4% -9.2% 1.2% -0.6% -0.3% 4.7% 5.7% -1.1% 0.5% 4.5% 0.8% 4.8% 80.2% SocialNumeric 9.4% -4.1% 3.6% 4.2% -3.3% -2.3% -10.0% -16.1% -18.8% -2.5% 8.2% -4.4% 3.2% 5.0% 5.9% 7.2% 8.3% 2.4% -0.6% 1.8% 0.1% 7.1% 80.2% SocialCaring 8.3% 2.4% 8.3% 2.5% -4.3% -7.3% -16.7% -9.8% -20.2% -3.5% 2.1% -6.1% -3.6% -9.5% -14.1% 1.9% 6.1% -1.5% 3.6% 6.8% 2.4% 1.8% 80.2% HumsArtsAcademic 13.8% 10.3% 13.2% 11.2% -4.5% 0.5% -9.9% 0.3% -8.8% 0.3% 8.6% -3.3% 5.7% 1.4% 3.3% -1.7% 2.7% -5.4% -0.9% 2.8% -0.7% 9.2% 80.2% HumsArtsProfessopnal 6.4% -0.1% 4.9% -6.3% -7.8% -8.4% -21.3% -8.9% -18.0% -5.4% 3.9% -10.7% -1.7% -10.2% -13.5% 2.8% 3.6% -2.8% -2.6% 4.4% -3.3% -2.3% 80.2% HumsArtsLangiage 11.6% 3.7% 8.7% 4.7% -2.6% -3.9% -13.1% -6.9% -13.4% -4.0% 7.5% -9.4% 2.1% -1.2% -4.0% 3.1% 5.8% 0.7% -2.6% 4.7% -1.8% 3.2% 80.2%

Excellent: Grand Mean Excellent = 35.0% BioAcademic -6.7% -8.1% 7.8% 11.3% -4.8% -8.8% -14.9% -14.3% -14.8% -3.8% 11.9% -0.8% 6.7% 3.9% -0.7% 12.6% 11.7% 7.6% 2.0% 11.7% 3.2% 8.3% 35.0% Bioprofessional -6.3% -6.6% 7.7% 11.2% -4.4% -9.3% -12.7% -12.4% -13.3% -2.3% 9.9% -0.3% -3.1% -5.4% -8.4% 13.9% 14.5% 7.4% 9.1% 22.2% 10.6% 6.4% 35.0% PhysicalAcademic -5.8% -7.4% 9.5% 12.3% -4.5% -3.2% -11.5% -11.3% -10.3% 0.5% 17.1% 3.1% 6.9% 7.7% 3.2% 16.2% 15.3% 9.3% 1.0% 8.1% 4.5% 11.3% 35.0% Physicalprofessional -9.3% -11.5% 0.2% 3.5% -4.5% -5.3% -14.3% -13.1% -11.5% -3.8% 10.6% -1.0% 3.8% 2.4% -3.6% 17.1% 18.7% 7.9% 0.5% 8.2% 3.9% 2.1% 35.0% PhysicalConstruction -10.0% -5.9% 6.8% 4.8% -8.0% -12.1% -12.1% -9.4% -11.7% -5.7% 7.7% -2.7% 1.7% -1.3% -7.4% 12.4% 11.3% 0.9% 1.4% 11.0% 2.6% 0.8% 35.0% SocialAcademic -4.9% -5.1% 10.0% 7.3% -2.6% -6.7% -11.9% -7.2% -11.7% -4.2% 10.8% -2.3% 9.5% 4.3% 0.8% 4.8% 5.5% 0.3% 0.0% 8.1% 1.9% 6.3% 35.0% SocialProfessional -9.1% -13.3% 0.8% 0.1% -1.8% -7.7% -12.8% -12.2% -13.9% -7.3% 7.8% -4.1% 5.4% 3.1% -1.0% 12.4% 12.3% 4.0% 1.6% 11.1% 2.9% 1.9% 35.0% SocialNumeric -6.2% -11.4% 2.1% 3.7% -0.4% -3.1% -8.0% -9.8% -10.4% -3.8% 13.2% 0.1% 8.6% 10.4% 6.3% 16.4% 17.2% 8.3% 0.2% 7.2% 1.5% 5.8% 35.0% SocialCaring -10.1% -11.9% 7.5% -1.0% -3.5% -10.6% -14.3% -9.9% -14.9% -5.6% 5.0% -2.9% -1.8% -5.6% -11.6% 6.6% 7.4% 0.4% 6.5% 14.5% 5.4% 0.3% 35.0% HumsArtsAcademic 1.2% 3.7% 20.8% 16.7% -3.9% -3.2% -10.2% 0.2% -6.5% 1.6% 16.4% 2.8% 13.0% 7.2% 4.1% 3.3% 6.5% 1.2% 2.8% 11.9% 4.5% 12.5% 35.0% HumsArtsProfessopnal -12.7% -12.2% 3.5% -8.1% -6.5% -12.8% -17.8% -9.9% -14.6% -8.3% 6.6% -6.3% 1.3% -5.6% -11.8% 6.5% 6.5% 0.8% -1.0% 10.7% 0.1% -5.1% 35.0% HumsArtsLangiage -5.2% -8.0% 8.8% 4.2% -1.2% -6.3% -12.2% -7.7% -10.7% -3.3% 13.1% -2.6% 5.6% 3.0% -4.4% 7.9% 10.5% 6.0% 1.7% 14.6% 3.9% 1.8% 35.0% HumsArtsPerformance -6.0% 0.7% 15.0% 3.9% -6.6% -8.1% -10.6% -2.3% -7.6% -1.9% 9.4% 0.4% 4.3% -2.9% -9.2% 10.2% 10.7% 0.7% 4.1% 11.4% 3.8% 2.8% 35.0% HumsArtsPerformance 7.4% 4.4% 8.4% 1.0% -7.7% -6.7% -14.7% -4.4% -11.9% -2.4% 3.6% -4.1% 0.0% -7.9% -12.3% 3.6% 5.1% -2.5% -1.3% 2.3% -2.0% 0.4% 80.2%

Poor: Grand Mean NOT Poor = 91.8% BioAcademic 5.1% 3.0% 5.0% 4.4% -4.8% -2.5% -12.3% -12.6% -12.8% -2.1% 2.3% -1.7% -1.0% -1.5% -0.7% 0.3% 0.6% 2.8% 1.6% 3.5% 2.6% 2.3% 91.8% Bioprofessional 4.4% 3.3% 4.9% 3.9% -4.6% -3.8% -11.8% -11.8% -11.8% -1.3% 1.7% 0.1% -4.3% -6.9% -6.2% 1.2% 2.9% 3.1% 3.5% 5.7% 4.5% 1.4% 91.8% PhysicalAcademic 5.0% 2.4% 5.0% 4.6% -3.6% 0.6% -7.9% -8.6% -7.5% 0.1% 3.7% 0.0% -1.2% 0.3% 0.7% 2.0% 2.0% 3.3% 1.5% 2.9% 3.2% 2.8% 91.8% Physicalprofessional 2.5% -0.6% 2.3% 1.8% -3.9% -1.3% -12.3% -12.3% -10.7% -0.9% 2.2% -0.8% -1.9% -2.3% -3.1% 2.7% 3.2% 0.9% 1.4% 3.1% 2.6% -0.4% 91.8% PhysicalConstruction 2.9% 1.9% 4.1% 2.3% -6.8% -5.5% -10.6% -8.7% -9.2% -0.8% 1.8% 0.1% -2.5% -3.6% -5.2% 1.2% 1.1% -0.8% 1.3% 4.0% 3.0% 0.8% 91.8% SocialAcademic 5.0% 3.1% 5.0% 3.6% -3.9% -1.0% -9.2% -5.3% -9.2% -3.1% 1.8% -3.1% -0.4% -1.5% 0.1% -5.1% -1.8% 0.4% 1.2% 2.9% 2.1% 1.6% 91.8% SocialProfessional 4.0% -0.1% 2.9% 1.0% -4.1% -2.7% -11.5% -12.3% -13.6% -3.3% 1.3% -3.5% -2.2% -2.0% -0.7% -0.4% 1.0% 1.3% 2.2% 3.9% 2.8% 1.1% 91.8% SocialNumeric 4.4% 0.3% 3.3% 3.4% -3.1% -0.1% -6.0% -10.8% -10.5% -0.8% 3.6% -0.5% -0.2% 2.0% 3.2% 1.8% 2.7% 2.9% 2.4% 3.2% 3.1% 2.9% 91.8% SocialCaring 3.2% 1.6% 4.6% 1.2% -4.3% -4.3% -12.7% -7.3% -12.1% -2.7% -0.5% -2.1% -4.7% -8.4% -10.2% -2.5% 1.4% 1.9% 2.8% 5.0% 3.4% -1.5% 91.8% HumsArtsAcademic 5.7% 4.9% 6.4% 5.0% -3.9% 0.8% -6.4% -0.8% -4.3% -0.6% 3.3% -1.3% 0.7% -1.0% 1.1% -5.2% -0.6% 1.4% 1.5% 3.3% 2.8% 2.8% 91.8% HumsArtsProfessopnal 1.6% 0.5% 2.8% -3.2% -5.8% -4.2% -15.2% -5.8% -9.3% -3.4% 1.3% -3.6% -4.2% -9.4% -9.6% -0.8% 0.5% -1.1% 0.8% 3.5% 2.0% -3.4% 91.8% HumsArtsLangiage 4.7% 2.2% 4.9% 2.7% -1.8% -0.6% -7.3% -4.6% -5.8% -2.5% 3.3% -3.8% -0.9% -1.8% -2.3% -1.0% 1.4% 3.0% 0.5% 3.8% 2.2% 0.0% 91.8% HumsArtsPerformance 2.6% 2.2% 4.1% 0.2% -5.9% -3.4% -10.5% -3.5% -6.3% -1.9% 0.8% -1.0% -1.6% -6.4% -8.2% -0.5% 1.3% -1.6% 0.2% 3.1% 1.4% -2.0% 91.8%

Difference from group mean by question, subject group, measure Green: better than mean. Blue: worse than mean. Darker further from mean


Psychology

Crocodile Plot

Choose best HEI

from high

achieving p

by measure

24-Nov-16

.1

.2

.3

.4

.5

.6

.7

.8

.9 1.0

Nor

wFE

U

WE

Bris

t K

ings

tUn

EA

ngJo

i H

ertfo

rd

Der

by

Gla

sgC

al

Lond

Met

Fa

rnTe

ach

Live

rJM

o La

ncas

te

Sal

ford

B

ucks

New

N

orth

umb

Leed

sMet

Le

eds

StA

ndre

w

Bou

rnem

o C

oven

try

Wes

tmin

s C

antC

hris

B

risto

l Q

ueen

Bel

G

lam

org

Bla

ckbC

o G

olds

mit

Leic

este

W

estL

ond

Ess

ex

Bru

nel

StM

aryU

C

Sus

sex

Kee

le

Oxf

ord

Tees

side

Lo

ughb

or

Gly

ndŵ

r C

hest

er

Wirr

aMet

R

oeha

mpt

C

ardi

ffM

Not

tingh

S

outh

amp

Cam

brid

g B

irmin

gh

Por

tsm

ou

Roy

alH

ol

DeM

ontfo

Q

Mar

gare

D

unde

e U

EA

Eas

tLon

d B

ango

r S

unde

rla

Gre

enw

ic

Gla

sgow

S

tirlin

g Yo

rk

Wes

tSco

t B

ucki

ngh

Psychology:*ExcellentQ22**

Target UCL LCL 25%ile median 75%ile Hertford

.3

.4

.5

.6

.7

.8

.9

1.0

Nor

wFE

W

orce

ste

EA

ngJo

i Q

Mar

y D

erby

Li

verH

op

Lond

Met

La

ncas

te

Brig

hton

U

WE

Bris

t S

alfo

rd

Live

rpoo

Q

ueen

Bel

B

radf

ord

Gla

mor

g G

rimsb

yF Hul

l W

estm

ins

Buc

ksN

eM

iddl

ese

Sou

thB

an

Rea

ding

R

oeha

mp

Abe

rdee

n H

ertfo

rd

Bru

nel

Leed

s M

anch

Me

Sus

sex

Cen

tLan

c Le

ices

te

Sta

fford

N

ottin

gh

Bris

tol

Loug

hbor

D

urha

m

Sou

tham

Wes

tLon

d K

eele

A

berta

yD

Gol

dsm

it P

orts

mou

E

dgeH

ill

Oxf

ord

Uls

ter

Wirr

aMet

E

xete

r E

dinN

api

War

wic

k W

estS

cot

Cam

brid

g D

eMon

tfo

Birm

ingh

G

louc

est

Tees

side

D

unde

e Le

edsT

ri S

urre

y B

athS

pa

QM

arga

re

Psychology:*Good*Q22**


.60

.65

.70

.75

.80

.85

.90

.95

1.00

Nor

wFE

W

orce

ste

QM

ary

Bla

ckbC

o W

inch

est

Que

enB

el

She

fHal

l Lo

ndM

et

Brig

hton

S

heffi

el

Man

ches

t B

radf

ord

Hul

l E

Ang

Joi

Gla

mor

g B

ourn

emo

Ply

mou

th

Roe

ham

pt

Wes

tmin

s C

entL

anc

City

H

ertfo

rd

New

man

UC

B

edfo

rds

Abe

rtayD

S

outh

Ban

D

urha

m

Not

tingh

A

bery

stw

C

antC

hris

S

unde

rla

Rea

ding

S

usse

x A

nglR

usk

Sw

anse

a S

trath

cl

Kee

le

Exe

ter

Eas

tLon

d Yo

rkS

tJo

Sou

tham

p E

ssex

G

olds

mit

GYa

rmC

o P

orts

mou

N

ewca

stl

DeM

ontfo

C

ardi

ffM

Edi

nNap

i Te

essi

de

Man

chM

et

Sur

rey

StM

aryU

C

Dun

dee

Glo

uces

t U

CL

Bat

h B

irkbe

ck

Cum

bria

W

estS

cot

Psychology:*NOT*Poor*Q22**



Prediction of Overall Satisfaction, Q22: Influence of all other questions/features

! Measures " Excellent, good, poor " Raw probability, logit, probit

! Forest Plots " Non-parametric?? Differs by measure " Display influence colour diagram " % variance accounted for

! Best subset regression " Parametric " Display influence colour diagram " % variance accounted for " Effect size, significance


Forest Influence Diagram: All Qs

24-Nov-16

Subject Sno N_HEIfair fair fair

excellent good poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poor excellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poorexcellentgood poor

Anatomy 3 49 .814 .771 .554 71 84 35 50 48 6 68 24 15 32 31 72 6 3 25 12 13 23 6 14 31 9 4 8 18 6 9 54 49 38 29 11 22 54 24 8 9 4 10 28 20 46 100 100 100 5 2 7 3 2 12 3 3 15 54 13 27 20 8 36 13 10 15

Pharmaco 4 40 .829 .590 .625 27 50 32 53 27 33 49 100 57 68 23 23 26 11 23 14 10 33 5 23 100 4 4 18 8 16 24 83 42 12 100 76 61 95 87 16 7 6 17 42 55 62 50 63 43 5 18 6 6 17 5 7 31 16 95 40 17 46 55 20 40 30 52

Biology 12 84 .719 .662 .640 16 37 19 53 32 57 24 13 10 81 27 23 4 5 5 8 23 11 5 13 10 5 13 5 11 11 4 31 35 38 25 10 11 30 28 42 22 5 10 25 45 43 100 100 100 4 4 8 4 4 3 5 7 6 29 28 10 52 9 8 20 10 9

Zoology 13 29 .775 .660 .473 23 31 10 47 35 55 21 28 15 100 35 54 3 5 12 10 91 9 4 8 5 15 5 7 21 8 45 12 27 23 75 100 23 27 58 51 11 41 17 17 53 83 18 78 99 3 23 100 4 5 5 4 31 13 63 79 21 25 18 21 21 26 24

OthBio 17 24 .629 .466 .233 20 44 100 20 100 39 11 33 62 100 92 94 4 4 18 11 4 21 4 9 8 3 7 11 7 6 6 27 44 73 56 84 77 15 29 37 5 16 17 11 9 26 6 26 30 34 13 5 26 5 28 20 6 5 9 26 15 10 12 14 17 21 28

Psycholo 19 121 .799 .622 .641 52 100 24 33 32 23 49 40 12 71 70 49 11 6 17 18 6 16 20 5 17 14 5 7 8 10 30 100 20 65 85 27 12 66 13 21 7 3 9 11 25 49 83 54 100 6 6 16 4 4 7 7 11 15 30 19 21 33 7 13 27 21 40

Medicine 1 33 .872 .717 .651 100 24 29 42 33 40 24 17 11 8 6 17 37 35 56 12 44 69 4 3 5 4 5 7 14 33 45 71 100 31 88 19 18 100 83 100 28 19 16 21 36 12 63 83 65 2 2 2 3 6 3 3 2 4 12 18 7 4 5 12 2 31 9

Nursing 5 70 .871 .775 .809 45 95 100 100 34 36 21 9 17 31 14 20 13 39 35 26 16 15 3 5 5 5 8 4 9 15 15 29 56 54 16 13 8 53 41 31 3 7 5 5 14 16 83 64 45 3 2 3 2 2 4 3 4 5 28 100 50 10 6 5 17 25 22

Nutritio 7 34 .639 .564 .392 69 23 26 51 47 72 77 52 38 87 43 60 20 17 9 15 9 46 13 13 6 9 15 4 35 12 4 43 6 20 34 80 23 28 24 83 8 10 36 26 55 52 100 100 74 10 7 6 8 3 2 4 5 20 86 50 100 52 11 3 40 22 19

MedTech 10 29 .506 .159 .492 92 86 81 37 67 88 19 31 100 92 37 17 25 31 31 20 73 23 17 41 12 22 9 10 59 11 4 100 73 46 45 19 10 90 62 62 27 5 5 31 25 21 18 67 35 5 34 2 5 4 3 5 3 2 93 100 64 27 19 22 48 25 28

AllieMed 11 66 .694 .758 .600 31 57 59 31 100 51 19 33 59 24 17 30 8 6 15 27 18 41 9 4 10 4 8 17 15 4 8 38 44 74 16 24 100 100 82 68 12 7 15 26 19 45 60 31 64 4 3 6 3 3 5 8 2 6 17 17 36 14 15 11 10 13 20

SportsS 18 84 .756 .639 .516 93 50 24 95 64 83 48 32 51 79 100 100 9 14 55 6 13 17 9 9 43 5 13 18 3 6 25 41 38 23 31 13 19 60 34 42 19 10 8 38 22 33 100 92 65 7 8 29 7 10 8 4 9 7 11 52 17 30 10 10 6 28 18

Agricult 24 21 .540 .575 .349 12 25 16 13 4 41 48 8 9 22 6 12 15 7 8 18 2 31 7 4 12 18 7 8 5 11 17 21 3 25 33 25 44 42 19 22 7 9 79 32 40 53 13 80 100 62 38 10 39 10 14 61 13 26 45 46 83 64 39 50 100 100 45

MolecBio 16 58 .632 .653 .555 60 100 88 10 45 30 13 10 17 27 45 62 9 4 14 12 9 13 4 3 10 9 4 11 4 2 14 38 18 71 100 20 100 50 5 19 5 3 17 39 4 24 32 30 70 8 4 12 5 5 11 6 5 16 23 10 25 25 3 7 26 6 10

Chemistr 25 49 .672 .509 .073 14 83 68 10 95 76 9 35 50 97 79 51 5 9 18 13 16 36 23 14 30 12 10 29 20 9 34 51 71 100 82 31 58 100 100 52 7 30 49 14 8 27 20 43 34 9 16 13 10 11 6 19 14 39 7 28 29 21 14 7 31 42 18

Physics 26 44 .672 .532 .441 100 67 45 20 60 100 17 43 83 94 16 80 10 11 16 53 19 12 20 9 7 13 8 15 19 7 7 45 23 18 15 14 10 35 29 61 10 6 11 6 10 44 14 100 43 6 9 26 3 6 4 7 10 32 35 42 16 10 6 11 29 23 28

ForenArc 27 43 .781 .629 .700 71 28 50 40 57 27 100 94 51 94 75 100 7 6 19 16 25 31 12 15 10 17 4 7 16 4 10 62 53 20 24 19 14 55 23 8 54 7 1 35 16 10 52 100 51 5 4 4 7 5 6 7 15 10 8 4 7 13 4 4 11 8 9

Geology 28 28 .725 .764 .655 77 68 78 42 87 70 22 64 74 24 68 30 3 21 29 5 66 42 9 33 15 14 10 18 5 14 15 100 24 22 12 66 91 52 85 30 10 10 13 4 59 55 43 100 100 3 4 9 5 4 12 6 16 63 29 67 67 39 76 16 35 67 18

PhyGeog 31 67 .638 .642 .588 100 84 41 75 94 27 72 19 14 56 41 25 19 19 61 20 40 26 8 13 12 20 50 100 9 13 90 35 100 96 9 9 13 33 9 6 19 7 13 17 54 26 62 57 47 6 6 4 7 7 9 9 8 8 75 36 20 60 38 26 22 54 51

Mathemat 32 69 .774 .458 .412 39 36 40 28 28 22 30 27 21 21 16 30 2 20 20 10 38 25 4 7 8 8 55 13 11 100 46 49 53 100 22 39 25 53 50 40 4 15 7 11 10 34 100 40 37 2 7 9 2 11 10 3 15 11 16 63 19 7 30 23 16 62 88

Computer 35 110 .848 .664 .578 19 26 46 13 21 54 3 17 24 100 40 49 3 4 6 10 15 12 2 4 7 2 3 5 2 5 5 51 35 26 25 17 9 13 4 14 4 6 5 25 39 12 63 100 100 3 5 9 6 9 10 4 5 9 15 8 9 4 3 10 29 26 44

GenEng 36 23 .629 .425 .125 24 91 99 6 43 62 9 100 31 100 32 90 4 6 31 17 13 71 4 10 18 5 9 23 11 11 25 42 75 100 41 25 35 5 27 10 6 13 35 11 41 71 44 64 66 8 20 35 10 12 8 20 13 40 28 30 18 10 8 15 36 12 48

MechaEng 37 68 .807 .817 .736 15 42 57 19 83 18 30 34 59 88 48 55 13 3 7 7 18 33 2 4 25 6 4 4 4 3 21 100 84 80 24 6 6 37 14 16 10 14 5 22 37 16 43 100 100 7 4 3 8 4 3 29 8 7 57 15 23 28 10 10 79 10 29

AerosEng 38 27 .708 .500 .578 32 50 39 14 100 97 19 38 35 52 42 33 11 16 17 29 40 22 16 7 6 7 5 16 26 16 41 69 12 46 32 16 12 100 19 25 10 6 21 29 20 100 42 42 23 15 16 21 29 3 13 42 10 12 98 23 93 27 18 53 60 38 11

ElectEng 40 72 .753 .649 .616 13 33 48 28 29 20 27 15 37 28 23 58 10 6 7 20 7 13 7 20 5 7 5 5 5 14 8 42 27 37 33 10 7 63 5 15 7 24 8 29 49 11 100 100 100 6 5 13 22 8 18 11 6 12 66 27 16 11 8 19 21 27 79

CivilEng 41 51 .757 .588 .344 13 24 86 15 100 68 59 48 100 50 24 47 3 5 21 9 34 44 4 5 63 3 4 63 4 2 61 18 23 50 8 3 87 26 17 54 8 6 14 78 15 16 100 27 64 4 11 11 9 12 38 10 3 44 27 10 11 10 11 24 15 15 57

ChemiEng 42 24 .749 .603 .407 36 100 12 20 9 33 32 83 15 89 24 18 3 19 33 13 25 43 40 6 20 4 15 18 15 19 32 39 42 100 23 3 30 80 15 47 9 3 4 47 29 6 100 40 94 13 11 13 14 5 3 37 12 12 70 6 15 33 2 23 43 7 72

OtheTech 46 27 .557 .590 .508 48 22 18 18 100 61 44 16 14 100 44 38 11 15 12 8 6 10 10 8 37 6 3 3 7 10 15 11 8 19 5 15 17 24 15 38 19 7 30 16 16 30 96 85 100 15 7 10 5 6 6 38 3 8 26 59 92 11 28 20 23 41 23

Architec 47 50 .709 .539 .528 13 28 55 14 70 100 41 70 79 19 93 51 49 19 12 41 9 26 4 12 25 21 9 12 30 47 72 69 41 81 10 9 41 41 21 64 8 19 31 19 31 38 33 42 48 8 19 6 16 12 8 60 5 4 100 45 18 52 35 9 26 100 44

Building 48 34 .858 .698 .716 15 37 9 13 100 55 100 40 18 66 24 66 6 60 24 18 24 36 9 52 31 7 9 72 5 30 87 24 50 27 31 52 5 42 24 2 47 16 28 15 46 18 86 38 100 21 6 8 4 2 7 4 4 5 79 22 72 18 13 13 29 55 16

Planning 50 23 .570 .485 .218 75 85 95 39 100 38 20 62 37 65 74 100 6 7 7 6 27 21 4 8 8 19 7 27 25 9 16 12 48 20 35 59 28 32 5 5 3 15 15 8 25 23 46 23 55 4 8 5 5 9 55 18 8 12 100 9 35 27 16 36 37 29 29

Politics 53 76 .702 .603 .501 99 89 40 44 87 28 45 16 39 43 100 67 8 9 6 7 18 100 9 8 20 13 9 23 26 17 20 52 61 81 65 39 39 28 91 76 8 6 12 14 24 26 44 44 50 6 5 7 9 15 6 8 10 7 100 46 25 29 61 34 33 84 41

Sociolog 54 97 .589 .590 .535 74 60 25 100 52 73 42 14 14 42 59 23 12 10 12 14 13 6 16 9 11 13 6 6 26 7 11 46 20 20 71 19 25 25 14 24 14 7 9 22 51 59 95 100 100 11 7 12 9 7 8 12 7 9 28 11 11 40 8 11 56 21 8

SociPoli 55 35 .769 .539 .303 66 20 30 100 100 100 38 42 43 65 28 21 6 10 24 11 9 47 3 11 12 9 7 21 5 5 8 25 16 87 24 36 88 31 79 31 7 14 87 26 19 38 27 33 87 4 6 25 2 2 21 4 7 10 15 12 29 9 12 16 14 26 17

Anthropo 56 31 .657 .497 .537 26 53 30 100 100 25 52 64 20 26 89 76 5 8 5 7 60 64 3 9 15 5 4 16 7 7 31 14 19 46 7 11 47 15 10 100 13 12 5 7 41 17 10 13 9 4 12 3 8 7 5 8 4 2 56 12 40 14 11 31 44 14 24

SocGeog 59 50 .604 .419 .383 68 59 20 100 100 50 53 54 100 81 49 80 13 10 21 37 27 17 14 39 25 48 39 60 31 29 49 80 43 56 46 41 18 41 44 42 21 11 32 12 28 25 57 32 36 9 8 7 13 8 9 24 10 14 47 37 98 23 29 23 54 77 78

Law 60 107 .713 .643 .546 48 64 42 50 44 100 36 65 11 41 17 39 5 8 9 12 16 21 7 5 9 12 10 30 5 13 28 46 39 27 62 14 8 100 36 46 19 10 8 18 8 13 86 52 82 8 8 13 6 5 8 6 5 5 48 27 18 10 15 11 53 100 31

Business 61 109 .753 .715 .673 63 28 15 100 80 64 42 40 43 94 100 100 16 9 10 10 16 12 11 20 23 15 14 19 9 7 15 14 11 25 18 10 18 61 16 14 24 7 6 39 25 16 100 89 37 5 3 5 7 5 4 13 4 5 43 29 10 13 16 9 58 28 26

Marketin 62 69 .658 .549 .489 100 92 44 23 50 100 34 41 67 24 86 80 10 32 12 18 18 17 8 7 11 9 14 20 5 9 8 73 37 21 29 49 21 29 30 12 14 17 30 12 23 35 40 52 61 6 7 9 5 8 10 5 8 10 22 39 32 49 100 26 47 54 80

Manageme 63 84 .751 .710 .674 90 20 45 73 24 88 41 16 56 83 100 96 17 7 8 10 11 11 7 4 24 5 6 9 14 5 13 67 10 23 19 9 5 100 5 12 13 5 19 8 12 29 19 42 41 4 3 5 6 4 4 4 4 4 32 45 14 14 14 8 83 60 100

HumanRel 64 25 .370 .423 .280 88 37 29 16 25 5 40 100 7 31 13 55 10 38 27 13 86 19 8 25 7 50 11 19 39 9 10 100 36 47 74 68 40 31 77 51 18 21 13 18 20 7 93 23 17 11 20 3 6 13 5 23 13 11 43 14 9 16 9 100 20 4 38

Tourism 67 59 .695 .676 .605 47 95 43 23 87 69 34 100 20 38 60 100 5 13 13 4 6 7 9 8 6 11 8 14 9 26 23 18 20 6 6 11 5 6 10 15 6 5 7 54 47 20 100 69 18 5 4 4 5 6 3 5 9 7 13 23 20 10 22 24 17 50 26

Economic 52 72 .801 .596 .501 80 100 46 87 49 100 46 71 92 100 35 50 7 8 12 11 31 73 5 9 23 5 14 36 8 11 26 95 45 55 60 18 44 45 31 81 24 14 10 20 16 40 71 17 29 8 6 10 5 4 9 19 6 16 10 60 74 6 17 35 34 37 81

Finance 65 65 .765 .585 .347 68 83 49 86 90 31 73 25 9 59 94 100 41 9 12 21 30 18 32 14 16 9 10 10 6 29 18 48 22 33 48 12 28 28 100 43 8 16 12 18 11 7 100 88 39 8 21 11 21 36 12 24 23 8 57 45 12 21 8 7 39 17 52

Accounti 66 97 .767 .635 .527 100 100 100 10 30 77 17 78 55 55 88 50 11 6 13 18 9 8 17 9 9 10 5 5 24 16 9 28 44 99 60 12 15 53 32 42 7 11 4 25 28 9 100 21 20 8 15 7 9 16 11 9 7 5 13 7 63 11 5 7 16 7 16

SociWork 58 91 .805 .813 .728 65 52 23 49 40 32 71 100 37 20 39 33 4 5 6 6 4 5 11 8 10 23 16 25 4 9 13 100 14 27 31 47 37 65 99 100 4 5 8 30 16 26 64 63 90 3 6 3 5 5 4 4 4 6 23 12 13 14 27 11 17 21 9

AcadEduc 105 70 .812 .566 .615 17 55 33 9 62 100 9 32 15 8 46 14 5 7 7 32 13 17 5 6 5 3 11 28 8 11 43 100 100 69 27 19 12 74 26 40 15 13 11 33 40 25 90 75 27 2 8 8 3 17 7 3 10 11 10 59 30 6 11 6 8 30 11

InTchTra 108 57 .862 .795 .596 53 90 37 36 32 24 62 48 7 45 52 100 11 6 9 12 19 12 11 12 12 22 5 9 16 10 7 37 100 65 12 22 20 100 50 18 7 13 8 18 24 43 47 35 31 8 4 6 7 3 3 8 3 2 21 81 6 6 8 6 64 20 17

English 75 108 .737 .638 .617 28 25 100 80 31 80 67 7 21 55 20 33 9 17 8 8 20 20 6 3 7 9 7 13 6 9 10 33 25 48 33 9 19 28 39 26 15 11 11 25 34 37 32 100 51 9 6 9 10 6 7 8 5 13 100 45 94 76 33 16 51 27 31

Classics 78 24 .639 .167 .426 12 10 6 100 63 48 20 9 12 50 89 100 4 17 21 5 32 37 4 39 8 9 39 31 67 100 38 22 19 19 19 11 97 30 57 45 6 7 19 15 17 37 28 20 57 3 33 25 14 15 12 10 24 12 42 25 38 40 55 10 27 12 8

History 90 99 .786 .614 .484 77 100 15 69 46 37 43 26 18 31 18 100 9 10 58 24 33 42 15 6 15 29 6 11 52 32 73 56 50 33 22 11 16 64 10 47 13 11 26 30 9 86 100 73 58 5 7 14 5 13 8 5 7 24 83 11 14 26 9 26 38 17 91

Archaeol 91 25 .659 .603 .564 73 43 11 100 20 14 10 41 4 13 100 32 2 10 5 3 20 7 17 16 24 9 10 11 17 21 50 18 53 28 10 47 54 26 51 100 41 5 4 11 11 2 11 15 11 5 3 7 9 5 8 10 5 5 8 14 3 3 10 5 31 17 18

Philosop 93 53 .633 .569 .456 42 38 100 100 85 68 9 43 38 25 92 24 9 11 6 24 10 28 4 14 12 7 7 7 5 20 12 36 44 31 28 21 29 31 31 18 22 6 12 11 6 6 55 9 73 6 7 13 11 6 10 19 6 12 52 85 37 36 100 40 37 14 17

Theology 94 36 .616 .466 .507 55 27 7 51 63 23 19 100 46 35 81 88 8 7 35 16 15 17 8 6 22 4 7 32 25 21 24 56 17 100 100 95 62 73 18 26 11 6 7 9 20 9 72 23 62 11 33 28 10 14 6 12 8 10 26 22 27 42 21 26 27 12 30

Publicit 71 21 .780 .719 .620 7 87 29 7 26 38 10 17 9 2 8 3 32 12 17 100 4 9 10 6 11 8 8 19 27 3 22 24 67 53 43 3 4 40 75 24 7 23 5 7 78 100 81 100 78 2 3 2 1 3 1 5 8 9 20 10 8 10 16 7 10 25 4

Journali 73 56 .650 .815 .822 57 45 17 51 43 47 32 25 18 94 100 100 12 5 5 9 7 25 5 2 3 9 2 2 11 7 8 33 30 50 63 6 9 50 12 13 12 2 8 9 6 19 100 24 15 8 2 4 15 2 5 18 3 4 38 17 21 19 6 19 34 9 8

French 79 50 .832 .653 .398 79 31 70 30 31 28 49 8 39 69 37 95 3 3 24 7 11 22 7 3 21 3 11 34 4 10 67 100 24 76 30 8 100 24 5 21 14 8 30 16 17 38 57 100 88 7 2 10 4 3 22 7 2 8 76 26 46 32 6 10 64 27 26

GermSca 80 29 .675 .506 .458 15 20 52 71 31 59 39 8 84 45 37 42 3 4 15 2 4 22 6 5 21 43 12 12 100 16 26 17 35 100 14 7 4 45 11 10 14 5 38 4 5 74 10 10 44 18 4 24 9 4 18 15 10 12 41 27 11 62 100 40 16 44 66

Iberian 82 43 .728 .748 .631 23 38 49 51 100 76 21 41 92 39 68 100 9 7 12 17 13 48 6 13 6 7 4 35 11 15 65 12 27 25 7 28 12 14 3 15 4 3 8 9 56 9 80 31 11 3 6 9 8 14 21 15 5 14 93 23 22 38 22 8 100 39 12

OthEuLan 83 29 .750 .611 .701 36 55 95 20 41 65 39 41 25 32 5 35 6 6 10 23 12 11 3 4 34 3 9 15 5 19 55 23 30 100 8 23 25 38 26 66 6 7 4 3 23 10 36 100 68 3 8 11 4 8 16 5 4 10 37 44 20 100 28 4 56 74 21

Linguist 84 28 .710 .379 .547 47 100 29 28 28 100 53 8 28 47 61 38 10 27 32 76 29 65 25 11 21 5 18 15 6 10 11 59 51 22 11 77 12 83 15 9 12 8 5 8 11 9 26 28 32 80 17 5 11 12 9 100 15 6 44 11 14 28 10 5 28 51 21

Media 69 95 .738 .760 .680 51 100 97 68 54 99 35 25 25 100 85 100 6 4 7 7 12 17 5 4 5 5 7 21 12 7 16 61 16 16 11 15 9 31 10 25 17 4 7 7 7 10 27 43 58 4 3 9 3 2 7 7 3 6 12 9 17 10 3 10 30 15 47

FineArt 95 75 .752 .689 .648 25 56 26 24 91 30 34 25 29 39 100 100 8 5 9 17 7 20 10 12 26 8 4 5 11 24 20 56 23 24 16 12 14 100 20 20 8 8 5 15 12 10 48 38 21 7 6 6 6 3 5 6 4 8 38 48 21 58 15 13 74 17 24

Design 96 95 .770 .787 .716 17 76 100 55 100 53 30 28 30 43 22 70 7 4 10 5 5 19 6 9 9 7 8 10 15 7 11 46 76 67 38 17 23 24 51 38 9 7 10 12 5 13 18 39 47 5 2 4 3 2 4 3 7 50 100 49 28 82 11 7 83 32 36

Music 97 85 .793 .788 .607 52 12 46 77 99 43 36 100 100 57 45 78 4 6 10 3 4 6 4 2 4 5 6 7 6 9 15 40 46 44 20 17 27 97 28 13 10 5 7 21 12 24 100 71 91 3 4 5 4 3 3 3 3 6 34 27 41 34 14 14 15 22 39

Drama 98 95 .837 .754 .812 13 76 32 11 44 40 21 11 7 43 73 15 8 4 10 6 11 3 8 4 3 4 14 12 4 13 12 100 93 50 17 36 12 94 39 6 7 6 2 38 9 4 21 100 100 7 4 2 3 4 2 4 3 2 47 33 35 61 17 7 50 62 16

Dance 99 31 .743 .634 .499 19 22 22 91 48 56 45 12 34 84 47 22 34 25 10 40 70 100 6 7 15 17 10 6 12 11 15 100 43 62 8 6 14 34 36 19 40 9 30 90 51 11 25 62 29 5 13 5 6 5 8 9 15 18 57 66 19 30 100 22 93 83 12

Cinemati 100 80 .814 .717 .652 27 22 19 57 100 42 63 31 16 16 70 94 5 14 14 5 5 7 4 13 41 6 11 26 22 12 21 27 57 100 11 20 9 100 10 25 10 18 18 16 22 9 57 80 56 5 3 2 2 3 4 4 3 4 34 26 17 11 13 5 17 9 25

ImagiWrit 101 44 .763 .684 .674 100 63 16 37 100 18 30 97 38 45 19 57 16 18 12 28 12 7 6 16 14 8 11 9 12 4 25 54 13 100 20 12 7 40 16 48 18 22 40 28 9 25 60 56 99 3 3 7 3 6 7 3 5 7 27 28 63 11 27 41 22 12 23

OthCreat 102 25 .905 .868 .536 25 65 24 100 94 100 24 29 28 52 15 60 6 9 12 76 16 16 23 16 22 50 15 5 41 52 61 81 100 42 6 12 18 88 66 30 11 12 16 5 5 14 6 7 39 3 2 4 2 2 3 3 3 10 31 69 62 7 24 22 36 24 46

Q07prompt

Q01explain

Q02interest

Q03enthuse

Q04stimulate

Q05criteria

Q06notify

Q08detail

Q09clarify

Q10 Q11advice contact

excellent

Q16library

Q17IT novelty

Q21proportion6variance Q18labs

Q19confidence

Q20comms

Q15smooth

Q12choice

Q13timetable

Q14


Resources Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioural, and biomedical sciences. Behavior Research Methods, 39(2), 175-191. http://www.gpower.hhu.de/ UCLA. Probability and statistics EBook Retrieved 17/12/2014, 2014, from http://wiki.stat.ucla.edu/socr/index.php/Probability_and_statistics_EBook Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. A catalogue of reporting guidelines for health research. European Journal of Clinical Investigation, 40(1), 35-53. http://dx.doi.org/10.1111/j.1365-2362.2009.02234.x Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. (2014). [Research Support, Non-U.S. Gov't]. Syst Rev., 3:37.(doi), 10.1186/2046-4053-1183-1137. Survey & Questionnaire Design. (2010). Retrieved from http://www.statpac.com/surveys/index.htm#toc Arsham , H. (2009). Questionnaire Design and Surveys Sampling. Retrieved from http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm#rssss


Documents

Quantitative Analysis of Survey Data & Related Design Issues...Results and Implications "Expert, lay "Presentation "Analysis 24-Nov-16 Kornbrot Quantitative Methods for Surveys . Project