1

Today's program

• The Utz et al. Paper

• Our follow up survey

• Related matters– from research question to research question– selection effects (and what to do about it)– Multi-level analysis on such kinds of data

The Utz et al. paper(and follow up questions)

2

[show Utz summary and follow-up questions]

3

New questions should really follow-up on the scientific discussion, not just “be related”.

4

For instance: Do the results still hold: (1) when the weaknesses are taken into account? (2) under slightly different circumstances? (3) given different kinds of respondents? But you have to have a reason why you think this might be so! An this potential difference should be interesting.

In Utz et al, for instance: come up with a reason why we do not replicate the Kim finding.

5

Our follow-up survey

• Take our follow-up survey (now)…

• Concentrate, don’t interact with others

• … also because these will be the data that you will work with at the exam

• … and because you will need to invite other people to participate

http://tue.nfshost.com/index.php?ref=StudentID&p=ParticipantID

6

For your own run, use your own StudentID and p=000

For your invitations use your own StudentID and different p-values per invitation!

<go through the survey here>

7

8

Selection effects

My sample might not be representative for the population

That is: for which group of people are my results supposed to hold?

9

Selection effectsKiezen van de doelgroep vs non-response.

• eBay vs non-eBay users• computer literate vs not• ...

• those who participate and those who don't– carefully consider what the difference might be– compare the early with the late responders– ask something to the non-participants

• And ... what was so good about experiments again? ...

• ... non-response might screw that up!

10

Selection effects (cont’d)

11

To Do

• We're going to run the survey you just completed

• Come up with 15 persons to invite. Vary the characteristics. When you did not submit follow-up questions you have to come up with 25

(use the Excel template online)

• Create an invitation list. Do not invite anyone yet. Submit this list to CS on Thursday latest. Start inviting this Friday.

• We will keep track of who indeed participates, and who does not

• These data can also be part of the exam data