2007 Yngve Lindvig, 2007 Yngve Lindvig, The Learninglab Norway, Overview 15/01 13-15 – Introduction, Validity and Reliability. 16/01

2007 Yngve Lindvig, 2007 Yngve Lindvig, The Learninglab Norway, www.laeringslaben.no

Overview

15/01 13-15 – Introduction, Validity and Reliability.16/01 14-16 – What is science 1 Facts, observation,

induction and falsification.19/01 09-13 – What is science 2 Paradigms, programs,

against method, realism and anti-realism. (Rom 234) 22/01 13-15 – Introduction, qualitative research23/01 14-16 – Conducting qualitative research25/01 09-13 – Introduction Quantitative research29/01 13-15 – Quantitative research30/01 14-16 – Quantitative research01/02 09-13 – Quantitative measurement

Seminarrom U31 Helga Engs hus


Data collectionDevelopment of:-Observation log-Interview guide-QuestionnaireImplementation

The process of empirical research

Generalised knowledge/theory/the Generalised knowledge/theory/the typical/constructed constructstypical/constructed constructs

Pre-understanding- Experience- Scientific theory- Subject theory- Method comprehension

Writing-Developing categories-Summary -Test-Back to the outset

Analysis-Confirming-Exploring-Structuring- 1 d- 2 d- N d

Reflection- Validity- Reliability- Researcher - object

Operationalising- Planning- Confronting the domain- Concept orientation- Test dialogue/interviews- Design

CommunicationCommunication


Reliability and validity

• Reliability – random errors

• Validity - systematic errors


Validity in social sciences

1. Internal validity

2. External validity

3. Construct validity


Internal validity

To what extent does the research design permit us to reach casual conclusions about the effect of the independent variable on the dependent variable?


Internal validity

The independent variable: Causes effect

The dependent variable: Is affected


External validity

To what extent can we generalize from the research sample and setting to the populations and settings specified in the research hypothesis?


Construct validity

To what extent are the constructs of theoretical interest successfully operationalized in the research?


Construct validity• Coheres between the theoretical construct

and the operationalization of the construct

Theoretical construct

O1 O3O2 systematic errors

random errors


Construct validity

Construct Theoretical level

Empirical level

Operationalization of construct

Measurement of constructEmpirical domain


Sampling

1. Random samples

2. Stratified random samples


Qualitative research

1. Research question

2. Selecting subjects

3. Collecting of data

4. Analyze

5. Conceptual and theoretical work

6. Writing


Participant observation

1. Active or passive

2. Complete participant to complete observer


Interviewing

1. Unstructured or semi-structured


Focus groups

1. Size

2. Organisation


Structuring data in qualitative analyze

1. Research question 2. Theoretical sampling3. Collection of data4. Coding – looking to constructs and concepts5. Comparison – looking to categories6. Exploring relationships - looking to the hypothesis and the research

question7. Exploring new hypothesis 8. New sampling9. New collection10. Test of hypothesis


Main models of administration of a survey

Survey

Structured interview Self-completion questionaire

Face-to face

TelephonePaper

CAPI Paper

CATI

Supervised

Postal

Internett

WebE-mail

Embedded Attached


Structured interviewing

1. Standardized

2. Pre-coded and fixed choice

3. Goal to give each respondent the same interview stimulus


Possible errors in Structured interviewing

1. Poorly worded question2. The way the question is asked by the interviewer3. Misunderstanding on the part of the interviewee4. Memory problems on the part of the interviewee5. The way the information is recorded by the

interviewer6. The way the information is processed, either

when the answers are coded or when data are entered into the computer


Important in Structured interviewing

1. Clear instructions

2. Question order

3. The way we ask questions

4. The way we recording answers

5. Probing

6. Leaving the interview

7. Training


Problems with Structured interviewing

1. Pre understanding

2. Characteristics of interviewers

3. Yea-saying or NO-saying

4. Culture

5. Meaning

6. Asymmetrical relation


Self-completion questionnaire

1. No interviewer

2. Postal or web-based

3. Fewer open questions

4. Easy to follow design


Advantages related to Self-completion questionnaires

1. Cheaper

2. Quicker

3. Absence og interviewer effects

4. No interviewer variability

5. Convenience for respondents


Disadvantages related to Self-completion questionnaires

1. Cannot prompt2. Cannot probe3. Cannot ask possible boring questions4. Cannot ask to many open questions5. Questionnaire can be read as a whole6. Do not control who answers7. Cannot collect additional data8. Reading skills may influence the result9. Risk of missing data10. Lower response rate


Improve response rate on Self-completion questionnaires

1. Good covering letter

2. Return postage

3. Follow up

4. Not to long

5. Clear instructions and design

6. Start with the most interesting

7. Money

8. Constructs that not provoke


Asking questions

1. Open

2. Closed


General rules for Asking questions

1. Don’t forget your research questions

2. Focus on structures

3. Test the questions on your self and others


Specific rules for Asking questions1. Avoid word as often or regularly, be specific

2. Avoid long questions

3. Avoid double-barred questions

4. Avoid very general questions

5. Avoid two questions in one

6. Avoid questions that include negatives

7. Avoid technical terms

8. Avoid questions the respondent don’t know something about

9. Avoid unbalanced categories

10. Don’t rely on the respondent memmory


Ethic

1. What are the beneficial consequences of the study?

2. How can the informed consent of the participating subjects be obtained?

3. How can the confidentiality of the subjects be protected?

4. What is the structure behind the study? Financing, who want the study and who want what result?

5. [email protected]


What to know? Validity and Reliability ScalesDesignSamplingStarting an analyse in SPSSOrganizing and Graphing dataDistributions and Central TendenciesCrosstabulationCorrelationRegression analyzeSignificanceEffect sizeFactor analysisSum variablesHypothesis testingT-testStructural Equation Modeling


Quasi-experimental and surveys designs

Design 1:

Group 1 x O1

------------------------------

Group 2 not-X O2



Design 2: Pretest-posttest

Group 1 O1 x O2

-------------------------------------------------

Group 2 O3 not-X O4



Example

High

Low

Post-test

Pre-test

Electric consumption for hot water

O1

O4

O3

O2 People who did not install solar panels

People who install solar panels


Scales or levels of measurement

1. Nominal

2. Ordinal

3. Interval

4. Ratio


Nominal

1. Different categories

2. No numerical meaning

Example:

Gender? Male female


Ordinal

1. Different categories that can be ordered by rank

2. Arithmetic meaning (more or less)

Example:

Do you have autonomy?

Little moderate much


Interval

1. Scale and equal quantities2. Numerical meaning

Example:

Is it Correct of USA to go to war against Irak?

Totally disagree 1 2 3 4 5 Totally agree


Ratio

1. Same as interval an a true zero

Example:

How old are you? _______ years


Starting an analyse in SPSS

Designing a questionnaire

Coding of questions

Entering data

Cleaning data

Describing variables


Null and research hypotheses

Ho (null hypothesis) The population mean of group x is equal to the population mean of group y.

H1 (research hypothesis) The population mean of group x is not equal to the population mean of group y.


Type 1 and type 2 errors

Type 1 Correct

Decision

Correct

Decision

Type 2

Population

Ho is true Ho is false

Reject Ho

Accept HoDec

isio

n


q3976 Do you have a good time with your fellow students?

5 ,5 ,5 ,5

36 3,6 3,6 4,1

240 23,9 24,0 28,1

720 71,6 71,9 100,0

1001 99,6 100,0

4 ,4

1005 100,0

1 Not in any class

2 In some classes

3 In many classes

4 In all casses

Total

Valid

SystemMissing

Total

Frequency Percent Valid PercentCumulative

Percent



q3971 Are you interested in learning at school?

4 ,4 ,4 ,4

216 21,5 21,5 21,9

478 47,6 47,7 69,6

305 30,3 30,4 100,0

1003 99,8 100,0

2 ,2

1005 100,0

1 Not in any class

2 In some classes

3 In many classes

4 In all casses

Total

Valid

SystemMissing

Total

Frequency Percent Valid PercentCumulative

Percent



Statistics

1003 1001

2 4

3,08 3,67

3,00 4,00

3 4

,728 ,568

,529 ,322

Valid

Missing

N

Mean

Median

Mode

Std. Deviation

Variance

q3971 Areyou interestedin learning at

school?

q3976 Doyou have agood timewith your

fellowstudents?


Correlations

1 ,395**

,000

1003 995

,395** 1

,000

995 996

Pearson Correlation

Sig. (2-tailed)

N

Pearson Correlation

Sig. (2-tailed)

N

q3971 Are youinterested inlearning at school?

q3974 Do youwork as well as youcan at school?


school?

q3974 Doyou workas well asyou can atschool?

Correlation is significant at the 0.01 level (2-tailed).**.


Correlations

1 ,136**

,000

1003 996

,136** 1

,000

996 998

Pearson Correlation

Sig. (2-tailed)

N

Pearson Correlation

Sig. (2-tailed)

N

q3971 Are youinterested in learningat school?

q3978 Do you have agood time in breaksbetween classes?


school?

q3978 Doyou have agood timein breaksbetweenclasses?

Correlation is significant at the 0.01 level (2-tailed).**.


Rotated Component Matrixa

,054 ,778

,034 ,779

,126 ,780

,789 ,136

,786 ,118

,744 ,022

-,441 -,010

q3971 Are youinterested in learning atschool?

q3974 Do you work aswell as you can atschool?

q3975 Do you enjoydoing schoolwork?

q3976 Do you have agood time with yourfellow students?

q3978 Do you have agood time in breaksbetween classes?

q3979 Do you havemany friends at school?

q3983 Do fellowstudents often harassyou?

1 2

Component

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 3 iterations.a.


Chi-square = 121,121Degrees of freedom = 8Probability level = ,000

Model fit:

CFI=0,983

IFI=0,983

TLI=0,968

RMSEA=0,052

Model - students learning culture

Confirmatory Factor Analysis

Miljø

Motivasjon

sosial trivsel

tilhørighet

mobbing

interesse for læring

motiverende lærere

innsats og utholdenhet

f1

f2

f3

f4

f5

f6

,69

,89

-,30

,87

,53

,62

,73

-,45

,96

,49

,85

-,79

,32

Friends at school

School culture

School environment Harassment

School environment

Interest in learning

Motivating teachers

Work effort

Motivation


•Formulate a hypothesis, discuss the possibilities of errors and discuss the following T-test output:

Assume that your pre-understanding was that

there is a relation between gender and factor 2.

Group Statistics

672 2,8046 ,57332 ,02212

309 2,9385 ,52790 ,03003

q4051 SEX1 Male

2 Female

factor2N Mean Std. Deviation

Std. ErrorMean

t-test for Equality of Means

t dfSig. (2-tailed)

Mean Difference

Std. Error Differenc

e

95% Confidence Interval of the

Difference

Lower Upper

factor2 Equal variances assumed

-3,484 979 ,001 -,13395 ,03845 -,20940 -,05849

Equal variances not assumed

-3,591 645,532 ,000 -,13395 ,03730 -,20718 -,06071


Coefficientsa

2,996 ,064 46,988 ,000

-,126 ,054 -,075 -2,330 ,020

(Constant)

q3983 Do fellowstudents oftenharass you?

Model1

B Std. Error

UnstandardizedCoefficients

Beta

StandardizedCoefficients

t Sig.

Dependent Variable: factor2 Motivationa.

Coefficientsa

4,029 ,051 78,565 ,000

-,326 ,043 -,236 -7,512 ,000

(Constant)

q3983 Do fellowstudents oftenharass you?

Model1

B Std. Error

UnstandardizedCoefficients

Beta

StandardizedCoefficients

t Sig.

Dependent Variable: fector1 Friendsa.

Documents

2007 Yngve Lindvig, 2007 Yngve Lindvig, The Learninglab Norway, Overview 15/01 13-15 – Introduction, Validity and Reliability. 16/01