MA Quantitative Methods Measuring Instruments Peter Ratcliffe

MA Quantitative Methods

Measuring Instruments

Peter Ratcliffe

Concepts and Indicators

Key issue: translation of underlying concepts (which are not directly observable/measurable) into indicators (which are)

Then assess external (or construct) validity - extent to which the indicator really does adequately ‘measure’ the underlying concept.

Examples

• Empirical measure of ‘housing classes’• Decomposition of ‘housing needs’ into

‘current’, ‘special’ and ‘future’ (Bradford 1996). [Key concept -‘housing need’: multi-dimensional involving current needs (quality, space, social/physical environment..), special needs (‘cultural’, health/disability....), and future needs (spatial and social mobility, fertility/fecundity, household transformation, net in-migration.......)]

Reliability and (internal) validity

• Reliability - replicability of the measurement process, i.e. measuring the ‘same thing’ each time. May relate to question wording, interviewer bias and/or recording error

• Validity is concerned with whether a particular ‘indicator/variable’ measures what you want it to measure.

[In studies repeated at regular time intervals – also question of comparability]

Examples(a) Income • Gross or net? Basic or ‘total’? (In cases of variable income)

last week or last month? Does it include ‘benefits’? Does it include ‘unearned’ income? etc.

(b) Rooms • Definition?: Four walls, a ceiling and a floor?! Is R.03/04

(ground floor) one room or two? In the domestic sphere the number of ‘rooms’ important given a concern with overcrowding levels, e.g. is a kitchen a ‘room’, etc..

(c) Ethnic Group [Extremely complex! – next week]

So - want BOTH reliability and validity (and, indeed, comparability)

The Structured Interview

DEFINITION: Questions are listed on an interview schedule and asked in the same order in the case of each respondent. [Most extreme version - all questions are pre-coded.]

Different theoretical perspectives:

• Positivist• Phenomenological• Feminist.

Positivist

‘The case for formal interviewing is simple. Only if all respondents are asked exactly the same questions in the same order can one be sure that all the answers relate to the same thing and are strictly comparable. Then, and then only, is one justified in combining the results into statistical aggregates.’ (Moser and Kalton, 1971: 296)

Positivist (cont.)

All theoretical approaches stress importance of interaction between interviewer and respondent.

Positivists - ‘interaction’ can be modelled

statistically, in the following manner:

Response = f[ Individual True Value(ITV), Interviewer Effect, Interaction Error]

Positivist (cont.)

Solution? Standardise interview process by defining the respective roles of interviewer and respondent: ‘(i)f complete uniformity could be achieved and interviewers acted like machines, answers could be regarded as independent of the way the questions were asked’ (ibid.: 275-6)

But could they? This is the classic defence of the structured interview as social scientific method par excellence.

Positivist (cont.)Reliability vital, hence concern devoted to

achieving uniformity.

Response Errors: how do they occur? Are errors systematic (rather than ‘random’)? How can they be countered?

• Random Errors, e.g. coding (box-ticking) error - unlikely to be systematic

• Interviewer Bias - systematic errors• Interviewee Bias, may be systematic over the

whole sample or simply across a sub-sample (say men/women, young/elderly).

Positivist (cont.)

Sources of interviewer error:

• Interviewer’s opinions• Interviewer’s expectations. Not quite

so obvious. Three sorts: attitude structure, role expectations, and probability expectations.

Phenomenological

One view is that ‘responses obtained are produced in part by dimensions of individual differences relevant to all social encounters.’ (Derek Phillips)

Phillips uses a symbolic interactionist (Goffmanesque) approach to analyse the process underlying the survey interview……..

Phenomenological (cont.)

Aaron Cicourel is especially critical of conventional positivist accounts of interviewing, in particular those which place great emphasis on fixed-choice schedules or questionnaires.

He argues that the latter may be adequate for ‘factual’ information, but for data on social processes they might force a respondent to provide precise responses to events and issues about which s/he may be ignorant or vague.

Phenomenological (cont.)Three core arguments:

• Documents constitute ‘grids for the distortion of social process’.

• He asks: How much do we need to know about ‘language and cultural meanings’ and ‘structures of social action’ before a successful questionnaire can be written?

• What is the role of theory in coding and scaling fixed-choice responses?

Much ‘spurious precision’ at the analysis phase of a study.

FeministAnn Oakley shares the view of writers such as Phillips

that role asymmetry is a pivotal issue, BUT goes further in suggesting that this is an explicit replication of male power and patriarchy.

• Rejects concerns about ‘over-rapport’• Rejects the view of the ideal interview

schedule/process as a ‘scientific measuring instrument’

• Argues that interviews should be occasions for sharing information, not for the extraction of data from the respondent by the interviewer.

Feminist (cont.)A number of concerns:

(1) The implications of her approach for sociological research as a whole?

(2) An overly rigid, deterministic division of male/female ‘characteristics’/social roles?

(3) An underlying ethnocentrism [who are the ‘we’ when she talks about ‘women’?]

Questionnaire Design Two principal types of document:

The Interview Schedule. Interviewer’s interests are paramount. Layout must be well designed and clear; particularly where sections of the schedule apply to only sub-groups of the overall sample. (Mistakes can be both fatal to the research and acutely embarrassing for the interviewer!) Computerisation has helped here.

Mail questionnaire. Has to be relatively brief, clear in design and simple to complete.

Questionnaire Design Criteria

(1) Should flow like a ‘normal conversation’ Questioning should develop in a logical way, and any change of direction should be flagged up clearly, with some explanation. ‘Classification’ questions should generally come at the end.

(2) Certain things should be avoided:• vague or leading questions• rambling or overlong questionnaires• ‘impractical’ questions (e.g. requiring long memory)(3) People need to be able to answer the questions

(don’t like admitting ignorance), and willing to do so. (NB. willingness doesn’t imply response validity)

Questionnaire Design Criteria(4) Can broadly distinguish between ‘factual’ and ‘non-factual’ (or

opinion) questions

‘Factual’ questions: issue of I.T.V. once again‘Non-factual’ questions. There are a number of problem areas:

• May be no clear ‘knowing’ of the answer by the respondent• Possibility of no one ‘correct’ answer• May wish to measure intensity of opinion (how?)• These questions may be particularly sensitive to wording,

emphasis, sequence, etc.• In general, no possibility of consistency checks (these may be

available for ‘factual’ questions). Moser & Kalton suggest checks against ‘measurable’ behaviour (?)

Question wording/Question Type

• Need to be sufficiently specific (to avoid multi-dimensional response)

• Use simple language (but not too simple!)• Avoid ambiguity, and vague words (fairly,

generally, often, why.....)• Avoid leading questions• Avoid presuming questions (often require

a ‘filter’) But, may wish to presume knowledge/behaviour if people are reluctant to discuss certain sensitive matters (?)

Question wording/Question Type

• Avoid hypothetical questions• Need to give special thought to researching

embarrassing/difficult issues. Can use indirect questioning, sentence completion, try to increase anonymity, use ‘check lists’ (or ‘randomised response technique’)

• Take care when studying periodical behaviour

• Special measures may be required if questions require memory (aided recall techniques, diaries, bounded recall, list of ‘available’ answers....)

Open-ended v. Pre-coded Questions

Open-ended questions. Problem areas: • loquacity, • compression into codes • difficulties in achieving verbatim record

Pre-coded questions • importance of piloting • categories need to be mutually exclusive and

exhaustive• produce ‘forced choice’ answers (cf. Cicourel)

Pre-coded Questions: types• A Likert scale: e.g. a range of answers from

‘Strongly agree’ through to ‘Strongly disagree’.• A semantic differential scale: A range of

positions between two extremes of a continuum (e.g. ‘Caring’ through to ‘Uncaring’).

• A checklist: e.g. a list of leisure activities.• A ranking of items: e.g. placing the most

important attributes of a potential partner in order.• A choice between statements: e.g. a choice of

responses to the knowledge that one’s best friend’s partner is being unfaithful.

(Mail) Questionnaire

All questionnaires: Important to give thought to issues of questionnaire length and question order. Even more so for the mail (or self-completion) questionnaire.

• Latter is best considered in balance sheet format (advantages /disadvantages compared with the interview-based approach)

Advantages • Cost• Can cover widely dispersed populations• Quick (but problem of late returns)• Obviously, avoids interviewer errors/biases

but.....(?)• Useful when intra-household consultation

(or considered response) is required• Useful when asking embarrassing

questions (more truthful responses?)• Problem of ‘non-contacts’ eliminated

Limitations • Questions need to be sufficiently simple and

straightforward• Answers need generally to be accepted as

final (therefore, lack of flexibility)• Inappropriate when spontaneity is required• Answers to different questions cannot be

seen as independent• Cannot be sure the correct person answers

(e.g. survey on ‘pig keeping’!)• No chance to supplement data with

observational material

Principal Limitation

Some of these can be overcome by combining the method with interviewing, but this undermines two of the major advantages of the method (speed and low cost).

• Main limitation is its typically low response

rate. Can be as low as 10 per cent. So:• What factors affect response rates? How

can we improve them?

Response rate issues• Must give careful thought to design,

content and length of questionnaire• Can seek ‘prestigious/respectable’

sponsorship• Give special thought to covering letter

(including method of salutation, and assurance of anonymity and confidentiality)

• Include s.a.e. (?)• Offer payment for completion (?), and• (Most importantly) plan follow-ups carefully

The mail questionnaire and sociological research

In sociological research: probably most useful

• EITHER for the collection of ‘factual’ data

• OR as part of a multi-phase study to locate ‘rare populations’

• (OR BOTH). EXAMPLE: Heidi Mirza, Young, Female and Black

Coding and Data Analysis Coding (even if done pre-fieldwork) is a ‘theoretical’ task.

Positivistic ‘solution’ is to go for the apparently ‘objective’ solution. Rather than suggest that the interviewer is the best person to ‘get into the respondent’s head’, Moser and Kalton argue that well trained coders are in the best position to make a dispassionate assessment of what respondents actually meant:

‘.....any summarising or coding can be carried out uniformly in the office, uninfluenced by the circumstances of the interview or the reaction of the respondent to the interviewer’ (p. 342). Do you agree?

Coding is also a ‘theoretical’ task in that it needs to reflect the initial hypotheses/research problem.