Data Collection methodology and process by SS KHAN

Data collection In this chapter we discuss sources of data, the pros and cons of using primary and secondary data, the problems associated with collecting your own data, sampling techniques, questionnaire design, the importance of pilot studies and deciding how you will analyse the resulting data. There are examples of questionnaires for analysis.

1 Approach to study We address the issues relevant to data collection principally through the case studies which accompany this chapter. ....................5

2 Quality, sources, and collection of data Existing sources may tell us what we need to know, but sometimes the only way to obtain the information required is to collect it specifically for the purpose. The method we use to do this is determined by whether we are looking for objective or subjective data. There are issues concerning the scale of the data collection and ease of obtaining the data. Decisions have to be made as to whether all of a set of data may be collected, or whether sampling will be required. ...................................................................................11

3 Sample design and methods Choices have to be made as to the appropriate sampling strategy for obtaining the data required: probability or non-probability sampling. Further choices include what sampling method will be used to obtain the data and, if a questionnaire is used, how will it be administered?.................................................................................21

4 Questionnaire design There is more art than craft in designing a questionnaire but we guide you through the main features and around the pitfalls. .............30

5 Pilot studies and pre-pilot studies It is extremely important to ensure that a questionnaire is thoroughly tested before widespread use. A pre-pilot study can give an early indication of badly worded questions. The pilot study itself should be a test of the whole process of a study. It is a further check for badly worded questions, should provide more information on appropriate categories for closed questions and indicate differences in the terms of reference of researchers and respondents. .......................................................................................34

6 Analysis of survey results Having collected satisfactory data, there are a number of ways to process it .............................................................................................38

2 │ DATA COLLECTION

The collection of data is some people’s business. Whole armies of people are involved in the design, collection and analysis of social and marketing data on a large scale. For example, National Census, Retail Price Index, Market Research Data, and the household names, MORI (Market and Opinion Research Institute) and NOP (National Opinion Poll).

Such data is the foundation for many significant decisions. National Census data forms a basis for many resource allocation decisions in education, housing, social services and health provision at local and national government level. The Retail Price Index, as the basis for the calculation of inflation, not only influences government decisions about allocation of resources, for example, welfare benefits, but also drives wage bargaining and plays a significant role in the economy as a whole. Many large companies invest considerable energies in the acquisition of market research data on which to base decisions about advertising, product design, and so on.

It is unlikely that many of you will be directly involved with data collection on such a large scale, although you may have cause to use such data. However, it is quite likely that you will become involved in a smaller scale exercise, possibly as the commissioner of a study, possibly with direct responsibility for an investigation. Whatever the circumstances, it is important that you are aware of the relevant issues.

Objectives By the end of this chapter you should be able to make recommendations for the design and conduct of a reasonably large-scale data collection exercise and to comment on such proposals. This should include a knowledge of:

potential sources of secondary data and how to access these

possible approaches to the collection of primary data (recording, observation, questioning), the advantages and disadvantages of each

DATA COLLECTION │ 3

the design of sample surveys. The use of probability and non-probability sampling methods and the advantages and disadvantages of each

the advantages and disadvantages of different approaches to questioning (self-administered questionnaire, face-to-face interview, telephone interview, etc)

the design of questionnaires

tools for the analysis of data.


1 Approach to study As each topic is introduced and discussed you will be asked to relate it to the cases – after spending time thinking about an issue you should compare your suggestions with mine. However, I should begin by making clear that there is not necessarily a right or even a best way of proceeding in any particular circumstances. Data collection almost always involves a compromise of some kind, for example, ‘Is it worth spending more money, or taking longer, to obtain more, or better quality, data?’ Also, it often brings unexpected problems – for example, the data, or list of potential interviewees, which you thought would be easily accessible, may turn out to be, in someone else’s view, highly confidential.

Before you read any further with this chapter spend time on the following activities which are designed to get you thinking about different approaches to data collection.

Read through the following case studies. Spend about 15 minutes thinking about each of them, bearing in mind the following questions:

What data/information is needed to help the manager in each case?

If it already exists, where would you find it?

If it doesn’t exist, how might you go about collecting it?

What problems might you encounter?

Make a note of your thoughts. You will reconsider each of these questions as the discussion develops during the course of this chapter, so you will find it useful to be systematic with your notes.

Case study A

Your small consultancy company, which is highly renowned for its investigation work, has just been awarded a contract by the government for £150,000 to investigate the effectiveness of policing and the image of the police force in a particular city. You are


scheduled to attend a meeting next week at which you will present the approach you propose to take. In addition to describing your proposed approach you should explain your reasons for deciding to do things this way and comment on alternative approaches which you have rejected.

Case study B

Aldebaran Coaches operates an express service between major towns and cities in Scotland. A few prominent people on the board of directors have been pushing for the introduction of a no-smoking policy on the coaches, on the grounds that it will be seen to promote a health conscious image and will differentiate them from the competition. Other board members are concerned that it will lead to a net loss in custom.

Currently the coaches have 25% of their seating capacity made available for passengers who wish to smoke, but they have no idea to what extent this is used by smokers. You have been asked as a consultant to advise Aldebaran Coaches on how they should carry out an investigation of the impact on their operations of the introduction of a no-smoking policy.

Case study C

MacDrain Ltd is a firm of emergency plumbers which operates across the central belt of Scotland; it has a base in each of the major cities. It responds, on average, to between 100 and 200 calls each day. The company has grown substantially from a small family business, to which most of the customers were known personally, to its current level. The managers of the company feel that they have lost touch with their customers; they have no feel for whether or not they are satisfied with the service they receive. Thus, they would like to carry out a survey of customer satisfaction, with a view to instituting a process for regular feedback. You have been asked to help them design this survey.

Case study D

In the late 1980s unleaded fuel became available in the UK. New cars were designed to run on the unleaded fuel and owners of recent cars were urged to have their cars ‘converted’ to run on unleaded fuel (this was a simple job, taking no more than 15 minutes or so), encouraged by lower fuel prices and ‘environmental’ factors. The effect on the cars’ performance, it was said, would be negligible. However, some of the owners of cars which had been converted claimed that the petrol consumption of their car was significantly increased. A motoring organisation set out to investigate this claim, drawing on the


experiences of members of the public (they had no funds to carry out sophisticated tests). What options were available to them to carry out such an investigation?

Case study E

The management of a large organisation which has recently undergone substantial change (this might be a city council which has recently been restructured, or a company created by a merger) is concerned about levels of absenteeism and general morale. They would like to carry out a staff attitude survey to enable them to better understand the motivation of staff at all levels of the organisation and to help them in the design of a new benefits package. Can you offer any advice to them in planning this investigation?

Case study F

Since the deregulation of the telecommunications industry, personal customers have had a greater freedom of choice in purchasing telephone equipment for use in the home, and mobile phones. Business users also have a wider choice of equipment and can choose a range of services.

You have been commissioned by an independent body to design and conduct an investigation of the impact of deregulation on customers’ perceptions of the services offered. They are concerned about perceptions of choice, cost and quality of all aspects of the service, including connection, installation, distribution, maintenance and billing. How might you carry out such an investigation?

Case study G

A local education authority, covering a large city and surrounding suburban and rural areas, is concerned about the high numbers of personnel leaving the teaching profession and citing stress as their reason for doing so. The problem is particularly acute in the city schools.

They have obtained a questionnaire, shown in Figure 1, designed by a group of consultant psychologists, which is purported to measure stress levels. They are considering paying the consultants for the use of this questionnaire in a study of employees. You are asked to advise them on how to judge the appropriateness of this questionnaire for their purposes.


Figure 1 How often do you experience these feelings?

Never Once Rarely Sometimes Often Usually Often

1 Being tired 0 1 2 3 4 5 6 2 Feeling depressed 0 1 2 3 4 5 6 3 Having a bad day 0 1 2 3 4 5 6 4 Being physically exhausted 0 1 2 3 4 5 6 5 Being emotionally exhausted 0 1 2 3 4 5 6 6 Being wiped out 0 1 2 3 4 5 6 7 Feeling pushed around 0 1 2 3 4 5 6 8 Being unhappy 0 1 2 3 4 5 6 9 Feeling trapped, closed in 0 1 2 3 4 5 6 10 Feeling cynical 0 1 2 3 4 5 6 11 Feeling worthless 0 1 2 3 4 5 6 12 Wanting to quit 0 1 2 3 4 5 6 13 Being cold, callous or hostile 0 1 2 3 4 5 6 14 Feeling disillusioned about

people 0 1 2 3 4 5 6

15 Feeling bored 0 1 2 3 4 5 6 16 Feeling hopeless 0 1 2 3 4 5 6 17 Feeling resentful about

people 0 1 2 3 4 5 6

18 Feeling pessimistic about outcomes

0 1 2 3 4 5 6

19 Feeling listless 0 1 2 3 4 5 6 20 Feeling anxious 0 1 2 3 4 5 6

Your total score ……………………………...

I imagine the group average score is ……..

What is the objective of the investigation?

This is the first question to ask in designing a data collection exercise.

It is important to begin with a clear idea of why we are collecting data/information and what information we would ideally like to collect. The focus here is on information for decision making, but within that context, the nature of both the information and the decision can be wide ranging, as the case studies suggest.

For example, information may be required to support a one-off decision process which needs to incorporate the views of many people. This could be the relocation of a company’s offices, the introduction of


a company benefit package, the decision to move to a new way of working, such as the decisions of NHS primary care groups to become primary care trusts from 2000/01, or of universities to move to a semester system.

The decision may be on a much lesser scale, for example, the suitability of an individual for a particular job. We may wish to find out about the person’s personality and skill level. A standard way of proceeding would be for the individual to be interviewed by a number of appropriately skilled people and the decision to be based on their judgement. However, many standard ‘measurement instruments’ (tests, if you like to call them that) are now in widespread use to measure both personality and aptitude, particularly in situations where large numbers of applications are received for just a few posts. For example, the GMAT test is widely used as a measure of the suitability of applicants for MBA programmes.

Information may be required for purposes of monitoring, and may or may not indicate a need for action. Such data may be largely subjective, for example, perceptions of quality of service as provided by customer satisfaction surveys.

On the other hand, the data may be objective, for example, relating to:

faults arising in a production process

the time taken to process orders

time taken to settle accounts

worktime lost through illness or injury.

Information may be required to find out who engages in a particular activity, or purchases a particular product, and why.

This may inform decisions about product design or service provision, or form a basis for the design and targeting of information or advertising. The majority of market research falls into this category.

Clearly, this is not an exhaustive account of the types of information one might want to collect, nor of the types of decision one might face, but it gives an indication of the broad spectrum.


2 Quality, sources, and collection of data

Quality of data

Before going on to examine various potential sources of data, we consider a number of issues relating to the quality of data, for example, reliability, validity, error and bias.

Reliability of the data This refers to the accuracy and consistency of the measuring instrument (which could range from a sophisticated piece of electronics to a simple questionnaire). Data is said to be reliable if the same results would be obtained if the sampling procedure were to be repeated in precisely the same way. Whilst engineering data is generally very reliable, social statistics tend to be unreliable – people lie, exaggerate or simply forget!

Validity of the data Data is said to be valid if it is a good measure of what it is supposed to be measuring. Data can be reliable but not valid for its purpose. The issue of validity arises when we attempt to use surrogate measures to reflect ‘unmeasurable’, or difficult to measure, concepts (for example, quality, satisfaction, stress). For example, is recorded crime rate a valid measure of actual crime? Is IQ a valid measure of intelligence?

Error This refers to the amount by which a measurement differs from reality. A good measuring instrument should yield few errors, but those which occur should be randomly distributed. Precision and bias are the


measurable faces of reliability and validity. Measurements can be very precise, that repeated measurements show very little variability, but widely inaccurate, in the sense that they are way off the ‘true’ value – see cartoon.

Bias This occurs when errors which occur tend to be mostly in the same direction.

Ideally a measuring instrument should be reliable, bias-free, yield minimal errors and be a valid indicator of the concept of interest. The extent to which it is likely to be so should be addressed at an early stage in an investigation, otherwise considerable resources could be wasted. One of the aims of a pilot study should be to assess these issues. The notion of reliability will be revisited later in this chapter.

Sources of data

We must discover if the required data already exists in some form, or whether it will have to be collected specifically for the investigation. Examples of sources of existing, secondary, data are given below.

Primary or secondary data? Once you have an idea of what data you require, a first question to ask is whether it already exists in some form, or whether it has to be collected specifically for this purpose. Data which is collected specifically for your purpose is known as primary data. Data collected for another purpose, whatever the source, is known as secondary data.

In any study it is well worth investigating if there is secondary data which may be of use, as it could yield substantial savings in time and effort. We first look at sources of secondary data and issues in its use. We then look at the various approaches to the collection of primary data.

Sources of secondary data

There are many sources of secondary data, for example:

internal company records on production, personnel, accidents

official government sources such as the Annual Abstract of Statistics, published by the Office for National Statistics, which provides information on most aspects of economic, social and industrial activity in the UK

non-official sources such as the Economist Intelligence Unit reports, and

commercial sources.

Much information can be found from electronic sources, including commercial online databases, the internet, and do not forget the value


of the press. The following section gives a selective listing of sources of secondary data.

Official sources

Yearly UK Annual Abstract of Statistics, Social Trends, Digest of Energy Statistics, Highway Statistics, Family Expenditure Survey, International Financial Statistics (IMF), International Trade Statistics Yearbook (UN), …

Quarterly Housing and Construction Statistics, Transport Statistics, Quarterly Oil and Gas Statistics (OECD), …

Monthly Monthly Digest of Statistics (similar to the Annual Abstract of Statistics), Economic Trends, Financial Statistics, Overseas Trade Statistics of the UK, Monthly Statistics of Foreign Trade (OECD) …

Most countries have a national statistical office which publishes statistics in some form.

Non-official sources of statistical information

Commercial sources such as Potato Statistics Bulletin, from the Potato Marketing Board, SMMT Monthly Statistical Review, from the Society of Motor Manufacturers and Traders, Quarterly Copper Statistics from the World Bureau of Metal Statistics …

Subscription services such as Mintel Market Intelligence – a monthly report mostly concerned with consumer products, The Economist Group – incorporating the Economist Intelligence Unit (EIU) publishes regular reports on certain industrial sectors and special reports on topics of current interest, Media Expenditure Analysis (MEAL) – information on advertising expenditure across all media.

Yearbooks and directories such as Key British Enterprises, Kompass UK.

Special financial analyses such as Key Note Reports – market size and trends, financial data, etc for about 200 consumer and industrial markets.

Electronic sources The range of information available from on-line databases is immense and includes competitor intelligence, company profiles, market research reports, mergers and acquisitions, export opportunities, credit checks, air and rail timetables and the complete text of today’s newspapers.


General, non-statistical sources

National press – The Economist, Financial Times, Guardian, Independent, The Times …

Trade press – Chemist and Druggist, The Grocer, Motor Trader …

The above list is an indication of the key sources of data available in the UK. There are many more. A local business library will be able to give you help in deciding appropriate sources for your purpose.

Pros and cons of secondary data

If appropriate secondary data is available to help in your investigation, then it can be a great bonus, giving substantial savings on time and effort. In some circumstances there may be no alternative, for example, if you want to examine how something has changed over time.

However, there are a number of things to be beware of.

Will it ever arrive? If you are using published data then this should not be too much of a problem. However, if the data is internal to an organisation, often you are reliant on someone else to supply it. No matter how well-intentioned they are, if there is no real incentive for them to supply the data it is likely to be low on their list of priorities, especially if considerable effort is involved in producing it. It may involve photocopying a substantial amount of material, producing a special computer printout, disguising data to protect confidentiality, waiting for a committee meeting to obtain approval to release the data, or a host of other potential hold-ups.

In what form will it arrive? It is possible that your data will arrive just as you need it – but more likely that you will have to do further work to extract what is required. We recently asked an organisation if it would be possible to have a list of their members working in building societies; they very kindly said, yes, that would be no problem. However, when it arrived it was a list of everyone working in a financial institution, numbering almost 1000, of whom about 5% worked in building societies. It wasn’t too time-consuming a task to extract what I wanted from this data. However, a colleague tells a story of a PhD student in the department who waited six months to obtain data for his project on manpower planning in the health service – eventually he received 400 pages of data rather than the ten pages he was anticipating, furthermore, it wasn’t at all clear if his ten pages were to be found somewhere in the 400!

How was it collected? Although on the face of it the secondary data you have collected may match your requirements, it is prudent to be cautious. You may not know how the data was collected, how reliable it is, or on what assumptions it is based. One often hears


stories of people employed to do market research filling out endless questionnaires in the comfort of their own homes (although bona fide market research agencies carry out checks to guard against the possibility of this).

Why was it collected? Knowledge of why the data was collected may give you some idea of how reliable it is likely to be for your purpose. For example, many surveys collect data on the age of respondents – in some cases age may be a key variable in the study and care should have been taken to collect accurate data. In others the respondent’s age may be estimated by the interviewer and used simply to ensure a good cross-section of respondents was achieved. The latter would not be a good place to start to obtain a good picture of the age distribution of the population of the region.

Collecting primary data

Often, the only way to obtain the information required is to collect it specifically for the purpose – this is primary data.

Primary data may be objective, data that is in some way measurable using appropriate recording processes. Alternatively it may be subjective – that is, it concerns people’s perceptions, opinions, attitudes, beliefs or priorities.

Objective or subjective data? Objective data may relate to external, uncontrollable events (for example, weather data). It may be output from a controllable process (for example, production rates or service times). It may be behaviour patterns reflected in large-scale activity (for example, traffic movements, electric power consumption, level of investment or demand for consumer goods). It may be details of individual behaviour (for example, which magazines are read by which social groups, or how do individual managers spend their day). There are many possible ways of accessing objective data, unlike the collection of subjective information, which must directly involve people.

Ways of collecting primary data

Primary data can be collected in a variety of ways: automatic recording recording by an independent observer recording by the subject observation questioning multiple criteria analysis cognitive maps.

These are discussed below.


Automatic recording

It may be possible to establish an automatic procedure for the collection of objective data, using appropriate instrumentation and control methods. Such data is usually the premise of scientists and engineers, but may be useful in a managerial context.

Increasingly sophisticated technology is enabling the extension of data collection processes; smart technologies can monitor and provide automatic feedback on the wear and tear on buildings and other structures – information which can allow for more effective planning and management of maintenance; sophisticated monitoring systems can provide phenomenally detailed information on patients under intensive care; computers can monitor their users’ every interaction.

Data which is readily available in such large volumes poses new problems in how to analyse and make best use of it. Expert systems, and more recently, neural networks (computer systems which mimic the way the nervous system works in order to learn about the behaviour of a system) have been applied in these contexts. Essentially such systems are searching massive data sets, looking for patterns in the data and identifying exceptions. Such systems have been used, for example, to monitor performance indicators with a view to identifying poor performance and to monitor the use of credit cards in an attempt to identify fraudulent use.

Such sophisticated approaches are not always possible – smart technology, expert systems and neural networks are beyond the scope of this course, although you may encounter them elsewhere in the MBA programme. We are concerned with more ‘everyday’ methods of data collection.

Recording by an independent observer

Recording may rely on far more basic technology than that mentioned above and on the co-operation of the subject. For example, regular surveys are carried out at airports, by airlines and air transport organisations, to gather information about the weight of passengers. The measuring instrument is an everyday set of weight scales (accurately calibrated, of course) and an individual’s weight is recorded manually – the real skill involved is in persuading the passengers to be weighed.

Recording by the subject

Studies which call for longitudinal data, ie data collected over an extended time period, often involve a means by which the subjects of the study record their own activities. Market research often draws on such data. Examples might include consumer panels (individuals, or families) who record every purchase they make using bar-code readers, and audience research panels, families who are recruited to


monitor who watches which TV programmes. Diary-keeping exercises are often used to monitor how individuals at work spend their time, or how they utilise particular facilities. Exercises such as these clearly raise issues about the reliability of the data recorded, but there is often no alternative way of gathering information – observation may be too costly or simply impractical.

Between these two extremes of hi-tech and basic means of recording data are a host of other possibilities. However, in some situations there may not be a suitable instrument for recording data, or it may not be possible to put one to use. In such cases you need to adopt other approaches to data collection – simply watching or asking.

Observation

It is sometimes argued that observation is the only way of obtaining truly objective data. It has the advantage that it does not rely on the honesty or memory of the subject. However, it does have its pitfalls.

Observation may be overt or covert. Attempts at covert observation have led to some ingenious schemes and some quite bizarre antics. Hidden microphones and video cameras feature strongly, disguised investigators appear in unexpected places, domestic refuse has been hijacked to provide market research data, to mention just a few tactics. Some approaches show initiative, for example the local radio station which had an arrangement with garages in the area to record which station the car radios were tuned into when they were brought in for service (Chisnall, P.M. (1992) Marketing Research: analysis and measurement. 4th edn. McGraw-Hill). Covert observation, an extract from a French magazine, shows that some tactics leave something to be desired.

Covert observation

Les Français s’en lavent les mains

Un Français sur trois ne se lave jamais les mains après avoir été aux toilettes. Chez les plus jeunes (collégiens …), ils ne seraient même qu’un sur deux à le faire. Encore que laver soit un bien grand mot, car, pour 70% de ceux qui font une escale devant le lavabo, ils’agirait plutôt d’un rinçage sous un filet d’eau sans savon. Un rafraîchissement aussi bref qu’inefficace, l’opération ne prenant que neuf secondes en moyenne. Cette petite étude a été effectuée par l’Association européenne de promotion de l’hygiène. Les enquêteurs s’étaient déguisés en plombiers pour «ne pas influencer le comportement des usagers»!

The French wash their hands of it

One in three French people do not wash their hands after going to the toilet – amongst younger people it's not even 1 in 2. Even then ‘washing’ is too grand a word. For 70% of those who go to the wash hand basin, it is nothing more than a rinse under a stream of water … without soap. A wash so brief as to be ineffective – it takes no more than nine seconds on average. This small study was carried out by the European Association for the promotion of hygiene. The investigators were disguised as plumbers so as ‘not to influence the behaviour of the users’


Some of the problems associated with overt observation are well known, in particular, the Hawthorne effect which suggests that the very act of observing someone can bring about a change in behaviour. This takes its name from an experiment carried out at the Hawthorne Electrical Plant in the USA in the late 1920s. The investigators were interested in the effect of working conditions on productivity, making a number of improvements to the environment and each time recording an improvement in productivity. Finally, everything was restored to initial conditions – and productivity increased once again!

Can observation be truly objective? The fact that data is recorded by observation does not wholly remove subjectivity or scope for error from the process. The observer has to interpret what they are observing, often making some subjective judgement or estimate. Examples of optical illusions abound, for example this illustration.

Even though a ruler demonstrates that the two lines are the same length, the lower one continues to look longer than the upper.

How often have you argued with someone about the colour of an object or piece of clothing – is it blue or green? Red or brown? How good are you at estimating someone’s age? At judging whether they are tired or bored? How good would your estimate of the floor area of your home or your office be?

These questions highlight some of the potential problems with observation and the need to use trained observers if you are reliant on such data. You may not use such data personally, but it is an important component of many jobs: customs officers are skilled observers, looking to identify travellers bringing in illegal imports; estate agents’ estimates of house prices are based substantially on their observations of a property; the Council Tax is largely based on estimates of property values carried out on a limited budget.

Observation does not yield explanations A major limitation of observation is that, without follow-up, it does not provide explanations for why someone behaved in a certain way. Thus, market research data which simply provides information on buyer behaviour, without explanation of why a particular product was chosen, does not provide a basis for decisions on product improvements or how to target advertising.

Time and cost factors Whether overt or covert, observation is generally costly and time consuming. Even if data is initially collected by automatic means, for example video, the review and analysis of that data generally requires substantial human effort.

Questioning Probably the first things that spring to mind when you think of questioning as an approach to gathering data are the numerous


questionnaires that arrive through the letter box or the ubiquitous people with clipboards one dodges in the High Street on a busy shopping afternoon. These are usually attempts to gather data on a large scale. Although these activities represent a large part, and the most public face, of data gathering by questioning, they are by no means the only approach relevant to decision making.

It is helpful to think of the data to be collected along two dimensions: the amount of detail (or depth of information) required, and the volume (or quantity) of information required. The kind of study referred to above falls into quadrant B – high volume, relatively shallow data as illustrated.

Often when we are seeking information to support a decision-making process we require detailed information from a few individuals – categorised by quadrant C. For example, if a company is looking to appoint a new managing director to guide it through a period of expansion into overseas potential candidates. Frequently such information is gathered on an ad hoc basis through informal interviews with the interested parties. There are more formal approaches which may be used to facilitate the elicitation and analysis of in-depth, qualitative information of this nature. It is beyond the scope of this course to cover these approaches in detail, but some examples are described below.

Multiple criteria analysis

As the name suggests, this is an approach to the detailed evaluation of alternative courses of action, or strategies, which seeks to take explicit account of multiple, often conflicting, criteria.

Cognitive maps

These are a tool for the representation of qualitative data. They are used in a range of contexts, but in particular as a component of SODA (Strategic Options Design and Analysis) – an approach to the development of strategy pioneered by Professor Colin Eden, which you will study as part of implementation issues in the module Making Strategy.

Information generated by these processes is not amenable to statistical analysis, but to analysis which examines the structure of the data, looks for patterns in the data which might shed new light on the problem, and which may facilitate a group of people coming to a shared understanding of the issue.

Often, in practice, we are faced with a compromise between the two extremes characterised by quadrants B and C. Ideally, we would like


to obtain in-depth information from a large number of individuals (ie quadrant D), but practical consideration of the time and resources taken both to collect data and to analyse it force us to sacrifice detail in order to increase quantity. Sometimes, an in-depth investigation may be the first step of a larger-scale study. If a problem is stated in very general terms it may be best to begin by working with a few individuals to get a better understanding of the issues before setting out to collect data from a larger group.


3 Sample design and methods

Large-scale data collection Collection of data on a large scale raises a number of practical problems related to how it should be collected and from whom. We will begin with a consideration of the latter question. Sometimes it may be possible to question everyone whose opinion you are interested in – for example, following a lecture course we seek feedback from all the participants on course content, delivery, etc. Some of undergraduate lecture classes involve over 500 students, so such data collection could hardly be called small scale. However, the students are relatively easily accessible and the cost of collecting the data is minimal – the effort involved in collating and analysing it is not so insignificant. Often it is not possible to survey everyone of interest; it is necessary to sample. Before progressing further it is useful to introduce some definitions.

Population – This is the body of people, or objects, which is of interest to us. Sometimes the relevant population is easy to define, for example, in Case study C (on page 5), the population of interest is the customer base of MacDrain Ltd. In Case study A we are asked to research the image of a police force – in this case the population is more difficult to define. We might define the relevant population as residents who live in the city protected by the police force; however, this would overlook people who work but do not live in the city; it also ignores visitors to the city, any of whom may have cause to interact with the police. A manufacturing company may want to find out about the reliability of one of its products; in this case, the relevant population would be all the products of that particular type.

Census – A census is a study of the entire population. The course evaluation described above is a census. The national census, carried out in the UK every 10 years attempts to obtain data on everyone resident in the country.


Sample – If the population is very large, or if resources are limited, it is not practical to obtain information from every unit in the population. In such cases a sample is taken. A sample is simply a subset of a population.

Sampling fraction – The size of sample divided by the size of the population is referred to as the sampling fraction.

Designing a sample We are forced to take a sample because it is impractical to survey, or study, the whole population. However, we hope that the sample we take will give us a good picture of the population. We could regard a sample as a snapshot of the population of interest – a different sample would give us a slightly different picture. Our aim in selecting a sample is to obtain as representative a picture of the population as we possibly can, given the resource limitations to which we must work.

There are two distinct approaches to selecting a sample:

probability sampling (below) and

non-probability sampling (see page 26).

Probability sampling

Probability sampling ensures that each member of the population has a calculable, non-zero probability of being included in the sample. The different methods of probability sampling are:

simple random sampling

systematic random sampling

cluster sampling

stratified random sampling

multi-stage sampling.

These are explained below.

Simple random sampling

The most straightforward form of probability sampling is known as simple random sampling. The easiest way to conceptualise simple random sampling is to imagine that a piece of paper corresponding to each item in the population is put into a hat and that a number of papers, equal to the size of the sample required, is drawn at random out of the hat. Those items whose number is drawn out of the hat will be surveyed. For this reason, simple random sampling is often referred to as the ‘Top hat method’. Simple random sampling ensures that each member of a population has an equal chance of being included in the sample.


Clearly this would be a very tedious way of going about things. Random number tables, or random number generators (you should find one in the spreadsheet you use, and possibly also on your calculator) are a more practical way of proceeding. The simplest random number generators give values between 0 and 1, which you must transform to meet your requirements. For example, if you want to select from a population of 2,500 people, you need to generate random numbers between 1 and 2,500. To do this, first generate the required number of random values between 0 and 1, multiply each of those values by 2,500 and round to the nearest integer (if you get a zero, treat it as a 2,500 to maintain the equal chance of any number occurring). Some spreadsheets (for example, Excel) allow you to specify the range of numbers you require.

To be able to put the above procedure into practice, you need to have a list of members of the population. Often it is impossible to find a list which exactly matches the population of interest and investigators resort to lists which are easily available. Commonly used ones are the electoral roll, the telephone directory, memberships lists, registration records and so on.

Sample frame – This is a list chosen to represent the population.

Desirable characteristics of a sample frame include the following:

Adequacy: How well does the sample frame match the population of interest?

Completeness: Are any parts of the population not covered?

No duplication: No multiple entries.

Accuracy: Is the list up to date? Does it contain non-existent units?

Convenience: Of the list itself – is it easily accessible? And is the way in which it is arranged appropriate – especially if we are looking to develop stratified or cluster samples, as described below.

Using simple random sampling, we generate random numbers to indicate the units to be sampled and go through the list selecting the appropriate units (eg 5th, 11th, 22nd, etc according to random numbers).

Systematic random sampling

If the sample frame (list of units) is arranged in random order, then a slightly more convenient way of proceeding is to use systematic random sampling (also known as pseudo-random sampling). Systematic random sampling selects every N’th it from the sample


frame, where N is determined by the size of the population (P) and the size of sample required (S). N is equal to P divided by S. The first item is chosen at random from 1 to N.

For example, if we have a population of size 10,000 and we want to select a sample of size 1,000, then we would select every N = 10,000/1,000 = 10th person from the list. We begin by generating a random number between 1 and 10, say 9. We then select the 9th, 19th, 29th, 39th, etc unit from the list.

There are more sophisticated approaches to random sampling, for example, cluster sampling which can facilitate the collection of data, and stratified sampling which is a means of obtaining improved estimates of population characteristics.

Cluster sampling

One of the potential problems with probability sampling is that the selected units may be scattered over a wide area – a particular problem if you propose to gather data by interview. Cluster sampling is a means of overcoming this problem. The population is grouped into clusters which define convenient units – clusters may be towns, universities, units of a manufacturing operation. It is important that each cluster is equally representative of the overall population. True cluster sampling selects a number of clusters at random and each member of the selected clusters is sampled. However, if the number in each cluster is large this is not practical, so more often a multi-stage approach is adopted. The first stage of sampling selects clusters at random, a second stage selects a simple or systematic random sample from within each cluster.

Stratified random sampling

As mentioned above, stratified sampling is a means of obtaining better estimates of population characteristics. It is used when it is felt that the population is non-uniform in some respect, for example, between males and females, across age groups or socio-economic groups, or between public and private sector organisations. The population is divided into the relevant strata before the sample is taken. Independent samples are then drawn from each strata. Note that this places extra demands on the sample frame: it must be structured in a way, or include information, which allows the population to be divided into the relevant strata.


Multi-stage sampling

This is an example using clusters and strata. A large scale study sought to investigate the effect of aircraft noise at night on residents around one of the UK’s major airports. The residents are the population of interest. It was carried out in three stages.

1 It was anticipated that the effects would differ according to the exposure to aircraft noise and so the population was stratified according to aircraft noise level. Noise maps helped identify the strata, as illustrated here. Contours indicate points at the same noise level – the areas between the contours define the strata.

2 The survey was carried out using both postal questionnaire and interviewer administered questionnaire, thus it was desirable for the survey units to be located in a concentrated area. To achieve this, within each noise band (stratum) clusters were identified using postal code groups. A number of clusters were randomly selected within each stratum.

3 The population within each cluster was still too large to allow a census, so a systematic random sample, based on the electoral register, was drawn within each cluster.

Clearly a lot of effort is involved in determining a sample this way – and so far we have only decided which units to survey, we haven’t tried to find them and persuade them to respond to our solicitations. An alternative approach, which is widely adopted in practice is non-probability sampling.


1 In the UK, the electoral register is one of the most widely used sampling frames. The National Opinion Poll (NOP) use it in carrying out a weekly Random Omnibus Survey, one of the few surveys carried out by a polling organisation which uses probability-based rather than quota sampling. The results of this survey are published in NOP Review – any client can buy space on the survey at a fixed rate per question.

What are the desirable characteristics of a sampling frame?

Does the electoral register satisfy these?

What other sampling frames might be used for such a survey? Comment on their advantages and disadvantages.

2 What is a sample frame? – You have been asked to design a survey, using probability sampling, to investigate attitudes of university employees and students to extended teaching hours (eg evening and Saturday lectures).

Briefly discuss what you might use as a sample frame for this study, and comment on any perceived inadequacies of that sample frame.

3 What is the benefit of cluster sampling? – Briefly describe how you might design a cluster sample to investigate the take-up of satellite TV in the central belt of Scotland.

4 What is a sample frame? – Suggest appropriate sample frames for carrying out a statistical survey on each of the following topics, giving a brief justification for each:

public opinion on the postal service

attitudes of a company’s employees to the introduction of a no-smoking policy

a study of the impact of information technology on the management accounting profession.

5 Give a brief explanation, in general terms, of:

a sample frame

circumstances in which you would need to use a sample frame

desirable characteristics of a sample frame.

The publishers of a popular management magazine (such as Management Today) are planning a survey to obtain information on their readers. They are proposing to use the list of subscribers for 2000 (the most up-to-date information which they have access to) as their sample frame. Comment on this in the light of your responses to the above.


Non-probability sampling

The people with clipboards wandering up and down the High Street we mentioned earlier will be engaged in non-probability sampling. This is because, in contrast to the methods just discussed, it is impossible to determine the probability of selection for any individual unit and also impossible to ensure that all units have a non-zero chance of inclusion.

The most common methods of non-probability sampling (also referred to as non-random sampling, although intuitively it would seem that the opposite is the case) are as follows:

Convenience sampling

As the name suggests, this refers to sampling in any way that is convenient – for example, standing in an appropriate place and interviewing anyone who passes.

Quota sampling

This is more structured than convenience sampling, in that the investigator specifies quotas of people to be interviewed or units to be studied, in an attempt to ensure a representative sample. Quotas may specify, for example, the numbers of males and females to be interviewed, or that the sample includes a certain number of people in each age group, or a specified number of staff at different grades within an organisation. The illustration below shows a sample quota sheet.

There are a number of ways in which the quotas are determined. The simplest is to specify quotas which represent the distribution of the population, ie the sampling fraction is the same for all groups. For example, if the population is 49% female, then 49% of the sample should be female. Alternatively the quota size may reflect the importance of a particular group, the sampling fraction being higher for important groups, eg larger companies.


Judgement sampling

The investigator selects the objects of study on the basis of their ability to contribute. This is more appropriate in the case of small-scale, in-depth studies, when it is important to include knowledgeable and/or important people.

Pros and cons of different sampling strategies As the above discussion probably suggests, non-probability sampling methods are generally more straightforward. There is no need to find an appropriate sample frame, you are not faced with the problem of tracking down specific individuals, of persuading them to participate. A probability sample ensures an unbiased selection of individuals – unfortunately, it does not guarantee an unbiased response, as there will inevitably be selected individuals who do not respond, possibly for reasons which cause the findings to be biased. On the other hand, it is difficult to ensure a representative sample using non-probability methods – samples are subject to two types of bias:

Availability bias: some members of the population may never pass by the scene of the data collection at the time interviewers are there

Interviewer selection bias: interviewers tend to approach people they feel are more likely to be co-operative, although the major market research companies argue that they are skilled in avoiding this.

A major advantage of a probability sample is that it enables you to carry out analysis which gives an idea of how good a picture of the population is provided by the sample, drawing on the results of sampling theory. To be more specific, it enables you to predict characteristics of the population using information about the variability inherent in the sample.

You are now invited to consider the case studies in the light of the discussion of sampling strategies.

Spend some time now thinking about the case studies (on pages 4-6), reviewing your earlier notes in the light of what you have just read, and for each case answer the following questions.

What is the relevant population?

Is it necessary to sample?

If yes, would it be feasible to carry out a probability sample? What sample frame would you use?

If you were to opt for a non-probability sample, how would you obtain your sample?


Sampling methods So far we have only considered who is to be sampled, or the sampling strategy. We now go on to look at how, the sampling method.

The most widely used sampling method is the questionnaire. This may be distributed by post, fax or e-mail, and is wholly self-administered by the respondent. Alternatively, it may be administered by an interviewer, either face-to-face, or over the telephone. Or it may be somewhere between these two situations – for example, an investigator may hand the questionnaire to the respondent for self-completion, but remain there to answer questions if necessary and to collect the completed document.

There are clear advantages and disadvantages associated with the different methods. The self-administered questionnaire is generally cheaper and can be used to obtain larger samples than a questionnaire administered by interviewer. The interviewer administered questionnaire can, on the one hand, be viewed as a means of obtaining more in-depth information, but on the other, it can be viewed as introducing interviewer bias. Non-response rate tends to be higher for self-administered surveys. Telephone surveys have some of the advantages of face-to-face interviews:

the personal contact and opportunity to follow up the answers to questions

travel costs, which may be high if the sample is widely spread, are eliminated

responses are obtained more quickly than by postal survey.

However, there is the cost of the telephone bill to take into account! The approach which is best suited to any particular case is largely dependent on the resources available – it may be the case that plenty of interviewers are available, but cash is limited, meaning that a postal questionnaire is out of the question, but an inter-viewer-administered survey is feasible.

A well-designed questionnaire is clearly essential in any effective data-collection process. However, questionnaire design is very much an art; it is easy to criticise a badly designed questionnaire, but far more difficult than most people imagine to construct a good one. The following section gives some guidelines to good practice, but before moving on to that, spend some time on the following exercise.


You have been given a number of questionnaires which I have gathered over the years. (These can be found on the intranet under questionnaires.) Study these and for each questionnaire answer the following questions.

What do you think was the objective of the questionnaire?

Do you understand all the questions asked?

Would you be able to answer all the questions?

Is the questionnaire well laid out?

How easy/difficult do you think it would be to analyse the information gathered?

Are there any particularly good or bad features?

The last of the questionnaires, on the use of petrol, is an excellent example of how not to construct a questionnaire. I do not know its source. Study it – it is actually quite amusing – and make a list of your criticisms.


4 Questionnaire design Some guiding principles of questionnaire design are as follows:

respondents must be able to understand the question

they must be able to provide the information requested

they must be willing to provide the information requested.

Questions should:

be phrased in simple language

be economically worded

avoid jargon

avoid phrases or words which have different meanings to different groups; differences may also arise between nationalities, regions or socio-economic groups. A question which asked a Scot how many apartments he had might elicit a response of five or six (meaning rooms) – an answer which would give the average Belgian, to whom an apartment is a flat, the impression that the Scots are very well off. In the UK the same word is used to describe a different vegetable in different regions – in the North a turnip is a large orange vegetable, in the South it is a small white one. When I travel abroad I am never quite sure where the first floor in a building is – in the USA it is the level at which you enter the building (ground level), in the UK that’s called the ground floor, the first floor is the one above that. I’m sure you can think of many other potential sources of confusion

be well defined – ie avoid ‘Do you come here often?’. It is better to ask ‘How often do you come here?’ and perhaps offer specific response categories, such as, every day, once a week, and so on

avoid ambiguity – eg ‘Do you often enjoy take-away pizzas?’ (Is the researcher trying to find out how often you buy take-away pizzas or if you enjoy them when you do buy them?)


avoid leading questions – eg ‘You do exercise regularly, don’t you?’

avoid double negatives – eg ‘Is it not true that you did not vote in the last general election?’ Better to ask simply ‘Did you vote in the last general election?’

avoid embarrassing questions, or if necessary leave to the end of the questionnaire (if faced with such questions early on the respondent may refuse to continue – this is particularly relevant in the case of interviewer-administered questionnaires)

avoid hypothetical questions if possible – eg ‘If a drinks machine were available, how often would you use it?’

not place too much strain on the memory

avoid asking for others’ opinions (although this can be a way of tackling potentially embarrassing questions)

avoid assumptions – eg ‘What do you do for a living?’

Question sequence should be such that the co-operation of the respondent is gained early on. Take care to ensure that answers to early questions do not cause bias in responding to those later in the questionnaire. The ‘funnel technique’, beginning with general questions and working towards the more specific, can help avoid this. As mentioned above, place sensitive or embarrassing questions towards the end of the questionnaire to reduce the likelihood of losing the respondent’s cooperation early on. As a general rule. leave demographic/classification questions to the end of the questionnaire – unless using quota sampling. It may be helpful to reassure the respondent of confidentiality.

The Laurie Taylor article (see next page) is a light-hearted reflection on question wording.

Types of question

Open-ended questions – Example: ‘What are your views on the government’s transport policy?’

These have the benefit of allowing greater freedom of thought and are useful in the early stages of a study when trying to build a picture of the issue under study. However, the responses can be difficult and very time consuming to analyse. When used in interview surveys, inaccurate recording, interviewer intervention and attitude can lead to biased results.


Laurie Taylor article A survey at Southampton University has revealed that some academic staff were incapable of answering straight questions – THES 5 April 1991.

University of Poppleton Car Parking Questionnaire (1991/6/778/C)

Please answer all questions by placing a ring round your preferred answer.

Do you have a car? Yes/No

If you mean do I own a car then technically the answer is ‘No’, in that my wife and I are co-owners of a car. If, however, you are asking whether or not I drive a car (without any implications of ownership) then the answer would be ‘Yes’.

Do you regularly drive your car into work? Yes/No

If you are asking whether I drive my car in an habitual, steady, uniform, manner which could be said to conform to established driving practice rather than in an irregular, eccentric fashion (for example, by engaging third gear, immediately after selecting first or by indicating ‘right’ when turning ‘left’) then I would say ‘yes’. If, however, your reference is not to the regularity of my mode of driving but to my tendency to drive my car at ‘regular’ intervals of time, then my answer would obviously depend on what you mean by ‘regular’, upon how ‘regular’ is ‘regular’. I wouldn’t, therefore, at this stage of the questionnaire, like to commit myself one way or the other.

Thinking about parking over the last year, how would you describe your own experience of car parking arrangements at the university:

Satisfactory Fairly satisfactory Unsatisfactory

I’m afraid that your grammatically unsatisfactory formulation poses an impossible phenomenological task. In common I suspect with others among your respondents I have certainly not been ‘thinking’ about parking over the last year’, and I am not therefore able to cast myself into such a remembered frame of mind in order to proceed to the second experiental part of your question.

Taking everything into account, would you say that on the whole it was fair and reasonable that senior members of the administrative staff should have reserved parking spaces on campus? Yes/No

Now please turn over

If this is to be read as an acrobatic injunction then I am inclined to disobey; if however …

© The Times Supplements Limited, 1991

Closed questions – These can take a number of forms as you will have seen from the questionnaires (on the intranet) you studied earlier. For example: a simple yes/no answer (make sure the option of don’t know is

allowed if it is a reasonable response), eg National Breakdown Recovery Club Question 2, Greenpeace Question 2

a number of categories of response, eg National Breakdown Recovery Club Question 6, Dillons Question 11


a response according to a semantic scale (for example, 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree), eg Avis Question 6, Greenpeace Question 3, Institute of Manpower Studies G.

Closed questions have the advantage of being easier to analyse. However, careful piloting of the questionnaire is required to ensure that sufficient categories are provided for respondents to provide valid answers.

However careful one is in designing a questionnaire, unpredictable effects can occur, as the following story illustrates.

In 1975 a referendum was held to establish the UK population’s view on whether or not the country should remain a member of the EEC. Prior to the referendum NOP carried out some research which, amongst other things, looked at the effects of question phrasing on response. The following two question formulations were used to assess whether the UK should remain in the Common Market or withdraw:

Do you accept the Government’s recommendation that the UK should come out of the Common Market?

Do you accept the Government’s recommendation that the UK should stay in the Common Market?

Of the sample who were asked the first question the difference between those in favour of staying in the Common Market and those against was 0.2%, ie the two groups were almost equal in size. However, for the sample who were asked the second question the difference in percentages was 18.2%.

This example highlights the need to exercise caution in interpreting responses and the need to incorporate ways of cross-checking data into the design of a study.

Questionnaire layout

The appearance of a questionnaire is important. If it is to be self-administered, then it should be clearly laid out, but on as few pages as possible. Where filter questions are used, the respondent must be given clear instructions on which question to answer next.

If the questionnaire is administered by an interviewer, then it should ease the smooth flow of the interview.


5 Pilot studies and pre-pilot studies Pilot surveys can highlight problems with the sampling frame, for example, an interviewer may arrive at 2 Long Drive to interview Mr Small only to discover that 2 Long Drive is a 30 storey block of flats with no indication of which of 150 or so dwellings is occupied by Mr Small. They can also give an indication of non-response rate, and highlight any problems which may arise in the analysis of the data. A pilot analysis often reveals data which has been collected, but is not of any use, and other data which would be useful, which has not been collected.

Quality of data revisited Earlier in this chapter we discussed reliability and validity as aspects of the quality of data. Any approach to data gathering which relies on questioning is potentially unreliable. A person’s memory of an event may be inaccurate. Questions often require the respondent to assess quantities or frequencies to an accuracy far exceeding what would be normal for them. People may respond according to what they believe to be socially acceptable, or what they think the interviewer wants to hear rather than telling the truth. Such difficulties may be overcome to some extent by a well-designed questionnaire, but cannot be completely overcome. Examples of ways to check the reliability and validity are described below.

Parallel approaches – In large-scale studies it is common to collect data by more than one means, possibly a postal questionnaire backed up by a smaller-scale survey using an interview-administered questionnaire. The results obtained are checked against each other.

Secondary data or alternative sources – Sometimes it may be possible to use secondary data as a check on the data collected from individuals. For example, a city council is undertaking a study of the utilisation of ‘Green Cones’, a waste reduction system similar to a compost heap. It wants to assess the potential of this method for


reducing the volume of collected waste; this clearly depends on the extent to which households use the system. Data is being collected primarily from households involved in the study, but this will be compared with data on the total amount of waste collected from the area of the study.

Duplicate questions – In a complex questionnaire the same question may be asked more than once, but phrased in a different way, to act as a check on responses.

Measures of reliability Some questionnaires, such as the one to assess stress level associated with Case G (on page 35), use a large number of questions relating to the same issue to build up a score for, or a profile of, the respondent. Popular examples of such questionnaires can be found in many magazines – eg ‘Are you a hypochondriac?’, ‘Are you racist?’ etc. Designers of questionnaires of this nature, often psychologists, employ a number of methods for measuring reliability, outlined briefly below.

Test-retest method – The questionnaire is administered to the same group of subjects on two occasions and the results are compared. This method has obvious drawbacks: firstly, respondents may remember the answers they gave on the first occasion; secondly, the passage of time between the two events may have led to a genuine change of attitude.

Parallel forms technique – This involves the use of two questionnaires which can be considered parallel, ie measuring the same concept. Both forms are administered to a group of respondents and the results compared. This has the obvious difficulty of assessing whether or not the two forms are indeed parallel, together with the additional effort required to design two, rather than one, questionnaires.

Split-halves method – If all the elements of a questionnaire are measuring the same concept, then these can be split into two groups (eg odd and even questions). A score is computed for each subset of questions and these are compared as a test of reliability.

Such questionnaires may also include questions to catch out ‘dishonest’ respondents. The sense of questions seeking responses on a scale ‘Strongly agree’ to ‘Strongly disagree’ can be reversed to identify people who tend to agree with everything, no matter what. It is possible to identify people who are trying to respond in a socially acceptable manner rather than being honest by including questions


seeking a response to statements such as ‘I never tell lies’ and ‘I have never stolen anything’.

It is argued that strong agreement with such statements tends to reveal respondents who are seeking to present a socially desirable image.

If the investigator does not want to reveal the true purpose of the survey then questions completely unrelated to its purpose may be included to put the respondent off the scent, or the questionnaire may be deliberately designed to appear more general than it really is. For example, in the study to investigate the effects of aircraft noise at night on residents around a UK airport, which we discussed in the context of designing a multi-stage probability sample, the questionnaire addressed all forms of possible disturbance – road traffic, railways, neighbours, etc – and was introduced as part of a general study on sleep disturbance.

How large a sample A number of factors need to be taken into account in determining sample size.

Accuracy – Clearly, provided the process adopted is generating a sample which is representative of the population, the larger the sample, the better the picture we obtain. However, the gain is not directly proportional to the size of the sample. Sampling theory, which we will be studying in more depth in Chapter 8, tells us that to increase the accuracy of an estimate based on a sample by a factor of N, we must increase the sample size by a factor of N2 – for example, to double the accuracy, we must quadruple the sample size. So if you wanted to know the proportion of smokers in the population, an estimate based on a representative sample of size 1,000 would be accurate to within about ± 3%, one based on a sample of 10,000 to within ± 1%: ie accuracy is increased by a factor of around three by increasing the sample size by a factor of 10 (32 = 9).

Response rate – The only thing which can be certain about response rate is that it is unpredictable. Response rate is generally lower for postal surveys than for interview-based surveys. Action can be taken to boost response rate: reply-paid envelopes, incentives to respond and reminders are commonly used. The design and wording of the questionnaire and covering letter can have a significant impact. Respondents may feel that it is not worth returning a questionnaire if they are unable to answer all the questions – if a partially completed questionnaire is still of use, then that should be made clear. The offer of a copy of a report summarising the findings of the study may encourage response but would preclude anonymous response;


moreover, handling requests for these reports could lead to a substantial workload and expense. If carrying out an interview face-to-face, or by telephone, an introductory letter explaining the purpose of the survey, timed to arrive a few days before making contact, may increase response rate.

However, problems and surprises often arise. For example, contrary to expectations one postal survey on a subject which was topical and of central concern to the recipients achieved zero response, despite telephone reminders. In another survey a very low response rate (under 20%) was anticipated and consequently a large number of questionnaires were sent out. This survey was on behalf of a company which sold domestic central heating systems and wanted to find out from customers who had rejected their services why they did so. Unexpectedly, the response rate was 80%, giving the company substantially more data and work than it had bargained for.

Cost The size of sample which can be obtained may be simply constrained by the resources, both cost and time, available.


6 Analysis of survey results Responses to open-ended questions will have to be categorised by the researcher and possibly cognitive mapping used. A computer can take the answers of closed questions and show the resulting data in tables or, for example, graphs. Computer-based analysis can range from simple arithmetic manipulation of the data using a spreadsheet to complex multivariate analysis using a specialist statistical package.

Types of analysis will be discussed in detail in later chapters.

Commonly used computer packages are:

SPSS (Statistical Package for the Social Sciences): This is probably one of the most widely used of these packages. However, it is viewed with reserve by some statisticians because of its tendency to produce ‘an answer’ without offering guidance to the non-expert user in interpreting the output.

MINITAB: This is another widely used package. It was widely used in teaching and is now used by many organisations.

Review In this chapter we have explored the main issues relating to data collection. The key questions to be addressed in designing a data collection are summarised in the illustration below.

In determining the most appropriate design for a particular investigation we must also be aware of issues of validity and reliability. Inevitably we will be constrained by the resources available.

You may wish to read further, there is a wealth of work in this area and it is impossible to cover all the detail in this unit. Chisnall (1992) provides an excellent coverage of the topic and many references to specific studies and other texts.


To consolidate what you have learned in this chapter, work through the two activities. Then try questions 6, 7 and 8 which are examples of exam questions from previous years.

Review questions

Do you think a quota sample can ever be as representative of a population of interest as a probability based random sample?

Do you think market research panels and audience research panels are a good means of collecting reliable and accurate data? Comment on any problems you envisage.

Guidelines for conducting a survey

Imagine that you are employed by, or have decided to set yourself up in business as, an organisation specialising in survey research. On the basis of material presented in this chapter, and your own experience, draw up a set of guidelines for conducting a survey. These should provide a checklist, beginning with the definition of the purpose of the survey and ending at the stage at which data is collated ready for analysis.


6 The research department of a dairy products processing company for which you work has come up with, what to your mind, is a breakthrough – instant powdered cream that tastes like the real thing. The marketing department is reporting back on their hall tests. These involved renting a hall in Birmingham city centre and asking passers-by if they would be willing to take part in a test for five minutes for market research, for a fee of £5.00. The report found that the blind tests were quite favourable with regard to taste, but 98.5 per cent of respondents stated they would not buy the product instead of fresh cream. The marketing manager advises you to drop the project on the basis of these results. When you queried the composition of the sample, you were told it was composed of 60 men aged between 45 and 65 years.

(i) What concerns might you have over the validity of the results?

(ii) How would you approach the problem if you were able to devote a much larger market research budget to investigating the market for powdered instant cream?

7a Compare and contrast the interview and questionnaire method of obtaining information about attitudes and opinions. When would the questionnaire method be a more appropriate data collection technique?

7b Two consulting organisations have each developed a measure of ‘leadership and decision style’. You are considering using one or the other as a way of evaluating new MBA-qualified entrants to your organisation. What questions would you ask the consultants in order to evaluate the merits/demerits of each of the questionnaires? What characteristics should the ideal questionnaire show?

8 Sound and Vision, a company in the home entertainments market, is considering expanding its business to cover the production of videos for rental. It would like to obtain further data on the size and nature of the video rental business before committing itself. The company’s management have decided to carry out a survey of video rental stores in Scotland using probability-based (random) sampling methods.

You have been commissioned to advise them on the following:

the sampling strategy to adopt (eg simple random sampling, cluster, stratified, multi-stage); suggest a design that would be appropriate and convenient for their purposes


choice of a sample frame – include comments on the adequacy of your recommended sample frame for this purpose and any limitations you are aware of

possible sampling methods (eg questionnaire, face-to-face interview, telephone interview, etc) including comments on the appropriateness of each for this purpose

write a brief report outlining your suggestions and comment on any other important aspects of survey research they should pay attention to.

Documents

Data Collection methodology and process by SS KHAN