Instruments in Quantitative Research

Presented to: Dr.Muhammad Saeed

Presented by: Zahra Rasheed

Questionnaire

• After the careful formulation of research questions/hypothesis and sample selection next step in research chain is developing data collection instrument.

The two most commonly used research instruments in quantitative research studies include

• Questionnaire and

• Tests.

Validity and reliability of instruments:

• Validity is the degree to which an instrument measure what it is purports to measure. Invalid instruments can lead to erroneous research conclusions, which in turn can influence educational decisions.

• Reliability is the internal consistency or stability of the measuring device over time (Gay, 1996).

• A questionnaire is a research instrument consisting of a series of questions and other prompts for the purpose of gathering information from respondents. Questionnaire is widely used esp. in descriptive survey studies(Borg & Gall, 1983).

Advantages of Questionnaire

• Can reach a large number of people relatively easily and economically.

• Provide quantifiable answers.

• Relatively easy to analyze.

• Less time consuming than interview or observation.

(Bailey,1982)

Steps in Questionnaire Construction:

A. Reviewing the Literature

B. Deciding what information should be sought

C. Knowing respondents

D. Constructing Questionnaire Items

E. Reexamination and revision of the questions

F. Pretesting questionnaires

G. Editing the questionnaire and specifying procedure for use

A. Reviewing the Literature:• Before constructing the questionnaire the

researcher must review all the related literature to see if an already prepared questionnaire is available to similar his/her topic of study. This will save time and effort required to construct an entirely new questionnaire. Changes can be made as the study demands.

B. Deciding what information should be sought:

• List specific objectives to be achieved by the questionnaire.

• Methods of data analysis that will be applied to the retuned questionnaire should also be kept in mind

C. Knowing respondents:

• Researcher must know his target population in relation to occupation, special sensitivities, education, ethnicity language etc.

D. Constructing Questionnaire Items:• Each item on the questionnaire must be

developed to measure a specific aspect of objectives or hypothesis.

• Researcher must be able to explain in detail why a certain question is being asked and how the responses will be analyzed.

• Making “dummy tables” that show how item-by-item results of the study will be analyzed is a good idea(Borg & Gall, 1983).

Rules for constructing items

1.Both open as well as closed ended questions can be used however closed ended questions are preferred

2.Clarity of all the items is necessary to obtain valid results.

3.Short items are preferable to long items as they are easier to understand.

4.Negative items should be avoided as they are often misread by many respondents.

5. Avoid “double-barreled” items, which require the subject to respond to two separate ideas with a single answer.

6. Avoid using technical terms, jargons or big words that some respondents may not understand.

7. When a general and a related specific question are to be asked together it is preferable to ask general question first. Otherwise it will narrow the focus of the general question if specific question is asked first.

8. Avoid biased or leading questions. (Borg & Gall, 1983)

Question Formats

BasicOpen-ended

Questions

ProbingQuestions

ClarifyingQuestions

Open-endedQuestions

DichotomousQuestions

Single-coded

Multiple-coded

Multiple-ChoiceResponses

Closed-endedQuestions

LabeledQuestions

UnlabeledQuestions

ScaledQuestions

BasicQuestion Formats

Open ended questions: Basic open-ended questionsQ. What do you particularly like about Lipton Tea?_______________________________________________________________________ Probing Questions:What do you particularly like about Johnson & Johnson baby oil?____________________________________________________________________ PROBE: Anything else Close ended questions: A. Dichotomous questions with no neutral responseQ Do you have a cellular phone?1 Yes ...................................... without neutral2 No ...................................... response b. Dichotomous questions With Neutral ResponseIs it likely that you will purchase a cellular phone in the next six months?1 Yes ...................................... with neutral2 No ...................................... response3 Not Sure ..............................http://www.authorstream.com

• Single- and multi-coded multiple choice questionOn an average, how much do you spend on newspapers, books and

magazines in a month? (Please check one from the following responses.)

• 1 Less than $15 ................................... Single-coded

• 2 Between $16 & $30 ......................…. question• 3 Between $31 & $45 .......................…• 4 Between $46 & $60 .....................…..• 5 $60 or more .................................…..

Which of the following household appliances does your household have?

• (Please check as many responses that are applicable to you.)• 1TV ………… 5 VCR …… … Multi coded

question• 2 LCD …………. 6 Microwave ….…… • 3 PC ………… 7 Cellular phone………. • 4 Fax ………… 8 Others ……….• Specify

________________________________________________________________________________________________________________

Scaling questions

Scaling questions are special types of closed-ended questions. They include following categories of questions.

• Behavioral/Attitudinal questions• Agree-Disagree questions• Preference questions• Ranking questions

The questions can be labeled or unlabeled

Labeled and Unlabeled Scaling Questions:

Unlabeled scaled-response question• On a scale of 1 to 7 how would you rate the IBM Thinkpad on ease of operation?Advantages: • Allows respondents to express the degree of his/her intensity of feelings.• Easy to administer and codeDiasadvantage:• Respondent may not relate to the scale well.Labeled scaled-response question e.g “Do you disagree strongly, disagree, agree, or agree strongly with the statement,

‘IBM laptops are a better value than Compaq laptops’?”Advantages:• Allows a respondent to express the degree of his/her intensity of feelings. • Easy to administer and code. • Respondents can relate to the scale.Disadvantage:• Scale may be “forced” or overly detailed.

http://www.authorstream.com

Questionnaire format

Borg and Gall (1983) state following rules for questionnaire format:

• Attractive for the respondent. • Organize such that it’s easy to complete.• Number questionnaire items and pages.• Instructions should be brief, clear & in bold type.• Organize questions in a logical sequence.• Name and address of the person to whom questionnaire

is to be returned should be mentioned in the beginning as well as in the end.

• Use examples before any item that might be confusing or difficult to understand.

• Begin with a few interesting and non-threatening items.

• Do not put important items at the end of a long questionnaire

• Use of words like “questionnaire” and “checklist” should be avoided as some people might be prejudiced against these words.

• as short as possible. • for questionnaire items to be meaningful to

respondents enough information should be included.

E. Reexamination and revision of the questions:After questionnaire formulation revise it which involves:• supplementation of one’s effort by the critical opinion of

experts (should represent different approaches and belong to different social backgrounds)

• reviewing by representative of different groups such as minorities, racial groups, women etc.

• questionnaire should be scrutinized for any technical defects (Selltiz et al., 1976).

F. Pretesting questionnaires:

• According to Selltiz et al ., (1976) pretest helps in identifying and solving the unforeseen problems in administration of questionnaire such as phrasing, sequence of questions or its length, identifying the need for any additional questions or elimination of undesired ones.

• sample from the population used to pretest questionnaire should be similar in characteristics to those who will be included in actual study.

• Data collection technique should be same as planned for actual study.

• After making necessary changes second pretest should be conducted. Sometimes, in fact a series of three or four or even more revisions and pre-testing is required (Selltiz et al., 1976)

G. Editing the questionnaire and specifying procedure for use:

• A final editing by the research staff is done to ensure that every element passes inspection: the content, form and question sequence, spacing arrangement and appearance(Selltiz et al., 1976).

Tests as Research Instrument

• A means of measuring the knowledge, skill, feeling, intelligence or aptitude of an individual or group.

• Produce numerical scores that can be used to identify, classify or evaluate test takers(Gay,1996).

Types of tests

• Norm-reference and domain-reference test:Norm-reference test produces a score that tells how

individual’s performance compares with other individuals. Tables of norms based on scores obtained by relevant group of subjects tested by the test developer are provided by manual. Interpretations based on relative performance are very useful for most of the characteristics studied in behavioral sciences such as anxiety, creativity, dogmatism or racial attitudes. It describes performance, such as achievement, in relative terms.

Example: How does the overall achievement of student in class A compare to the students in class B? (Borg & Gall, 1983)

• Domain-reference test measures the learner’s absolute level of performance in a precisely defined content area or domain. It is being used increasingly to measure achievement –related performance. Domain reference tests estimate individual’s domain status i.e. precisely what is his level of performance and specific deficiencies in the domain covered by the test.

Example: what percentage of addition problems john can solve correctly? (Borg & Gall, 1983)

Individual versus group tests:• A group test is designed so that a sample of subjects

can take the test all at one time whereas individual test measures one individual at a time.

• Group tests are more of objective type. Individually administered measures are used when researcher is interested in studying the process rather than the product and should be used only if they make an important contribution to the research.

• Example: Tests developed in clinical settings such as Rorschach Inkbolt technique and Thematic Apperception Test often are individually administered so that clinicians can measure not only a subject’s response but also learn why subject gave a particular response.

Development of tests:

• Reviewing literature• Define objectives• Define target population • Review related measures • Develop an item pool • Prepare a prototype • Evaluate the prototype • Revise measure

Reviewing literature:• Before developing a new test review the available

literature in order to seek a test already available that can be used for the study as test development is an extremely complex process and require training.

Define objectives:• Give careful thought to the specific outcomes that

measure is to achieve. E.g. construction of achievement tests requires careful description of the knowledge or skill that the test should measure.

Define target population:• definition of target population is required as characteristics of target

population must be considered in many of the decisions on such questions as item type, reading level, test length and type of directions.

Review related measures:• A careful review of tests that measure similar characteristics provide

ideas on methods for establishing validity, application of different type of items, expected level of validity and reliability and possible formats.

Develop an item pool:• Before starting to write test items researcher needs to make

decisions regarding type of items that should be used and amount of emphasis that should be given to each aspect of characteristics or content area to be measured.

Prepare a prototype:• The first form of the test puts into effect the earlier decision made regarding

the format, item type etc. through tryouts.Evaluate the prototype:• Obtain a critical review by at least three or more experts in test construction.

This review identifies needed changes and the prototype is field-tested with a sample from target population.

• After collecting data item analysis is conducted to identify good and bad items. Analysis and interpretation depends upon nature of test. E.g. in developing a norm-referenced achievement test item analysis is usually concerned with difficulty level of each item and its ability to discriminate between good and poor students.

Revise measure:• On the basis of field test experience and results of item analysis prototype is

revised and again field tested. This cycle may be repeated several times in order to develop an effective test.Collect data on test reliability and validity.

Standardized test:

• Constructed by experts

• Individual test items revised and analyzed to meet standards of quality

• Directions for carrying out test available

• Objective

• Existence of validity and reliability test

Types of standardized test

Test Example

Intelligence test Wechsler Adult Intelligence Scale(suitable for testing late adolescents and adults)

Aptitude test Modern Language Aptitude Test

Achievement test Wide Range Achievement Test(in areas of reading, spelling and arithmetic)

Diagnostic test Stanford DIAGNOSTIC reading Test

Measures of Creativity Word Fluency(person writes word each containing a specified letter)

Projective techniques Holtzman Inkbolt Technique and Rorschach Test.

Measures of self concept Tennessee Self Concept Scale(include areas as self-criticism, physical self, personal self and social self)

Attitude scale Thurston Type Scale(individual expresses agreement or dis agreement with a series of statements about attitude object)

Measures of vocational interest Career Assessment Inventory

Self report measures of personality:

a. General inventories Minnesota Multiphasic Personality Inventory (most suitable in late adolescents and adults, measure variables as response sets, scales of ego strength, anxiety and repression-sensitization in addition to original scales e.g. depression, schizophrenia)

b. Specific inventories Rokeach Dogmatism Scale (designed to measure the variable of closed-mindedness, often used in educational and psychological research as a measurement of authoritarianism)

c. Checklist The Adjective Checklist (measure adjectives such as imaginative, stubborn, relaxed ) (Gay,1996; Borg & Gall, 1983)

Use of tests in research:

• Tests are used in co relational studies e.g. to determine relationship between quantitative ability and achievement in science among high school students, the researcher may decide the quantitative ability will be defined as scores School and College Ability Test Series III and science achievement will be defined as scores on science sections of sequential Test of Educational Progress (STEPIII).

• Experimental studies also use different tests such as Youth Program Quality Assessment, Kansas City Youth Net Standards etc.

References

• Bailey, K. D. (2nd Ed.). (1982). Methods of social research. London: Collier

• Macmillan Publishers.• Borg, W.R. & Gall, M .D. (4th Ed.). (1983). Educational research: an • introduction. NewYork: Longman Inc.• Gay, L. R. (5th Ed.).(1996). Educational research: competencies for

analysis and • application. New Jersey: Merrill• Selltiz, C., Wrightsman, L.S., & Cook, S.W. (3rd Ed.). (1976).Research

methods • in social relations..NewYork: Holt Rinehart & Winston.• http://www.authorstream.com/Presentation/aSGuest37105-315810-concept-

• questionnaire-research-education-ppt-powerpoint/