Reliability and Reliability Analysis (Business Research Methods).ppt

7/30/2019 Reliability and Reliability Analysis (Business Research Methods).ppt

1/30

Business Research Methods

Reliability & Reliability Analysis


2/30

Introduction The scores are the subject's responses to items on an instrument (e.g., a

mail questionnaire).

Observed scores may be broken down into two components: the truescore plus theerror score.

The error score, in turn, can be broken down into systematic error(non-random

error reflects some systematic bias, as due, for instance, to the methodologyused -- thus also called method error) and random error(due to random traitsof the subjects.

This is also called trait error.

The greater the error component in relation to the true score component, thelower the reliability, which is the ratio of the true score to the total (true + error)score.

Reliability is used to measure the extent to which an item, scale, orinstrument will yield the same score when administered in different times,locations, or populations, when the two administrations do not differ inrelevant variables.

Reliability coefficients are forms of correlation coefficients.

The forms of reliability below measure different dimensions of reliability andthus any or all might be used in a particular research project.


3/30

So Multiple-item or multiple-observation scales are often developed

to assess characteristics about individuals.

One important element in deciding the value of such a scale is itsreliability and validity.

A number of methods can establish a scales reliability includingtest-retest, equivalent-forms, and internal consistency estimatesof reliability.

With test-retest reliability, individuals are administered a measure ontwo occasions with some time interval between them.

Equivalent-forms estimates are based on a similar methodology,except an equivalent form is administered on the second occasionrather than the same measure.

For either of these methods, the easiest way to compute a reliabilitycoefficient is through the use of the Bivariate Correlation procedures.

In these cases, the reliability estimate is the correlation between thescores obtained on the two occasions.


4/30

With an internal consistency estimate of reliability,individuals are measured on a single occasion usinga scale with multiple parts.

The parts may be items on a paper-and-pencil measure,

responses to questions from structured interview, multipleobservations on an observational measure, or some otherunits of a measure that are summed to yield scale scores.

For ease of discussion, we will frequently refer to items ratherthan describing the analyses in terms of all types of parts.

The reliability procedure computes estimates of reliability

based on the consistency among the items (parts). Here, well look at two internal consistency estimates,

split-halfand

coefficient alpha.


5/30

Split-half estimates and coefficient alpha maybe used to estimate the reliability of the totalscore if a scale has multiple items and themultiple items are summed to obtain a totalscore. If a measure consists of multiple scales, separate

internal consistency estimates should be computedfor each scale score.

In some instances, you may need to transform oneor more items (or whatever the parts are) on ameasure prior to conducting the analyses so thatthe total score computed by the Reliabilityprocedure is meaningful.


6/30

Well look at two types of applications, which vary depending onwhether or how items are transformed:

No transformation of items. If the responses to these items are in thesame metric, and if high scores on them represent high scores on theunderlying construct, no transformations are required.

The Reliability Analysis procedure uses the untransformed item scores. Reverse-scoring of some item scores. This is the case when all items

on a measure use the same response scale, but high item scoresrepresent either high or low scores on the underlying construct.

Low item scores that represent high scores on the construct need to bereverse-scaled. Such items are commonly found on attitude scales.


7/30

Applying the Reliability Procedure No Transformation of Items

Sarah is interested in whether a measure she developed hasgood reliability. She has 83 students complete the 20-item

Emotional Expressiveness Measure (EEM). Ten of the itemsare summed to yield a Negative Emotions scale, and theother 10 items are summed to produce a Positive Emotionsscale. Sarahs SPSS data file contains 83 cases and 20items as variables. These 20 items are the variablesanalyzed using the Reliability program. She computes an

internal consistency estimate of reliability (split half orcoefficient alpha) for the Negative Emotions scale andanother internal consistency estimate for the PositiveEmotions scale.


8/30

Reverse-Scoring of Some Items Janet has developed a 10-item measure called the Emotional

Control Scale. She asks 50 individuals to respond to theseitems on a 0 to 4 scale, with 0 being completely disagree and4 being completely agree. Half the items are phrased so thatagreement indicates a desire to keep emotions under control(under control items), while the other half are written so thatagreement indicates a desire to express emotions openly(expression items). Janets SPSS data file contains 50 casesand 10 item scores for each. The expression items need to bereverse-scaled so that a response of 0 is transformed to a 4, a1 becomes a 3, a 2 stays a 2, a 3 becomes a 1, and a 4 is

transformed to a 0. The scores used by the Reliability Analysisprocedure contain the scores for the five under-control itemsand the transformed item scores for the five expression items.She computes an internal consistency estimate of reliability forthe 10-item Emotional Control scale.


9/30

Understanding Internal Consistency Estimates The coefficients for split-half reliability and alpha assess reliability based

on different types of consistency.

The split-half coefficient is obtained by computing scores for two halves ofa scale.

With SPSS, scores are computed for the first and second halves of the scale.

The value of the reliability coefficient is a function of the consistency betweenthe two halves.

In contrast, consistency with coefficient alpha is assessed among items. The greater the consistency in responses among items, the higher coefficient

alpha will be.

If items on a scale are ambiguous and require individuals to guess a lot ormake unreliable responses, there will be a lack of consistency between halves

or among items, and internal consistency estimates of reliability will be small. Both the split-half coefficient and coefficient alpha should range in value

between 0 and 1. Values close to 0 indicate that a measure has poor reliability, while values

close to 1 suggest that the measure is reliable.


10/30

Assumptions Underlying Internal ConsistencyReliability Procedures Assumption 1: The parts of the measure must be

equivalent For split-half coefficients, the partstwo halves of the

measuremust be equivalent.

With equivalent halves, individuals who score high on one half ofthe scale should score high on the other half of the scale, andindividuals who score low on one half of the scale should alsoscore low on the other half of the scale if the halves of the scalecontain no measurement error.

You can add the odd-numbered items together to create onehalf and add the even-numbered items to create the otherhalf.

You can then use these two halves to compute split-halfcoefficients.


11/30

For coefficient alpha, every item is assumed to be

equivalent to every other item.

All items should measure the same underlying dimension.

Differences in responses should occur only as a function of

measurement error.

It is unlikely that this assumption is ever met

completely, although with some measures it may be

met approximately.

To the extent that the equivalency assumption is

violated, internal consistency estimates tend tounderestimate reliability.


12/30

Assumption 2: Errors in measurement between parts areunrelated

A respondent's ability to guess well on one item or one part of a testshould not influence how well he or she guesses on another part.

The unrelated-errors assumption can be violated a number of ways.

Internal consistency estimates (split half or coefficient alpha) should notbe used if respondents' scores depend on whether they can complete thescale in an allotted time.

For example, coefficient alpha should not be used to assess thereliability of a 100-item math test to be completed in 10 minutesbecause the scores are partially a function of completing the test.

Second, sets of items on a scale are sometimes linked together.

Neither coefficient alpha nor split half measures should be used asa reliability estimate for these scales since items within a set arelikely to have correlated errors and yield overestimates of reliability.


13/30

Assumption 3: An item or half test score is a sum of its

true and its error scores

This assumption is necessary for an internal consistency

estimate to reflect accurately a scale's reliability.

It is difficult to know whether this assumption has beenviolated or not.


14/30

The example Data Set The data set used here contains the results of a survey of 50 respondents. Half the

items are phrased so that agreement indicates a desire to keep emotions undercontrol (under control items), and the other half are written so that agreementindicates a desire to express emotions openly (expression items).

Variable DefinitionItem 1 I keep my emotions under control.

Item 2 Under stress I remain calm.Item 3 I like to let people know how I am feeling.

Item 4 I express my emotions openly.

tem 5 It is a sign of weakness to show how one feels.

Item 6 Well-adjusted individuals are ones who are confident enough to express their true

emotions.

ltem7 Emotions get in the way of clear thinking.

Item 8 I let people see my emotions so that they know who I am.Item 9 If I am angry with people, I tell them in no uncertain terms that I am unhappy with

them.

Item 10 I try to get along with people and not create a big fuss.
http://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Reliability%20Analysis-1%20(Business%20Research%20Methods).savhttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Reliability%20Analysis-1%20(Business%20Research%20Methods).sav


15/30


16/30

The Research Question The research question can be phrased, "How

reliable is our 10-item measure of emotionalcontrol?

Conducting a Reliability Analysis Before conducting any internal consistency

estimates of reliability, we must determine if allitems use the same metric and whether any items

have to be reverse-scaled. All items share the same metric since the response

scale for all items is 0 to 4 (completely disagree tocompletely agree).

However, the five items in which high scoresindicate a willingness to express emotion must bereverse-scaled so that high scores on the totalscale reflect a high level of emotional control.

These items are 3,4, 6, 8, and 9.

You may want to peek at how to reverse-scaleitems for a Likert scale.

Here, I reverse-scale items 3, 4, 6, 8, and 9 beforegoing through the steps to compute coefficientalpha and split-half internal consistency estimates.


17/30

Computing Coefficient Alpha(1) Click Scale, then click Reliability

Analysis. You'll see the ReliabilityAnalysis dialog box.

(2) Hold down the Shift key, and clickitem1, and then click item9 to select all

the items.(3) Click to move them to the Items

box.

(4) Click Statistics. You'll seethe Reliability Analysis: Statistics dialogbox.

(5) Click Item, click Scale in theDescriptives for area, then clickCorrelations in the Inter-Item area.

(6) Click Continue. In the ReliabilityAnalysis dialog box, make sure thatAlpha is chosen in the box labeledModel.

(7) Click OK.


18/30

Selected SPSS Output for Coefficient Alpha As with any analysis, the descriptive statistics need to be checked to

confirm that the data

Have no major anomalies. For example, are all the means within the range of possible values (0 to 4)?

Are there any unusually large values of variances that might indicate that a

value has been mistyped?

In general, are the correlations among the variables positive? If not, shouldyou have reversed-scaled that item?

Once it appears that data have been entered and scaled appropriately, thereliability estimate of alpha can be interpreted.

The output reports two alphas, alpha and standardized item alpha.

In this example, we are interested in the alpha. The only time that we would be interested in the standardized alpha is if the

scale score is computed by summing item scores that have beenstandardized to have a uniform mean and standard deviation (such as z-scores).
http://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Output-Reliability%20Analysis-1%20(Business%20Research%20Methods).spohttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Output-Reliability%20Analysis-1%20(Business%20Research%20Methods).spo


19/30


20/30


21/30

Computing Split-Half Coefficient Estimates SPSS computes a split-half coefficient by evaluating the consistency in responding between the

first half and the second half of a measure. It is important to carefully choose which items toinclude in each half of a measure so that the two halves are as equivalent as possible. Differentitem splits may produce dramatically different results. The best split of the items is the one thatproduces equivalent halves (see Assumption 1).

For our example, we chose to split the test into two halves in the following fashion: Half 1: Item 1, Item 3, Item 5, Item 8, and Item 10

Half 2: Item 2, Item 4, Item 6, Item 7, and Item 9

We chose this split to take into account the ordering of items (with one exception, no twoadjacent items are included on the same half) as well as the two type of items, under controland expression items (2 items of one type and 3 of the other on a half).

To compute a split-half coefficient, follow these steps:

(1) Click Statistics, click Scale, then click Reliability Analysis.

(2) Click Reset to clear the dialog box.

(3) Hold down the cntl key, and click the variables that are in the first half: item!, item3, item5,item8, and item10.

(4) Click to move them to the Items box.

(5) Hold down the cntl key, and click on the variables that are in the second half: item2, item4,item6, item7, and item9.

(6) Click ~ to move them to the Items box in the Reliability Analysis dialog box.

(7) Click Statistics.

(8) Click Item and Scale in the Descriptives for area.

(9) Click Correlations in the Inter-Item area.

(10) Click Continue.

(11) Click Split-half in the drop-down menu in the Reliability Analysis dialog box.(12) Click OK.


22/30


23/30


24/30


25/30

Selected SPSS Output for Split-Half Reliability The descriptive statistics need to be checked to confirm that the

data have no anomalies as described in our earlier discussion ofcoefficient alpha.

The descriptive statistics associated with the split-half coefficient

are identical to the descriptives for coefficient alpha.

The most frequently reported split-half reliability estimate is theone based on the correlation between forms.

The correlation between forms is .78, but it is not the reliabilityestimate.

At best, it is the reliability of half the measure (because it is thecorrelation between two half-measures).

The Spearman-Brown corrected correlation, r= .87, is thereliability estimate.
http://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Output-Reliability%20Analysis-2%20(Business%20Research%20Methods).spohttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/Data-In_Class/Output-Reliability%20Analysis-2%20(Business%20Research%20Methods).spo


26/30


27/30

If there were an odd number of items, a split

would produce an unequal number of items in

each half.

Under these conditions, the value for theUnequal-length Spearman-Brown should be

reported because it will likely differ from the

Equal-length Spearman-Brown value.

APA-Style Results Section
http://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/APA-Style_Results_Sections/APA-Style_Reliability_(Business_Research_Methods).htmlhttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/APA-Style_Results_Sections/APA-Style_Reliability_(Business_Research_Methods).htmlhttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/APA-Style_Results_Sections/APA-Style_Reliability_(Business_Research_Methods).htmlhttp://faculty.pba.edu/callisto/courses/quantitative_methods/research_methods/APA-Style_Results_Sections/APA-Style_Reliability_(Business_Research_Methods).html


28/30


29/30


30/30

Documents

Reliability and Reliability Analysis (Business Research Methods).ppt