S519: Evaluation of Information Systems

Week 14: April 7, 2008

2

Announcement

Guest lecture next week Director Jonathan Plucker from the Center for

Evaluation and Education Policy

3

Data Analysis Report

Report the results and interpretation of the analysis of the data you collected last week

Explain what kind of analysis was conducted and why

The analysis may require statistical analysis or may involve qualitative methods, such as analytic induction or content analysis

Include the raw data set in an appendix Do not include all the tests you run, but do include

ones that you find important 2-3 single-spaced pages

4

Inferential Statistics

Descriptive statistics = summarize characteristics of sets of data & relationships between various sets of data

Inferential statistics = make estimates of how likely it is that such characteristics or relationships exist in the universe (generalization)

5

Inferential Statistics

Inferential statistics differ from descriptive statistics in two fundamental ways: They are usually concerned with the strength of

relationships (associations or differences) among two or more sets of measured variables

They have accompanying tests of statistical significance (See Salkind Ch 8) which indicate how likely it is that a particular relationship would occur by chance, given a certain sized sample

6

Relationships

Relationships are important to examine because: answering research questions to examine, e.g.,

relationships between independent variables and dependent variables

Suggesting new research hypotheses and/or Qs

7

Relationships

We are looking for answers to the two questions, when asking about associations and/or differences in sample data: How strong are the associations and/or

differences? How likely is it that these associations and/or

differences exist in the universe (population) from which the sample was drawn?

8

Statistically Significant

Statistically “significant” means that any difference between the attitudes of the two groups is due to some systematic influence and not due to chance

9

Significance Level

Significant findings occurred at the 0.05 level (p < .05) there is 1 chance in 20 (or 5%) that any differences found were not due to the hypothesized reason, but to some other

5% of allvalues

95% of all values

Do not reject null Reject null

Critical value

10

How Inference Works

1. Select representative samples from two different groups (e.g., mac users vs. pc users)

2. Each user is administered a survey to assess his/her attitude. The mean scores for groups are computed and compared using some tests

3. A conclusion is reached as to whether or not the difference between the scores is the result of chance or the result of “true” and statistically different differences between the groups

4. Based on the results of analysis of the sample data, an inference is made about the entire population

11

Relationships

12

Relationships: T-test

Difference???

Treat b

13

T-test: Tests between the Means of Different Groups

t =

(n -1)s +(n -1)s

n + n - 2 n n

n + n

X - X1

1

1

1 1

1

2

2

2

2

2

222

X is the mean for group 1

n is the # of participants in group 1

s is the variance for group 1

1

1

12

The difference between the means

The amount of variation within & between eachof the two groups

14

T-test: Tests between the Means of Different Groups Compare two groups: one using PDA and the other

using paper-based in a hospital These two populations do not overlap The variable is normally distributed in each of the two

populations The variances of the normally distributed test variable for

the two populations are equal

Use the “TTEST” function or Data Analysis ToolPak select “Two-Sample

Assuming Equal Variances”

15

T-test: Tests between the Means of Related Groups

Compare means of related groups E.g., pre- and post- IT implementation Comparison of means from each group Focus on the differences between the scores

t = D

nD - (D)

(n-1)

2 2

D is the sum of all the differences between groupsn is the # of pairs of observations

16

T-test: Tests between the Means of the Same Group

Use the “TTEST” function or

Data Analysis ToolPak select “Paired Two Sample for Means”

17

T-test: One Sample

Use it when you only have one sample Useful for comparing with a hypothetical mean or

previous evaluation score

Unfortunately, there is no easy way to calculate this, so if you want, you can use SPSS

18

T-test: One Sample

19

T-test: One Sample

20

T-test: One Sample

21

More Than Two Groups: Analysis of Variance (ANOVA)

Simple ANOVA One factor or one treatment variable (e.g. group

membership) being explored More than two groups within this factor

Simple ANOVA is used where: There is only one dimension or treatment There are more than 2 levels of the grouping factor One is looking at differences across groups in averages

scores

Excel: Data Analysis ToolPak select “Anova: Single Factor”

22

Determining Importance (Davidson, 2005)

Dimensional evaluation E.g., DeLone & McLean’s IS Effectiveness Model

Component evaluation E.g., policies, programs (e.g., Teen program in a

library) Holistic evaluation

Personnel, product, service

23

6 Strategies for Determining Importance (Davidson, 2005)

Have stakeholders or consumers “vote” on importance

Draw on the knowledge of selected stakeholders Use evidence from the literature Use specialist judgment Use evidence from the needs and values

assessments Use program theory and evidence of causal linkages

What are pros and cons of these strategies?

24

Basic Steps in Data Analysis (adapted from LeCompte & Preissle, 1993)

1. Review proposal and research question2. Scan

1. Check data for completeness [go back to the original informants]

2. Jot notes

3. Write summaries, e.g., research log, post-it4. Speculate5. Determine categories/ identify themes 6. Notice relationships7. Theorize and eliminate rival theories

(interpretations). Begin, often with typologies8. Try to confirm and to disconfirm interpretations