Fluency with Information Technology: From rubric to Assessment

From Rubric to Assessment

Fluency with Information Technology

2/18/2005

Gail Matthews-DeNatale, Ph.D.Senior Instructional Designer, Simmons College

Bruce Tis, Ph.D.Computer Science and Information TechnologyFIT Task Force Chair, Simmons College

A Liberal Arts Approach to FIT

TECHNOLOGY• Computer Science

• Telecommunications

OriginalFIT domain as

defined by NAS*

* National Academy of Sciences




INFORMATION LITERACY• Library Science

• Research

Yet more and more information is “born/found

digital”





• Research

MEDIA LITERACY• Semiotics / Linguistics

• Art / Media Studies• Communications

Another layer: understanding

multimedia





• Research



• Philosophy / Ethics• Cultural Studies

• Anthropology• Sociology • Education

… and much more





• Research



• Philosophy / Ethics• Cultural Studies

• Anthropology• Sociology • Education

Implications for Assessment

Need to assess three dimensions (technology, information, and media literacy)

No off-the-shelf assessment filled the bill

Composed questions and drew from:

IT Fluency: NAS / Washington State University Information Literacy: California State University Media Literacy: Media Awareness Network (CA)

The Baseline Assessment

Part I: Background Information Demographics Access to computers Experience with computers Technology skills (self-assessed, scale of 1-10)

Part II: Fluency (Developed out of Rubric) Emphasis on critical thinking, problem-solving,

and ethical reasoning

The Results

Ample Access at Home …

Mean = 2.5

SD = 1.5

How many computers in household?

How many computers in household?

876543210

Perc

ent

40%

30%

20%

10%

0% 35

11

25

29

25

… and at / for School

Do you have a computer with you at school?

Do you have a computer with you at school?

noyes

Perc

ent

120%

100%

80%

60%

40%

20%

0%

96

96% of incoming students own a computer that can be used for schoolwork

This includes those who live off campus

Type of internet access at home

Type of internet access at home

otherdon't knowDSLcabledial-up

Perc

ent

40%

30%

20%

10%

0%

12

2727

34

97% of students have internet access at home

How often do you check email?

How often check email?

several times a daydailyw eeklymonthly or less

Perc

ent

50%

40%

30%

20%

10%

0%

22

43

29

5

98% of incoming students had email before Simmons

Access and Familiarity ≠ Fluency

Despite considerable access to computers and the Internet, most incoming students are poorly prepared to live, learn, and work in a technology-rich,multimedia saturated society.

Why?

Social vs. Intellectual / Academic

Prior use of technologyPrior experience with technical problem-solvingHas been largely social and/or informal

Example:

Q: What do you do when you have a computer Problem?

A: 96% “Ask a family member or friend for help”

What’s wrong with that? Basic technical understanding

Example: How email works

Only 41% the students surveyed knew how email messages get sent

Lays a foundation for more sophisticated thinking Example: Distinguishing between valid and fraudulent email

Only 13% of the students surveyed knew how to best deal with a fraudulent email message requesting personal bank account information

Discrepancy Between Self-Assessment ...

Rate your technology skills

Rate your technology skills (1=Poor, 10=Excellent)

10.09.08.07.06.05.04.03.02.01.0

Perc

ent

30%

20%

10%

0%

13

18

21

23

12

65

Mean = 5.6

SD = 1.8

and Actual Insight, Understanding,

Example:

Students who believed they were highly proficient were less likely to say that there were "legal and ethical issues to consider" in:

Copying and pasting extended passages from a web site into a term paper; and

Buying software and letting your friends install it on their computers.

Ability, and …

Example:

Q: "You’re interested in career opportunities in nursing. What would you type into the Web search engine to find the most information?“

A: Only 11% selected the correct response

Awareness

Example:

33% of the students surveyed did not know that computer viruses spread through files on disk and on CDs

Their Greatest Difficulty?

Integrating Multiple Dimensions of Analysis

56

34

2926

0

10

20

30

40

50

60

Selected Fake Site (incorrect) Selected NSF (correct) Selected NSF for AppropriateReasons

Not Sure

You are doing background research for a science project. In addition to perusing library sources, you decide to go online to see what's available.Only 24% selected the most credible site for appropriate reasons

24%22%

49%

29%

Incoming Student Misconceptions

has a .org domain name (more credible than .gov)

has lots of “information” / links

has a “shorter,” “easy,” “straightforward” URL

has a “professional “look”

includes information on how to “send feedback” and/or contact the webmaster/designer

“comes from the United States”

“claims to be unbiased”

is “official,” or has a name that “sounds official” (such as “Centers” and “Organizations”)

has an online store

links to other “well-known organizations”

includes advertising

includes a search option

includes press releases

A web site is more likely to be credible if it

In a Nutshell

Incoming students need the most help with:

problem diagnosis and solving; media literacy (as consumers, researchers, and

authors); and grappling with ethical challenges.

In the same way that incoming students need to learn systematic processes for writing and research,

they also need to learn how to work, think, create, and problem-solve systematically with technology.

FIT Breakout GroupsDivide into groups (4-5 faculty members per group). Please take notes – you will report back to all. Each group has the FIT rubric, baseline assessment with explanations, and tabulated results.

Given the volume of information before you, you may decide that you need to focus on one issue or theme. That’s okay. Considering the rubric and assessment information, discuss the following questions:

What do you think are the most critical learning needs of first year students?

What are you already doing to address these learning needs? How could you be more explicit in connecting what you're already doing with the rubric?

What's one thing you'd like to add or change in your courses?