Information and Technology Literacy Institutional Learning ... · appropriate technologies to locate, access, select and manage the information. Examples of mastery of this ILO include,

1 | P a g e

Information and Technology Literacy

Institutional Learning Outcome (ILO) Evidence Team Report

Spring 2012

Team Members

Sandra Bierdzinski (Library) Team Leader

Carmen Montanez (Business Education) Note Taker

Leslie Mosson (Library) Report Writer

Alfredo Koch (Agribusiness)

Jennie Robertson (Institutional Research and Planning)

Wilma Sukrad (English)

2 | P a g e

PROJECT BACKGROUND In 2007 after a multi-year process with campus-wide participation, Allan Hancock College identified

seven institutional learning outcomes (ILOs) – outcomes that we expect students to develop through

experiences at AHC. The ILOs cover a variety of areas – the focus for this evidence team is information

and technology literacy. The knowledge, skills, abilities and attitudes incorporated in this ILO are

integrated in a variety of courses and student services at the

college.

As the college was exploring teaching and assessing student

progress in attaining ILO skills, a five-year U.S. Department of

Education grant (El colegio de aprendizaje: The Learning College)

was awarded to AHC (2007). One of the grant’s primary activities

was assessment of the seven institutional learning outcomes;

evidence teams were included to accomplish this goal. The

Learning Outcomes and Assessment Committee (LOAC) drafted an

institutional assessment plan providing for the collection and

analysis of ILO data by interdisciplinary evidence teams comprised

of faculty, administrators, students and other relevant staff. After

testing the evidence team process with the communication ILO

during spring 2011, the information and technology literacy ILO

was selected to be assessed during fall 2011-spring 2012.

This report provides a description of the process conducted by the

information and technology literacy ILO evidence team, the results

of their assessment and recommendations to ensure our students

successfully gain technology and information literacy skills.

THE INFORMATION AND TECHNOLOGY LITERACY ILO -- BACKGROUND The information and technology literacy ILO states that upon

graduation Allan Hancock College students will be able to “Define

what information is needed to solve a real‐life issue then use

appropriate technologies to locate, access, select and manage the

information.” As stated in “The Digital Divide: Information

Competency, Computer Literacy, and Community College

Proficiencies”1, “For over fifteen years, since the popular explosion

of the Internet in the early 1990’s, computers and online information resources have been evolving

from cutting-edge instructional enhancements into an essential aspect of lifelong learning and daily

1 Cox, Cathy. “The Digital Divide: Information Competency, Computer Literacy, and Community College Proficiencies.”

Academic Senate for California Community Colleges. March 2009. Web 18 Apr. 2012.

INFORMATION AND TECHNOLOGY

LITERACY

Define what information is needed

to solve a real‐life issue then use

appropriate technologies to locate,

access, select and manage the

information.

Examples of mastery of this ILO

include, but are not limited to:

* Use a computer to perform basic

functions appropriate to the

classroom and workplace.

* Select and use technology

appropriate for the task.

* Determine the nature and extent

of information needed.

* Locate, access, manage, and

evaluate information from multiple

sources.

* Use information ethically and

legally.

* Develop the ability to understand

the applications and implications

of technology in society

3 | P a g e

life.” To ensure students gain these “essential” skills, aspects of the information and technology ILO

are embedded in various courses and student services at the college. An indication of which courses

include information and technology literacy content can be seen in the mapping between courses

and ILOs – information available from the Institutional Research and Planning Office. This mapping

does not paint a complete picture since we were originally limited to the number of ILOs that can be

linked to a course student learning outcome (SLO). Several classes assessed by this evidence team do

not “officially” link to the information and technology ILO but do require these skills, at some level, in

the assignments evaluated by the team.

When this ILO was developed it was decided to combine technology and information literacy skills

into one ILO, a practice at some other colleges and universities and partially a result of the common

use of the term “information technology.” To fully understand this ILO, it is useful to look at the

backgrounds and expected competencies for both information and technology literacies.

Information Literacy

At a state and local level, colleges have struggled to decide how to ensure students gain information

literacy skills. In 2002 the CCC Board of Governors attempted to implement changes to Title 5 to

include information literacy as a community college graduation requirement and the Statewide

Academic Senate has passed resolutions in support of this change. But, as an “unfunded mandate,”

no statewide action has been taken. Some California community colleges have a graduation

requirement and a systematic method to ensure students gain the skills. Cal Poly SLO has chosen to

include information literacy in its curriculum as part of their university learning objectives for lifelong

learning: “Lifelong Learning (LLL) is comprised of the foundational attributes that enable graduates to

navigate a constantly changing world in which success is highly contingent upon the continuous

intake, evaluation, and deployment of information.”2 Information literacy skills are also included in

the model standards for the California state K-12 system although, as seen in the results of evidence

assessment, these skills seem to be taught inconsistently in the statewide system. Standard 3.1 of

the Model School Library Standards is “Demonstrate ethical, legal, and safe use of information in

print, media, and online resources” and Standard 3.3 is: “Use information and technology creatively

to answer a question, solve a problem, or enrich understanding.”

At Allan Hancock College there are a variety of methods for teaching information literacy. Many

classes have research requirements and the skills are gained in a variety of ways – through the LBRY

170 course, library “one-shot” orientations, students consulting with librarians, taught by classroom

instructors, or trial and error. While many classes across the disciplines include research papers or

projects, without a common definition or rubric the student learning outcomes and expectations for

these projects necessarily differ.

2 “Lifelong Learning.” Cal Poly San Luis Obispo. 12 May 2010. Web. 24 April 2012. <ulo.calpoly.edu/

facultyandstaff/lifelong_learning.html.

4 | P a g e

The college libraries specifically address information literacy. Their resources and services are guided

by student learning outcomes (SLOs) that address information literacy skills and that are based upon

the “Information Literacy Standards for Higher Education” developed by the Association of College

and Research Libraries (ACRL)3 (NOTE: the ILO rubric dimensions are also based upon these

standards). The AHC library SLOs are:

1. Navigate a physical or virtual library to find a variety of resources to answer information needs.

2. Use the research process to efficiently locate and use the most appropriate source of information for a problem or question.

3. Evaluate the quality of information sources for bias, currency, authority, accuracy, and coverage.

4. Use information ethically by citing sources, not representing work attributable to others as their own, and requesting permission for use of copyrighted materials.

A two-unit transferrable class, LBRY 170 (“Library Research Methods”) also includes student learning

outcomes based upon the ACRL standards:

1. Student defines an information need by formulating and stating a research question, problem, or issue in a manner that others in the discipline can readily understand.

2. Student assesses information resources and selects those that most efficiently and effectively meet their particular information need.

3. Student uses information technology tools and a variety of delivery mechanisms to locate and retrieve information relevant to their research question, problem, or issue.

4. Student evaluates the accuracy and bias of information in the context of an information need.

5. Student demonstrates an understanding of the ethical and legal issues surrounding using information and information technology.

6. Student produces a product that integrates a variety of timely and appropriate sources.

English 101 also has specific goals for information literacy. Several years ago an additional unit was

added to the course outline of record for English 101 and the class now includes the following course

goals:

Conduct research effectively including investigation, collection, evaluation, and documentation, and present the findings in acceptable written form.

Access and use information ethically and effectively.

Identify both discipline specific and other information technology resources. Technology Literacy

For years at the state and local level there has been a focus on technology. The California Community

Colleges Technology II Strategy Plan 2000-2005 (published in September 2000) discusses preparing 3 Association of College and Research Libraries. “Information Literacy Competency Standards for Higher Education.” Web

25 April 2012. www.ala.org/acrl/standards/informationliteracycompetency.

http://www.ala.org/acrl/standards/informationliteracycompetency

5 | P a g e

students for a “knowledge-based society” and the need for students to be able to use technology “to

be successful in their academic careers, as citizens and as workers in a knowledge-based society

(11).”4 Six years later the technological needs of students were a focus of attention in California

Community Colleges System Strategic Plan -- Education and the Economy: Shaping California’s Future

Today. Under the heading “Needs and Issues” the report stated:

The use of technology is a necessary skill that students will need to learn to stay competitive in the current academic and work environment, and a tool for providing access to college education both in remote areas of the state and to part-time or fulltime working students who are not able to fit into the regular college calendar. The California Community Colleges can increase the use of technology to help students succeed in their academic careers and ultimately participate in a knowledge based society. (26)

In addition to this focus at the community college level, the California K-12 system has focused on

technology and preparation of students for our technological world. In a recent communication, Cliff

Rudnick (an administrator in the Instructional Resources Unit for the California Department of

Education) discusses how technology is incorporated into K-12 curriculum:

In the K-12 environment there are no separate set of California standards, or a separate framework for technology as a subject, there are standards with technology components woven throughout other content areas, as well as technology standards contained within the Model School Library Standards. For example, in the California Common Core State Standards for English-language arts (ELA), adopted by the State Board of Education in August 2010, Standard 3. W 6 reads, “With guidance and support from adults, use technology to produce and publish writing (using keyboarding skills) as well as to interact and collaborate with others.”

Jose Ortega (an administrator at the Education Technology Office for the California Department of

Education) commented on K-12 assessment of these skills (personal communication, April 2012):

“absent legislative mandates and requirements, the state cannot require a statewide assessment of

those skills. Schools can however, and often do, measure those skills locally.” With a varying degree

of skill building and assessment in the K-12 environment, the community colleges face a challenge in

determining the level of technology literacy of entering students and developing curriculum to

prepare students for course work, the workplace, and transfer. Presently, at Allan Hancock College,

there is no pre-testing to determine technological competency. There is also no post-testing prior to

graduation to determine if students are competent with technology and are prepared for challenges

in the workplace or in transfer studies.

As with information literacy skills, technology skills are required in a variety of AHC courses although

the specific skills used in classes vary and the requirement may not always be clearly documented in

the course SLOs.

4 California Community Colleges Office of the Chancellor. California Community College Technology II Strategic Plan 2000-

2005. ERIC Document: ED 453 867. September 2000.

6 | P a g e

ASSESSMENT METHODOLOGY Introduction In an effort to capture the broadest picture of student skills, the evidence team examined both direct

and indirect evidence. Direct evidence includes student behaviors or products that demonstrate how

well students mastered the expected outcomes. Since we could not monitor behaviors, the direct

evidence used were 78 “products” students created for eight classes at AHC.

Indirect evidence includes “reported” (or “second hand”) perceptions or observations about student

mastery of the learning objectives. These observations can be collected in published reports, survey

results, and in a variety of other ways. The team looked at a wide variety of indirect evidence before

choosing evidence that was appropriate for this assessment.

Direct Evidence Assessment Since technology and information literacy are core skills across college disciplines, the team

determined that evidence could be gathered from a wide variety of courses. In fall 2011 evidence

team members contacted instructors in various disciplines and asked if they were willing to share

their students’ work. All courses selected were offered in fall 2011 and instructors were asked to

supply the assignment, grading rubric (if used), and the work from all the students in a class.

Instructors were reassured that student work would be anonymous; the institutional research office

redacted names from all student work before making it available to the team.

Course work was gathered from: AG 102 (Introduction to Viticulture), BIO 128 (Microbiology), CBIS

101 (Computer Concepts and Applications), CBOT 333 (Business Desktop Publishing), ENG 101 (2

sections-one in which the instructor brought the class to the library for an orientation and one that

did not), LBRY 170 (Library Research Methods), MUS 101 (Music History: Ancient-Baroque) and NURS

109 (Medical/Surgical At Risk Population). The assignments varied but all required some level of

information and technology literacy. The chart below illustrates the diversity of assignments that

were collected and assessed.

Artifacts Collected and Assessed

Course Assignment Number collected

Number assessed

AG 102 Vineyard Development Paper 20 9

BIO 128 Research paper on biofilms, quorum sensing or microbial symbiosis

48 10

CBIS 101 Formatted research paper and newsletter for “Memphis Primary Materials”

20 0

CBOT 333 Business Brochure 10 10

English 101(A) Research Paper 14 10

7 | P a g e

English 101(B) Argumentative Essay 21 9

LBRY 170 Research Pathfinder 16 10

MUS 101 Research Paper 21 10

NURS 109 Problem Solving Situation Case Study (works cited page only was included as an artifact)

41 10

Total Number of Artifacts 211 78

The first step in assessing the direct evidence for this ILO was to develop a rubric that could apply to

the variety of disciplines and assignments. In creating the rubric, the team began by determining

what students should be able to do by the end of their education at AHC. During several meetings the

team analyzed a variety of rubrics from other institutions and discussed several variations, finally

agreeing on a 4-point rubric with a rating scale that included the levels of “beginning,” “developing,”

“competent,” and “professional/advanced.” The ILO rubric focused on five dimensions:

The research question and/or thesis

Access and retrieval of information from a variety of sources

Critically evaluating information and sources

Using information and technology to create a final product

Accessing and using information ethically and legally

Rubrics used by the evidence team are in Appendix I. The appendix includes both the original and

modified rubrics used during the assessment process (the rubric change is described below).

Throughout the assessment it was important to ensure “inter-rater reliability,” the degree of

agreement between the various team members. If we saw discrepancies we knew that our inter-rater

reliability needed to be checked. The team first tested the rubric and our inter-rater reliability with

the Music 101 research assignment. We selected one student paper and assessed it as a group –

discussing why we rated each dimension the way we did. We then individually assessed a second

artifact and compared results. Ratings were very similar (within a point of each other) and we

discussed the differences. The team was especially conscientious when discussing a discrepancy

between “developing” and “competent” ratings since this is the difference between meeting and not

meeting the standard.

After testing the rubric, the team randomly selected 10 artifacts from each course (drawing numbers

to select the numbered artifacts). For CBOT 333 there were only 10 artifacts so we assessed them all.

CBIS 101 was eliminated as evidence since the assignment had specific formatting directions, involved

no research, and all student work looked the same. Since the assignment did not include the skills

rated in the rubric, it was inappropriate to evaluate the evidence from the course.

Early in the assessment process we realized that the rubric dimension “Uses information and

technology effectively to create a final product within the specifications of the assignment” rated two

distinct skills so we divided the dimension into two columns on the scoring sheet. There were

8 | P a g e

originally six criteria for this dimension. The first two became one rubric column and the last four

were scored in the second column. This change was made after MUS 101 was assessed and we did

not go back and reassess the two parts of the dimension – as a result the final results are duplicated

in the two columns for MUS 101.

A few artifacts could not be used. Two AG 102 artifacts were not from the correct semester and year

and another was a duplicate artifact. The team randomly selected replacement artifacts. For ENG

101-B, an artifact was eliminated because it appeared that pages were missing – pages that included

the works-cited page. An additional artifact was not substituted.

During the assessment process the evidence team was divided into two groups with three members

in each group (Group 1 included a librarian, research analyst, and agribusiness instructor; Group 2

included a librarian, English instructor, and CBOT/CBIS instructor). Each group assessed five artifacts

for each course. Assessment was conducted individually and then members of each group met to

agree on the final ratings for each artifact. In a continuing effort to “norm” the assessment, all six

evidence team members reviewed one common artifact for Music 101 and CBOT 333 before moving

on to assess the other artifacts individually.

Early in the process team members decided that to consistently use the rubric we needed to evaluate

artifacts as if they were “capstone” projects – projects assigned to students who were expected to

have learned the various skills embedded in the Information and Technology ILO. Even though these

artifacts are not all capstone projects this was the only way to assess where students are in gaining

the skills. It must be clear that the results are NOT an assessment of faculty and their teaching only

an assessment of the skills that their students currently exhibit based upon the rubric developed for

this ILO.

In summary, a total of 78 artifacts were assessed. The assignment instructions and student artifacts

are available upon request. See the summary of direct evidence assessment results, by course, in

Appendix II. In addition individual team member ratings (valuable for checking inter-rater reliability)

are in Appendix III.

Indirect Evidence Assessment The evidence team gathered a wide variety of indirect evidence. After analyzing the evidence, the

following were selected for inclusion in this study:

CBIS Computer Skills Survey Results - 2010

Community College Survey of Student Engagement (CCSSE) results – 2010

Community College Faculty Survey of Student Engagement (CCFSSE) results – 2010

EOPS Survey – Spring 2011

Library SLO Survey Results – Spring 2009

Registration Survey – Spring 2011

Student Help Desk Count Log – Fall 2011

Student Help Desk Monthly Count – March 2012

9 | P a g e

FINDINGS – DIRECT EVIDENCE The rubric ratings for all artifacts were submitted to the Office of Institutional Research for analysis.

Their report (compiled by Kelly Brune) is included in Appendix IV. This report includes assumptions,

methodology, results and conclusions. The report clearly shows student performance for each

dimension. The evidence team was pleased that this report was consistent with the team’s

impressions of the data collected. In reading the report it is important to note that the institutional

research analysis assumes the division of the fourth dimension into two dimensions – the first one

labeled analytical and the second one labeled technical (as discussed earlier) so the report includes

six dimensions instead of the original five.

As seen in the chart below, the results of the institutional research analysis indicate that students

from the various courses seem to perform similarly in the first four dimensions. Their performance in

the last two dimensions is also similar but ratings are lower than for the first four dimensions. The

overall data indicates a need for improvement in AHC’s student information and technology literacy

skills. Our target is that 70% of students are competent or professional/advanced and we did not

meet that target in any of the six dimensions. As shown in this table, at least half of the artifacts

were rated as ‘competent’ or better for the first, third, and fourth dimensions. The weakest areas

were the last two dimensions, with only about 25% of artifacts rated as “competent” or better.

Statistical Analysis from Institutional Researcher Report

Students competent or

professional/advanced Average Rating

Standard

deviation

Uses a clearly expressed

research question and/or

thesis to determine the extent


54.5% 2.51 0.78

Accesses and retrieves needed

information from a variety of

appropriate resources

42.3% 2.32 0.76

Critically evaluates

information and its sources 54.5% 2.43 0.75

Uses information and

technology effectively to create

a final product within the

specifications of the

assignment (analytical)

55.3% 2.51 0.75





assignment (technical)

24.7% 1.99 0.77

Accesses and uses information

ethically and legally 27.9% 2.07 0.78

10 | P a g e

The last two dimensions need more study to determine why ratings are so low -- student scores were

well below “competent.” In order to perform “competently,” the dimension “Uses information and

technology effectively to create a final product within the specifications of the assignment

(technical)” (restated by the institutional researcher when the original fourth dimension was split into

two dimensions) requires students to create their project with minimal grammatical and spelling

errors, a readable font, mostly identifiable headings and sections, and minimal format errors for their

document style. Faculty emphasis on these criteria varies and, if not an important part of the

instructor’s scoring, would not be important to the student.

Low scores on the dimension “accesses and uses information ethically and legally” does not

necessarily indicate a high level of cheating. The criteria for “competent” performance in this

dimension include: “most sources are documented properly,” “recognizes common knowledge vs.

ideas requiring attribution,” and “reflects a good understanding of plagiarism.” Students who

documented their sources incorrectly and had difficulty distinguishing between common knowledge

or words that required attribution did not score well on this dimension. Once again, if the activity

was not designed to measure the rubric criteria (and students did not know of these criteria), it is

perhaps unfair or unreasonable to assess the students’ competency.

The institutional research report includes other findings including a disparity of ratings from course to

course. The team has looked at “outlying” courses (especially CBOT 333 and MUS 101) to determine

if there may be reasons for these differences other than student competency. For example, CBOT

333 had low ratings in a couple of dimensions that can attributed, at least in part, to the nature of the

assignment – the assignment did not require everything that the rubric did. Some MUS 101 ratings

were low because the team rated the papers based on criteria for a works-cited page not for the

comprehensive bibliography that the instructor assigned. Although the team’s self-assessment

indicates there may be a variety of reasons for low scores, the institutional researcher indicated that

low ratings do not necessarily mean that there is a problem with the rubric or scoring. Sometimes it

means that students are not “competent” in the skills embedded in the ILO.

FINDINGS – INDIRECT EVIDENCE Information Literacy In spring 2009 the library conducted a student survey to assess progress on our library student

learning outcomes. We learned that students who come to the library find the information they need

most of the time (86%). Fifty-six percent have come to the library with a class for a formal library

orientation. In responding to questions about seeking information for class or personal use, students

indicated that they use books and “other internet sites such as Wikipedia” more often than library

databases (ProQuest, SIRS, AccessScience, etc.), newspapers, magazines and journals. But 60% of

students are able to access and use ProQuest, SIRS, AccessScience, etc. The remaining students are

probably either not required to use these college-level resources or are unaware of them. To see if

11 | P a g e

student library use and practices are changing, the evidence team recommends repeating this survey

within the semester or two.

In the 2010 CCSSE AHC students were asked two questions that focused on their experiences with

information literacy at AHC. When asked how often during the school year they were asked to “work

on a paper that requires integrating ideas or information from various sources” (key information

literacy skills including using a variety of sources and integrating the ideas of others to produce a

product), 62% of students responded that they practiced these skills “often” or “very often.” Thirty-

seven percent responded they “never” or “sometimes” practiced these skills. In the faculty version of

the survey (CCFSSE) faculty were asked how often these same skills were used in their classes and

responses were very similar. Fifty-eight percent responded “often” or “very often” and 41%

responded “never” or “sometimes.”

A second CCSSE question emphasized “synthesizing and organizing ideas, information, or experiences

in new ways.” Students were asked how much their “coursework at this college emphasized making

judgments about the value or soundness of information, arguments, or methods.” Forty-five percent

responded “quite a bit” or “very much.” Fifty-six percent responded “very little” or “some.” Faculty

seem to believe these skills are practiced more often – 62% responded “quite a bit” or “very much

(17% more than the students’ response) and only 38% responded “very little” or “some” (an 18%

difference from student responses).

Technology Indirect evidence provided information on the types of technology that students use, their

technology challenges, and faculty expectations on the use of technology by students. Students at

AHC use technology in a variety of ways – from emailing their instructors to registering for classes.

According to student help desk statistics, students have the most questions about registration –

indicating difficulty with this process.

In 2010, CBIS conducted a survey to determine the technology skills faculty consider important for

our students. Although 69% of faculty responded that “fundamental computer skills (file

management, word processing, spreadsheets, presentations, internet)” are definitely required for

success, 20% responded that although the skills were “generally helpful” students “could get by

without” them. Eight percent responded that “computer skills are not needed.” In this survey faculty

were also questioned about what skills they consider necessary for student success in class work, the

work place and for transfer. When asked what skills are vital for students to succeed in their

discipline only about 50% responded that the listed technological skills were necessary (skills included

“create/modify/delete computer files and folders,” “web navigation and searching”, etc.) Numbers

were slightly higher (averaging 53-54%) when asked about the necessity of these skills for transfer.

Some skills rated very high for success at AHC (email, web navigation and searching, etc.) but not

nearly so high for success for transfer or working in the discipline. The team found it interesting that

students are using technology skills for coursework that faculty seem to think have less application

for transfer or the workplace.

12 | P a g e

Perceptions of AHC’s emphasis on the use of technology vary between students and faculty. On the

Community College Survey of Student Engagement (CCSSE) and the Community College Faculty

Survey of Student Engagement (CCFSSE) conducted in 2010, 73% of students and 91% of faculty who

responded indicated that AHC emphasizes the use “quite a bit” or “very much” – a difference of 18%.

Responses are much more similar when asked about the college’s contribution to the student’s

knowledge, skills and personal development using computing and information technology. Fifty-six

percent of students and 59% of faculty acknowledge that the college’s contribution is “quite a bit” or

“very much.”

FINDINGS – TEAM ASSESSMENT At the conclusion of assessing the direct evidence, a survey of team members captured feedback on

both the findings and the process (see “Process Review” section later in this document for comments

on the process). When asked what surprised/shocked them most in reviewing the evidence, team

members commented on the lack of consistency across campus in research requirements and

expectations. For example, instructors use a variety of format and citation styles – making evaluation

across disciplines difficult. Another commented that although information and technology literacy

are an institutional learning outcome, we saw a lot of “minimal” proficiency and were surprised that

although you might expect that microbiology students would be more “content-focused” in their

research papers, student in English classes with information literacy course objectives were not

significantly more proficient in this ILO. In addition, team members noted the difficulty in assessing

technology skills through artifacts. This may be an area in which direct observation is more

appropriate.

RECOMMENDATIONS The Institutional Learning Outcome – Information and Technology Literacy

In assessing “technology competency” the team noted that institutionally we do not have a clear and up-to-date definition of what it means, in 2012, to be technologically competent. Since early discussions of this ILO seven years ago, technology has radically changed (social networking, etc.) and skills can now range from basic keyboarding and internet navigation to use of publishing software, student portal (MyHancock), learning management systems, and social networking tools. The institution needs to define “technology competency” so we can both ensure that students are ready for transfer and/or the workforce and we can assess the correct skills (do all students need “all” skills?). In addition, we may want to define technology competency so that skills required at AHC more closely mirror those required for transfer or in the work place.

The institution should consider the relationship between technology and information literacy – are they separate outcomes? Team members noted that with the growth in importance of our use of technology, it really seems to be an applicable outcome across all ILOs – not just

13 | P a g e

this one. The evidence team recommends that the information and technology ILO be split into two ILOs.

Assessing the ILO

The rubric should be revised before it can be used across the curriculum to evaluate the ILO. Before refining the rubric, follow up with faculty who provided artifacts to incorporate their input on the rubric and how it works for their assignments -- their impressions of our scoring would also be valuable. In response to the IRP report recommendation that it is “important to record the team member ratings and provide them for inter-rater reliability analysis,” complete an inter-rater reliability check before the rubric is revised. Finally, split the fourth dimension to clearly separate the two skill sets (information and technology) into two dimensions. The team suggests: “Uses information effectively to create a final product within the specifications of the assignment” and “Uses technology effectively to create a final product within the specifications of the assignment.”

For both technology and information literacy, consider some kind of entry-level pre-testing so that we have benchmarks and can mark progress as students proceed through their coursework. We need entry-levels if we are to measure progress. Subsequent post-testing could show progress.

Refine the process for selecting courses to assess. Considerations should include what course level students should be in for the rating (for example, what courses should have an expectation of “competent” work) and the prerequisites and advisories for the courses.

Another recommendation for course selection is included in the report from Dr. Brune: “One of the assumptions is that the courses included in the study are a representative sample of all the courses at AHC that are linked to the ILO being assessed. The goal of ILO analysis should be to include randomly selected courses linked to the ILO. If the LOAC decides that every course linked to a particular ILO should be included in the ILO analysis once every six years, perhaps randomly assigning numbers 1-6 to each course and then assessing all 1s in year 1, 2s in year 2, etc... Then, at the beginning of the next 6-year cycle, randomly assign 1-6 again so that the same courses aren’t always assessed together every cycle. This process will ensure that the assumption of random selection is met.”

Conduct the AHC Library User Survey (last conducted in spring 2009 and discussed in “Indirect Evidence” section of this report) to determine if users’ perceptions about the library student learning outcomes and information literacy have changed.

Include reports from eLumen as part of future ILO assessment since these reports reflect direct assessment by the course instructor. In addition, it is very important for course instructors to review and discuss the ILO rubric and perhaps adjust their assignment rubrics to incorporate information and technology literacy skills.

Gather more information from student services on their observations of student competencies and their role in teaching the ILO skills.

Teaching the ILO

14 | P a g e

Clarify our institutional expectations for information and technology literacies at various levels of the curriculum. For example, what proficiency level should be required in 100-level classes? We know what information literacy skills we want students to have when they graduate but are not systematic on when and where this skill-building takes place and how it is assessed. We do not seem to be clear on what technology skills to include in an ILO. Are these decisions that should be part of curriculum development and review?

Teaching information literacy needs to be integrated on a wider institutional level and requires more planning, coordination, cooperation and reflection. “One-shot” library instruction is probably not adequate to ensure students are gaining the necessary skills to find, evaluate, and use information to gain knowledge. In addition, classroom instruction is not consistent in incorporating and requiring these skills.

Develop and conduct professional development for teaching and service faculty and college staff on what information and technology literacies are and how to include these skills in the curriculum and student services.

Other Recommendations

Review course mapping to ILOs and allow more than one ILO to be linked to a course -- allowing more comprehensive mapping between ILOs and courses. This is an option on eLumen and, if implemented, would provide a more complete picture of where the ILO is taught.

To overcome potentially poor ratings because of differences between the assignment and the ILO rubric, encourage faculty to provide both rubrics with an assignment or redesign their rubrics to include the technology and information literacy skills.

PROCESS REVIEW After completing the artifact evaluation the evidence team was surveyed about the assessment

process and results. Team members were pleased with the variety of classes included and also that

faculty assign projects that include information and technology literacy skills.

Team members were asked about the process used to assess direct evidence. Participants

commented on how time intensive artifact assessment is – “especially since we were not subject

experts in the various fields and felt that, although important to the institution, this type of

assessment might not be sustainable.” Another participant said: “It was surprising…there isn’t some

mechanism in place that allows for an efficient collection of artifacts.”

When asked what changes team members would make to the process participants commented:

“Hopefully using eLumen in the future to capture the direct evidence directly from the faculty will help. If we can share the benefit of teaching to that rubric, the data collected would be pretty accurate.” Another commented: “eLumen data should be available to evidence teams. If eLumen data was available, the evidence team could

15 | P a g e

determine how faculty rated their students on the artifacts – probably useful information in the analysis.”

“Design the assessment process in such a way so that to the extent possible, departments are accountable for collection/provision of select(ed) artifacts that the evidence teams need for conducting the ILO assessment.”

“I thought it was very helpful to have an IRP staff member on the team.” CONCLUSION The evidence team’s assessment provides snapshots of our students’ current skill levels but in no way

reflects upon the instructors who assigned these projects. Since there is no overall institutional plan

for how students move from beginning to advanced levels, the skills are gained or not gained in a

haphazard manner. One of the evidence team’s overriding recommendations is that there needs to

be more dialogue about the information and technology literacy ILO on campus. The evidence team’s

work provides a baseline for these conversations and the team recommends development of an

institution-wide plan.

Information literacy skills are needed to create a high-quality research project and we have a clear

picture of what these skills are (accessing and evaluating information, using information ethically and

legally, etc.) but we often expect students to gain them on their own or through library one-time

orientations (“one shots”). Waiting for English 101 is often too late since students frequently take

classes requiring research before taking English 101 and, if students arrive in Eng 101 with little or no

familiarity with information literacy, they will probably not proceed to a competent level in one

semester.

As an institution we do not have a clear picture of the technology skills we expect our students to

possess upon graduation. Indirect and direct evidence do not give us a complete picture of student

(or faculty and staff) skills in technology literacy. The definition of mastery of technology skills is

vague so it is difficult to measure. When the ILO was developed we talked about word-processing,

PowerPoint and other skills related to earlier use of technology. But the world of technology has

changed since the ILO was approved in 2007 and various new skills undergird much of what our

students do. In addition, technological skills vary by discipline. We must clarify our expectations and,

perhaps, need to evaluate this crucial area separately from information literacy if we are to ensure

our students gain the skills they need for transfer, work and life.

APPENDIX I

Team Rubrics

(Original And Revised)

Information and Technology Literacy ILO Rubric (Original)

Rev. 01-27-2012

Beginning Developing Competent Professional/Advanced Uses a clearly expressed research question and /or thesis to determine the extent of information needed.

Unable to formulate a clear research question and/or thesis. Unclear understanding of the topic

Can partially formulate appropriate research question or thesis. demonstrating some understanding of topic.

Can formulate appropriate research question or thesis, showing a clear and focused understanding of topic.

Question is focused, clear and demonstrates complete understanding of the topic.

Accesses and retrieves needed information from a variety of appropriate resources

Sources used are inappropriate. There is no variety, limited number and/or not relevant to the research topic. Did not use appropriate information technologies to gather information

Some sources used are appropriate. There is some variety, limited number and some are relevant to the research topic. Uses a few appropriate information technologies to gather information

Most sources used are appropriate. There is a variety of sources, adequate number and the sources are relevant to the research topic. The sources satisfy the requirement. Uses appropriate information technologies to gather information.

All sources used are appropriate and exceeded expectations in variety, number and relevance to research topic. Exceeds expectations in the use of appropriate information technology to gather information.

Critically evaluates information and its sources

Shows no evidence of source evaluation for credibility, relevance, accuracy, timeliness, bias, coverage and context. Little or no information matches project criteria.

An attempt has been made in the evaluation of sources for credibility, relevance, accuracy, timeliness, bias, coverage and context; however, significant errors in judgment are present. Some information matches project criteria.

The sources have been evaluated for credibility, relevance, accuracy, timeliness, bias and context with minimal errors in judgment. Information matches project criteria.

The information has been thoroughly analyzed, conclusions have been drawn; main ideas have been synthesized in the most effective manner. Information exceeds project criteria.

Uses information and technology effectively to create a final product within the specifications of the assignment.

No attempt at analyzing/ drawing conclusions based on the information gathered; minimal/ ineffective attempt at synthesizing the information. There is absolutely no organization in the presentation of the information, data, etc.

There is minimal attempt at

analyzing/drawing conclusions

based on the information

gathered; good but ineffective

attempt at synthesizing the

information.

Student had only a basic and sometimes incomplete understanding of appropriate

The information has been analyzed, conclusions have been drawn; main ideas have been synthesized I a fairly effective manner. Student addressed well purpose of study and had a clear understanding of key words

The information has been thoroughly analyzed, conclusions have been drawn; main ideas have been synthesized in the most effective manner. Student extensively and perfectly addresses the purpose of study. Student had excellent

Information and Technology Literacy ILO Rubric (Original)

Rev. 01-27-2012

Beginning Developing Competent Professional/Advanced and the main purpose or objective in the study is vaguely addressed. Numerous grammar and spelling errors. Hard to read fonts Difficulties identifying headings and sections Does not use the correct format for document style (MLA/APA)

keywords and ideas, and these were not completely clear. Some grammar and spelling errors. Slightly difficult to read font Headings and sections somewhat identifiable Some errors on format for document style (MLA/APA)

and ideas. Minimal grammar and spelling errors. Readable font Headings and sections mostly identifiable Minimal errors on format for document style (MLA/APA)

understanding of appropriate keywords and ideas and used them properly to convey specific purpose.

Grammar and spelling accurate. Easily readable font Headings and sections clearly identifiable. Follows the correct format for document style.

Accesses and uses information ethically and legally

Most sources are not or improperly documented. Confuses common knowledge vs. ideas requiring attribution. Doesn’t reflect a clear understanding of plagiarism including copying or improper use of paraphrasing.

Some sources are not documented or documented improperly. Several citations were missing data. Sometimes recognizes common knowledge vs. ideas requiring attribution Beginning to reflect an understanding of plagiarism including copying or improper use of paraphrasing.

Most sources are documented properly. Recognizes common knowledge vs. ideas requiring attribution. Reflects a good understanding of plagiarism.

Excellent use of bibliography in which all resources are annotated and correctly cited in MLA/APA format. Always recognizes common knowledge vs. ideas requiring attribution Clearly reflects an understanding of plagiarism.

Information and Technology Literacy ILO Rubric (Revised)

Rev. 02-17-2012

Beginning Developing Competent Professional/Advanced Uses a clearly expressed research question and /or thesis to determine the extent of information needed.

Unable to formulate a clear research question and/or thesis. Unclear understanding of the topic

Can partially formulate appropriate research question or thesis. demonstrating some understanding of topic.

Can formulate appropriate research question or thesis, showing a clear and focused understanding of topic.

Question is focused, clear and demonstrates complete understanding of the topic.

Accesses and retrieves needed information from a variety of appropriate resources

Sources used are inappropriate. There is no variety, limited number and/or not relevant to the research topic. Did not use appropriate information technologies to gather information

Some sources used are appropriate. There is some variety, limited number and some are relevant to the research topic. Uses a few appropriate information technologies to gather information

Most sources used are appropriate. There is a variety of sources, adequate number and the sources are relevant to the research topic. The sources satisfy the requirement. Uses appropriate information technologies to gather information.

All sources used are appropriate and exceeded expectations in variety, number and relevance to research topic. Exceeds expectations in the use of appropriate information technology to gather information.


Shows no evidence of source evaluation for credibility, relevance, accuracy, timeliness, bias, coverage and context. Little or no information matches project criteria.

An attempt has been made in the evaluation of sources for credibility, relevance, accuracy, timeliness, bias, coverage and context; however, significant errors in judgment are present. Some information matches project criteria.

The sources have been evaluated for credibility, relevance, accuracy, timeliness, bias and context with minimal errors in judgment. Information matches project criteria.

The information has been thoroughly analyzed, conclusions have been drawn; main ideas have been synthesized in the most effective manner. Information exceeds project criteria.

Uses information and technology effectively to create a final product within the specifications of the assignment.

No attempt at analyzing/ drawing conclusions based on the information gathered; minimal/ ineffective attempt at synthesizing the information. There is absolutely no organization in the presentation of the information, data, etc.

There is minimal attempt at

analyzing/drawing conclusions

based on the information

gathered; good but ineffective

attempt at synthesizing the

information.

Student had only a basic and sometimes incomplete understanding of appropriate

The information has been analyzed, conclusions have been drawn; main ideas have been synthesized I a fairly effective manner. Student addressed well purpose of study and had a clear understanding of key words

The information has been thoroughly analyzed, conclusions have been drawn; main ideas have been synthesized in the most effective manner. Student extensively and perfectly addresses the purpose of study. Student had excellent

Information and Technology Literacy ILO Rubric (Revised)

Rev. 02-17-2012

Beginning Developing Competent Professional/Advanced and the main purpose or objective in the study is vaguely addressed.

keywords and ideas, and these were not completely clear.

and ideas.

understanding of appropriate keywords and ideas and used them properly to convey specific purpose.

Numerous grammar and spelling errors. Hard to read fonts Difficulties identifying headings and sections Does not use the correct format for document style (MLA/APA)

Some grammar and spelling errors. Slightly difficult to read font Headings and sections somewhat identifiable Some errors on format for document style (MLA/APA)

Minimal grammar and spelling errors. Readable font Headings and sections mostly identifiable Minimal errors on format for document style (MLA/APA)

Grammar and spelling accurate. Easily readable font Headings and sections clearly identifiable. Follows the correct format for document style.

Accesses and uses information ethically and legally

Most sources are not or improperly documented. Confuses common knowledge vs. ideas requiring attribution. Doesn’t reflect a clear understanding of plagiarism including copying or improper use of paraphrasing.

Some sources are not documented or documented improperly. Several citations were missing data. Sometimes recognizes common knowledge vs. ideas requiring attribution Beginning to reflect an understanding of plagiarism including copying or improper use of paraphrasing.

Most sources are documented properly. Recognizes common knowledge vs. ideas requiring attribution. Reflects a good understanding of plagiarism.

Excellent use of bibliography in which all resources are annotated and correctly cited in MLA/APA format. Always recognizes common knowledge vs. ideas requiring attribution Clearly reflects an understanding of plagiarism.

APPENDIX II

Summary of Direct Evidence

Assessment Results

Scoring (See rubric for dimension criteria): 1 = Beginning; 2 = Developing; 3 = Competent; 4 = Professional/Advanced

INFORMATION / TECHNOLOGY LITERACY ILO ASSESSMENT RESULTS

Course being Assessed: Agribusiness 102 (Intro to Viticulture)

Artifact #

Uses a clearly expressed research question and /or thesis to determine the extent of information needed.

Access and retrieves needed information from a variety of appropriate resources


Uses information and technology effectively to create a final product within the specifications of the assignment

Accesses and uses information ethically

4 3 3 3 3 3 2

5 2 2 2 3 2 2

7 3 1 1 2 2 1

8 2 2 2 2 2 2

9 3 2 1 2 2 1

10 4 3 3 4 3 3

12 3 2 3 3 1 3

13 3 2 3 3 1 1

20 2 1 1 1 1 1

Comments: Artifact #2 was not used – it was not an example of student work from this semester



Course being Assessed: Biology 128 (Microbiology)

Artifact #






3 2 2 2 2 3 2

7 3 3 3 3 2 3

13 3 3 3 3 3 3

16 3 2 3 3 1 1

19 3 3 3 2 2 2

20 4 3 3 3 2 3

38 2 3 3 3 2 2

39 3 2 3 3 2 2

40 3 3 3 4 3 2

43 2 1 2 3 3 1

Comments:



Course being Assessed: Computer Business Office Technology 333 (Intro to Word Processing)

Artifact #






1 2 2 2 2 1 2

2 3 3 3 3 2 2

3 2 2 2 2 1 2

4 1 2 1 1 1 2

5 2 2 2 2 1 2

6 3 3 3 3 1 3

7 3 3 3 3 2 2

8 1 2 2 2 1 2

9 1 1 2 2 1 2

10 1 2 1 – 2 1 - 2 1 - 2 2

Comments: All team members scored #10. Where 2 scores are shown, the subgroups had minor differences, however, all thought the work

was below standard.



Course being Assessed: English 101-A (Freshman Comp)

Artifact #






1 3 3 3 2 2 2

2 3 3 2 3 2 3

3 2 3 2 3 2 2

4 2 2 2 3 2 2

5 4 3 3 3 3 3

9 3 3 3 3 3 3

10 2 1 1 1 1 2

11 2 2 3 3 2 2

12 3 2 3 3 2 3

14 3 1 3 3 2 3

Comments:



Course being Assessed: English 101-B (Freshman Comp)

Artifact #






1 2 2 2 2 2 2

2 3 2 2 3 2 1

5 2 2 3 3 2 2

6 2 2 3 3 2 2

8 4 3 3 3 3 3

9 1 1 1 1 1 1

13 2 1 1 2 1 2

16 N/A N/A N/A N/A `N/A N/A

20 2 2 3 2 1 2

21 1 3 2 2 1 1

Comments: Artifact #16 was not used since it was missing the works cited page. There was evidence of citation in the paper but the last page

could not be found. The team did not want to review an incomplete specimen.



Course being Assessed: Library 170 (Library Research Skills)

Artifact #






1 2 2 2 3 2 2

2 3 3 3 2 3 2

3 3 3 3 3 2 3

5 3 3 3 2 2 2

8 3 4 4 3 3 4

11 2 1 1 1 1 1

12 3 3 3 3 3 1

13 3 2 2 2 2 1

14 4 4 4 4 4 4

15 3 3 3 3 2 1

Comments:



Course being Assessed: Music 101 (Music history)

Artifact #




Uses information and technology effectively to create a final product within the specifications of the assignment **


2 3 3 3 2 2 3

8 2 2 2 2 2 1

9 2 3 2 2 2 2

11 3 4 3 4 4 3

12 2 2 2 2 2 2

16 3 2 2 3 3 3

18 3 2 3 2 2 3

19 1 1 1 1 1 1

20 2 2 3 2 2 1

21 3 2 3 3 3 2

Comments: **This criteria wasn’t separated into two rating parts until after this course was reviewed, therefore the same score is used for

each part.



Course being Assessed: Nursing 109 (Medical/Surgical at Risk Populations)

Artifact #






1 N/A 2 2 N/A 2 N/A

9 N/A 1 1 N/A 1 N/A

15 N/A 2 2 N/A 1 N/A

21 N/A 2 2 N/A 2 N/A

22 N/A 3 3 N/A 1 N/A

27 N/A 2 2 N/A 3 N/A

28 N/A 3 3 N/A 2 N/A

33 N/A 3 3 N/A 3 N/A

34 N/A 3 3 N/A 3 N/A

39 N/A 3 3 N/A 2 N/A

Comments:

APPENDIX III

Artifact Ratings by

Individual Team Members



Team Member Name: Leslie, Jennie, Alfredo

Course being Assessed: Ag Bus 102

Artifact #

Scorer Uses a clearly expressed research question and /or thesis to determine the extent of information needed.





4 LM 4 2 3 4 4 3

JR 2 2 1 2 2 1

AK 2 3 3 3 3 3

5 LM 3 2 2 3 3 2

JR 1 1 2 2 1 1

AK 2 2 2 2 2 2

7 LM 3 1 1 3 3 2

JR 3 1 1 3 2 1

AK 2 2 1 2 2 1

8 LM 3 2 2 3 2 2

JR 2 2 1 2 1 1

AK 1 2 1 2 2 2

2 (not used)

LM

JR

AK Comments:



Team Member Name: Wilma, Sandra, Carmen

Course being Assessed: Ag Bus 102

Artifact #






9 WS 3 1 2 1 2 1

SB 3 2 1 2 2-3 1

CM 3 2 1 2 2-3 1

10 WS 4 4 3 4 2 4

SB 4 3 3 4 3 2

CM 4 3 3 4 3 2

12 WS 3 2 3-2 3 1 3

SB 3 2-3 2-3 3 3 3

CM 3 2 2-3 3 1 3

13 WS 3 2 3 2 1 1

SB 3 2 3 3 1 2

CM 3 2 3 3 1 1

20 WS 2 1 1 1 1 1

SB 2 2 1 1-2 1 1

CM 2 1 1 1 1 1 Comments:




Course being Assessed: Bio 128

Artifact #






3 LM 2 2 2 2 2 2

JR 2 3 2 2 3 2

AK 3 2 1 3 2 3

20 LM 4 4 3 4 2 3

JR 4 3 3 3 3 3

AK 3 3 3 3 2 3

38 LM 3 3 3 3 2 2

JR 1 2 2 2 2 2

AK 2 3 3 3 2 3

39 LM 4 2 3 3 3 2

JR 2 3 2 2 1 2

AK 2 3 3 3 2 2

43 LM 3 1 2 3 3 3

JR 2 2 2 2 2 2

AK 3 3 3 3 3 3

Comments:




Course being Assessed: Bio 128

Artifact #






7 WS 4 2 3 4 3 4

SB 2 3 2 2-3 2-3 2-3

CM 2 3 2 2 2-3 2

13 WS 4 3 4 3 2 2

SB 3 3 3 3 3 3

CM 3 3 3 3 3 3

16 WS 4 3 3 3 1 1

SB 3 2 3 3 1 1

CM 3 2 3 3 1 1

19 WS 4 4 4 2 2 2

SB 2-3 2-3 3 3 2-3 1

CM 3 3 3 3 2-3 1

40 WS 3 4 4 4 3 2

SB 3 3 2-3 4 3 2

CM 3 2-3 3 3-4 2-3 2 Comments:




Course being Assessed: English 101 A

Artifact #






1 LM 3 3 3 2 2 2

JR 3 3 3 2 2 2

AK 3 3 3 2 2 2

2 LM 3 3 2 3 2 3

JR 3 3 2 2 2 3

AK 3 3 2 2 2 3

3 LM 2 3 2 3 2 2

JR 2 3 2 3 2 2

AK 2 3 2 3 2 2

4 LM 2 2 2 3 2 2

JR 2 2 2 3 2 2

AK 2 2 2 3 2 2

5 LM 4 3 3 3 3 3

JR 4 3 3 3 3 3

AK 4 3 3 3 3 3

Comments:



Team Member Name: Sandra, Wilma, Carmen

Course being Assessed: English 101 A

Artifact #






9 SB 3 3 3 3 3 3

WS 3 3 3 3 3 3

CM

10 SB 2 1 1 1 2 1

WS 2 1 1 1 1 2

CM

11 SB 3 2 3 3 2 2

WS 2 2 3 2 2 2

CM

12 SB 3 2 3 3 2 3

WS 3 2 3 3 2 2

CM

14 SB 4 2 3 3 3 3

WS 3 1 2 3 2 3

CM

Comments:




Course being Assessed: English 101 B

Artifact #






1 LM 2 2 2 2 2 2

JR 2 2 2 2 2 2

AK 2 2 2 2 2 2

2 LM 3 2 2 3 2 1

JR 3 2 2 3 2 1

AK 3 2 2 3 2 1

5 LM 2 3 3 3 2 3

JR 2 3 3 3 2 3

AK 2 3 3 3 2 3

6 LM 2 2 3 3 2 2

JR 2 2 3 3 2 2

AK 2 2 3 3 2 2

8 LM 4 3 3 3 3 3

JR 4 3 3 3 3 3

AK 4 3 3 3 3 3

Comments:



Team Member Name: Sandra, Wilma , Carmen

Course being Assessed: English 101 B

Artifact #






9 SB 1 1-2 1 2 1 1

WS 1 1 2 1 1 1

CM

13 SB 2 1 1 2 1 1

WS 2 1 1 1-2 1 2

CB

16 SB N/A N/A N/A N/A N/A N/A

WS

CB

20 SB 2 2 3 2-3 1 2

WS 2 2 3 2 1 1-2

CB

21 SB 1 2-3 2-3 2 1 2

WS 2 2-3 2 2 1 1

CB

Comments: Artifact 16 did not have a works cited page attached although it was evident that the student used sources. Team decided not to

use this artifact for scoring.




Course being Assessed: LIBR 170

Artifact #






1 LM 2 2 2 3 2 2

JR 2 2 2 3 2 2

AK 2 2 2 3 2 2

2 LM 3 3 3 2 3 3

JR 3 3 3 2 3 3

AK 3 3 3 2 3 3

3 LM 3 3 3 3 2 3

JR 3 3 3 3 2 3

AK 3 3 3 3 2 3

5 LM 3 3 3 2 2 2

JR 3 3 3 2 2 2

AK 3 3 3 2 2 2

8 LM 3 4 4 3 3 4

JR 3 4 4 3 3 4

AK 3 4 4 3 3 4

Comments:



Team Member Name: Sandra, Wilma, Carmen

Course being Assessed: LIBR 170

Artifact #






11 SB 2 1 1 1 1 1

WS 2 1-2 1 1 1 1

CM

12 SB 3 3 3 3 3 1

WS 3 3 3 3 3 1

CM

13 SB 3 1 2 2 2 1

WS 3 2 2 2 2 1

CM

14 SB 4 4 4 4 4 4

WS 4 3 3 4 4 1

CM

15 SB 3 3 3 3 2 1

WS 3 3 3 3 2 1

CM

Comments:




Course being Assessed: Music 101

Artifact #






16 WS 2 2 2 2 3

SB 3 2 2 2 3

CM 3 4 3 4 3

18 WS 3 2 3 2 3

SB 3 2 3 2 3

CM 4 3 3 3 2

19 WS 1 1 1 1 1

SB 1 1 1 1 1

CM 1 1 1 1 1

20 WS 2 2 3 2 1

SB 2 2 3 2 1

CM 3 3 2 3 3

21 WS 3 2 3 3 2

SB 3 2 3 3 2

CM 3 3 2 2 2 Comments:




Course being Assessed: Music 101

Artifact #






2 LM 3 3 3 2 3

JR 3 3 3 2 3

AK 3 3 3 2 3

8 LM 2 2 2 2 1

JR 2 2 2 2 1

AK 2 2 2 2 1

9 LM 3 3 2 2 3

JR 2 3 2 2 2

AK 2 3 2 3 3

11 LM 3 4 3 4 3

JR 3 4 3 4 3

AK 3 4 3 4 3

12 LM 2 2 2 2 2 2

JR 2 2 2 2 2 2

AK 2 2 2 2 2 2 Comments:

Jennie: : I realize I probably shouldn’t have used halves, but I found it hard to know about how to weigh certain pieces of the rubric…like they

may have done a portion of the rubric really well, but other pieces not so good…how do you rate that?




Course being Assessed: NUR 109

Artifact #






1 LM n/a 2 (1 non-textbook, little

variety 2 (age-see assign) n/a 2

(formatting

inconsist)

n/a

JR n/a 3 2 n/a 3 n/a

AK n/a 2 2 n/a 3 n/a

9 LM n/a 1 1 (needs more) n/a 1 n/a



15 LM n/a 2 3 n/a 2 n/a



21 LM n/a 2 (age, etc) 3 n/a 1 n/a



22 LM n/a 3 3 n/a 1 n/a






Course being Assessed: NUR 109

Artifact #






27 WS n/a 3 2 n/a 3 n/a

SB n/a 2 2 2 3 n/a

CM n/a 2 2 2 2 n/a

28 WS n/a 3 3 n/a 2 n/a

SB n/a 2 2-3 3 3 n/a

CM n/a 2 2 3 2 n/a

33 WS n/a 3 3 n/a 2 n/a

SB n/a 3 3 3 3 n/a

CM n/a 2 3 3 3 n/a

34 WS n/a 3 3 n/a 3 n/a

SB n/a 2-3 3 3 3 n/a

CM n/a 3 3 3 3 n/a

39 WS n/a 3 3 n/a 2 n/a

SB n/a 2 3 2-3 2 n/a

CM n/a 2 3 2 2 n/a Comments:

APPENDIX IV

Course Based Analysis of

Directly Collected Evidence

From: Kelly Brune, KB Statistical Consulting To: Information and Technology Literacy Evidence Team Date: May 8, 2012 RE: Analysis of Information and Technology Literacy ILO collected evidence

Course Based Analysis of Directly Collected Evidence

Background:

Direct evidence for the ILO “Information and Technology Literacy” was collected in 9 courses. To

assure a comparable assessment of the 9 different courses, 6 separate rubrics measuring 6 different

dimensions of Information and Technology Literacy were developed by the evidence team. For each

course, a random sample of 10 direct evidence artifacts (from 10 different students) was evaluated

(team members evaluated 9 direct evidence artifacts from AG 102 and ENG 101-B). The final scores

are combined in Table 1.

COURSE


research question and /or

thesis to determine the

extent of information

needed.

Access and retrieves

needed information from a

variety of appropriate

resources


information and its sources

(4) (3) (2) (1) (4) (3) (2) (1) (4) (3) (2) (1) AG 102 1 5 3 0 0 2 5 2 0 4 2 3

BIO 128 1 6 3 0 0 6 3 1 0 8 2 0

CBOT 333 0 3 3 4 0 3 6 1 0 3 5 1

ENG 101-A 1 5 4 0 0 5 3 2 0 6 3 1

ENG 101-B 1 1 5 2 0 2 5 2 0 4 3 2

LIB 170 1 7 2 0 2 5 2 1 2 5 2 1

MUS 101 0 5 4 1 1 2 6 1 0 5 4 1

NUR 109 N/A N/A N/A N/A 0 5 4 1 0 5 4 1

Total 5 32 24 7 3 30 34 11 2 40 25 10

COURSE


technology effectively to

create a final product within

the specifications of the



technology effectively to

create a final product

within the specifications of

the assignment (technical)

Accesses and uses

information ethically

(4) (3) (2) (1) (4) (3) (2) (1) (4) (3) (2) (1) AG 102 1 4 3 1 0 2 4 3 0 2 3 4

BIO 128 1 7 2 0 0 4 5 1 0 3 5 2

CBOT 333 0 3 5 1 0 0 2 7 0 1 9 0

ENG 101-A 0 8 1 1 0 2 7 1 0 5 5 0

ENG 101-B 0 4 4 1 0 1 4 4 0 1 5 3

LIB 170 1 5 3 1 1 3 5 1 2 1 3 4

MUS 101 1 2 6 1 1 2 6 1 0 4 3 3

NUR 109 N/A N/A N/A N/A 0 3 4 3 N/A N/A N/A N/A

Total 4 33 24 6 2 17 37 21 2 17 33 16

Table 1


The rubric scores were defined as follows

(1) – Beginning

(2) – Developing

(3) – Competent

(4) – Professional/Advanced.

If the dimension was not applicable to a specific artifact, a rating of ‘N/A’ was given.

Assumptions:

A random sample of students is not practical to analyze ILOs; therefore, cluster sampling is used in

this study with clusters being courses at AHC. All assumptions related to cluster sampling apply to

this study. Another assumption in the analysis of ILOs is that the courses included in the study are a

representative sample of all the courses at AHC that are linked to the ILO being assessed, in this case

Information and Technology Literacy. Also, it is assumed that the student artifacts were randomly

selected for assessment for each course and that this sample is representative of the population of

credit students at AHC.

Method of Analysis:

Since this analysis is the premier analysis of the Information and Technology Literacy ILO, analyses

that consider time as a variable are not needed, here. In the future, to assess how students are

progressing in Information and Technology Literacy over the years, it may be more appropriate to

consider analyses that assess performance over time. Given that the criteria for success identified by

the Title V project are that 70% of students are competent or professional/advanced in each

dimension of information and technology literacy, simple analyses of central tendency, frequencies,

and percentages provide adequate analyses. The artifact ratings are considered to be scaled,

meaning that the difference between 1 and 2 is the same as the difference between 2 and 3, so that

means and standard deviations may be calculated.


Figure 1

7

24

32

5

-10%

0%

10%

20% 30%

40%

50%

60%

Beginning Developing Competent Professional/ Advanced

Uses a clearly expressed research question and /or thesis to determine the

extent of information needed.

11

34 30

3

-10%

0%

10%

20%

30%

40%

50%

60%



Results:

The results are presented separately for each dimension as well as side-by-side so that the Learning

Outcomes & Assessment Committee may examine each dimension on its own, or compare the

dimensions of Information and Technology Literacy. First, each dimension’s analysis is detailed.

The first dimension is “Uses a clearly

expressed research question and/or thesis

to determine the extent of information

needed.” There were 68 ratings from 7

courses to assess this dimension of

Information and Technology Literacy.

Figure 1 shows the percentages and

numbers of ratings given for each rating

category (i.e. beginning, developing,

competent, and professional/advanced).

Thirty-seven (54.5%) of the students

attained ratings of competent or advanced/professional. Looking back at Table 1, CBOT 333 was a

major contributor to the lower ratings compared to other courses with 40% of students having

‘beginning’ ratings. Most of the ratings for the courses examined for this dimension were split

between ‘developing’ and ‘competent.’

For the second dimension, “Access and

retrieves needed information from a variety

of appropriate resources,” there were 78

ratings from 8 different courses. Figure 2

shows that 33 (42.3%) artifacts met or

exceeded college level expectations for this

dimension. The distribution of course

ratings was very similar across courses with

most of the ratings in the ‘developing’ and

‘competent’ categories. Two of the courses

examined had a total of 3 advanced/

professional ratings, which were the only

ratings given in that category for this

dimension.

Figure 2


10

25

40

2 0%

10%

20%

30%

40%

50%

60%



Figure 3

Figure 4

21

37

17

2 0%

10%

20%

30%

40%

50%

60%


Uses information and technology effectively to create a final product within

the specifications of the assignment (technical)

6

24

33

4 0%

10%

20%

30%

40%

50%

60%


Uses information and technology effectively to create a final product within

the specifications of the assignment (analytical)

The third dimension is “Critically evaluates

information and its sources.” Team

members rated samples from 8 courses for

a total sample size of 77. Forty-two (54.5%)

artifacts met or exceeded the ‘competency’

level. As with the second dimension, the

distribution of course ratings was very

similar across courses with most of the

ratings in the ‘developing’ and ‘competent’

categories. LIB 170 was the only course

with ratings in the ‘professional/

advanced’ category.

There were 7 courses totaling 67 artifact

ratings for the dimension, “Uses

information and technology effectively to

create a final product within the

specifications of the assignment

(analytical).” The original dimensions was

split into dimensions 4 and 5 since the first

two parts of the original dimension dealt

with information analysis and organization,

and the second part dealt with the more

technical aspects (i.e. grammar, fonts,

headings, and document style). The team

found that assessing these pieces separately

worked better for courses like MUS 101.

Thirty-seven (55.3%) artifacts met or

exceeded college level expectations for

competency. The majority of the artifacts

were given ratings of 2 or 3. MUS 101 had

the greatest proportion of lower ratings

with 70% of ratings below competency.

The fifth dimension is “Uses information

and technology effectively to create a final

product within the specifications of the

assignment (technical).” Team members

Figure 4

Figure 5


16

33

17

2 0%

10%

20%

30%

40%

50%

60%



rated samples from 8 courses for a total sample size of 77. Nineteen (24.7%) artifacts met or

exceeded the ‘competency’ level. Many of the courses had their greatest number of ratings in the

‘developing’ and ‘beginning’ categories. CBOT333 was the only course with no ratings in the

‘professional/advanced’ or ‘competent’ categories.

There were 7 courses totaling 68 artifact

ratings for the dimension, “Accesses and

uses information ethically.” Nineteen

(27.9%) artifacts met or exceeded college

level expectations for competency. The

majority of the artifacts were given ratings

of 2. CBOT 333 had the greatest proportion

of lower ratings with 90% of ratings below

competency. ENG 101-A had the greatest

percentage of ‘competent’ ratings, but only

half of the artifacts were judged to have

achieved ‘competent’ status.

Figure 7 provides a look at all Information and Technology Literacy dimensions. The red line at 70%

shows the criteria for success identified by the Title V project. The figure shows that none of the

dimensions achieved these criteria, with the highest percentage (55.3%) for the ‘Analytical’

dimension (i.e. dimension 4) and the lowest percentage (24.7%) for the ‘Technical’ dimension (i.e.

dimension 5).

Figure 6


Figure 7

7.4% 3.8% 2.6% 6.0% 2.6% 2.9%

47.1%

38.5%

51.9% 49.3%

22.1% 25.0%

35.3%

43.6%

32.5% 35.8%

48.1% 48.5%

10.3% 14.1% 13.0% 9.0%

27.3% 23.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Uses a clearly expressed research

question and /or thesis to determine

the extent of information needed.

Access and retrieves needed information

from a variety of appropriate resources

Critically evaluates information and its

sources

Uses information and technology effectively

to create a final product within the

specifications of the assignment (analytical)

Uses information and technology effectively

to create a final product within the

specifications of the assignment (technical)


Information & Technology Literacy Rating Percentages by ILO Dimension

Beginning

Developing

Competent

Professional/ Advanced

70% of students

should meet or

exceed

competency


Another way to view the data is by creating distributions to examine the means and standard

deviations in graphical form. Figure 8 shows a normal distribution with a mean of 0 and a standard

deviation of 1 (the black curve). If attending has an effect on the Information and Technology

Literacy skills of the general student body, the distributions of the various dimensions should shift

toward the right of the black initial curve. In addition, one would expect that the dimension curves

be narrower than the initial curve, showing that the variation between students is smaller than the

theoretical general population.

Figure 8

Conclusions:

Students seem to perform similarly when comparing the ratings of the first four dimensions of

Information and Technology Literacy. The ratings of the last two dimensions are similar to each

other, but the ratings are lower than the first four dimensions. This similarity in performance is most

evident when looking at Figure 8. The peaks of the normal distribution lines at almost the same point

for the first four and last two dimensions, indicating that the means are almost the same (also see

Table 2), the width of the distributions are very similar, indicating that the students scored similarly

across dimensions (i.e. the scores across dimensions have similar standard deviations). Scoring

similarly, in this case, means that there were roughly the same percentages of 1s, 2s, 3s, and 4s for

each dimension (also shown in Figure 7). In addition, the distributions were narrower than the initial

black curve (i.e. the standard deviations were smaller), meaning that team members rated artifacts

more similarly than one would expect in the population. The smaller the standard deviation (thus,

the narrower the distribution), the more similarly the artifacts were rated. A perfect scenario would

be that all artifacts were given ratings of 4 (professional/advanced), which would result in a standard

0.0

0.1

0.2

0.3

0.4

0.5

0.6

-4.00 -2.00 0.00 2.00 4.00 6.00

f(x)

Mean

Normal Distributions of Ratings by Dimension

Normal

DIM 1

DIM 2

DIM 3

DIM 4

DIM 5

DIM 6


deviation of 0. This scenario illustrates that the higher the average rating and the smaller the

standard deviation, the better is the outcome. For all 6 rubrics, the directly collected and assessed data

shows evidence for the need for improvement in AHC’s student Information and Technology Literacy skills. The

percentage of students meeting competency did not reach the targeted 70% for all 6 dimensions. As shown in

Table 2, at least half of the artifacts were rated as ‘competent’ or better for the first, third, and fourth

dimensions. The weakest areas were the last two dimensions, with only about 25% of artifacts rated as

‘competent’ or better.

Students competent or

professional/advanced Average Rating Standard deviation


research question and/or

thesis to determine the extent


54.5% 2.51 0.78

Accesses and retrieves needed

information from a variety of

appropriate resources

42.3% 2.32 0.76


information and its sources 54.5% 2.43 0.75






55.3% 2.51 0.75





assignment (technical)

24.7% 1.99 0.77

Accesses and uses information

ethically and legally 27.9% 2.07 0.78

Table 2

Discussion and Future Direction:

One of the assumptions is that the courses included in the study are a representative sample of all the courses at AHC that are linked to the ILO being assessed. The goal of ILO analysis should be to include randomly selected courses linked to the ILO. If the LOAC decides that every course linked to a particular ILO should be included in the ILO analysis once every six years, perhaps randomly assigning numbers 1-6 to each course and then assessing all 1s in year 1, 2s in year 2, etc... Then, at the beginning of the next 6-year cycle, randomly assign 1-6 again so that the same courses aren’t always assessed together every cycle. This process will ensure that the assumption of random selection is met.


Another assumption is that students are randomly selected within classes. The team drew numbers corresponding to the artifact number to select artifacts for courses, so this assumption was met. To ensure that this assumption is always met, a team member or the research analyst, not an instructor, should randomly select students from the rosters of the courses included in the study (as done in the present study). It may be difficult for an instructor to submit a less than desirable artifact or refrain from submitting an artifact that was exceptional. In addition, some instructors may feel compelled to submit artifacts that represent a range of abilities even though their sample may not be representative of the abilities in their classes. In order to avoid the human aspect of selection, the best method of randomly selecting artifacts would be to randomly select students from the rosters of the classes included in the analysis and give the names to the instructors. The instructors would then give the sample from the selected students to the team for assessment. The artifacts would remain anonymous since the artifacts would not have identifying information on them. Another reason for knowing which students’ artifacts are being assessed is to be able to compare sample demographics to overall AHC demographics so that readers may decide if the sample is representative of the student population at AHC. Institutional research could provide demographics (i.e. ethnicity, age, gender) for the sample, class (from which samples were collected), and AHC credit students.

To rate the artifacts, the team split into two groups of three and each team assessed 5 of the 10 artifacts from each course. For two courses, MUS 101 and CBOT 333, all six members rated all 10 artifacts. The team members rated the artifacts individually and then the members got together to agree on the final ratings. Rating the artifacts individually is the proper method of scoring the artifacts, but it is also important to record the team member ratings and provide them for inter-rater reliability analysis. It is acceptable to say that the team agreed upon ratings; however, showing that all members individually scored an artifact with the same ratings is a good demonstration of reliability. If the inter-rater reliability is low (i.e. ratings vary widely across raters), then it is possible that the rubrics need to be defined better or the team members need more training on using the rubrics. Also, to ensure that one team isn’t rating artifacts easier than another team, both teams should rate two or three artifacts from each course to see if their ratings match. Another method is to divide the courses among the two teams and have all members on team 1 rate the artifacts for half of the courses and all members on team 2 rate the artifact for the other half of the courses. For example, if you have artifacts from 10 courses, team 1 would rate all artifacts from 5 courses and team 2 would rate all artifacts from 5 courses. This way, team members only have to get experienced in using rubrics for 5 of the courses instead of 10. Shuffle team members so that team 1 and team 2 are not always the same groups of people, if possible, to lessen the possibility of team 1 and team 2 rating differently. Lastly, a greater sample size would increase reliability of the ratings. The sample of artifacts is supposed to represent the credit population at AHC, which is roughly 10,000 students. In the future, the team should asses either more courses or a greater sample of artifacts per course, or both to improve reliability of ratings.

As for the ratings themselves, it is important that when the team is in the process of rating the artifacts that the members compare ratings across courses to see if they are similar. If the ratings are not similar for a course or courses, it does not necessarily mean that there is a problem with the ratings. The outlying courses should be checked to see why ratings are much higher or lower than other courses. Mentioned in the results, CBOT 333 had a greater proportion of lower ratings for two


of the dimensions than the other classes rated. Also, some courses had no low ratings at all. There are several possibilities why the ratings were lower for CBOT 333. The best the team can do is to make sure that the ratings are not lower because of reasons other than the artifacts lacking evidence of meeting or exceeding college level expectations. Other potential reasons for extremely low or extremely high ratings are inconsistent raters (i.e. “easy” or “hard” raters for some class artifacts), artifact type (i.e. subjectively vs. objectively scored), and discrepancy between rubrics and assignment goals. An example of a discrepancy would be that the goal of the assignment was to assess student mastery of an accounting concept where font type was not indicated in the rubric, but then using the assignment to assess choosing the correct font type as part of the ILO. It is easy to see why student artifacts might not receive good ratings on assignments that were rated with ILO rubrics that are vastly different than assignment rubrics. To overcome potentially poor ratings because of assignment vs. ILO rubric differences, it is important to provide students with both rubrics so that students know on what they should focus.

Particularly good parts of the team’s actions were the sample selection and rating procedure. The team had the right idea rating the artifacts individually and then comparing their ratings to see if they agree. In addition, the team sampled a variety of course types to assess information and technology literacy and then took steps to choose a random sample of artifacts from each course. By looking at various disciplines, the team is able to get a representation of students at AHC. In the future, a greater sample size would increase the reliability and ensure that the ratings represent all credit students at AHC.

Documents

Information and Technology Literacy Institutional Learning ... · appropriate technologies to locate, access, select and manage the information. Examples of mastery of this ILO include,