Skills Assessment Manager (SAM)assets.cengage.com/pdf/wp_sam.pdfSkills Assessment Manager (SAM) is an online learning environment that helps learners master Microsoft® Office skills

White PaPer

Skills Assessment Manager (SAM): Personalization, Productivity & Preparation

to Increase Student Outcomes March 2013

This report highlights the results of an efficacy study on the impact of

using Cengage Learning’s Skills Assessment Manager (SAM) product

within college-level computer skills development classes. Project

Tomorrow, a national education nonprofit organization with research

expertise in the use of emerging technologies within K-12 and higher

education, facilitated this study on behalf of Cengage Learning.

The study was conducted during the fall academic term of 2012, August

through December. Participants were instructors teaching introductory

computer skills courses. In total, 14 SAM instructors agreed to participate

in the study, and 917 of their students submitted data in the initial wave

of data collection. Six non-SAM instructors using the same textbook as

the group using SAM agreed to participate in the study, and 129 of their

students submitted data in the initial wave of data collection. In total,

20 instructors representing 20 colleges and universities, and more than

1,000 students, agreed to participate in the study.

Study Participants

*Not all survey participants responded to all survey questions.

About SAM

Skills Assessment Manager (SAM) is

an online learning environment that

helps learners master Microsoft®

Office skills and computer

concepts that are essential to

academic and career success. SAM

engages students in self-paced

learning of Word®, Excel®, Access®,

PowerPoint®, Windows®, Internet

Explorer®, and Outlook®, as well as

technology concepts and issues.

At the same time, SAM reduces

instructors’ workloads with auto-

graded assignments and exams,

and easy-to-use course setup and

management tools. SAM provides a

simulated environment for Microsoft

(MS) Office products that integrates

an interactive learning experience

with the tools and knowledge to

prepare students to use these

products in real-world settings.

Learn more online at www.cengage.com/samoffice20132

about the Study

higher education institutions —

9 four-year colleges and universities;11 two-year or

community colleges

states representing allgeographic regions

of the country:CA, FL, IN, KS, MD, MAMN, MO, NH, NY, NC,

OH, RI, TX and VA

college instructors and

20 15

20* 1,046students enrolled in

introductory computerskills courses.

Key findings

This study explores both the instructor and student perspective on the value of

SAM as an instructional tool. The key findings include:

• Nine out of 10 instructors (90%) believe that SAM enhances their personal productivity, naming the auto-grading capabilities as their most valued benefit.

• SAM increases the proficiency level of students in the use of the MS Office applications with the greatest increase in using Access.

• Eight out of 11 instructors (73%) and 69% of students say that SAM increases student interest and engagement in course content.

• Nine out of 11 instructors (82%) and 81% of students identify SAM as a key factor in preparing students to use the Office applications in real-world settings.

• All 11 instructors (100%) would recommend SAM to colleagues; 77% of students would tell a friend to take a class that used SAM.

• Students’ top three benefits from using SAM are anytime access, step-by-step instructions, and the ability to learn at their own pace.

SAM instructors consider digital resources instrumental to student learning

SAM instructors appear to leverage technology to improve their own productivity

and enhance their professional networks and personal development. The SAM

instructors in this study also demonstrated a sophisticated grasp of technology

within instruction:

• Eight out of 10 SAM instructors (80%) noted participating in webinars and online conferences on a regular basis.

• Conversely, only four out of the 10 non-SAM instructors exhibit similar behavior patterns.

• Ten out of 15 (67%) noted being very comfortable using digital content or resources within their lessons or class activities; an additional four (27%) assessed their level as comfortable.

• Nine out of 10 SAM instructors (90%) are more likely to be using a Learning Management System (LMS) than the instructors in our control group; only two of the five in that cohort are LMS users.

SAM instructors are also more likely to see digital products playing a significant

role in the learning process, both in and out of the classroom. As noted in Table

1, which compares the perceptions of SAM and non-SAM instructors on the

role of digital tools and resources within instruction, SAM instructors place a

higher premium on the use of technology to stimulate student engagement in

class content and support the development of knowledge and skills.

SAM: Personalization, Productivity & Preparation to Increase Student Outcomes — March 2013

aboutProject tomorrow

Project Tomorrow®, the

national education nonprofit

organization dedicated to

empowering student voices

in education discussions,

prepared this program

evaluation for Cengage

Learning. Project Tomorrow

has 16 years of experience in

the K-12 and higher education

sector and regularly provides

consulting and research

support to school districts,

government agencies,

business and higher education

institutions about key trends

and research in science, math

and technology education.

3

4 Learn more online at www.cengage.com/samoffice2013

Table 1

Value of digital tools and resources within instruction

The SAM instructors also place a high value on the student outcomes associated

with using digital tools within the instructional process. For example, eight out

of 10 of the SAM instructors believe that the use of technology allows students

to gain a better understanding of class material through “trial and error” learning

opportunities. And seven of the 10 SAM instructors cite student self-motivation,

ownership of learning process, and better organization as key outcomes as well. Not

surprisingly, these outcomes are well supported by the inclusion of SAM within their

class environments.

Specific to the use of SAM, eight of 10 instructors in the treatment group had prior

familiarity with using the product in other courses and say that they used SAM at

least once a week during the fall semester, with five of 10 accessing SAM daily. This

combination of sophisticated technology knowledge, high familiarity with SAM,

and frequent usage provides an excellent foundation for analyzing the instructors’

perceptions on the value of SAM.

SAM enhances instructor productivity

Regarding the impact of SAM, 90% (nine of the 10) of the SAM instructors considered

SAM an important tool for supporting or enhancing their productivity. When asked

to identify the features or functionality that provided the most significant benefit to

them, eight out of 10 instructors chose “auto-grading capabilities.” The relative value

of SAM’s benefits is also interesting to evaluate and informative for understanding

the instructor perception on SAM’s productivity value.

The top benefits as noted by the 10 SAM instructors are:

• Auto-grading capabilities (80%)

• Easy-to-schedule coursework and assignments (60%)

“Turnaround time for grading

projects has been reduced

tremendously! SAM also makes it

easier to catch students cheating.”

Christy LopezEast Carolina University

40%

60%

Benefits of using technologywithin my course

SAM Instructors(Treatment Group)

Non-SAM Instructors(Control Group)

90%

40%

70%

60%

100%

60%

20%

Extends learning processbeyond the classroom

Helps students developproblem-solving and

critical-thinking skills

40%Improves students’ ability tounderstand abstract concepts

Makes class more interesting

Resources are more timely,accurate, and relevant


5

• Complements in-class activities (50%)

• Time savings (50%)

• Tracking student achievement (40%)

Instructors’ comments about the value of using SAM in their course overwhelmingly

supported the auto-grading capabilities (and its accompanying benefits) as the

strongest benefit.

SAM increases student interest and engagement

Research tells us that engagement in course content is an important predictor of

student success in that course. While student-rated course engagement is more

informative, the instructor perspective is also important for understanding the

relative importance of SAM as an instructional tool. To examine this perspective, we

first asked SAM instructors to evaluate the interest level or level of engagement of

their students in their course content:

• Seven out of 10 SAM instructors assessed student engagement as high;

• Three out of 10 assessed engagement as moderate.

Given that the majority of the instructors already had high levels of student engagement in their class, the more interesting finding is around the role of SAM in that student interest level:

• Eight of 11 (73%) SAM instructors pointed to SAM as a key factor in increasing student engagement in their course.

• Two of 11 (18%) identified SAM as a very important factor (Figure 1).

The high value that instructors place on SAM in increasing student interest or

engagement supports the vital role of SAM in establishing a solid foundation for

course instruction goals.

Figure 1

Instructors — How important is SAM to increasing

student engagement in your course?

“The time savings in not having

to grade tests is the biggest

advantage.”

Theresa LuiHoward Community College

“SAM’s greatest asset is the auto-

grading of exams and projects.”

Kathleen GowdeyBerkshire Community College


Very Important

Important

0% 15% 30% 45% 60%

55%

18%


Instructors and students identify SAM as a key factor in preparing students for the real world

Within this study, the goals for the instructors were universal — to prepare their

students to use the MS Office suite of applications in real-world workplace settings.

In some cases, this course was a student prerequisite for more advanced computer

science or technology courses; in other cases, it was required for specific majors or

even graduation.

Thus, it makes sense that 10 of 11 SAM instructors (91%) made full use of the guided

instruction, tutorials, and projects within their course as these provide students with

hands-on opportunities tied to MS Office tools. When asked to comment on the

value of SAM in the preparation of their students to use Office tools, 10 of 11

(91%) SAM instructors said that their students are better prepared to use the Office

applications in real-world settings because of their use of SAM within this computer

applications course. Equally notable is that none of the instructors saw SAM as a

detriment to their students’ preparation to use Office.

All instructors would recommend SAM to colleagues

The final value proposition factor summarizes SAM’s impact on the instructors’

personal productivity, and how well SAM helped the instructor achieve key

instructional goals within their course.

In general, individuals — and college faculty in particular — are often reluctant

to assume that the personal value proposition they have with technology products

can extend to others; the assumption is that everyone’s needs are different and

that it is hard to extrapolate value. Thus, it was highly relevant to ask the SAM

instructors about the likelihood of recommending the product to another instructor

or professional colleague as this could indicate a potentially important finding.

• Within this study, 100% of the SAM instructors, 11 of 11, noted that, based on their experiences, it was likely they would recommend SAM to a fellow instructor.

• Five of the 11 instructors (45%) amplified that finding, saying they were very likely to make a recommendation.

“My course experience using SAM

has been great. It has actually

helped me more in class because

it lets us do an unlimited amount

of training. If we do need help with

something, we can just log onto

SAM and do it.”

StudentWestchester Community College

SAM Instructors:Are your students better prepared to use MS Office applications in real-world settings because of your use of SAM this semester?

10 of 11 SAM instructors say “YES”!

10 11outof


7

Given that our SAM instructor pool was very familiar with sophisticated uses of

digital content, tools, and resources within instruction, this finding particularly

supports the high efficacy of SAM within college computer skills courses.

Use of SAM increases the proficiency level of students

The student pool for the study initially consisted of 1,046 college students representing

20 higher education institutions; 917 of those students were enrolled in computer

skills classes that were using SAM; 129 were enrolled in similar courses but without

access to SAM.

To examine the productivity value of SAM from the student perspective, we examined

three factors:

1. SAM features used by the students;

2. changes in the students’ self-assessed proficiency using Office tools from the

beginning of the semester to the end of the semester; and

3. comparisons between treatment and control groups.

While the SAM students reported regularly using the practice tests (75%), the projects

(67%), and the guided instruction and tutorials (57%) throughout the semester,

the SAM feature that they believed was the most helpful to course success was the

guided instruction and tutorials (41%). Several student comments from the focus

groups and the open-ended responses on the surveys underscore the importance of

the self-paced tutorials.

The students recognized the high value of the projects in preparing them to use the

Office applications. The practice tests were especially valuable by providing students

ways to self-remediate and drive their own learning process.

A key component of the study was to explore students’ self-assessed ability to do

certain tasks within the Office suite of applications and to effectively use Word,

Excel, PowerPoint, and Access. Across the board, students in the SAM courses rated

their proficiency in these tasks and products higher at the end of the semester with

the exception of their proficiency using Word.

• As discussed in the focus groups, Word is an application that the students perceived that they knew how to use well before coming into this course, only to learn that there was much more that they did not know. Their preconceived self-assessment did not live up to their expectations.

We saw a similar pattern in the data from the control group students. Among those

students, 95% assessed their skills with Word as proficient in the pre-survey; 83%

gave themselves the same grade in the post-survey. The most significant increase in

proficiency was in the use of Access, a product that students typically do not know

before taking this kind of class (Figure 2).

“Using SAM helped me

tremendously. There were times

when I would miss class and I

could go to SAM, do some practice

tutorials and understand in a

shorter time span than in class.”

StudentWestchester Community College



Figure 2

SAM students’ comparative pre- and post-assessment of their

proficiencies

This discussion of Access proficiency is also interesting in light of the data from

the control group students. In their post-survey, 77% of the control group students

evaluated their ability at the proficient level when building a query using Access,

while 86% of the SAM-using students chose that identifier for the same skills. Within

the focus groups, the students specifically pointed to SAM’s Access projects as being

important in their development of proficiencies using this application.

Another way to understand the differences between the control group and the

treatment group is to look at the students who rated their skills as very proficient

and the change in that rating from the beginning of the semester to the end. SAM

students exhibited a higher rate of movement into the very proficient category for

the majority of Office applications and tasks measured.

• For example, the proportion of SAM students who self-rated their skills in using Excel as very proficient grew by 46% between the beginning and the end of the semester, while the corresponding increase in the control group was only .4%.

SAM increased students engagement in the course

Parallel to their instructor’s perspectives, students in the SAM courses credit SAM

for increasing their interest or engagement in the class.

• Two-thirds (69%) of the students labeled SAM as important to their course engagement, with 34% rating SAM as very important.

“SAM has improved my learning

because it has hands-on projects

that I can practice my skills on.”

StudentGaston College

“[SAM] improved my course

experience by letting me do the

work at home at my own pace.”

StudentSoutheastern Community College

(Based upon data from 212 students)

Access

PowerPoint

Word

Excel

0% 20% 40% 60% 80% 100%

Post-Survey Pre-Survey


9

Interestingly, among certain sub-groups, the engagement value of SAM was markedly

higher. More than three-quarters (77%) of the non-traditional college students (those

over 24 years old) saw SAM as a key factor in driving their personal engagement

with this course. Not surprisingly, the students identified as high (83%) or moderate

(74%) technology users also placed a higher premium on the value of SAM to keep

them engaged in the class.

SAM prepared students for real-world scenarios

Given course objectives to prepare the students for real-world application use, it is

important to understand from the students’ perspectives their assessments of their

abilities to use these applications beyond the conceptual classroom practices. As

noted above, nine of 11 (82%) SAM instructors believe the product better prepared

their students to use MS Office applications in real-world settings. Some may

question the objectivity of the instructors’ statement since SAM provides them with

tangible productivity and time-savings benefits as well; the students’ assessment of

the preparation value of SAM is therefore a critical input. Within the SAM test

group of students, 81% believed that the product better prepared them to use Office

applications, legitimizing their instructors’ perspective. Interestingly, among the

students we identified as moderate technology users, almost nine out of 10 (88%)

believed that SAM was the key to their better preparation. They obviously placed

a high value on the hands-on use of the tools to support their emerging technology

skills. Comparatively, only 56% of the students in the control group, who did not

have access to SAM, believed that they were well prepared to use the Office suite of

applications in real-world settings.

Figure 3

Students’ responses: I am well prepared to use Office applications

in a real-world setting

“[SAM] told me all of the mistakes

I made, and when I saw the

mistakes, it allowed me to use

critical thinking of what I could do

to make that problem

100% correct.”

StudentHutchinson Community College


(Based upon post-survey data from 246 students)

Non-SAM Students SAM Students

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

0% 10% 20% 30% 40% 50% 60%


AM leads to better outcomes

Another analysis of the preparation value was to look at students’ expectations for

their final grades. In a survey taken near the end of the course, two-thirds of the

students using SAM (64%) felt that their final grades would be higher. Following the

pattern noted previously, an even stronger cohort of moderate tech-using students

(72%) credited SAM as the driver for their potentially higher final grades. And seven

out of 10 of the non-traditional students highly valued SAM for its preparation value

within the context of a higher final grade.

Subsequent to the end of the semester, we collected the grade distributions from a

subset of participating instructors — both those using SAM and those not. Looking

at small classes (31 students or fewer) at community colleges, more students received

A’s in the SAM classes than in the non-SAM classes.

Students would tell their friends to take a class using SAM

The student experience using SAM within their computer applications courses was

overwhelmingly positive. This overall perspective is supported by the students’

increased proficiencies in MS Office applications, their higher engagement levels in

the course, and their preparation to use the applications in future real-world settings.

However, as with their instructors, we wanted to evaluate the intensity of that positive

perception and see if their personal value proposition extended to recommending

SAM courses to their friends. Over three-quarters of the students in the SAM courses

(77%) would likely recommend a SAM course to a friend, with 34% of the students

very likely to do the same. Notably, only 4% of the SAM students said they would not

make such as recommendation, a low number, considering we could not account for

teacher instructional style or other classroom-specific issues that influence students’

overall satisfaction with courses.

“Without SAM, I wouldn’t have

been so motivated. It motivates

me to do my best and get 100 on

everything.”

StudentGaston College

“It has improved my course

experience because I don’t think

my grade would have been as high

if I didn’t have SAM.”

StudentBerkshire Community College


SAM Classes Non-SAM Classes

45% of these studentsearned an “A” grade

in this course


in this course

SAM Classes Non-SAM Classes


in this course


in this course


Students’ top benefits include anytime access, step-by-step instructions, and the ability to learn at their own pace

The students’ rankings of the benefits provide insights into SAM’s overall value

proposition. The statements can be divided into two categories: the value of certain

SAM features, and how SAM influences learning. Figure 4 provides the top five benefits

for each category.

Figure 4

The most significant SAM benefits: Student perspectives (n=212)

From the student perspective, we see SAM’s value extends beyond the functionality and

the features that develop proficiency in Office applications. The students themselves

see SAM as enabling self-directed learning that fits their preferred learning style, and

the ability to put into practice what they have learned. This combination of factors —

the feature value and the learning value — explains the high recommendation value

from the students and their instructors.

11

Access SAM any time I want

I can learn at my own pace

Value of SAM features How SAM influences learning

1

SAM provides step-by-step instructions2 Ability to practice

what I have learned2

Easier to see my mistakes and take corrective action3 I can self-test my

knowledge3

Easier to keep trackof assignments4 This fits my

learning style4

77%

59%

53%

48%

35%

52%

50%

36%

32%

31%Ability to submit project multiple times5 I am more engaged in

my course materials5

1

Source Code: 13M-AA0043

ISBN: 978-1-133-36740-6

Cengage Learning is a leading provider of innovative teaching, learning, and

research solutions for academic, professional, and library markets worldwide. The

company’s products and services are designed to foster academic excellence and

professional development, increase engagement, improve learning outcomes, and

deliver authoritative information to people whenever and wherever they need

it. Through the company’s unique position within both the library and academic

markets, Cengage Learning is providing integrated learning solutions that bridge

from the library to the classroom. www.cengage.com

Copyright ©2013 Cengage Learning. All Rights Reserved.

Documents

Skills Assessment Manager (SAM)assets.cengage.com/pdf/wp_sam.pdfSkills Assessment Manager (SAM) is an online learning environment that helps learners master Microsoft® Office skills