Upload
jasminesing
View
766
Download
0
Embed Size (px)
Citation preview
Evaluation Plan for “Teaching and Learning
with VMware Workstation Software”
TLT476: Technology Program Evaluation
Tonya B. Amankwatia, Ph.D.
Prepared by: Lifang Chang
Lehigh University, Spring 2008
1
Introduction and Background
Introduction
This evaluation plan describes the procedures that will be used to evaluate
“VMware Workstation (VMW)” implemented by the department of Networking, Security
and System Administration (NSSA) at Rochester Institute of Technology. VMware
Workstation is a virtual machine software suite for x86 or x86-64 computers and it allows
one physical machine to run multiple operating systems simultaneously. Currently, the
department of Networking, Security and Systems Administration (NSSA) at RIT uses
VMware Workstation to provide its computer science students remote access to multiple
virtual operating environments for distance learning. The document begins with
background information on the VMware workstation, followed by the purposes of the
evaluation and the audiences with whom the evaluation results will be shared. Next, the
intended decisions based on the evaluation results and questions to be addressed during
the evaluation process are described. In addition, the proposed methods and
instrumentation, the selected sample and inherent limitations, logistics and timeline are
explained including who, where, when, and how will be doing what. The final section in
this document proposes a budget for the evaluation.
This evaluation plan has been prepared by Lifang Chang, a graduate student in the
program of Teaching, learning and Technology at Lehigh University. The plan has been
informed by discussions with the client, Dr. Jorge Diaz-Herrera, Dean of the College of
Computing and Information sciences, the supervising instructor, Dr. Tonya Bevedare
Amankwatia of the program of Teaching, learning and Technology at Lehigh University.
Background
2
New York’s Rochester Institute of Technology (RIT), with an enrollment of more
than 15,000 students, is known for its professional, science and technology programs.
The Department of Networking, Security and Systems Administration (NSSA) is one of
six departments in the Golisano College of Computing and Information Sciences at RIT.
The department of NSSA has five computing labs that serve nearly 400 students, many of
whom are working and can’t be on campus every day. The IT (Information Technology)
administrators and managers chose VMware virtualization software to provide a way for
students to remotely access the multiple operating environments they normally work with
in the lab.
VMware Workstation (VMW) provides couple of virtual systems (such as
Windows and Linux) and configurations to multiple students with a single server. Using
personal computers, students can remotely log in and instantly be connected to an
available virtual machine rather than having to be physically present in the lab to log on
to individual systems.
As shown in the following screen capture of the system interface, VMM can run
Windows under Linux (P1),
3
(P1)
run Ubuntu (one type of Linux) under Windows (P2),
(P2)
run Windows Vista under Mac OS/X (P3),
4
(P3)
and also can support a virtual machine running Windows XP on a Windows Vista host
(P4).
(P4)
The VMW software suite is distributed by electric download from the VMware
website with discount cost to academic communities.
Purposes
5
The purpose of this evaluation is to collect data which will provide decision
makers at the College of Computing and Information Science in RIT with the timely,
accurate information required to support decisions regarding the extension, management,
and marketing of the VMware Workstation software.
The specific sub-purposes:
1. To collect information regarding the acceptability, applicability, and
effectiveness from the target audience of students using the virtualization software.
2. To collect information regarding the implementation of the various components
of VMware Workstation initiative into the overall teaching and learning environment,
with a goal of providing information that can be used to improve the future
implementation in the entire college.
Audiences
The primary audiences for the evaluation will be the dean council including the
college dean, Dr. Jorge Diaz-Herrera and other 16 council members. Important secondary
audiences include supporting faculties and staff of the VMware Workstation operating
environment in the department of NSSA. Potential audiences include the Information
Technology managers and administrators from other departments and organizations at
Rochester Institute of Technology.
Decisions and Questions
The evaluation will be used to inform two major decisions to be made by the dean
council and supporting faculties and staff. Also, most of these decisions will be made
regardless of the quantity and quality of information available to the decision makers.
The following decisions are assumed:
1. Should the department continue the use of VMware virtualization software?
6
2. What techniques and strategies should be used for effectively integrating
VMware virtualization system across the curriculum?
3. How should VMware workstation software be used for enhancing students’
real-world problem-solving skills?
4. How should VMware workstation software be used for improving server
utilization?
5. How should VMware workstation software be used for enhancing extending
access to and use of information resources?
6. What techniques and strategies should be used for integrating the VMware
Workstation virtualization solution in the wider Computing and Information Science
College?
7. How should people and other resources be deployed to support an everywhere
computing environment based upon the VMware workstation solution?
In order to have the information needed to guide the types of decisions noted
above, many questions must be addressed:
A. What are student reactions to using VMware Workstation system?
• Does the student enjoy using the module?
• Does the student think that the general skills learned with the VMware
software were relevant to their current research environment or future
career plans?
• What is their level of satisfaction with the learning experience in the
VMware virtualization environment?
• What do they learn from the VMware software?
7
B. What are reactions of the faulty and computing labs’ staff to the VMware
Workstation system?
• What is their expert opinion of the cost-effective of the VMware software?
• What is their expert opinion of the usability of the VMware software?
• What is their expert opinion of flexibility of the VMware software?
C. How can the department of NSSA best share its experience with VMware
virtualization software with other departments looking to implement similar programs?
D. What would the NSSA department like to work with the other VMware
virtualization technologies?
Methods
Couple of descriptive and interpretive methodologies will be used to assist the evaluators
in gathering data to answer the questions listed above as well as other questions that will
surely revealed during the evaluation time. The data collection methods include:
1. Interviews with faculties who have integrated VMware Workstation with their
curriculums.
2. Interviews with computing lab personnel
3. Observations in the computing lab
4. Questionnaires to be completed by students who have utilized LTTS modules
5. Expert review of the VMware Workstation software from three types of experts
(content, usability, and flexibility)
The primary source of data will be collected through interviews with faculties and
technology staff in the department. The evaluator will go to the computing lab making
the observations. At least three interviewees in the department of NSSA will be
interviewed based on their permission. Interview protocols include questions about
8
virtualization deployment environment, networking security, students’ academic
performance and open-ended questions (see Appendix A). Information will also be
gathered via a questionnaire what will be given to students (see Appendix B). The
questionnaire will be distributed on the web site with SurveyMonkey.com.
One final method of gathering information will be expert review. This evaluation
requires three different types of experts in content, usability, and distance learning area.
Ideally, an expert who can cover three different areas at the same time is preferred.
The relationships among these five methods and the evaluation questions
identified in the previous section are described in the evaluation matrix that appears
below.
Method What are student reactions to using VMM?
What are reactions of the faulty and computing labs’ staff to the VMW?
How can the department best share its experience with other departments?
What would the NSSA department like to work with the other VMware technologies?
Faculty Interview X X Staff Interview X X X Lab Observation X X Student Questionnaire X X Expert Review X X
Sample
Data will be gathered from several sources during the evaluation. The participants
for this evaluation will be:
• 25 – 30 students who has used VMware software from the NSSA department.
The participants in the evaluation include both undergraduate and graduate students, and
9
also represent a variety of age groups. The students are randomly selected with female
and male.
• 3 expert reviewers from other departments such as Computer Science,
Information Technology and Distance learning. Accordingly, they will focus on the
reviews on content, usability and distance learning area.
• At least 3 faculties who has used VMware software with their curriculums.
• 2 representatives from the department’s computing labs
Instrumentation
The instruments used to gather data for the evaluation include:
• A sample User Questionnaire that the students participating in this evaluation
will complete after they have used the VMware virtualization software. The questionnaire
will be administered during the last ten minutes of one class period. Moreover, the user
questionnaire will be reviewed by the supervising instructor, Dr. Tonya Bevedare
Amankwatia to help assure wording is appropriate.
• A protocol for the User Interviews that will be conducted with selected faculties
who have used the VMware Workstation software in their class.
• A protocol for the User Interviews that will be conducted with selected
computing lab staff who have supported the VMware Workstation operating environment.
• A description of the in-lab observation that will record student and instructor
interactions with the VMware operating environment.
• A expert review checklist that will be conducted with selected experts in
different area such as content, usability and distance learning.
10
• Brief resumes of six experts from whom a panel of three will be selected to
review the VMware software.
All instruments are subject to major revisions, and should only be considered as
rough drafts of the actual instruments to be developed and refined during the real
evaluation.
Limitations
Several constraints on this evaluation must be clarified because they constitute
limitations on the interpretation and utility of this evaluation. First, many of the
participants in this evaluation study are students with a wide range of technology skills.
Different levels of technology skills bring up different sets of needs and expectations
with respect to the technology itself and the context of the operating systems. Therefore,
the results from this evaluation will be affected by the background of the participating
students. On the other hand, a relatively small sample of students is also a limitation.
Students may have varying levels of interest in the topic of networking security,
operating system principle, and software programming depending on their personal
interests and future job plans. It can be anticipated that randomly selected students will
not represent the whole department student populations.
Second, only one virtualization solution will be evaluated. The results cannot be
generalized to all VMware virtualization software. Currently, the Department of NSSA
installs “Remote Laboratory Emulation Systems” on the computing lab. The evaluation
results may not be appropriate for the future ones that will be developed.
Third, the data for this evaluation will be collected over a brief period of time
during the Summer Semester of 2008. Given the brevity of this evaluation, it will be
especially difficult to find out how effective the software are in influencing transfer to
11
teaching and technology practices in the colleges. Consequently, longer term evaluations
will be needed in the future.
Fourth, the evaluator, Lifang Chang, is a novice undertaking this evaluation for
the first time. Although she will be supervised by an experienced evaluator- Dr. Tonya
Bevedare Amankwatia, it is still a student project and the results should be viewed from
that perspective.
Logistics and Time Line
Couple of people will participate in this evaluation, including faculty and graduate
students from Lehigh University. Their roles and responsibilities are displayed in the
following table. Roles and responsibilities for the Rochester Institute of Technology
participants will be clarified after consultation with the personnel there.
Personnel Role Responsibilities Lifang Chang Principle Investigator Direct the overall project effort;
Prepare initial evaluation plan; Direct and participate in data
collection and analysis; Prepare evaluation reports; Deliver evaluation updates info
to various audiences. Tonya Bevedare Amankwatia Co-Investigator Provide expert evaluation
consultation; Assist with project reports; Deliver evaluation updates info
to various audiences. To be determined IT Graduate Assistant Assist with the preparation of
instrumentation; Assist with data collection and
analysis; Assist in preparing evaluation
reports.
The majority of the evaluation work will take place at Rochester Institute of
Technology. The College of computing and Information Science has agreed to sponsor
12
the evaluation project at Lehigh University. Space and equipment for the IT graduate
assistant will be located in the RIT campus.
This evaluation plan will be revised based on an earlier draft prepared in April
2008. The final evaluation plan will be reviewed, revised, and approved by May 2008.
The initial data collection phase of the evaluation itself will commence June, 2008. The
first interim report will be delivered by July 1, 2008. The final report of this evaluation
will be submitted by July 10, 2008.
Budget
Some parts of the budget are hypothetical since resources are not available to pay
for this evaluation. The detailed proposed budget is showed in the following table.
Item 4/15/08~7/10/08 Personnel Proposed Budget Actual Expenditure Rate Hours Amount
Planning 20h $800 Conducting 4h $160 Analysis 10h $400
Evaluator-Chang
$40/hour
Reporting 20h $800 $2160
Planning 2h $400 Analysis 1h $200
Consultant- Dr. Amankwatia
$200/hour
Reporting 2h $400 $1000
Conducting 3h $195 Analysis 1h $65
IT Graduate Assistant
$65/hour
Reporting 0.5h $32.5 $292.5 Total Personnel $3452.5 Duplication
Clerical costs $150
13
Total Duplication $150 Misc. Administration
Participation Incentives $400
Evaluator Travel $200 Total Misc. Administration $600
Total Evaluation Budget $4202.5
14
Appendix A
Preliminary Interview Questions for faculties Note: This protocol is represented in this proposal as an example of the type of tool we might use in the VMware Workstation evaluation project. 1. Which courses are you teaching in RIT? a. For which courses do you use the VMware system? b. For which courses do you use VMware to assign homework? c. For which courses do you use VMware for in-class activities? d. Is it necessary to use VMware for these classes or is the use of VMware optional? e. Are there any courses that you never use VMware? Please list some? 2. What types of assignments and lab activities do you use VMware for (e.g., creating virtual local networks, programming exercises, etc.): a. on campus? b. off campus? c. Is it necessary to use VMware system for these activities or is the use of virtual system optional? 3. Which operating system do you use in your courses? 4. How could you provide help when students work on VMware system?
a. on campus? b. off campus? … … …
[Continued…]
15
Appendix B
Student Experience Survey Note: This survey is represented in this proposal as an example of the type of tool we might use in the VMware Workstation evaluation project. Circle all choices that apply to you.
1. Participate description: a. Undergraduate student b. MS student c. PhD. student d. Female e. Male
2. VMware system experience: a. I do not know how to use b. Less than a year’s experience c. 1-3 years d. Over 3 years … …. 3. Student use of “Remote Laboratory Emulation System”
a. on campus for in-class activities b. on campus for advanced exploring c. on campus for final assignments d. off campus for final assignments e. off campus for personal interests. f. …
[Continued….]