Upload
lediep
View
238
Download
1
Embed Size (px)
Citation preview
156
CHAPTER 5
SYSTEM IMPLEMENTATION AND TESTING
5.1 Introduction
This chapter describes the implementation and evaluation process conducted on the e-
BSC system. In terms of implementation, the development environment, tools,
development platform, database used and levels of system users are also discussed.
Meanwhile for testing, the type of testing procedure used, participants and analysis
made on the results for the testing procedure are explained.
5.2 Development Environment
The suitable development environment has to be established to ensure that the
implementation process runs smoothly. The following describes the software and
hardware requirement for the development process.
5.2.1 Hardware Requirements
Table 5.1 describes the hardware requirements of the e-BSC system.
Table 5.1
Hardware Requirements
Hardware Description Processor Intel Centrino 1.6 Ghz Processor or higher
or other equivalent processors Memory At least 512 MB
Recommended: 1GB or more Hard disk space At least 50MB Others Internet access
157
5.2.2 Software Requirements
Table 5.2 describes the software requirements for the e-BSC system.
Table 5.2
Software Requirements
Software Description Operating System Microsoft Windows XP or higher Web Server Apache 2.2.6 or higher Relational Database Management System MySQL 5.0.45 or higher Internet Browser Microsoft Internet Explorer Programming languages PHP, Javascript, HTML Web design and Development tool Adobe Dreamweaver MX 2004 (formerly
Macromedia Dreamweaver MX 2004) Image Editor Adobe Photoshop
5.3 Development Tools
5.3.1 PHP
To develop the proposed e-BSC system, PHP (acronym for Hypertext Preprocessor)
was selected as the server-side scripting language. PHP was first developed by Rasmus
Lerdorf in 1995 (Kent et al., 2004), to track the online access of his personal resume.
Since then, the creator improved the language with additional features such as database
support and for web applications development. Subsequently, the creator felt the need
to release the source codes to enable other programmers to contribute in how the
language can be improved. Soon after, many threw in different ideas and even helped
rectify the errors in the system while adding new functionalities to the language.
Now, PHP is managed by the PHP Group and is widely used around the world due to
its numerous strengths. The very fact that it is an open source language, its extensibility
is limitless. In other words, developers around the world can easily write new
extensions to further improve the functionality of the language. Besides that, its syntax
which has been designed to support object-oriented programming makes it an attractive
option for developers who prefer that concept. In addition to that, PHP can be easily
158
embedded into HTML codes to create interactive and dynamic web pages, which
makes it especially appropriate for web development where the PHP code is executed
in the web server which in turn creates the desired web page based on the codes. The
platform compatibility of PHP is so vast that it can be used in most web servers,
operating systems and can work in unison with majority databases. A sample of the
programming codes for the e-BSC system is attached in Appendix I.
5.3.2 Javascript
Javascript is the client-side scripting language used during the development phase. In
certain circumstances, Javascript had to be used instead of PHP. One such instance
includes displaying popup windows to alert users of errors in data entered during
validation of input or possible loss of data when a delete operation is executed. The fact
that the language works in a run-time environment which is especially true in web
browsers, makes Javascript suitable for offering the required services that a server-side
language cannot. However, Javascript is not used to pass sensitive data such as
passwords in view that the codes are executed at the client side where the system can be
vulnerable to malicious attacks.
5.3.3 HTML
HTML, acronym for HyperText Markup Language, is a markup language used to
describe the formatting of text in a document. It is useful in the sense that it allows text
to be structured according to its purpose, namely as a heading, paragraph and so on.
This is accomplished by writing the HTML in ‘tags’ that describes to the web browser
how the text is to be displayed. A scripting language such as PHP and JavaScript can be
easily embedded in HTML to enhance the functionality of HTML.
159
5.3.4 Adobe Dreamweaver MX 2004
Adobe Dreamweaver MX 2004 was selected as the web design and development tool
as it provides an easy way to build a site where graphics and webpage components can
be ‘dragged and dropped’ to the appropriate location. At the same time, Dreamweaver
automatically generates the corresponding code where additional code manipulation
can be made by the programmer. While using HTML alone is enough to build a
website, other technologies such as scripting languages and database supported in
Dreamweaver increases the functionality of the website and its dynamics. The
development environment provided by Dreamweaver provides a simple and easy way
of combining the strengths of PHP, Javascript, HTML and MySQL.
5.4 Development Platform
The Windows XP operating system was selected as the platform for the development of
the e-BSC system. The fact that the potential users are already using and are familiar
with the Windows XP environment played an important role in the selection. Besides
that, a simple observation on the computer labs in the university displayed that the
operating system selected is widely adopted throughout the educational institution.
Since the inception of the operating system, the developers of Windows XP have
continually improved its stability while providing a user-friendly environment.
5.5 Database
MySQL is a relational database management system (also known as an SQL Database
Server) which is widely used around the globe due to it being open-sourced. Most SQL
servers provide reliability but not ease of use unlike MySQL. MySQL is also mostly
platform independent which means it can run on most operating systems such as
Windows and Linux. When using PHP to develop a website, Macromedia
160
Dreamweaver MX 2004 only supports a MySQL database connection as both PHP and
MySQL are tightly connected. Since PHP complements MySQL very well and given
the fact that both technologies are widely used, it is can be certain that thorough testing
has been done.
5.6 Levels of Users
As identified in Chapter 4 there are four types of potential users of the system, namely
SA, faculty deans, appraisers and the academic staff. In this section, the responsibilities
of every user in the admin and faculty module are discussed in detail while
demonstrating how the roles of each relate to the testing procedure that will be
discussed in Section 5.7.
5.6.1 System Administrators (SA)
SAs are responsible for system management and maintenance. The key role played by
the SA involves adding new, editing or removing details of:
KPIs as set at the corporate level.
Evaluation score range values (edit only as these are permanently required by
appraisers to evaluate staff).
Faculties and corresponding information such as departments attached.
Academic staff.
Appraisers (except insertion and deletion as this is the responsibility of the
respective faculty dean).
Other system administrators.
Notices.
161
Since the editing and deleting operations require seeking the correct data, whether KPI,
faculty, academic staff and so on, search features have been implemented to ease this
process. The testing procedure on the faculty module will also involve checking that the
search features work as expected. The insertion, updating and deletion of data have to
be implemented while ensuring referential integrity to make certain that there is data
consistency while avoiding data redundancy. Furthermore, constraints have to be
enforced during insertion and updating processes to avoid illegal data from being
entered.
5.6.2 Faculty Dean
The faculty module provides the functionalities for the faculty dean in terms of faculty
performance planning and management. As the head of the faculty, the dean is
responsible for ensuring that the whole unit works in alignment to corporate strategies
and achieves the expectation of the management thus contributing to the advancement
of the university. This can be accomplished if corporate level goals are well understood
while preserving the alignment of management level KPIs to the faculty’s KPIs. To do
so, the faculty module includes providing features where the dean can select corporate
KPIs to be cascaded into the faculty scorecard with the weight each carries, expected
targets and measures. Hence, the testing procedure for this module requires that this
feature be checked for errors where constraints such as maintaining that the total weight
on the faculty scorecard is equal to 100, changes cannot be made on the scorecard when
the contracting period has started.
The dean is also responsible for assigning the first appraiser for each department
attached to the faculty. Staff has to be first added by the SA in the admin module before
the assignment process can take place. In the testing procedure for this feature, it has to
162
be made certain that each department can only have one first appraiser and therefore
disabled when the assignment has already taken place. Besides that, as the head of the
faculty, the dean has the authority to assign an appraiser to a different department.
Lastly, the dean should be able to view the performance of individual staff, department
or the entire faculty. To view the performance of an academic staff, a search has to take
place first and foremost. Consequently, the performance of the respective individual
can be viewed. The role played by the dean as the second appraiser for evaluating the
performance of all academic staff in the faculty is covered in the appraiser module.
5.7 System Testing
Before beta system testing was initiated, an alpha test was conducted on the integration
of all four modules to ensure that the complete functionality of the e-BSC system can
be presented and tested. Subsequently, a user acceptance test (beta testing) was
conducted in the month of July 2008 in FCSIT and the Chancellery / Administration
building with potential end users of the system, who are namely the lecturers, dean of
FCSIT, members of the human resource department, representative from SPU,
personnel from the IT centre and a BSC expert.
5.7.1 Testing Procedure
Beta Testing was done to demonstrate the functionality of each module and how the
responsibilities of the target users can be accomplished using the system. Besides that,
the purpose of the procedure was to gather feedback through a system evaluation
questionnaire distributed to the participants with regards to the system’s usability and
fitness for purpose.
163
Prior to conducting the test, a brief explanation of research objectives and descriptions
of system modules with the roles and responsibilities played by each type of user was
done. Consequently, a system demonstration was conducted to show the functions and
features of the system. The participants were then presented with a system evaluation
questionnaire (appended in Appendix J) to express their feedbacks towards the
developed prototype. Subsequently results from the user acceptance test are used to
substantiate the research in terms of improving PM and planning for lecturers as well as
performance management and planning at the faculty level.
5.7.2 System Evaluation Questionnaire Format
Participants for the user acceptance test were required to complete the system
evaluation questionnaire that attempts to assess the usability of the prototype while
verifying that the system truly delivers its business functions required. Subsequently,
the results from the system evaluation test will determine if the e-BSC can indeed
improve the performance measurement of the lecturers in UM. The questionnaire is
divided into the two sections, where:
i. Section 1 is divided into 4 parts each pertaining to the four system
modules. Each part consists of 5 to 8 questions which require the
participants to evaluate specific features of the system based on a 5-level
Likert scale with the values 1 for ‘Strongly Disagree’, 2 for ‘Disagree’, 3
for ‘Not Sure’, 4 for ‘Agree’ and 5 for ‘Strongly Agree’. The charts in
Figures 5.1 to 5.3 and Tables 5.4 to 5.9 are based on the similar 5-level
Likert scale.
ii. Section 2 entails the overall evaluation of the e-BSC system, specifically
the system’s rate of complexity and its effectiveness in measuring the
performance of the academic staff. In addition to that, the participants were
164
also required to rate the improvements, if any, of the system as compared
to the staff PM system currently used.
5.7.3 Participants of the System Evaluation Test
The following table lists the participants of the system evaluation.
Table 5.3
List of participants in e-BSC system evaluation
No. Name Role Date Tested 1 Respondent 1 Academic Staff 14 July 2007 2 Respondent 2 Academic Staff 21 July 2007 3 Respondent 3 Academic Staff 14 July 2007 4 Respondent 4 Academic Staff 18 July 2007 5 Respondent 5 Academic Staff / Appraiser 17 July 2007 6 Respondent 6 Academic Staff / Appraiser 17 July 2007 7 Respondent 7 Academic Staff / Appraiser 16 July 2007 8 Respondent 8 Academic Staff / Appraiser 17 July 2007 9 Respondent 9 Academic Staff / Appraiser / Dean 24 July 2007 10 HR Representative Human Resource (top
management) 25 July 2007
11 HR Representative Human Resource (top management)
25 July 2007
12 HR Representative Human Resource (top management)
25 July 2007
13 HR Representative Human Resource (top management)
25 July 2007
14 IT centre representative SA 25 July 2007 15 IT centre representative SA 25 July 2007 16 SPU representative SPU (top management) 16 July 2007 17 Respondent 10 BSC expert 25 July 2007
Though the participants held different roles, all of them were presented with the same
questionnaire to give their personal opinions with regards to each system module and
the overall e-BSC effectiveness in measuring the performance of the academic staff in
UM.
165
5.7.4 Test Data Analysis
An analysis on the results obtained from the e-BSC system evaluation was carried out
to determine the feedbacks from users with regards to the effectiveness and usability of
the system. As explained earlier, the system evaluation questionnaire is divided into 2
sections where the first contains 4 distinct parts with questions pertaining to the
respective system module while the second section contain questions with regards to
the overall performance of the system. Since this portion of the study focuses on the
faculty and system administration modules, only the results from these sections and the
overall evaluation of e-BSC will be discussed.
5.7.4.1 Results of the evaluation on the System Administration module
The chart in Figure 5.1 reveals results obtained from the evaluation on the system
administration module.
12
59
29
47
29
24
18
53
29
12
59
29
35
47
18
0%10%20%30%40%50%60%70%80%90%
100%
Q1 Q2 Q3 Q4 Q5
Questions
Evaluation on System Administrator Module
Not Sure Agree Strongly Agree
Perc
enta
ge (%
)
Figure 5.1
Results for the Evaluation on the System Administration Module
166
Based on the results, positive results were evident for Questions 1, 3 and 4 which
measured more agreement that the system displayed straightforward, sufficient of
warnings for drastic actions and constraints to prevent users from performing illegal
operations. Table 5.4 clearly displays the corresponding frequency distribution for the
chart in Figure 5.1.
Table 5.4
Results for the Evaluation on the System Administration Module Frequency
Distribution
Q1 Q2 Q3 Q4 Q5 Strongly Agree 5 4 5 5 3 Agree 10 5 9 10 8 Not Sure 2 8 3 2 6
Total 17 17 17 17 17
Based on Table 5.4, it is clear that more than half of the participants answered either
agreed or strongly agreed for Questions 1, 3 and 4. The results are further validated by
the mean of responses that were skewed towards agreement as shown in Table 5.5.
Table 5.5
System Administration Module Descriptive Statistics
Question Minimum Maximum Mean Std. Deviation
Q1 3 5 4.18 .636
Q2 3 5 3.76 .831
Q3 3 5 4.12 .697
Q4 3 5 4.18 .636
Q5 3 5 3.82 .728
However, almost half of the participants were unclear about the options for the search
functions and the sufficiency of features for system management and maintenance with
each having eight and six respondents who responded uncertainty that the system
167
displayed those features. One possible reason for the response is because these features
were not highlighted during the system demonstration which emphasized more towards
the value of the system in improving performance management and PM of academic
staff.
Another point to consider is the standard deviation showing the dispersion of data
collected compared to the mean value. For questions 2 and 5, it can be seen that the
standard deviation (Table 5.5) recorded higher than other values causing the
consistency of responses to be queried. However, the inconsistency of data which led to
the assumption will not be used as an excuse that the module does not need further
refinement. Instead, all responses obtained will be taken into account to further
investigate the possible weaknesses of the module and propose these for future
enhancements. Nevertheless, no negative results to disagree that the system delivered
specific purposes were noticeable among the responses.
168
5.7.4.2 Results of the evaluation on the Faculty module
Figure 5.2 and Table 5.6 illustrates the results of evaluation on the faculty module of
the e-BSC system.
66
65
24
12
59
29
12
65
24
12
59
29
18
59
24
12
59
29
18
59
24
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q1 Q2 Q3 Q4 Q5 Q6 Q7Questions
Evaluation on Faculty Module
Disagree Not Sure Agree Strongly Agree
Perc
enta
ge (%
)
Figure 5.2
Results for the Evaluation on the Faculty Module
Table 5.6
Results for the Evaluation on the Faculty Module Frequency Distribution
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Strongly Agree 4 5 4 5 4 5 4 Agree 11 10 11 10 10 10 10 Not Sure 1 2 2 2 3 2 3 Disagree 1 0 0 0 0 0 0
Total 17 17 17 17 17 17 17
Based on the results, it is clear that most of the participants agree or strongly agree that
the methods used in the module as sufficient for faculty performance management.
169
The result is further substantiated with the descriptive statistics shown in Table 5.7.
Table 5.7
Faculty Module Descriptive Statistics
Question Minimum Maximum Mean Std. Deviation
Q1 2 5 4.06 .748
Q2 3 5 4.18 .636
Q3 3 5 4.12 .600
Q4 3 5 4.18 .636
Q5 3 5 4.06 .659
Q6 3 5 4.18 .636
Q7 3 5 4.06 .659
In the table, the average responses for the questions ranged from 4.06 to 4.18 which
showed strong inclination toward agreement that the module fits its purpose.
Meanwhile, only a small proportion of the results showed uncertainty with regards to
the features provided in this module as illustrated in the chart for Question 1.
5.7.4.3 Results of the evaluation on the overall e-BSC system
Figure 5.3 shows the evaluation on the effectiveness of the overall e-BSC system.
Evaluation on e-BSC
6 6 66
2918
29 1847
59
12
59
59
59 4141
3524
53
2912
24 29 3518 18
29
0%10%20%30%40%50%60%70%80%90%
100%
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Questions
Strongly AgreeAgreeNot SureDisagree
Perc
enta
ge (%
)
Figure 5.3 Results for the Evaluation the overall e-BSC system
170
Table 5.8 displays the corresponding frequency distribution for the chart in Figure 5.3.
Table 5.8
Results for the Evaluation the overall e-BSC system Frequency Distribution
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Strongly Agree 5 2 4 5 6 3 3 5 Agree 10 10 10 7 7 6 4 9 Not Sure 1 5 3 5 3 8 10 2 Disagree 1 0 0 0 1 0 0 1
Total 17 17 17 17 17 17 17 17
In the results, most of the results agreed that the e-BSC system delivered its purpose.
However, uncertainty is obvious in the responses for questions 6 and 7, where eight and
ten respondents respectively answered “unsure” when evaluating the effectiveness of
the system in aligning strategies at the university level with that of the faculties and
supporting staff to develop positive work ethics.
As recorded in Table 5.9, the mean values for these two questions also recorded
indecision among the responses.
Table 5.9
e-BSC system Descriptive Statistics
Question Minimum Maximum Mean Std. Deviation
Q1 2 5 4.12 .781
Q2 3 5 3.82 .636
Q3 3 5 4.06 .659
Q4 3 5 4.00 .791
Q5 1 5 4.00 1.061
Q6 3 5 3.71 .772
Q7 3 5 3.59 .795
Q8 2 5 4.06 .827
It is important to note that the result for Question 6 does not concur with the result for
Question 2 for the faculty module evaluation as shown in Figure 5.2 where the former
records as much as 47 percent of uncertainty while the latter shows 88 percent (59%
171
+29%) agreement that the e-BSC is effective in organisation wide alignment of
strategies. Nevertheless, this result will be used to fuel efforts for future enhancements
on the system to make these features more obvious and easily understood by end-users.
Figure 5.4 illustrates the results for the evaluation on the complexity of the e-BSC
system.
Figure 5.4
Complexity (Learnability) of e-BSC System
From the results, it is clear that 100% of responses from the target users who are the
staff evaluated the system as average or moderately easy to use. Similar results are
observed for responses from users other than the human resource and IT centre
personnel. Meanwhile, the dean expressed that the system as easy to use. Only a
minimal percentage of responses from the human resource and IT centre
representatives rated the system as complex. Otherwise the system has proven to be
easy to learn as can be seen in the summary result for the complexity/learnability of e-
BSC as exhibited in Figure 5.5.
172
Figure 5.5 and Table 5.10 depicts the summary result for the complexity of e-BSC
without the separation of ratings from each category of participants.
Ea s y to useModera te / Average
ComplexVery Complex
S1
6%
82%
12%0%
0
20
40
60
80
100
Summary results for Complexity/Learnability of e-BSC
Perc
enta
ge (%
)
Figure 5.5
Summary results Complexity (Learnability) of e-BSC System
Table 5.10
Summary results Complexity (Learnability) of e-BSC System Frequency
Distribution
Complexity Frequency Percent (%) Easy to learn/use 1 6 Moderate/Average 14 82 Complex 2 12 Very Complex 0 0
Total 17 100
If compared to Figure 4.15 in Chapter 4 which illustrates the evaluation of respondents
on the PM system currently used in UM, e-BSC has demonstrated that none of the
participants of the system evaluation felt that the system is too complex to learn unlike
the current system which recorded 5% in that category. In addition to that, if the
positive results for the evaluation, in particular ‘easy to use’ and ‘moderate/average’,
are compared with the results for the current system, e-BSC responses totaled to 88%
173
(6% easy to use + 82% moderate/average) which showed a slight increased in
satisfaction with its deliverables while the current PM system only recorded a total of
85% (20% easy to use + 65% moderate/average).
Figure 5.6 displays the results for the ratings from every category of participants on the
effectiveness and quality of the e-BSC system. Most of the responses showed positive
results.
Figure 5.6
Quality (Effectiveness) of e-BSC System
174
Figure 5.7 illustrates the summary of the response. When contrasted with figure 4.16 in
chapter 4 which shows the evalution on the quality of the current PM system used, the
results for e-BSC showed significant improvements. Unlike the current system which
showed a high of 20% of respondents rating it as not suitable, the evaluation on e-BSC
did not record any negative feedbacks. Instead, some 12% (from two participants out of
the seventeen) even rated the e-BSC as very effective in delivering its purpose.
Figure 5.7
Summary results Quality (Effectiveness) of e-BSC System
175
Figure 5.8 below demonstrates whether e-BSC displayed any form of improvements in
comparison to the current PM system.
Figure 5.8
Comparison of e-BSC System to the Current PM system
From the results, most participants considered e-BSC to have some enhancement and
improvement in terms of effectiveness compared to the current system. Only a small 25
percent of staff and human resource respectively (one person from each category)
revealed uncertainty in their responses.
176
Meanwhile, the Figure 5.9 shows the summary of ratings with regards to the
improvements in e-BSC compared to the current PM system.
Figure 5.9
Summary Comparison of e-BSC System to the Current PM system
As observed, the pie chart shows a feedback with a high of 47 percent, from eight of
the participants who evaluated the system as delivering significant improvement in
contrast to the current system. Also, seven participants who make up 41 percent rated e-
BSC to have slight improvement over the current system. No more than 12 percent
(from two participants) recorded uncertainty for that query.
5.7.4.4 Comments from participants
During the e-BSC system demonstration, several comments from the participants were
also recorded as recommendations for future enhancements purposes. Most of the
comments were related to the appraiser, faculty and system administrator module where
enhanced features had to be provided to ensure that the system truly performs
alignment of all strategies while supporting user friendliness. Meanwhile, some
177
comments targeted on how to further ease data entry efforts. Please refer to Appendix
K for the detailed comments and the contributors.
At the same time, written comments on the system evaluation forms were also tabulated
for easy comprehension of participants’ expectations and feedbacks on the e-BSC
system. Table 5.11 summarizes the comments as follows:
Table 5.11
Summary of Written Comments from System Evaluation
Participant Category Comment(s)
Staff My response is only based on what has been demonstrated during the meeting. I don’t have prior knowledge and experience with the system before.
System is adequate (referring to question 10) but still have rooms for improvements. Good effort and good system.
Predefine the ISI journals’ titles and match that with staff’s records.
Very good. Appraiser A good effort. Well done. Particularly like the interface! More
variables and lookup tables needed to facilitate data entry. Well done.
Dean Well structured and very intuitive.
IT Centre Needs to integrate with existing information systems and to the corporate scorecard so that it is cascaded upwards.
SPU e-BSC must have features that will enable modifications to suit/accommodate feedbacks on the needs and requirements of users. System rigidity will kill it! Ease of use is of utmost importance.
178
5.8 Summary
In this chapter, the system development environment and programming tools used were
discussed in detail. Adding to that, the system evaluation procedure conducted on the
finished product was explained. From the results obtained, it is obvious that the e-BSC
is effective in meeting its objectives though there is still room for improvement to
further enhance the features of the system. Nevertheless, satisfaction with the e-BSC
was generally observed in the participants’ expression during the system demonstration
to show their contentment with its features. Compared to the current PM system used in
UM, e-BSC has shown to be suitable for measuring and managing the performance of
academic staff.