Communication requirements and network evaluation within electronic meeting system environments

Decision Support Systems 7 (1991) 13-31 13 North-Holland

Communication requirements and network evaluation within electronic meeting system environments

Alan R. Dennis, Tom Abens, Sudha Ram and J.F. Nunamaker Jr. Department of Management Information Systems, College of Business and Public Administration, Universi O, of Arizona, Tuc- son, AZ 85721, USA

One of the key issues in the design and implementation of Electronic Meeting Systems (EMS) is the identification of appropriate support for computer communications. Lack of adequate communicat ion support can create a bottleneck in the use of EMS. Our objective is to identify the communicat ion needs of several different EMS environments and to discuss an approach that can be used to evaluate the required communication support. We then use this approach to benchmark the performance of two Local Area Network (LAN) operating systems (IBM LAN program and Novell Netware) and three network servers (IBM PS/2 models 50, 60 and 80) in one EMS environment. While network operating systems have received comparatively less attention in LAN evaluation and design, fully 75% of the response time in some configurations was due solely to the network operating system. The Novell Netware software running on the IBM PS/2 model 50 server provided at least as great speed at lower cost than any other configuration tested. For the small files common to one style of EMS environment (i.e. under 2.5K), response time was not affected by the size of the file but rather was determined by the fixed overhead imposed by the network operating system.

Keywords: Electronic meeting systems, Group support systems, Local area networks, Performance evaluation

Alan Dermis is a doctoral candidate in Management Information Systems at the University of Arizona. He received a Bachelor of Computer Science from Acadia University and an MBA from Queens Univers i ty in Kings ton , Ontario, and was a 1987 winner of the AACSB National Doctoral Fellow- ship. Mr. Dennis has published articles in MIS Quarterly, Information and Management, Computers & Graphics, and DataBase. His current research interests include electronic meeting

systems and management graphics. * We would like to acknowledge the research assistance pro-

vided by Frank DiMaggio, Shiow-Jiuan Huang, Sandhya Sathe, and Jane Stodola, and the helpful comments of Joey George and Mark Pendergast on an earilier draft of this paper.

budgeting process.

Tom Abens is a Computer Applica- tions Specialist for the University of Arizona. He holds a Bachelor of Sci- ence in Finance and Economics (Magna Cum Laude) from Northern Illinois University, and is enrolled part time in the Master of Science in Managemen t Informat ion Systems pro.g(.am. Mr. Abens has held various positions in the financial and aeros- pace industries, and is currently devel- oping database systems to more effi- c ient ly m a n a g e the Univers i ty ' s

Jay F. Nunamaker, Jr. is Head of the D e p a r t m e n t of M a n a g e m e n t of Management Information Systems and is a Professor of MIS and Computer Science at the University of Arizona. He received a PhD from Case In- stitute of Technology in systems engineering and operations research. He was an Associate Professor of Com- puter Science and Industrial Adminis- tration at Purdue University. Dr. Nunamaker joined the faculty at the University of Arizona in 1974 to de-

velop the MIS program. He has authored numerous papers on electronic meeting systems, the automation of software con- struction, performance evaluation of computer systems, decision support systems for systems analysis and design, and has lectured throughout Europe, Russia, Asia, and South America. Dr. Nunamaker is Chai rman of the ACM Curriculum Commit- tee on Information Systems.

Sudha Ram is currently an Assistant Professor of Management Information Systems at the University of Arizona. She received her Ph.D. in Manage- ment Information Systems from the University of Illinois at Urbana- Champaign, in 1985. Dr. Ram has published in journals such as Com- munications of the ACM, Information Systems, Information Science, and Journal of Systems and Software. Her research interests are in the areas of distributed database and knowledge

based systems, semantic modeling, automated tools for database design, and application of knowledge based systems in business. Her research in these areas has been funded by IBM, US Army, The National Institute of Standards and Technology and The Marketing Science Institute. In configuring the networks, the default parameters were used for Novell Netware. For IBM PC LAN, parameters were set to use 10K network buffers at both the server and user workstations, with four buffers on the server. Time slicing on the server was set as recommended for a dedicated server [14, p. 12-171. One mega- byte was used for caching.

0167-9236/91/$3.50,0 1991 - Elsevier Science Publishers B.V. (North-Holland)

14 A.R. Dennis et al. / Communications requirements and network evaluation

1. Introduction

In recent years, there has been rapidly growing interest among both researchers and practitioners in the use of information technology to support group work [e.g. 10, 26]. Many different terms have been used to describe this use of information technology, including Group Decision Support Systems, and Computer Supported Collaborative Work [13,17,31]. While these systems are used to support collaborative work and to make decisions, they are also used to support a much wider variety of group tasks [10,17]. For this reason, we prefer the term Electronic Meeting System (EMS), which has been defined as an information technology- based environment to support group meetings, which may be dispersed in space and time [6].

Most early efforts to develop EMS met with very limited success [17]. In 1986, Kraemer and King identified three barriers to the widespread use of EMS technology [16]. First, developers and researchers had an incomplete understanding of group work processes. Second, professional qual- ity EMS systems were not readily available. Third, there were problems with the underlying technology to support EMS, including a lack of adequate systems to support computer communications. The importance of adequate network support for communications has also been noted by other researchers. In 1987, network efficiency was cited as one of three major factors inhibiting effective support of group work [20]. Providing adequate network support is a crucial issue in the design of EMS environments.

These barriers to success are gradually being overcome. First, while we still have an incomplete understanding of group work processes, our knowledge of computer support for group work processes is expanding [see 6, 17 for reviews of EMS literature]. Second, several new EMS systems have been developed [e.g. 8,11, 20, 31, 34]. Third, since 1986, the underlying technology to support microcomputer-based EMS has continued to evolve. Microcomputers have become faster and more powerful as has local area network hardware and software.

The purpose of this paper is to describe the communication needs of several different forms of EMS technology and to benchmark t hepe r fo r - mance of two local area network (LAN) software packages and three network servers for one EMS

environment. Section 2 discusses the EMS concept, examines several EMS environments and configurations, and presents general approach to the evaluation of communication systems to support EMS. In the third section, we examine previous LAN evaluations. The fourth section presents the communication requirements of one specific EMS environment, and describes the benchmarking methodology used in this study, which is somewhat different from the benchmarking strategies discussed elsewhere (e.g. 26). The fifth and sixth sections present the results and discuss their implications, respectively. We conclude with a summary of our observations, and a discussion of directions for future research.

2. Evaluating Communications Support in EMS Environments

2.1. The EMS Concept

One primary purpose of EMS is to enhance communication among group members [10,13,16]. While there are several ways that a EMS can enhance group communication, one major contri- bution is the addition of a new communication channel. With a EMS, each group member has a workstation (although it could be shared with another group member), linked via a computer network to other group members' workstations. Each member can use the workstation to com- municate with other group members, although in many EMS environments, group members can have verbal conversations as well.

While the tasks performed by groups (e.g. strategic planning, operational decision making, negotiation, collaborative writing) are different, they often share common activities, such as idea generation, idea organization and voting/decision making [13]. More recent EMS systems have been designed as tool kits, similar in concept to a Decision Support System (DSS) model base or tool set [28] EMS tool kits are collections of specific tools that support specific group activities, rather than entire business functions. Fig. 1 il- lustrates some activity-based tools that could be used to support specific business functions; this figure is certainly not complete, but does provide an illustration of the basic needs of several business functions. For example, strategic planning

A.R. Dennis et al. / Communications requirements and network evaluation 15

Business Funct ions

Strategic | I Planning ~

N e g o t i a t i o n ~

I Psro°lblie;

i Systems r ~

oo*'o eV / /

IPr°. pOsal V "

Fig. 1.

Group Ac t i v i t i es

i Idea I ~ Generatio 1

2g~_ Idea I U prganizatio 1

~ ' ~ Issue I ~ ~xploration I

Decision | Making I

\ ~, Impact | N Analysis I

Policy I !Formation I

often involves a wide variety of activities, from idea generation and organization to impact analysis and the drafting of specific policies. In contrast, writing a proposal will often involve fewer activities, perhaps just an initial idea generation phase where key issues are identified, followed by a more in-depth exploration of each issue that needs to be addressed in the proposal.

The key advantage provided by EMS tool kits is flexibility. There may be a variety of tools to support each activity. For example, the EMS developed at the University of Arizona provides several idea generation tools, such as Brainstorm- ing, Nominal Group Technique and Delphi. Each tool in the tool kit will have its own meeting

dynamics. One tool in the tool kit may support a highly structured interchange of ideas via the computer supported electronic communication channel, while another tool may encourage more verbal discussion. Thus the electronic communication requirements of a EMS will depend on the specific EMS tools used to support the group meeting.

2.2. EMS Environments

The environment within which EMS software tools are used also has a significant impact on the communications support required. Several classifi- cations for EMS environments have been developed [eg. 9, 10, 15]. Fig. 2 presents a taxonomy of EMS environments that integrates and builds on earlier EMS categorizations [6]. Although only the front six blocks are labelled (i.e. synchronous meetings), these labels also apply to their asynchronous counterparts.

With a Decision Room EMS environment, a small group of participants meets at one time in one room equipped with a EMS and wide screen computer video projection screens for public view- ing of comments. Face-to-face (e.g. verbal) group interaction is available as well as EMS-supported electronic interaction. Legislative Sessions differ from Decision Room sessions only in size (they accommodate larger groups) and in the degree of communication sophistication required of the EMS software. While verbal communication is still pos-

I

Group / - - .

Size I

,. :: : " All meet : at one Multiple One Multiple time

Individual Group Group Sites Site Sites

Group Proximity Fig. 2. EMS Environment Taxonomy.

Asynchronous "Meetings"

T i m e

D i s p e r s i o n

( f rom Dennis , et al., 1988)


sible with a large group, it is less effective. Either the equal participation of all group members is removed, or, if equal participation occurs, each participant has far less time in which to com- municate his /her ideas and opinions than he / she would in an equivalent small group meeting. Thus supporting a Legislative Session will place more of a load on a LAN than supporting a Decision Room session, as there is a higher volume of electronic communication.

A group of dispersed individuals could use a Local Area Decision Net to support small group work or Computer Conferencing for larger groups. Both Local Area Decision Nets and Computer Conferencing can be used by groups meeting at the same time, or at different times. The demands placed on the supporting LAN increase, as verbal communication is no longer possible. For other- wise equivalent meetings held in a Decision Room and Local Area Decision Net, the Local Area Decision Net will require more electronic communication.

When several groups (of any size) meet in separate locations, EMS Teleconferencing facilities are used. This is similar to non-EMS supported teleconferences, but with the addition of a EMS to facilitate intra- and inter-group communication. Each group works within itself, but transmits its results to the other groups for consideration in their subsequent sessions. The LANs supporting a EMS Teleconference need to be more sophisti- cated, as they must support both intra-group communication and inter-group communication. Long distance information transmission between LANs also becomes an issue.

2.3. Decision Room Meeting Processes

Most EMS research to date has focused on the use of decision rooms, although research is begin- ning to examine distributed environments [6]. De- velopers have taken rather different approaches to the basic design of EMS Decision Room technology. We have identified three types of EMS-supported group work process requiring network communication support, which we term a supported meeting process, graphically supported meeting process, and an interactive meeting process. Some EMS are capable of supporting two or even all three styles of meeting process, and thus it can be difficult to classify entire specific EMS into

one category. However, each type of meeting process places different demands on the supporting network software, and thus in evaluating network software, the specific meeting process should be considered. It should also be noted that meetings observed in field studies have used more than one meeting process at different stages of a given meeting to address different sub-tasks.

With a supported meeting process, each group member has a workstation that is used to com- municate ideas, comments and opinions to other group members. A large public display screen is also used to provide a repository for the group's information, thus providing an electronic version of the traditional blackboard, whiteboard or flipchart. The group verbally discusses the issues under consideration, with the electronic blackboard used to record and structure information the group deems appropriate. Any member of the group can work with the information on the blackboard, thus enabling some work to be done in parallel - i.e. two or more group members can add information simultaneously. The meeting pro- ceeds using a mixture of sequential and parallel processing; participants can work in parallel using the electronic channel, but work sequentially when using the verbal channel. This type of EMS potentially provides both formal / informal structure and an additional communication channel. Examples of supported meeting processes include work done at the University of Minnesota as reported in [8].

A graphically supported meeting process is similar to a supported meeting process, but differs in that the basic form of information and information presentation is graphical. Rather than the lists of textual information found in the supported meeting process, information is typically displayed with lines, boxes, windows, etc. visually displaying relationships, precedence, and interconnections. Examples of graphically supported meeting processes include work done at Xerox PARC as reported in [31].

An interactive meeting process is distinctly different from either a supported or graphically supported meeting process, in that the electronic communication channel provided by the EMS is used for most group communication. During an electronic meeting process, virtually no one speaks. All participants work in parallel, sending typed comments to other participants without waiting for others to finish "speaking." While an elec-

A.R. Dennis et aZ / Communications requirements and network evaluation 17

tronic blackboard may be provided, the group information is typically too large to fit on one screen, and thus group information is typically maintained so that all group members can access it electronically. This parallelism potentially en- ables more to be accomplished in less time. While typing comments into a computer is slower than speaking, reading is faster than listening [12]. In this case, the EMS can contribute structure, a new, fully parallel, communication channel, and a re- cording of the meeting. Examples of interactive meeting processes include work conducted at the University of Arizona as reported in [20,21,22].

We expect that as each of these different styles of meeting has different effects on the group work processes of the meeting, some may be better than others for specific tasks, groups, and organiza- tions. For example, in larger groups the major problem with a traditional non-automated meeting is communication among group members. For a meeting of a given length, the amount of time available for each member to speak is directly reduced as group size increases. Either everyone participates in the same proportion for less time, or participation becomes less equal, with a few group members dominating the meeting. The abil- ity of a supported or graphically supported meeting process to improve communication in a large group is more limited than that of an interactive meeting process as much communication is still verbal, and thus suffers from these same problems. Therefore, for a large group, we speculate that the use of an interactive meeting process would be more appropriate. For small groups, however, verbal communication is likely to remain a viable option, as there are fewer people compet- ing for "air time." As verbal communication is a more natural form of communication, we speculate that for small groups, a supported or graphically supported meeting process could be more appropriate.

Each of these three meeting processes differ in the network support they require. A supported meeting process often generates a small number of extremely short text messages. Messages can typically occur in bursts, as members work in phases of verbal and electronic communication (i.e. "let 's generate some alternatives now"). Graphically supported meetings are similar, except that the exchange of graphical images requires much larger message sizes. Interactive meetings, on the other

hand, have continuous exchanges of short to medium length text messages.

2.4. Factors for Evaluating Communication Support

In the previous sections, we have seen that EMS can provide support for several business functions. This support is provided by a collection of tools providing different meeting processes. In order to provide meeting support, one of the most important infrastructural facilities required is adequate electronic communication technology. Given the wide array of communication technology available in the market today, it is indeed very difficult to select the appropriate type of communication support. This section provides an approach for evaluating communication alternatives, specifically, local area networks.

Several factors can be used to evaluate LANs. These factors can be subdivided into three categories: technical factors that are generally applica- ble to local area networks, factors that need to be considered in the light of the specific application (in this case EMS tools), and miscellaneous factors (see table 1). The technical factors involve choice of topology, transmission technique, communication medium, and medium access protocol. These are the standard factors used in evaluating LANs [301.

However, it is also important to consider factors specific to the nature of the application [4,23]. Since we are dealing with a set of EMS tools, each tool may perform a different activity. Hence, the traffic patterns generated by each tool may be

Table 1 Factors for Evaluating LANS.

Technical Factors

Application Specific Factors (for EMS)

Miscellaneous Factors

Topology Transmission Technique Communications Medium Medium Access Protocol Type of EMS Traffic Patterns Size of Files Number of Users Group Proximity Cost Vendor Support Adherence to Standards Reliabihy Ease of Installation Security

18 A.R. Dennis et al. / Communications requirements and network eoaluation

different. For some EMS tools the traffic stems from the workstation to the network server and back. These transfers typically involve fairly small files that keep growing in size but not significantly. For other EMS tools, the file sizes may be large and flow of traffic may be from the network server to each workstation. The size of groups using the tools is yet another factor to be taken into account when evaluating the performance of the communication system. Currently, most EMS tools are used in a single location, usually one Decision Room. In the future, we expect group proximity to be an issue in evaluating the communication support. Several groups that are geo- graphically dispersed may be simultaneously using the tools. Intergroup communication becomes an important issue in such a situation. Further, the dispersed groups may each be using a different type of LAN. This would require linking heteroge- neous LANS together. Video, audio and teleconferencing also require transfer of voice and graphics (images) in addition to data and text. The nature of the tool determines the type of data transferred.

Evaluation of the communication and network system support should consider the effect of each of these factors on the performance of the system. Such a study would help to determine the relation- ship between the response time and the factors outlined above. However, it is not enough to choose the configuration that provides the mini- mum response time. While minimizing response time is important, we should not lose track of another set of miscellaneous factors that will help in choosing the appropriate communication system. These factors are vendor support, adherence to standards, and total cost of the hardware and software used for communication. The choice of an appropriate communication system is a multi criteria decision.

3. Previous LAN Evaluations

Traditionally, there have been three approaches to studying the performance of communication networks: analytical modeling, simulation, and benchmarking [25,33]. In the evaluation of LANs, analytical modeling, where mathematical models are built based on network characteristics, seems to be the most common approach [1,29]. In many

cases, models used in the evaluations do not apply to a specific LAN product, but rather apply to a class of LANs. For example, many articles com- pare medium access protocols (eg. token ring architecture to slotted ring architecture or CSMA- CD architecture) leg. 2, 30, 32].

However, as Bux observes:

Models of the above type are suitable to assess the performance characteristics of different access mechanisms (which usually im- ply a certain network topology) and thus are helpful in finding good network design. Such models, however, are not appropriate for determining application-oriented performance measures. [1, p. 351].

As well as being inappropriate for determining performance for specific applications (e.g. EMS), such models are also inappropriate for evaluating specific network products. Due to implementation issues in the network hardware and software design, products that appear to have the same network performance based on an analytical model or simulation, may have considerably different performance in practice [27]. Finally, these models do not reflect the kind of performance that an actual user could expect, as they do not address the needs of a specific application [3]. Therefore, in cases where we wish to examine specific network products for specific applications to determine the level of performance that a user could expect, direct performance benchmarking via a controlled experiment is most appropriate.

Although several experimental benchmarking studies of LAN performance have been done, most of these have examined the transfer of raw data between two operating systems, rather than examining the performance that a software application would have in these environments [3]. An exception is a study that examined the performance of an Ethernet network under " typical" traffic patterns in a specific application environment of networked printers, shared data bases, and down loading of data and programs [27]. Another study also examined application performance, but used an old implementation of an Ethernet network to examine general purpose LAN applications [3]. While these studies are useful, technology has dramatically changed in the years since they were conducted [24].

More recently, several experimental evaluations


of LANs have been conducted. One study examined the performance of the 3Comm Ethernet network using three different computers as the network server (an IBM XT, and IBM AT, and a 3Comm 3Server) [23]. Data was written to, and read from, each server in two groups (20 blocks of 1500 bytes, and 600 blocks of 50 bytes) by 1 through 12 workstations. Mean response time at each workstation increased exponentially with the number of workstations for both IBM servers, in both the read and write tests. After 4 workstations for the IBM XT and 6 for the IBM AT performance degraded sharply. In contrast, mean response times increased linearly with the number of workstations for the 3Comm 3Server.

In another study, four configurations were tested: (1) 3Comm Ethernet network hardware and software with a 3Comm 3Server network server; (2) 3Comm Ethernet network hardware and software with an IBM AT network: server; (3) 3Comm Ethernet hardware and Novell Netware software with an IBM AT network server; and, (4) IBM PC Network hardware and PC LAN software with an IBM AT network server [24]. Data was written to, and read from, the server in each configuration in three test sizes (750 bytes, 6K and 30K) by 1 through 15 workstations.

With the smallest data sizes, the Novell configuration performed the best, with mean workstation response times being essentially unaffected by increasing the number of workstations on the network. The pattern was similar for the pure 3Comm configuration until 10-11 workstations were added, whereupon response times began to increase. Both IBM configurations produced dramatically longer response times rising linearly with the number of workstations. With 15 workstations, the IBM PC Network configuration produced a response 20 times longer than that of Novell. For larger data sizes, response times increased more directly with the number of workstations on the network, but with the same general performance pattern among the configurations. Performance studies conducted by Novell have also shown Novell to outperform other network software [19].

The LAN evaluations discussed above provide some guidance in selecting an appropriate LAN for EMS applications, but most have not specifically addressed the data transfer needs of EMS. In summary, most previous evaluations have examined now obsolete LAN technology, focused on

the non-EMS applications, or tested LANs with a smaller number of workstations than are often available in EMS applications.

4. Benchmarking Methodology

4.1. Benchmarking Strategy

In evaluating the performance of LANs, it is important to conduct network tests in the same environment, and under the same conditions as a typical user of the target application [4,23]. The challenge is how to provide this environment in a consistent, yet realistic manner in all test conditions. Press [25] addresses this problem by adopt- ing a two tier strategy. First, a program is developed to simulate the typical background activities tha t occur in the network, such as file transfer, file opening and closing, etc. This program is then started on several "background" workstations to generate typical network traffic. Second, one "foreground" workstation runs the application being tested, and records the actual response times encountered.

In this case, we determined that it would be more useful to examine the actual application environment with all workstations participating in the tests, rather than use a series of simulated "background" workstations running simulated tasks. This also had the advantage of providing application specific timing data from all workstations, not just one "foreground" workstation. As Press [25] noted, and as we observed in this study, there can be considerable variation in the mean response time recorded at each workstation, even among seemingly identical trials, depending upon the application's and network's conflict resolution strategy. Thus collecting data from many workstations can increase the accuracy of the timing results.

In this case, we used a special utility that enabled us to control the keyboards of all workstations participating in the test from one control workstation. Using this remote keyboard controller, we were able to conduct the tests with all participating workstations running the specific application software and collecting timing data. As the remote keyboard controller itself was operating across the network underneath the application layer, some degradation in application perfor-


mance could be expected from the extra load caused by transmitting this one keystroke. How- ever, it would be miniscule. Likewise, due to propagation delay over the physical medium, not all workstations would truly be controlled simultaneously. However, the propagation delay would be too small to measure, and was consistent across all test configurations.

4.2. Communications Needs of One E M S Environ- ment

We have seen that different EMS technology configurations will have different communication support needs, and that different tools in those configurations that provide different meeting processes may place different loads on the supporting L A N technology. Defining a " typical" EMS session is impossible, as each category of environment has its own needs. One EMS that has been well accepted is the Group Systems EMS developed at the University of Arizona, which has been installed at more than 18 universities. Group Systems has been used for experimental research, as well as for field study research involving well over one hundred groups from the public and private sectors. It has also been installed at 33 corporate sites, where it is in daily use. Therefore, while identifying a " typical" EMS meeting session is impossible, the wide acceptance and use of this EMS suggests that using previous sessions supported by Group Systems would not be unrea- sonable. Group Systems provides many tools that can be used to support group work, each of which place different demands on the supporting LAN. However, the most commonly used tool is an idea generation tool called Electronic Brainstorming (EBS), which provides an interactive meeting process.

EBS is an idea generation tool that allows group members to simultaneously share comments via a series of networked computers. EBS provides an interactive meeting process. With EBS, each group member receives a file containing a brief question. Each member then enters initial comments and sends the file back out on the network in exchange for another file from another group member. H e / s h e reads this file, adds more comments, and exchanges the file once again. All group members work simultaneously and in parallel. EBS can be configured to run under any

category of EMS, but has primarily been used in Decision Rooms (with 4-12 users) and Legislative Sessions (with 12-24 users). While verbal communication is possible in these settings, past experience has shown that the electronic communication channel dominates the verbal channel [7,20].

What are the communication needs of a " typi - cal" EBS session? EBS is essentially a series of file transfers between user workstations and the network server. Initially, a short file is sent to each user workstation. Comments are appended to this file and it is sent back to the server, exchanged for another file, to which more comments are added. Thus, the size of files transferred in the system is initially small, and gradually grows. From a study of 20 EBS sessions, we determined that the average size of comments added to the file at each workstation was 250 bytes, with an average of 10 file exchanges per user during each 30-45 minute session.

While EBS is one of the cornerstones of the EMS, it is certainly not the only tool in regular use. Many other tools place similar demands on the supporting LAN: frequent transfers of small files to and from the network server. However, some tools, which are used less frequently during sessions, involve the transfer of larger files (primarily from the server to each workstation). For example, Issue Identification (an idea organization tool), typically begins with the transfer of all comments entered during EBS from the server to each workstation. This file of all EBS comments typically ranges in size from 50K to 100K. In other cases, medium sized files (typically 10K in size) are transferred to and from each workstation. In summary then, the most common data communications needs of the Group Systems EMS involves the transfer of small files, which grow in size from 250 bytes to 2.5K. With other tools, medium (10K), and large files (50K-100K) are also transferred across the network.

4.3. Identifying Test Configurations

The performance of a LAN is determined by many factors, as discussed previously. Network hardware, such as the network interface cards installed in the network server and network workstations, and the physical medium connecting the workstations to the server, plays a key role (eg. twisted pair or coax cable). In addition to network


hardware, performance is also influenced by the processors and disk storage media in the workstations and network server.

The network software also plays a key role. The network interface software manages the information transfer between the application software (the EMS), the operating system (DOS) and the network interface card. The network operating system is the operating system running on the network server(s), the computer(s) responsible for managing the operation of the network. In examining LAN performance, it is useful to examine the effects of network operating system characteristics, as well as specific network hardware and interface software, as, in some cases, it may be more efficient or effective to increase LAN performance by upgrading the network server's operating system, rather than upgrading the network hardware or installing a new network server.

4.3.1. Network Hardware and Network Operating

System

The LAN configurations to be tested were cho- sen based upon several subjective and objective measurements. In determining which LAN's to evaluate, one of the primary driving forces behind the decision was the LAN's compatibility with present and future EMS environments. Another important consideration was the customer base of various network vendors. EMS applications will be used within a business environment, as compared to manufacturing or engineering environments. To be considered, the network vendor had to be a leader in the field with a generally positive reputation among business users. Product support also had to be evident in the choice of EMS network configurations to evaluate. In a recent survey of network users by DataPro [5], IBM Token Ring and Novell were perceived by business users to be among the best networks available, in terms of network speed, reliability, vendor support, and overall performance. Novell has been a key figure in the PC LAN market for the last several years and consequently is well represented among the Fortune 500 companies, as well as smaller companies. IBM also has a considerable presence. A final consideration, was the commit- ment vendors have shown to supporting network products, to better ensure that the benchmarked systems would last into the future.

The results of previous evaluations also helped

determine the candidates for our test evaluation. We did not want to duplicate work already done, but rather build upon prior research. As mentioned earlier, two recent studies [23,24] evaluated network hardware for 3Comm Ethernet and the IBM PC Network. The network operating system evaluated in these studies was the Novell Ad- vanced NetWare, 3Comm 3Plus Network, and the IBM PC Network program. Novell's Advanced NetWare outperformed the other 2 network software packages on both hardware networks. Both Hardware configurations were of a C S M A / C D type. Would Novell perform as well in a Token Ring environment?

4. 3.2. Network Servers and Workstations

Microcomputer technology continues to ad- vance at a rapid pace. Development of the Intel 80286 and 80386 chips have led to a new standard in the PC market for business applications: IBM's PS/2 series. In selecting network servers and workstations for evaluation, the choice is essentially between the older P C / X T / A T technology and the newer PS/2 technology. We expect the PS/2 series to increase its presence in the business marketplace in the future. As our objective is to focus on the needs of present and future EMS applications, we chose to evaluate the performance of IBM PS/2 series.

4.4. Test Configurations

Therefore, the network hardware and software evaluated were the IBM PC LAN program, and Novell Advanced NetWare v2.0 running on the IBM Token Ring Hardware. The PS/2 models 50, 60, and 80 were used as network servers for the IBM PC LAN. As the version of Novell Netware we tested would not run on 80386-based microcomputers, only the Model 50, and Model 60 were used as network server for NetWare. We will refer

Table 2 Test Configurations.

Name Server Software

IBM/50 IBM PS/2 50 IBM LAN IBM/60 IBM PS/2 60 IBM LAN IBM/80 IBM PS/2 80 IBM LAN NOV/50 IBM PS/2 50 Novell NOV/60 IBM PS/2 60 Novell


to these configurations as NOV/50 (Novell software with the model 50 server), NOV/60 (Novell software with the model 60 server), IBM/80 (IBM software with the model 80 server), etc. (See table 2).

4. 4.1. Network Software Differences Novell's Netware integrates three software

modules into one comprehensive package. The three software modules are: The Network Operat- ing System; the Workstation Shell; and the Bridge. The Network Operating System is responsible for providing all network services including file, print and software protection services, as well as network security and messaging. The Network Oper- ating System is a multi-tasking (and multi- threaded) operating system. The Workstation Shell provides the means to incorporate the native operating system of the workstations with the network environment. The Bridge is necessary to provide for multiple network interconnections among dis- similar networks. Novell NetWare effectively requires a dedicated network server.

IBM's PC LAN Network does not require the use of a dedicated network server. The PC LAN Program consists of four basic components: PC- DOS 3.3; the Microsoft Redirector, the network server software and network utilities. While the PC LAN program offers some concurrent processing [14, p. 7-85], it is limited. Due to the reliance of DOS for file handling and file structure use and since DOS is a single-user, single-tasking operating system throughput diminishes quickly.

Another important difference between the IBM PC LAN Program and Novell's NetWare Program is the high level of flexibility that Novell offers due to its memory caching, hashing and disk access elevator seeking routines [19]. While both the IBM LAN program and Novell offer caching, Novell also indexes frequently used files and programs with a hashing scheme which reduces the overhead time needed to access the file or program. Another way in which Novell reduces the time required to access information involves an efficient scanning method called elevator seeking. Read/wri te requests are sorted into a priority list based on the current position of the read/wri te head, thus minimizing the disk head movement, t

Thus, a priori, one would expect Novell NetWare to provide better performance than the IBM LAN software in general in a LAN environ-

ment. The question is: is there a performance difference in a EMS application environment, and, if so, is it significant?

4. 4. 2. Network Server Differences Both the IBM PS/2 model 50, and model 60

use the 80286 microprocessor chip, whereas the model 80 uses the more powerful 80386 microprocessor. The model 60 and the model 80 share similar hard disks, while the model 50 has a slower hard disk. A priori, one would expect the model 80 to outperform the model 60, which in turn would outperform the model 50. Once again, the question is whether there is a performance difference in a EMS application environment, and, if so, whether it is significant. Another issue is cost. The cost of the model 80 is greater than the model 60, which in turn is greater than the model 50. Is the additional cost outweighed by the increased performance of the models 60 and 80?

4.5. Benchmarking Tests

As discussed previously, our objective was to measure the performance of specific LAN configurations supporting a typical EMS application. As such, the primary measure of performance was response time [3,18]. Of course, measures of response time will also include the processing time required for application software to interface with the network hardware and software, which is as it should be. The total response time faced by an actual user application involves more than just network transit time; it also includes the time required for the network hardware and software to accumulate the information at the workstation or server, and present the information to the user application for processing.

One key variable in the use of EMS systems is the number of user workstations. Past EMS sessions have involved group sizes of 3 to 24 user workstations. To provide an evaluation that would be useful to many EMS builders, we chose to evaluate the configurations with several different

1 In c o n f i g u r i n g the ne tworks , the d e f a u l t p a r a m e t e r s were

used for Nove l l N e t w a r e . F o r I B M PC L A N , p a r a m e t e r s

were set to use 1 0 K n e t w o r k bu f f e r s a t b o t h the server a n d

user w o r k s t a t i o n s , wi th f o u r bu f f e r s o n the server . T i m e

sl icing o n the server was set as r e c o m m e n d e d for a d e d i c a t e d

server [14, p. 1 2 - 1 7 ] . O n e m e g a b y t e w a s u sed for c a c h i n g .


group sizes. We balanced the desire for a limited number of group sizes against the need for a range of sizes by choosing 5 group sizes: 4, 7, 10, 15 and 20 user workstations.

Two general test categories were developed, EBS tests and DOS tests. The EBS tests involved testing the speed at which the actual EBS software operated on the specific LAN configuration. How- ever, while EBS is the principal software tool used in the system, it is not the only tool. To simulate the network requirements of other tools, and to further investigate LAN performance, a series of file transfer tests using DOS were also performed. The DOS tests were run independent of EBS, and were developed to test the networks' speed to copy files from and to the network server as is done when using other Group Systems tools. These tests can be easily replicated by other researchers and do not require the presence of Group Systems. They also provide a broader range of tests than the EBS tests, as files of different sizes could to be used to determine the effect of increased volume on network speed.

4.5.1. EBS Tests With EBS, each user enters comments into a

local workstation file. When a comment is complete, the user presses a function key, and that file is copied to the network server, and another file is read from the server to the workstation. The EBS software was modified to generate time stamps before a file was copied to the sever and after a file was read from the server. The difference between these two times was the time required by the network software to send the file to the server, transfer another file back to the workstation, and present it to EBS processing. This time difference was calculated by EBS and stored in a local log file at each workstation for later analysis. The EBS tests were designed to simulate an EBS session, based on statistics compiled from 20 actual EBS sessions. As mentioned previously, those statistics indicated that the average comment size was approximately 250 bytes, with an average of 10 comments per user per session.

Due to the constraints imposed by the operation of the EBS facility, EBS could not be installed under all five network configurations. Therefore, only the IBM/60, IBM/80 and N O V / 6 0 configurations were tested. Five sep-

arate, but identical, EBS tests were run for each configuration - one for each group size (4, 7, 10, 15, and 20).

The testing process went as follows. The EBS tool was started at each user workstation. Using the remote keyboard controller, a utility that en- ables one keyboard to control the keyboards of any or all workstations on the network, 250 bytes of information was entered on all workstations involved in the test. Then, using the remote keyboard controller, the function key instructing EBS to copy a file to the server and read a new file from the server was pressed. All workstations simultaneously attempted to transmit this information across the network. This process was re- peated 9 more times, thus giving a total of 10 comments entered, and 10 data points per workstation collected.

4.5.2. DOS Tests The DOS tests used batch files to copy files to

and from the network server. As with the EBS tests, the time was recorded in a local log file at each workstation before and after the execution of the copy command. To provide a range of file sizes "typical" for EMS applications, we choose to test the time required to transfer files of size 250 bytes, 1K bytes, 10K bytes and 100K bytes, for reasons discussed earlier. As with the EBS tests, we ran the DOS tests with 4, 7, 10, 15, and 20 users.

Separate batch files were created for each test and for each and every workstation. We ran three trials for each file s ize/number of users /network configuration combination to provide a more reliable set of data for statistical analysis. To test the network under a load, we ran the copy commands at each workstation simultaneously. The keyboard controller was again used to synchronize the tests.

The copy "from" (read) test tested the speed of the server and network in reading a file from the server's hard disk, transferring it across the network and storing it on each workstation. This process is affected by the speed of the network, and by the speed of the processor in the network server. The copy " to" (write) test tested the network speed, the processor speed of the server and, most importantly, the speed of the server's hard disk.


5. Results

5.1. EBS Tests

As shown in fig. 3, the N O V / 6 0 configuration outperformed both IBM configurations; N O V / 6 0 is faster than I B M / 8 0 (t = 13.24, p = 0.000) and I B M / 6 0 configuration (t = 16.25, p = 0.000). The I B M / 8 0 configuration is faster than the I B M / 6 0 configuration (t = 4.54, p = 0.000). This graph also shows the effect of adding more users to the network. The performance of the Novell network is only slightly affected by adding more users. In contrast, the response time of the IBM PC LAN program increases linearly with the number of users.

To better understand the relationships between the factors influencing network performance, we performed a linear regression analysis. This analysis used the original response time data from the EBS tests (expressed in seconds) as the dependent variable. The three network configurations were coded using three indicator (0-1) variables. Several transformations (eg. natural logs, exponentials, squares, square roots) and interaction terms were tested. The results of the linear regression analysis using the Minitab statistical package are presented in table 3. The best regression model includes a constant plus three interaction terms, one for each network configuration. In this model, the first interaction term (shown as User* I60 in table 3), represents the slope on the variable "number of users" for the I B M / 6 0 configuration. In other words, for the I B M / 6 0 configuration with between 4 and 20 users, we would expect average response time to increase by approximately 0.38 seconds for each user workstation added to the

10

IBM60 8 ~ J

0 c e 6 d n s 4 ~ ~ ~ IBMSO

. ~ NOV60 2 ~ ~

0 0 5 10 15

Number of users Fig. 3. Brainstorming Average Response Times.

2 0 2 5

Table 3 Regression Analysis of Electronic Brainstorming Response Times.

The regression equation is Time = 0.429 + 0.381 User * 160 + 0.279 User * I80

+ 0.0745 User* N60

Predictor Coef Stdev t-ratio VIF Constant 0.42886 0.06011 7.13 User* I60 0.381189 0.005683 67.07 1.7 User * I80 0.278990 0.005683 49.09 1.7 User* N60 0.074485 0.005683 13.11 1.7

s = 0.3342 R-sq = 97.7% R-sq(adj) = 97.6%

Analysis of Variance

SOURCE DF SS M S Regression 3 678.24 226.08 Error 146 16.31 0.11 Total 149 694.55

Key. VIF: Variance Inflation Factor, DF: Degrees of Free- dom, SS: Sum of Squares, MS: Mean Square.

network. Similarly, for the I B M / 8 0 and N O V / 6 0 configurations, we would expect response time to increase by 0.28 seconds and 0.07 seconds, respectively, for each workstation added to the network.

The t-ratios for the coefficients on each variable are statistically significant at p = 0.000. The MSE and R-Squared values indicate that the overall regression model is statistically significant and useful. Approximately 97.6% of the variation in processing time can be explained by the number of users and the network configuration. The remaining 2.4% is random variation or is due to other factors.

This regression model can also be used to pre- dict average response times for EBS. For example, for a 16 workstation EMS, we would expect an average response of 4.89 seconds for the I B M / 8 0 configuration, as compared to 1.62 seconds for the N O V / 6 0 configuration. Prediction is also a means by which to test the robustness of a regression model. In this case, the original response time data was randomly partitioned into two halves of 75 observations each. The first half was used to generate a new model of the same form as that shown in table 3. The coefficients for this new model were, of course, different from those of the model in table 3, but were remarkably similar - the same rounded to 2 decimal places. Using this model, predictions were made for the remaining 75 observations. The mean difference between the ac-


5 20

IBM50 4 ~

15 / z ~ 7 IBM60 S

3 . ~ . t - _ _ - IBMso C ~ ~ 0 10

n 2 d

~ ;'~ : : ~ s 5

1 ~ - = N O V 5 0 / 6 0

• IBM50

- I B M 6 0

I B M 8 0

. j ~ f ~ N O V 5 0 / 6 0

0 0 2 4 6 8 10 12 14 16 18 20 22 24

N u m b e r o f u s e r s

Fig. 4. Effect o f A d d i n g Users to N e t w o r k . ( R e a d 250 byte file

f rom server).

0 0 2 4 6 8 10 12 14 16 18 20 22 24

N u m b e r o f u s e r s

Fig. 5. Effect o f A d d i n g Users to N e t w o r k . ( R e a d 100K file f rom server).

tual observations and the predicted observations was 0.09 seconds (mean absolute difference of 0.20 seconds), indicating that the regression model in this form is remarkably robust, and therefore, should be a reliable predictor of network performance.

T a b l e 4

M e a n R e s p o n s e T imes o f C o p y F r o m Tests.

5.2. DOS Tests

5.2.1. Read From Tests As the read from and write to tests examine the

network configurations for different speed char-

M e a n Std n t- tests

N 5 0 N 6 0 I60 I50

180 3.437 2.566 60 180 2.57 2.51 - 1.73

N 5 0 2.295 2 .294 60 N 5 0 - 0.05 - 3.91

N 6 0 2.316 2.321 60 N 6 0 - 3.86

160 4.401 3.483 60 160

150 4 .752 4.401 60

- 2.00

- 3 . 8 3

- 3 . 7 9

- 0.48

T a b l e 5

Regress ion Ana lys i s of C o p y F r o m Tests R e s p o n s e Times•

The regress ion e q u a t i o n is

T ime = 1.06 + 0 .00515 U s e r Size + 0•0647 User * 180 - 0 .0324 User * N 5 0 - 0•0319 User * N 6 0 + 0 .157 User * I60 + 0 . t 88 Use r * I50

Predictor Coef Stdev t -ratio C o n s t a n t 1•06436 0.09031 11.79

User * Size 0 .00514863 0 .00007792 66.07

User * 180 0 .064737 0•009943 6.51

User * N 5 0 - 0•032380 0 .009943 - 3.26

User * N 6 0 - 0•031935 0 .009943 - 3.21

User * 160 0 .156658 0 .009943 15.76

User * 150 0 .187506 0 .009943 18.86

s = 0•7101 R-sq = 95.4% R-sq(ad j ) = 95.3%

A n a l y s i s of V a r i a n c e

SOURCE DF SS MS Regress ion 6 3040.70 506•78

E r r o r 293 147.73 0.50

To ta l 299 3188.43

VIF

1.1

1.6

1.6

1.6

1.6

1.6

Key. VIF: V a r i a n c e In f l a t ion Fac to r , D F : Degrees of F r e e d o m , SS: S u m of Squares , MS: M e a n Square•


acteristics, we will examine them separately. Fig. 4 graphically displays the effects of adding more users to the network configurations, when small files are read from the server. Novell provides the fastest time, with speed largely unaffected by adding more users to the system. NOV/50 and N O V /60 provide similar response times, as do the IBM/50 and IBM/60 configurations. As this is a test of server processor speed as well as network characteristics, it is not surprising that the IBM PS/2 Model 50 and Model 60 have similar performance - they both share the same 286 processor. Fig. 5 presents a graph of the same test, but with large file sizes. The pattern remains basically the same with the Novell configurations being faster than the IBM configurations.

Are these differences statistically significant? Table 4 presents the means and standard devia- tions for all configurations, as well as t-tests on the differences. The NOV/50 and NOV/60 configurations provide essentially the same response time, as do the IBM/50 and IBM/60 configurations. Both Novell configurations are faster than the IBM/80 configuration, which in turn is faster than the IBM/50 and IBM/60 configurations.

A regression analysis was again performed to better understand the effects of file size and number of users on network response time. Table 5 presents the best regression model for the " f rom" tests. The MSE is statistically significant at p = 0.000, indicating that the entire regression model is statistically significant. The R-squared value is approximately 95.3%, indicating that 95.3% of the variability in processing time can be explained by the number of users, file size and network configuration. The remaining 4.7% is random variation or is caused by other factors.

The model has six interaction terms, all of which are statistically significant. The first variable (User*Size) represents the total information volume flowing through the network. The remaining five variables indicate the effects of adding more users to each network configuration. The first of these five interaction terms (shown as User*I80), indicates that, on average, for the IBM/80 configuration with between 4 and 20 users, we would expect response time to increase by approximately 0.06 seconds for each user workstation added to the network, assuming that the total network volume remained constant. Simi- larly, for each user added to the network (holding

total network volume constant) we would expect response time to increase by 0.16 seconds for the IBM/60 configuration, and 0.19 seconds for the IBM/50 configuration.

The coefficients for the two Novell configurations are negative, which implies that response time decreases as we add users to the network, holding total network volume constant; in other words, for a given network volume, response time is better if that volume is spread across more users. While this could be interaction effects caused by the mathematics of the regression modeling technique (ie. multicolinearity), further analysis effectively ruled this out (see also the Vari- ance Inflation Factors (VIF) in table 5). It more likely indicates the performance benefits provided by Novell's multi-threaded operating system. With multiple threads, several user file read requests can be handled simultaneously. Spreading a given volume across more users provides better response time.

Once again, in order to test the robustness of the regression model, the original response time data was randomly partitioned into two parts of 100 observations and 50 observations, with the larger part used to generate a new model and make predictions for the remaining 50 observations. The mean difference between the actual observations and the predicted observations was 0.30 seconds (mean absolute difference of 0.51 seconds), thus indicating that the regression model in this form is robust, and therefore, should be a reliable predictor of network performance.

5.2.2. Write To Tests Figs. 6 and 7 graphically present the results of

the tests writing small and large files (respectively)

12

10 ~ SBM50

6 ~ ~ IBM60/80 /

2 NOVSO/60

0 0 2 4 6 8 10 12 14 16 18 20 22 24

N u m b e r o f use rs

Fig. 6. Effect of Adding Users to Network. (Write 250 byte file to server).


S IBM50

J J

J IBM60

J " ~ NOV60

d d d

50

40

30

20

10

0 0 2 4 6 8 10 12 14 16 18 20 22 24

Number of users

Fig. 7. Effect of Adding Users to Network. (Write 100K file to

server).

to the server. Due to disk space limitations, the " T o " tests using 100K files could not be performed with the N O V / 5 0 configuration. The patterns are similar to the read from tests. Novell is again faster than the IBM network, except for large files. The IBM PS/2 Model 60 and the Model 80 produce similar test times, as they have the same model of hard disk; the model 50 is slower, as it has a slower hard disk. The Novell configurations are less affected by hard disk speed (especially for small files), as the Novell software uses multitasking and disk caching to increase I / O throughput.

Table 6 presents the means, standard devia- tions, and t-test results for the " T o " tests. Two sets of data are presented: one for all file sizes (which omits the N O V / 5 0 configuration), and one for the 250 byte to 10K files sizes. The N O V / 5 0 and N O V / 6 0 configurations provide essentially

the same response time, as do the I B M / 8 0 and I B M / 6 0 configurations. Both Novell configurations are faster than the I B M / 8 0 and I B M / 6 0 configurations and the I B M / 5 0 configuration, but not significantly faster than I B M / 8 0 for files of 100K. I B M / 8 0 and I B M / 6 0 are faster than IBM/50.

Statistical regression analysis of the ' T o " tests was conducted in the same manner as that de- scribed above for the " F r o m ' ' tests. Table 7 presents the results of this regression. The MSE is again statistically significant at p = 0.OO0, and the R-squared value is 93.7%. On average, for the IBM/80 configuration with between 4 and 20 users, we would expect response time to increase by approximately 0.13 seconds for each user workstation added to the network, assuming that the total network volume remained constant. Simi- larly, for each user added to the network holding total network volume constant, we would expect response time to increase by 0.22 seconds for IBM/60 , and 0.65 seconds for IBM/50 . The t- ratios on the two coefficients for the Novell configurations are not statistically significant. There- fore, we can conclude that the Novell configurations are primarily influenced by total network volume, not the number of workstations on the network.

Once again, in order to test the robustness of the regression model, the original response time data was randomly partitioned into two parts of 100 observations and 50 observations, a new model generated, and predictions made. The mean difference between the actual observations and the

Table 6 Mean Response Times of Copy To Tests.

Mean Std n t-tests

N50 N60 160 150

Smaller File Sizes (250 bytes- 1 OK) 180 3.771 1.642 45

N50 1.316 1.118 45

N60 1.256 0.921 45

160 3.830 1.730 45

150 7.425 3.363 45

All File Sizes (250 bytes - lOOK) I80 6.003 5.368 60

N60 4.664 6.838 60

I60 7.030 7.184 60

I50 12.03 10.850 60

I80

N50

N60

I60

I80 N60

160

8.29 8.96 - 0 . 1 7 -6 .55 0.28 - 8.19 - 11.56

-8 .81 -11 .87

-6 .38

1.19 - 0 . 8 9 -1 .85

- 3.86 - 4.45

-2 .98


Table 7

Regression Analysis of Copy To Tests Response Times.

The regression equation is

Time = 0.729 + 0.0124 User * Size + 0.128 User * I80 + 0.0163 User * N50 + 0.0046 User * N60 + 0.222 User * I60 + 0.654 User * I50

Predictor Coef Stdev t -ratio Constant 0.7290 0.2589 2.82

User * Size 0.0123857 0.0002432 50.93

User * I80 0.12801 0.02824 4.53

User* N50 0.01634 0.02985 0.55

User * N60 0.00463 0.02824 0.16

User * 160 0.22185 0.02824 7.86

User * I50 0.65395 0.02824 23.15

s = 1.984 R-sq = 93.8% R-sq(adj) = 93.7%

Analysis of Variance

S 0 UR CE D F S S MS

Regression 6 16619.0 2769.8

Error 278 1094.2 3.9

Total 284 17713.2

V/F

1.1

1.6

1.4

1.6

1.6

1.6

Key. VIF: Variance Inflation Factor, DF: Degrees of Freedom, SS: Sum of Squares, MS: Mean Square.

predicted observations was -0 .23 seconds (mean absolute difference of 0.90 seconds), thus indicating that the regression model in this form is fairly robust, and should be a reasonable predictor of network performance.

6. Discussion of Results

Our first observation is about the performance effects of network operating systems. While network topology, transmission media and medium access protocols have received a great deal of attention in the design and development of communications networks, considerably less attention has been paid to network operating systems. This study has shown the dramatic performance effects that can be obtained from improved network operating system design. For example, as the NOV/50 configuration is 75% faster than the IBM/50 configuration, we can conclude that more than 75% of the response time in the IBM/50 configuration is due to the network operating system. For this configuration, increasing the speed of the network hardware would be ineffective; increasing network hardware speed by a factor of 100 could be expected to decrease response time by at most 25%.

Perhaps the most interesting observations have to do with the transfer of small files. As mentioned before, EMS applications typically involve the transfer of small files. With EBS, files start very small in size (eg. 250 bytes) and gradually grow, usually to about 2.5K in size. After examining the EBS response time data, an unusual pattern emerged: although the file sizes grew from 250 bytes to 2.5K, response times appeared to remain constant, regardless of file size. A series of 675 t-tests were performed to test this hypothesis of constant response time (every trial's mean response times versus every other trial for the same configuration). From this set, only 15 pairs were statistically significantly different (at p = 0.05), which is less than could be expected in a perfectly random distribution. This, combined with a lack of a pattern in the significant values, indicates that when using the EBS software, file size has no effect on response time.

To further confirm this finding, we examined the differences in response time for the DOS tests between the 250 byte and 1K file sizes. Across all network configurations and all numbers of users, the mean response time for reading a 250 byte file from the server was 0.785 seconds with a standard deviation of 0.925 seconds. For 1K files it was 1.978 seconds with a standarci deviation of 1.043


seconds. The difference in response time is not statistically significant (t = 1.70, p = 0.0892). Like- wise, there is no statistically significant difference between 250 byte files and 1K files when writing to the network server (t = 1.10, p = 0.2714). Therefore, we can conclude that for small files (in the 250 byte to 1K range), file size has no effect on response time when writing to the server, but may have a small effect when reading from the server.

This suggests that one important factor in network performance of particular interest in an EMS environment is the overhead imposed by the network operating system. For each data transfer across the network, the network operating system has to manage the token passing process, the allocation of disk space at the server a n d / o r workstation(s), and the opening, writing and closing of data file(s). For the small files typical of an interactive meeting process, this fixed overhead is much larger than the variable transit time due to the size of the file, and thus overhead dominates transit time. The bottleneck is at the server with the network operating system, not with the transit speed of the physical network media. This further suggests that in a Decision Room environment providing a supported meeting process or interactive meeting process, improving the physical media (by replacing twisted pair cabling with fiber optics for example) should have only a minor effect on the overall response time encountered by users. However, in an EMS environment providing graphically supported meeting process, or an environment with integrated voice and video transmission, the physical media may still play a key role, as the data and file sizes transmitted across the network are potentially much larger and could dominate the fixed overhead.

From these two observations (that network operating systems have significant effects on response times and that for small files, file sizes have very limited effects on response times) we can make another observation. For EMS environments providing supported a n d / o r interactive meeting processes, theoretical models of network performance and simulations based on them may not be appropriate, as they are based substantially on file size and network hardware speed. For example, the predicted network performance calculated by a token ring model [2] is many

orders of magnitude below the actual response times encountered by users.

7. Conclusions

The networking requirements of several EMS environments and the Group Systems EMS environment in particular, have been discussed. The principle requirement for many of the tools in this EMS and other EMS environments that provide an interactive meeting process is fast and accurate transfer of small groups of text, usually less than 5K in size. Occasionally, larger blocks of text (10K and 50-100K) are transmitted across the network. Previous evaluations have shown several network configurations, including the IBM Token Ring ha rdware /Nove l l software configuration

Table 8

Key Findings.

• In Electronic Bra ins to rming Tests, Nove l l Ne tware ran twice

as fast as the IBM PC LAN Program for 4 users, and 4

t imes faster for 20 users.

• In reading in fo rma t ion from the ne twork server, Novel l

Ne tware ran twice as fast as the IBM PC LAN program

overall.

• In wr i t ing in fo rmat ion to the ne twork server, Novel l Ne tware

ran three t imes as fast as the IBM PC L A N program overall .

• In all tests, Novel l Ne tware was less affected by increas ing

network vo lume or the n u m b e r of users for a given ne twork

volume. For very small files (1K and less), Novel l ' s perfor-

mance was unaffec ted by total ne twork volume or the

number of user works ta t ions .

• When using Novel l Netware , there was no pe r fo rmance

difference be tween an IBM P S / 2 Model 50 and a Mode l 60

server.

• When using the IBM PC LAN program, there was no

per formance difference be tween a Mode l 50 and a Mode l 60

server then reading files f rom the server.

• W h e n using the IBM PC L A N program, there was no per formance difference be tween a Mode l 80 and a Mode l 60

server when wr i t ing files to the server.

• In vi r tual ly all cases, Nove l l Ne tware on a Mode l 50 server

ran faster than the IBM PC LAN p rog ram on a Mode l 80

server.

• When dea l ing wi th the smal l files c o m m o n to EMS appl ica- t ions (2.5K and less), files size has no effect on ne twork

performance. Therefore, increases in ne twork ha rdware

speed are l ikely to have l i t t le effect on response t ime.


avai lable to provide a good level of pe r fo rmance across a wide range of appl ica t ions . Our EMS- specific test ing has p roduced similar results, as summar ized in table 8.

In short, the Novel l ne twork opera t ing system running on an IBM P S / 2 Mode l 50 or Mode l 60 server consis tent ly ou tpe r fo rmed the IBM PC L A N software running on an IBM P S / 2 Mode l 5 0 , Model 60 or Mode l 80 server. The Novel l sof tware was 2 - 4 t imes faster than the IBM software on the same server; N O V / 5 0 was 1½-3 t imes faster than I B M / 8 0 . One conf igura t ion outper - forms all o thers tested. F r o m a bo th a cost and a pe r fo rmance perspect ive, the best choice appears to be the IBM Token Ring h a r d w a r e / N o v e l l sof tware conf igura t ion running on an IBM P S / 2 Mode l 50. This conf igura t ion provides at least as great speed as any conf igura t ion tested, and the list price of the Mode l 50 processor with hard disk is subs tan t ia l ly less than the Mode l 60 or Mode l 80. Whi le the Mode l 60 and 80 provide more disk space than the Mode l 50, EMS app l ica t ions typical ly do not require a subs tan t ia l amoun t of server

disk space, as most sof tware is resident on the ind iv idua l works ta t ions , not the server.

F inal ly , and pe rhaps most impor tan t ly , the net-

work opera t ing system is a rguab ly the most sig- n i f icant factor in de te rmin ing ne twork performance in in teract ive E M S environments . Ra the r than increasing ne twork b a n d w i d t h to provide higher da ta t ransmiss ion speeds, a more effective s t ra tegy is to use a more efficient ne twork opera t - ing system. Increas ing da ta t ransmiss ion rates can be expected to p rov ide marg ina l impac ts on response time.

The overal l object ive is to de te rmine the ap- p ropr i a t e form of communica t ions suppor t within a var ie ty of EMS envi ronments . In this s tudy, we have focused on one set of factors (ne twork oper- a t ing system and ne twork server) within one EMS env i ronment conf igura t ion (a Decis ion R o o m EMS

prov id ing an interact ive meet ing process). The next s tep is to move into other environments , s tudying o ther sets of factors. The factor with the greatest effect on ne twork pe r fo rmance in a Decis ion R o o m env i ronment (ne twork opera t ing system), m a y not be as signif icant in o ther envi ronments , such as EMS Teleconferencing, once the d is tance between groups or g roup member s increases. In these en- v i ronments , for example , the need for voice and video signal t ransmissions , with their greater

b a n d w i d t h needs, may suggest the greater impor - tance of ne twork ha rdware speed. Similar ly, as the use of graphics in EMS becomes more wide spread, o ther pe r fo rmance factors m a y become more signif icant. Connec t iv i ty with o ther L A N ' s (in EMS teleconferencing, for example) and the abi l i ty to use exist ing o rgan iza t iona l resources, such as cor-

po ra t e da t a bases s tored on a m a i n f r a m e compu t - ing system, wilt become increas ingly i m p o r t a n t in the future. Thus factors o ther than pe r fo rmance may also have a major impac t on the communica - t ions requi rements for EMS in the future.

References

[1] W. Bux, "Performance Issues in Local-Area Networks", IBM Systems Journal, 23:4,1984, pp. 351-373.

[2] W. Bux, "Local-Area Subnetworks: A Performance Com- parison", IEEE Transactions on Communications, 29:10, October, 1981, pp. 1465-1473.

[3] L. Cabrera, E. Hunter, M.J. Karels, and D.A. Mosher, "User-Process Communication Performance in Networks of Computers", IEEE Transactions on Software Engineer- ing, 14:1, January, 1988, pp. 38-53.

[4] R. Cowart, and P. Feldmann, "Benchmarks for Network Ratings", PC Magazine, February, 195.

[5] Data Pro Reports on Communication: Network Architec- ture, Volume 1, Cll-010, June 1986. pp. 501-508.

[6] A.R. Dennis, J.F. George, L.M. Jessup, J.F. Nunamaker Jr. and D.R. Vogel, "Information Technology to Support Electronic Meetings", MIS Quarterly, 12:4, Dec 1988, pp. 591-624.

[7] A.R. Dennis, A.R. Heminger, J.F. Nunamaker Jr., and D.R. Vogel, "Bringing Automated Support to Large Groups: The Burr Brown Experience," Informa- tion & Management, 18:3, 1990, pp. 111-121.

[8] G. DeSanctis, and G.W. Dickson, "GDSS Software: A Shell System in Support of a Program of Research", Proceedings of HICCS Conference, 1987.

[9] G. DeSanctis, and R.B. Gallupe, "Group Decision Sup- port Systems: A New Frontier", Data Base, 16:2, Winter 1985, pp. 3-9.

[10] G. DeSanctis, and R.B. Gallupe, "A Foundation for the Study of Group Decision Support Systems", Management Science, 33:5, May 1987, pp. 589-609.

[11] R.B. Gallupe, and .I.D. McKeen, "Beyond Computer Mediated Communication: An Experimental Study into the Use of A Group Decision Support System for Face- to-Face Versus Remote Meetings", Administrative Scien- ces Association of Canada, 1988 Conference, pp. 103-116.

[12] S.R. Hiltz, K. Johnson, and M. Turoff, "Experiments in Group Decision Making Communication Process and Outcome in Face-to-Face Versus Computerized Con- ference", Human Communication Research, 13:2, Winter 1986, pp. 225-252.

[13] G.P. Huber, "'Issues in the Design of Group Decision Support Systems", M IS Quarterly, Sept 1984, pp. 195-204.


[14] International Business Machines Corporation, IBM PC Local Area Network Program User's Guide, Third Edi- tion, April 1987, Boca Raton, Florida.

[15] M.T. Jelassi and R.A. Beauclair, "An Integrated Frame- work for Group Decision Support Systems Design", Sys- tems, Objectives, Solutions in Information and Manage- ment, 13, 1987. pp. 143-153.

[16] K.L. Kraemer, and J.L. King, "Computer-Based Systems for Cooperative Work and Group Decision Making: Status of Use and Problems in Development", Proceedings of the 1986 Conference on Computer supported Collaborative Work, pp. 353-375.

[17] K.L. Kraemer and J.L. King, "Computer-Based Systems for Cooperative Work", Computing Surveys, 20:2, June 1988, pp. 115-146.

[18] K.A. Lantz, W.I. Nowicki, and M.M. Theimer, "An Em- pirical Study of Distributed Application Performance", IEEE Transactions on Software Engineering, 11:10, Oc- tober, 1985, pp. 1162-1173.

[19] Novell Inc., LAN Operating System Report 1956, Novell Inc., Provo, Utah, February, 1987.

[20] J.F. Nunamaker Jr., L.M. Applegate and B.R. Konsynski, "Facilitating Group Creativity with GDSS", Journal of Management Information Systems, 3:4, Spring 1987, pp. 5-19.

[21] J.F. Nunamaker Jr., L.M. Applegate and B.R. Konsynski, "Computer-Aided Deliberation: Model Management and Group Decision Support," Journal of Operations Re- search, November-December, 1988.

[22] J.F. Nunamaker Jr., D.R. Vogel, A.R. Heminger, B. Martz, R. Grohowski and C. McGoff, "Group Support Systems in practice: Experience at IBM", Decision Support Sys- tems, 5:2, 1989, pp. 183-196.

[23] G.S. Poo, and G.H. Ong, "Performance Evaluation of an Ethernet Network", Computer Communication, 9:3, June, 1986, pp. 128-134.

[24] G.S. Poo, and T.S. Tan, "Performance Comparison of PC

LAN File Servers - An Experimental Study", Computer Communication, 11:2, April, 1988, pp. 71-79.

[25] L. Press, "Benchmarks for LAN Performance Evaluation", Communications of the ACM, 31:8 August 1988, pp. 1014-1017.

[26] L.S. Richman, "Software Catches the Team Spirit", For- tune, June 8, 1987.

[27] J.F. Shoch, and J.A. Hupp, "Measured Performance of an Ethernet Local Network", Communications of the ACM, 23:12, December, 1980, pp. 711-721.

[28] R.H. Sprague, "A Framework for the Development of Decision Support Systems", MIS Quarterly, 4:4, Decem- ber 1980.

[29] W. Stallings, "Local Networks", Computing Surveys, 16:1, March, 1984, pp. 3-41.

[30] W. Stallings, Data and Computer Communications, second edition, Macmillan Publishing Company, New York, 1988.

[31] M. Stefik, G. Foster, D.G. Bobrow, K. Kahn, S. Lanning and L. Suchman, "Beyond the Chalkboard: Computer Support for Collaboration and Problem Solving in Meet- ings," Communications of the ACM, 30, 1, January, 1987, pp. 32A7.

[32] B.W. Stuck, "Calculating the Maximum Mean Data Rate in Local Area Networks", IEEE Computer, May, 1983, pp. 72-76.

[33] S.K. Tripathi, Y. Huang, and S. Jajodia, "Local Area Networks: Software and Related Issues", IEEE Transac- tions on Software Engineering, 13:8, August, 1987, pp. 872-879.

[34] D.R. Vogel, J.F. Nunamaker Jr., J.F. George, and A.R. Dennis, "Group Decision Support Systems: Evolution and Status at the University of Arizona", in R.M. Lee, A.M. McCosh, and P. Migliarese (eds), Organizational Decision Support Systems, Proceedings of 1FIP WG 8.3 Working Conference on Organizational DSS, North Hol- land, 1988, pp. 287-305.

Documents

Communication requirements and network evaluation within electronic meeting system environments