XNMS Performance DSOM Final

8/9/2019 XNMS Performance DSOM Final

1/16

1

Performance Evaluation of XML-based Network Management

Mi-Jung Choi1, So-Jung Lee

1, Dong-Hyun Kim

1, James W. Hong

1and Raouf Boutaba

21

Dept. of Computer Science and Engineering, POSTECH, Korea2School of Computer Science, University of Waterloo, Canada

{mjchoi, annie, dhkim03, jwkhong}@postech.ac.kr, [email protected]

Abstract

Recently, researchers and developers have been investigating the application of XML technologies to

network and systems management. Although the performance of XML-based network management

systems (XNMS) has been identified as an important issue, it has been difficult to obtain such information

due to a lack of implementations. In this paper, we provide a performance evaluation of XNMS. Our

performance evaluation is based on an XML-based manager managing network devices equipped with

SNMP agents through one or more XML/SNMP gateways. We show the performance evaluation of XNMS

from the perspective of network traffic, response time, and computing resources, namely CPU and memory

usage.

Keywords: Performance Evaluation, XML-based Network Management, XML, SNMP, XML/SNMP

Gateway

1.

IntroductionTo benefit from the advantages of XML [1], research on the use of XML in the network management area is

actively in progress [2, 3, 15, 16, 17, 18]. The use of XML in network management offers many advantages. XML

provides powerful modeling features for hierarchical management information in network management, and

XML-based network management (XNM) allows for the reliable transfer of a large amount of data over HTTP.

Also, XNM can be easily implemented using standard APIs and freely available software. This advantageous

applicability of XML technologies to network management alleviates many of the known problems of traditional

SNMP-based management [3].

During the past few years, some research and development have been done in the area of XML-based network

management. Much work has been done in the area of accommodating widely deployed SNMP devices,

specifically in the specification and interaction translation for the gateways between XML-based managers and

SNMP-based devices [4, 5, 6, 7]. Some work focused on extending Web-based management to exploit the

advantages of XML technologies [8, 9, 10, 11] while others focused specifically on the configuration management

of network devices and systems [12, 13, 14, 17, 18]. Also, some work has been done to provide an architectural


2/16

2

framework for XML-based network management [15, 16].

In our previous work [15], we proposed an architectural framework for the development of XML-based

network management systems (XNMS). The framework provides a detailed architecture for an XML-based

manager, agent and XML/SNMP gateway. It also provides methods for interacting between a manager and an agent.We have validated this architecture by implementing it and using it for managing network devices and systems [6, 7,

15, 16, 17, 18]. We evaluated computing resources such as the CPU and memory usage of an XML-based agent.

Also, we evaluated the performance such as network traffic volume and response time of an XML-based agent. We

compared the performance between the XML-based agent and SNMP agent. Then we showed that our XML-based

agent is as efficient as an SNMP agent in terms of computing resources and processing time [15].

Typically, a large amount of network bandwidth is required to transfer XML data due to its text-based nature

and the use of the HTTP/TCP protocols. An important issue is the ability of the XML-based manager and

XML/SNMP gateway to process information received from multiple managed devices. The development

experience of XNMS is not sufficient and also deficient for wide application to real network management.

Therefore, performance evaluation results for XNM have not been provided until now.

Most IP network devices currently deployed are basically equipped with SNMP agents, and the XML/SNMP

gateway approach is suitable to this situation. The XML-based manager manages the current SNMP-enabled

network devices through this XML/SNMP gateway. The XML/SNMP gateway translates and relays the messages

between an XML-based manager and an SNMP agent. We have applied our XML-based manager and XML/SNMP

gateway to the POSTECH campus network and evaluated the computing resources of the manager and gateway,

and network performance such as traffic volume and response time. We increased the number of SNMP agents from

1 to 100. Also, we increased the number of the XML/SNMP gateways from 1 to 3 and then measured the aboveperformance parameters. From these performance evaluation results, we show that XNMS does not create much

performance overhead from the perspectives of network traffic, processing time and computing resources. The goal

of our work is to validate the efficiency and scalability of XNMS.

The organization of this paper is as follows. Section 2 describes the testing environment of performance

evaluation in the POSTECH campus and the measurement methods. In Section 3, we present the performance

evaluation results of the XML-based manager and XML/SNMP gateway. In Section 4, we analyze and summarize

the performance test results. Finally, we conclude this paper and discuss some directions for future research in

Section 5.

2.

Measurement Environment and Methods

As depicted in Figure 1, we applied our XML-based manager and XML/SNMP gateway to manage various

network devices deployed in the POSTECH campus network. The XML-based manager and XML/SNMP gateway

are connected by a 100Mbps LAN. However, XML/SNMP gateways and SNMP agents are connected by not only


3/16

3

100Mbps LAN but also a gigabit Ethernet backbone network. Both XML-based manager and the XML/SNMP

gateway run on Linux servers with a Pentium-III 800 MHz CPU and 256M RAM.As a part of our tests to evaluate

the impact of the change in computing resources on the performance of a gateway server, we upgraded our server

running the XML/SNMP gateway to a Pentium-IV 2.8GHz CPU and 512M RAM.

XML-basedManager

XML/SNMP Gateway

100Mbps100Mbps

100Mbps

100Mbps

Linux severs with

Pentium-III 800 MHz CPU256 MB RAM

or Pentium-IV 2.8 GHz CPU512 MB RAM

100Mbps

100MbpsXML/SNMP Gateway

XML/SNMP Gateway SNMP Agents

Various Network Devicesin POSTECH campus

POSTECH Gigabit Ethernet Backbone Network

1Gbps

1Gbps

1Gbps

1Gbps

Switchinghubs

Backboneswitches

Internetrouters

Linux sever withPentium-III 800 MHz CPU

256 MB RAM

Figure 1. Performance Testbed

SNMP agents are running in various network devices deployed in the POSTECH campus network. POSTECH

has a gigabit Ethernet backbone network, which is composed of eleven gigabit backbone switches (Catalyst

6500/5500 series) and hundreds of 100 Mbps client switching hubs that are deployed inside the buildings, and two

Cisco 7513 and 7401 Internet routers are connected to the backbone network. We selected one hundred devices

among these Cisco backbone routers and switches that are equipped with SNMP agents and then we carried out

XNMS performance tests.

First, we measured the performance of XNMS using one XML/SNMP gateway. The solid arrow in Figure 1

illustrates this test scenario. To evaluate the XNMS scalability, we increased the number of SNMP agents from 1 to

100. We measured network traffic between the XML-based manager and SNMP agents through the XML/SNMP

gateway in relation to the number of SNMP agents. We separately measured the network traffic volume between the

XML-based manager and XML/SNMP gateway and the network traffic volume between the XML/SNMP gateway

and SNMP agents. The XML-based manager sends an HTTP request message to the gateway to retrieve

management information from multiple SNMP agents. The XML/SNMP gateway processes the HTTP requestmessage and sends multiple SNMP request messages to each SNMP agent. Since most of the currently deployed

network devices in the real world only support SNMPv1 agents, the gateway simultaneously sends multiple

GetNext requests to the agent for retrieving the group data using a thread mechanism.

Next, we measured the response times as a function of the number of SNMP agents. Also, we separately

measured the latency between the XML-based manager and XML/SNMP gateway, and the latency between the


4/16

4

XML/SNMP gateway and SNMP agent. We divide the latency between the gateway and SNMP agent into two

parts: 1) processing time of translating the message in the gateway, and 2) communication time between the

gateway and agent involving all processing required by the SNMP protocol stack in the gateway and the agent. The

purpose of this division is to determine whether the processing time or the communication time is taking longer inthe gateway. Also, we measured the response time and network traffic volume in relation to the XPath expression.

Various XPath expressions can be applied in the gateway. The complexity of XPath expressions impacts the

network traffic volume and processing time.

Finally, we measured the processing overhead of the XML/SNMP gateway in terms of CPU load and run-time

memory usage. To compare the processing time of the gateway based on the change in the computing resources, we

upgraded the CPU and memory of the gateway. Then, we performed the same performance tests and compared the

response time and processing overhead between the lower-performance gateway (Pentium-III 800MHz, 256MB)

and the higher-performance gateway (Pentium-IV 2.8GHz, 512MB).

In a second test scenario, we increased the number of XML/SNMP gateway from 1 to 3. We measured the

performance of XNMS in relation to the number of XML/SNMP gateways. The dotted arrows in Figure 1 illustrate

this test scenario. The performance evaluation metrics are the same as those used in the single XML/SNMP

gateway scenario. First, we measured network traffic between the XML-based manager and SNMP agents through

multiple XML/SNMP gateways in relation to the number of SNMP agents. We separately measured the network

traffic volume between the XML-based manager and multiple XML/SNMP gateways and the network traffic

volume between multiple XML/SNMP gateways and SNMP agents. The traffic volume between multiple

XML/SNMP gateways and SNMP agents represents the sum of the network traffic that each gateway

communicates with SNMP agents.Next, we measured the response time in relation to the number of SNMP agents. Also, we separately measured

the latency between the XML-based manager and multiple XML/SNMP gateways and the latency between multiple

XML/SNMP gateways and SNMP agents. The latency between multiple XML/SNMP gateways and SNMP agents

represents the entire time taken for all XML/SNMP gateways to finish communicating with SNMP agents. We

divided the latency calculation into two parts in the same way as done in the previous scenario. Finally, we

measured the processing overhead of XML/SNMP gateways, such as CPU load and run-time memory usage. We

measured the values for each XML/SNMP gateway and then averaged them. In this test scenario with multiple

XML/SNMP gateways, the later two gateways handle same load supported by one single gateway in the first test

scenario (i.e., 100 agents). The load is evenly distributed among the gateways. Assume that we are testing the

performance of XNMS using two XML/SNMP gateways. If one gateway receives information from 100 SNMP

agents, we divide the number of SNMP agents by two. Therefore, each gateway interacts with 50 SNMP agents.

Finally, we compare the performance evaluation results in relation to the number of XML/SNMP gateways. We

identify whether the performance is improved or not. If it has improved, we also identify how much it has improved


5/16

5

by increasing the number of XML/SNMP gateways.

The traffic pattern of POSTECH campus network is constant; there is always a large volume of traffic except

for the period from 4 a.m. to 8 a.m. Since we were not sure of the effect of network traffic volume to the

measurement of latency, we measured the latency at different times and calculated an average from measuredvalues. Also, for statistical significance, we conducted 10 measurements for each test and averaged them.

3. Performance Evaluation

First, we evaluate the network traffic volume at the IP layer. Table 1 shows the network traffic between the

XML-based manager and XML/SNMP gateway and between the XML/SNMP gateway and SNMP agents to

retrieve eight objects in the system group of MIB-II (system description, system object ID, system up-time, etc.). If

the manager retrieves the interface group information from the SNMP agent, the management information volume

varies because of the difference in the number of network interfaces of each network devices. However, the system

group information is almost the same for all network devices. Therefore, we chose the system group information for

our XNMS performance tests.

The manager sends an HTTP request message then the gateway processes the request message and sends

multiple SNMP request messages to each SNMP agent [15]. The network traffic between the manager and gateway

is the HTTP message including TCP control information and connection setup. The HTTP request message consists

of gateways information including SNMP information (i.e., gateways IP, SNMP community and version) and

network devices information (i.e., devices IP, XPath expression) [15]. In the case of retrieving 1 object or 8

objects from one device, the HTTP messages are almost the same. Merely, XPath expressions indicating retrieved

data are different. In the case of only 1 object in the system group, the XPath expression is //sysDescr or//sysObjectID, etc. In the case of 8 objects, the XPath is //system. So, the network traffic generated between the

manager and gateway while retrieving 1 object is slightly greater than that while retrieving 8 objects. Both

XML/HTTP traffic and SNMP traffic are divided into the request and response messages for the Get operation. The

sum of the XML/HTTP traffic and SNMP traffic is equal to the total network traffic volume for the XML-based

manager to retrieve management information from the SNMP agents through the gateway.


6/16

6

11455

9435

9316

8325

7268

6298

5315

4401

3462

2502

Req.

Manager Gateway

3

78148

70117

62305

54214

45951

38636

30936

22615

15305

7658

Resp.

65600

59040

52480

45200

39360

32800

26240

19680

13120

6560

Req.

Gateway Agent

65553

58938

53012

56345

40575

33998

27546

20605

13986

6854

Resp.

7814865600662681119678148656006485710936100 (800)

7011759040596249150701175904058248989590 (720)

6230552480536249068623055248052354880280 (640)

5421445200565988066542144520045786779570 (560)

4595139360410947012459513936039973675460 (480)

3863632800344826051386363280033457579850 (400)

3093626240279965073309362624027148481740 (320)

2261519680210184144226151968020356389530 (240)

1530513120143643208153051312013670296520 (160)

7658656086872264765865606688202210 (80)

744656110610781 (8)

958269110801 (1)

Resp.Req.Resp.Req.Resp.Req.Resp.Req.

Gateway Agent

Manager Gateway

Gateway Agent

Manager Gateway

21# of Gateways

# of Agents(# of MIB Objects)

Unit: bytes Table 1. Network Traffic of Get Operation for the MIB-II System Group

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

0 20 40 60 80 100

Number of SNMP Agents

Network

Traffic(Bytes)

Manager Gateway (Req.) : 1 Gateway

Manager Gateway (Resp.) : 1 Gateway

Gateway Agent (Req.) : 1, 2 or 3 Gateways

Gateway Agent (Resp.) : 1, 2 or 3 Gateways

Figure 2. Network Traffic of Get Operation for System Group

Figure 2 is the graph showing the network traffic in Table 1. The curves show the data using one gateway. The

dotted line with squares shows the request network traffic from the XML-based manager to one gateway. The solid

line with squares is the response traffic to the corresponding request. The network traffic of the request and

response message from the manager to gateway slightly increases as one more gateway is added. The dotted line

with triangles represents the request network traffic from the gateway to the SNMP agents for three different cases,

when the gateway is one, two, or three. As illustrated in Figure 2, the network traffic volume between the


7/16

7

XML-based manager and gateway increases linearly in relation to the number of SNMP agents, which is the

number of MIB objects. The network traffic volume between the manager and gateway increases to almost 500

bytes as the number of gateways increases shown in Table 1. However, the total traffic volume between the

gateways and SNMP agents remains constant no matter how many gateways are added. The sum of request andresponse network traffic volume between the XML-based manager and gateway is only 50~60% of that between

the gateway and SNMP agents.

Table 2 shows the response time of the Get operation for the system group in relation to the number of SNMP

agents. We divide the latency between the gateway and SNMP agents into two parts: processing time of XML

message and translation to SNMP message in the gateway and the communication time between SNMP stacks in

the gateway and agents. In the case of retrieving only one SNMP agent, the processing time of the manager is

almost same as that of the gateway. However, as the number of SNMP agents increases, most of the processing

overhead increases in the gateway. The time taken by the gateway to process interaction translation and to

communicate with SNMP agents takes about 80~90% of the total time for the XML-based manager to retrieve

system group information from the SNMP agents through one XML/SNMP gateway. However, the processing time

of the XML-based manager does not increase as much in proportion of the number of SNMP agents. In the gateway,

the processing of XML message and interaction translation occupies more processing time than that of the SNMP

communication does.

2542

2485

2398

2203

1726

1649

1625

1532

1468

1405

Manager Gateway

3

826

759

712

674

621

518

384

329

256

175

Comm.

1876

1765

1673

1591

1492

1386

1296

1253

1103

957

Proc.

GatewayAgent

702156424381015490531148100 (800)

6251502236468505826111390 (720)

5461435228732962586108480 (640)

469136320952942223098670 (560)

394130516041285199674460 (480)

35112631527814182071550 (400)

30412041512698161568840 (320)

26211531432597146365930 (240)

18910841364361127862320 (160)

1539121296236105855810 (80)

388325241 (8)

54045121 (1)

Comm.Proc.Comm.Proc.

GatewayAgentManager

Gateway

GatewayAgentManager

Gateway

21# of Gateways

# of Agents(# of MIBObjects)

Unit: ms

Table 2. Response Time of Get Operation for System Group


8/16

8

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

22000

0 20 40 60 80 100


ResponseTime(ms)

Manager Agent (1 Gateway)

Gateway Agent (1 Gateway)

Manager Agent (2 Gateways)

Gateway Agent (2 Gateways)

Manager Agent (3 Gateways)

Gateway Agent (3 Gateways)

Figure 3. Response Time of Get Operation for System Group

Figure 3 shows the response times in Table 2. The dotted lines illustrate the total response times between the

XML-based manager and SNMP agents through gateways, and the solid lines are the response times between the

gateway and SNMP agents. We summarized the processing time and communication time between the gateway and

agents and drew them as the solid lines. More specifically, the dotted line with triangles represents the total

response time two gateways take to retrieve the system group information from the SNMP agents. Similarly, the

dotted line with circles represents the total response time three gateways take to obtain the system groupinformation from the SNMP agents. The response time between the XML-based manager and gateway is omitted in

Figure 3 because it is insignificant and constant as the number of gateway increases. However, the response time

between the gateway and SNMP agents decreases as the number of gateway increases. The solid line with triangles

shows the response time between the gateway and SNMP agent when using two gateways and is almost half of the

response time between the gateway and SNMP agents when using one gateway. Similarly, the solid line with circles

shows the response time between the gateway and SNMP agent using three gateways. It is one-third of the response

time between the gateway and SNMP agents using one gateway. Because of the decrease in the response time

between the gateway and SNMP agents, the total response time between the manager and SNMP agents also

decreases as the number of gateway increases.

As shown in Figure 3, in the case of one gateway, the total response time increases linearly until the

XML-based manager retrieves information from 80 SNMP agents; that is, 640 MIB objects. However, if the

number of SNMP agents increases beyond 80, the total response time starts to increase rapidly. On the other hand,

in the case when we apply two or three gateways, the total response time continues to increase linearly until the


9/16

9

number of SNMP agents reaches 100. Therefore, we conclude that 80 SNMP agents are suitable for our XNMS

using a single gateway (Pentium-III 800MHz CPU and 256MB RAM).

We have also evaluated the CPU usage and memory usage of the XML-based manager and the XML/SNMP

gateway in relation to the number of SNMP agents. As shown in Table 3 and Table 4, memory usage increasesslowly in relation to the number of SNMP agents, whereas CPU usage increases almost linearly. Figure 4 and

Figure 5 show the graph of the CPU usage and memory usage in Table 3 and Table 4, respectively. To operate the

gateway, CPU is more utilized than memory. From the perspective of the gateway scalability, as shown in Figure 4,

the memory usage of the XML-based manager increases slowly in relation to the number of gateways. Also, the

CPU usage of the XML-based manager increases as the number of gateways increases because more processing is

required to distribute the requests to the XML/SNMP gateways and to merge the retrieved data from these

gateways.

10.2

5.1

10.2

5.1

10.2

5.1

1(8)

14.2

8.1

13.4

7.1

12.5

5.9

10(80)

Memory Usage

CPU Usage

Memory Usage

CPU Usage

Memory Usage

CPU Usage

# of SNMP Agents(# of MIB Objects)

18.4

30.3

17.2

28.0

16.3

25.6

80(640)

18.2

29.0

17.0

26.4

16.0

23.8

70(560)

17.6

27.0

16.6

24.4

15.4

21.7

60(480)

20.420.117.316.716.316.2

35.733.321.316.012.610.23

18.918.216.115.815.515.1

32.530.019.014.010.88.52

17.717.115.114.714.614.2

30.427.416.711.88.76.91

100(800)

90(720)

50(400)

40(320)

30(240)

20(160)

# ofGateways

Unit: %

Table 3. Resource Usage of XML-based Manager

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100


ResourceUtility(%)

CPU Usage (1 Gateway)

Memory Usage (1 Gateway)CPU Usage (2 Gateways)

Memory Usage (2 Gateways)

CPU Usage (3 Gateways)


Figure 4. Resource Usage of XML-based Manager

However, as shown in Figure 5, the memory usage of the gateway decreases slowly in relation to the number of


10/16

10

gateways. Also, the CPU usage of the gateway decreases as the number of gateways increases. Obviously, this is

due to the fact that each gateway interacts with a smaller number of SNMP agents as the number of gateways

increases. As illustrated in Figure 5, the dotted line with squares shows that the CPU usage of one gateway

increases linearly until the gateway handles 80 SNMP agents (640 MIB objects). However, when the number ofSNMP agents is above 80, the CPU usage of the gateway increases rapidly. This explains why the response time

between the gateway and SNMP agents increases when the gateway handles more than 80 SNMP agents.

12.1

3.1

12.1

3.1

12.1

3.1

1(8)

12.1

5.9

12.1

5.6

12.2

8.2

10(80)

Memory Usage

CPU Usage

Memory Usage

CPU Usage

Memory Usage

CPU Usage

# of SNMP Agents(# of MIB Objects)

12.4

13.2

14.1

19

17.3

39.5

80(640)

12.3

10.3

13.1

16.6

16.8

33.6

70(560)

12.3

9.7

12.4

14.2

15.4

28.5

60(480)

12.412.412.312.312.312.2

17.514.787.57.26.43

15.114.712.412.412.312.3

26.522.412.110.39.68.42

19.718.814.814.412.412.3

79.560.323.11814.610.31

100(800)

90(720)

50(400)

40(320)

30(240)

20(160)

# ofGateways

Unit: %

Table 4. Resource Usage of XML/SNMP Gateway

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100


Resource

Utility(%)

CPU Usage (1 Gateway)

Memory Usage (1 Gateway)





Figure 5. Resource Usage of XML/SNMP Gateway

As shown in Figure 3, the response time of one gateway sharply increases when handling more than 80 SNMP

agents (640 MIB objects). Also, as shown in Figure 5, the CPU usage of one gateway rapidly increases when

handling more than 80 SNMP agents. Hence, we conclude that the CPU has a significant impact on the processing

overhead in the gateway and we perform more evaluations after upgrading the computing resources from

Pentium-III 800MHz CPU and 256 MB RAM to Pentium-IV 2.8GHZ CPU and 512MB RAM. Table 5 shows the


11/16

11

comparison results of the response time on the two different computing environments.

1821

1694

1329

96 5

52 5

40 2

33 1

27 7

18 4

12 9

22

4

Comm.

1998

1757

1420

1167

93 2

76 2

64 6

58 1

54 6

47 1

34 2

20 3

Proc.

Gateway Agent

22951015490532295100 (800)

222768505826222690 (720)

216132962586216580 (640)

198429422230198670 (560)

148312851996148460 (480)

141081 41820140850 (400)

139169 81615139840 (320)

131459 71463131830 (240)

124736 11278124520 (160)

117623 61058118810 (80)

10323883 210341 (8)

1006540 410121 (1)

Comm.Proc.Manager Gateway

Gateway AgentManager Gateway

2.8GHz, 512MB800MHz, 256MBComputingResources of

Gateways# of Agents(# of MIB Ob jects)

Unit: msec

Table 5. Response Time According to Computing Resources

0

2000

4000

6000

8000

10000

12000

0 20 40 60 80 100


ResponseTime(ms)

Gateway Agent (Proc.) : 800MHz, 256MB

Gateway Agent (Comm.) : 800MHz, 256MB

Gateway Agent (Proc.) : 2.8GHz, 512MB

Gateway Agent (Comm.) : 2.8GHz, 512MB

Figure 6. Response Time between the Gateway and Agents according to Computing Resources

Figure 6 shows the graphical representation of Table 5. As shown in Table 5, the processing time between the

manager and the gateway with the lower performance hardware (Pentium-III 800MHz, 256MB) is almost the sameas that between the manager and the gateway with higher performance hardware (Pentium-IV 2.8GHz, 512MB).

Therefore, we do not present the response time between the manager and gateway in Figure 6. After the computing

resources have been upgraded in our case, the message processing time and translation time decreases by half and

the communication time between SNMP stacks in the gateway and agents also decreases by half. Moreover, the

total processing time using the higher-performance gateway increases slightly when the number of SNMP agents


12/16

12

exceeds 80 (i.e., more than 640 MIB objects).

Finally, we evaluated the message size and response time in relation to the different XPath expressions using the

800MHz, 256MB gateway. We used four types of XPath expressions as shown in Table 6. In the case of //ifTable

expression, the processing time of XPath and XML message is smaller than the communication time. Thecommunication time increases because the size of the transferred data is large. In the case of ifInOctets |

ifOutOctets, the processing time of XPath and XML message is almost the same as that of ifTable expression.

However, the transferred message size is a small portion of that of ifTable expression and the communication time

of the second expression is much shorter than that of the first expression. The third and the fourth are the same

XPath expressions. So the transferred message size and communication time between the SNMP stacks in the

gateway and agents are almost the same as one another. However, the processing time of the fourth XPath

expression is much higher than that of the third expression. Therefore, using the appropriate XPath expression is

essential to retrieve the necessary information from SNMP agents considering the processing time and message

size.

As shown in Table 6, the network traffic volumes and MIB objects in these tests are much bigger than those of

the system group tests in Table 1 and Table 2. However, the number of SNMP agents and the processing time in the

gateway are much smaller than in the previous system group tests because the gateway requires more time to

process longer XML messages of the system group tests.

41233024 716 582(# of MIBObjects)

41233024 716 582(# of MIBObjects)

82466049 433 016 4(# of MIBObjects)

80786268444436341810(# of MIBObjects)

//ifType/following-sibling::ifOutOctets

//ifOutOctets

//ifInOctets |//ifOutOctets

//ifTable

XPath

346082772020784138606888Res. (bytes)

337842706020254135306724Req.(bytes)

88073958 937 619 2Comm. (ms)

52714710396533552863Proc. (ms)Gateway Agents

346082772020784138606888Res. (bytes)

337842706020254135306724Req.(bytes)

89877752 637 919 6Comm. (ms)


6921655440414962772013776Res. (bytes)

6756854120405082706013448Req.(bytes)

18631291108078 639 3Comm. (ms)


684210530902376410304802155764Res. (bytes)

662396513976364408297988148420Req.(bytes)

18605148661125377704230Comm. (ms)


54321# of SNMP Agents

Table 6. Response Time according to XPath Expression


13/16

13

4. Discussion of Results

In summary, we evaluated the scalability of the XML-based manager and the XML/SNMP gateway to manage

existing SNMP agents. First, we measured the network traffic volume for XML/HTTP communication and SNMP

communication. The network traffic overhead in SNMP communication is twice as much as XML/HTTPcommunication. Because the SNMP agents supported by current network devices in the POSTECH campus

network implement the SNMPv1 protocol, the XML/SNMP gateway retrieves information from SNMP agents by

using GetNext operations. Consequently, the network traffic volume generated by SNMP communications is high.

The network traffic volume between the XML-based manger and gateway is becoming much smaller than that

between the gateway and SNMP agents as the number of SNMP agents increases. Even if we add more

XML/SNMP gateways, the total traffic volume between the gateways and SNMP agents remains the same. On the

other hand, the traffic volume between the XML-based manager and gateway increases by about 500~1000 bytes

for each added gateway.

Next, we measured the XML/HTTP and SNMP communications time. When we retrieve information from only

one SNMP agent, the response time between the XML-based manager and XML/SNMP gateway is almost the same

as the response time between the XML/SNMP gateway and SNMP agent. The latency between the XML/SNMP

gateway and SNMP agents increases rapidly as the number of SNMP agents increases, whereas the latency between

the XML-based manager and XML/SNMP gateway increases more slowly. The response time between the gateway

and SNMP agents is about 90% of the total response time between the XML-based manager and SNMP agents if

the number of SNMP agents is more than 80. From the perspective of the gateway scalability, if we use one

gateway, the total response time increases in scale by about 1000ms every time we add ten SNMP agents until the

number of SNMP agents is about 70~80. However, if the number of SNMP agents exceeds 80, the total responsetime rapidly increases by about 15000ms. In contrast, when two or three gateways are used, the total response time

increases almost linearly until the number of SNMP agents reaches 100. The processing time between the gateway

and SNMP agents decreases considerably as the number of gateways increases. Therefore, the total processing time

between the XML-based manager and SNMP agents decreases. The total response time when using two gateways is

almost half the response time when using a single gateway. When three gateways are used, the total response time

is almost one-third of that when a single gateway is used. 70~80 SNMP agents are suitable for one gateway

(Pentium-II 800MHz and 256MB) to manage.

Also, we measured the computing resources of the XML-based manager and XML/SNMP gateway. When the

network devices are managed through one gateway, the resource utilization of the XML-based manager increases as

the number of SNMP agents increases. The CPU usage increases by 1~5% every time ten SNMP agents are added,

whereas memory usage increases by less than 2%. The increase rate of memory usage is not large compared to that

of CPU usage, because the management information loaded to the DOM tree is almost regular although the number

of SNMP agents increases. In terms of the gateway scalability, CPU usage of the XML-based manager increases by


14/16

14

2~3% and memory usage of the XML-based manager increases by 1~2% every time we add one more gateway.

The resource utilization of the XML/SNMP gateway also increases as the number of SNMP agents increases.

When the network devices are managed through a single gateway, the CPU usage of this gateway increases almost

linearly by 3~6% until the number of SNMP agents reaches 80. However, it rapidly increases to more than 20%when the number of SNMP agents reaches 80. This is why the response time between the gateway and SNMP

agents increases considerably after the number of SNMP agents is above 80. The gateway simultaneously makes

requests for multiple MIB objects to multiple SNMP agents using multi-threading mechanisms. When the number

of SNMP MIB objects exceeds 640, the CPU cannot support more threads. Therefore, we also need to upgrade the

CPU of the gateway or add another gateway as the number of the SNMP agents grows. The memory usage of the

gateway increases by 1~2%, however, the change is negligible.

From the perspective of the gateway scalability, the amount of computing resources assigned to each gateway

decreases as the number of gateways increases. The CPU usage of the gateway decreases to a half when two

gateways are used in the management system. Also, the CPU usage of the gateway decreases to one-third when

three gateways are used. The memory usage of the gateway also decreases by about 1~4% each time a new gateway

is added.

We performed more tests after upgrading the computing environments from 800MHz CPU and 256MB RAM to

2.8GHz CPU and 512MB RAM. The processing time of the gateway decreased by a half after upgrading the

computing resources. Also, the processing time of the gateway with 2.8GHz and 512MB did not lead to a

significant increase when the number of SNMP agents is above 80. The upgrade of computing resources allows to

support more threads and hence to retrieve thousands of MIB objects simultaneously.

Finally, we evaluated the processing time of different XPath expressions. Based on different XPath expressions,the transferred message size and processing time vary significantly. Though they are the same XPath expressions,

the difference in the complexity of their XPath expressions leads to different processing times. Therefore, one must

carefully select an appropriate XPath expression for exactly specifying the target objects to retrieve and avoid

delivering unnecessary data to the manager. The XML/SNMP gateway must provide acceptable performance when

concurrent calls from managers are made. Therefore, it is more desirable to delegate these overheads of XPath

parsing to the managers, provided that most manager systems have sufficient computing resources. Also, it can be

considered that the XML/SNMP gateway supports limited XPath expressions, and time-consuming XPath parsing

is performed in the manager for reducing the overhead of the gateway.

Even though the network traffic volumes and the number of MIB objects are small, the processing time of the

gateway is much higher and larger when there are more SNMP agents due to the processing overhead of the XML

messages parsing in the gateway. The gateway calls the snmpwalk function to retrieve multiple MIB objects from

the SNMP agent and the snmpwalk function internally uses a series of GetNext operations. Calling GetNext

operations when there are a number of SNMP agents (i.e., 10 operations for each of 10 agents) takes more time than


15/16

15

when there are smaller number of SNMP agents (i.e., 100 operations for 1 agent), assuming the total number of

GetNext requests (i.e., 100 times) is the same in both situations.

5.

Conclusion and Future WorkIn this paper, we have presented a performance evaluation of XNMS and validated its scalability. From the

performance evaluation results, we showed that the performance overheads of XNMS is not severe. We have

deployed XNMS for managing various network devices in the POSTECH gigabit campus network. We monitored

100 Cisco routers and switches among the POSTECH network devices. Because these network devices are

equipped with SNMPv1 agents, we used an XML/SNMP gateway approach to manage them. As the number of the

gateway increases, the total processing time is reduced considerably. More the gateways are used, shorter the

response time is. Also, the selection of an appropriate XPath expression can reduce the transferred message size and

processing time of the gateway. The performance evaluation results show that the volume of network traffic

between the XML/SNMP gateway and SNMP agents is larger. Also, the latency between the XML/SNMP gateway

and SNMP agents is longer. Therefore, SNMP processing overhead between the gateway and agents is much higher

than XML/HTTP processing overhead between the manager and gateway.

To reduce the network traffic overhead and response time between the gateway and SNMP agents, one needs to

upgrade the SNMP agents to support SNMPv2 using GetBulk operations. However, most of the widely-deployed

network devices are equipped with SNMPv1 agents. So, it is difficult to employ SNMPv2 agents and obtain

performance gain of the network traffic in the current situation. An interesting future work is to implement a tuning

process to optimize the CPU of the XML-based manager and the gateway. Also, as a future work, we would like to

devise a method for replacing the DOM parser with a SAX parser in order to reduce memory usage.

Acknowledgements

The authors would like to thank the NMRG members at the NOMS 2004 NMRG meeting who gave us

valuable comments on our work for improving the quality of our paper.

References

[1] T. Bray, J. Paoli and C. M. Sperberg-McQueen, Extensible Markup Language (XML) 1.0, W3Recommendation REC-xml-19980210, Feb. 1998.

[2]

F. Strauss, and T. Klie, Towards XML Oriented Internet Management, Proc. IFIP/IEEE InternationalSymposium on Integrated Network Management (IM 2003), Colorado Springs, USA, Mar. 2003, pp.505~518.

[3]

J. Schonwalder, A. Pras, J.P. Martin-Flatin, On the Future of Internet Management Technologies, IEEECommunications Magazine, October 2003, pp.90~97.

[4]

Frank Strauss, et al, A Library to Access SMI MIB Information, http://www.ibr.cs.tu-bs.de/projects/libsmi/.[5]

Avaya Labs., XML based Management Interface for SNMP Enabled Devices, http://www.research.avayalabs.com/user/mazum/Projects/XML/.


16/16

Documents

XNMS Performance DSOM Final