BLACKBIRD MONITORING SYSTEM - Performance Analysis and

BLACKBIRD MONITORING SYSTEMPerformance Analysis and Monitoring in Information Systems

Joao P. GermanoVodafone Portugal, AV. D.Joao II, LT.1.04.01 Parque Nacoes 1998-017, Lisboa, Portugal

[email protected]

Alberto R. Silva, Fernando M. SilvaINESC-ID R. Alves Redol, 9 1000-029, Lisboa, Portugal

[email protected], [email protected]

Keywords: Application Monitoring, Systems Administration, Application Performance, Component, Adaptable.

Abstract: This work presents the BlackBird system, which is an analysis and monitoring service for data-intensive en-terprise applications, without restrictions on the targeted architecture or employed technologies. Monitoringsystems are an essential tool for the effective management of Enterprise Applications and the attainment of thedemanding service level agreements imposed to these applications. However, due to the increasing complexityand diversity of these applications, adequate monitoring systems are rarely available. The BlackBird monitor-ing system is able to interact with these applications through different technologies employed by the MonitoredApplication, and able to produce Metrics regarding the application service level goals. The BlackBird archi-tecture is composed by several Application Interface Modules, and by a central component responsible forMetrics calculation and presentation. Application Interface Modules interact with the target monitored Appli-cation in order to get performance data in a common format. These data are stored in a common repositoryand used for Metrics calculation and presentation. The BlackBird system can be specified through a set ofpre-defined Configuration Objects, allowing it to be extensible and adaptable for applications with differentarchitectures.

1 INTRODUCTION

As a provider of high technology services, Vodafoneis under constant pressure to implement new tech-nologies that will allow the diversification of the pro-vided services and the improvement of existing ser-vices. Like in most large scale and technology basedbusiness, the Information Technologies (IT) infras-tructure has become the main base of support to busi-ness processes, and many Enterprise Applications arenow considered mission-critical having a direct im-pact on the performance goals of the entire company.

The complexity and diversity of the business rulesand provided services, together with the pressure forfast implementation demand a vast portfolio of dif-ferent applications. These applications can be ex-tremely diverse, in terms of complexity, architecture,base technologies and application provider. Also, asresult of the fierce competition environment, all theseapplications are required to constantly evolve in orderto implement new business requirements and supportnew services. In organizations such as Vodafone theteams responsible for the operation and management

of these applications are faced with the challenge ofassuring the best possible quality of service and theattainment of the negotiated Service Level Agrement(SLA). For this task it is essential to have monitoringsystems capable of providing a comprehensive viewof the application status and the most critical com-ponents, in order to anticipate performance problemsand act before there is any impact on the quality ofservice. The currently available monitoring systemscan provide efficient monitoring on the network anddevice level, however, due to the complexity and di-versity of the applications, these systems are unableto provide the desired monitoring on the applicationlevel.

Most of the available monitoring systems specif-ically target applications or technologies that have alarge user base, are limited to a fixed architecture, andmonitor pre-determined system parameters and ex-pected system components. This kind of monitoringignores all the functionality that is developed over thebase application, even though, it is this added func-tionality that implements the business logic and pro-duces the most relevant contribution to the delivered

46

quality of service. The need for adequate monitor-ing applications is even more serious for applicationsdeveloped in house or when the Monitored Applica-tion results from an extensive customization of a baseapplication. In this case the only solution is to de-velop, also in house, the necessary monitoring sys-tems. However, this extra development effort will cer-tainly increase project cost and complexity and riskdevelopment delays.

The BlackBird monitoring system is intended forassisting in the effective management of the ex-tremely diversified set of applications from the Infor-mation & System Technologies department (DTSI) ofVodafone Portugal. It has two main features: i) canmonitor an extremely diversified range of applicationssuch as the one found at Vodafone; ii) provides com-plex Metrics that relate to the main application goals.The diagram in fig. 1 represents a general view of themonitoring system, it obtains data from a number ofservers using different technologies, calculates met-rics and presents the results to the Operators.

Figure 1: Monitoring System Overview.

1.1 Existing Systems

The need for monitoring systems was born from theneed to assure high availability of the first enterpriselevel systems and networks, and as they matured sodid the monitoring systems. Simple Network Man-agement Protocol (SNMP) (Task-Force, 2007) be-came the most widely used management protocol andis currently the base to most network and device man-agement systems, HP Openview (Hewlett-Packard,2007) and Nagios (Galstad, 2007). However, due tothe numerous programming languages and the almost

infinite number of architectures and purposes, appli-cation level monitoring remains an extremely diversi-fied field with no predominant protocols or method-ologies.

Application specific systems can provide the bestmonitoring of any application. However, develop-ing dedicated monitoring systems is a costly processthat can only be supported by large companies with asignificant application portfolio, and it becomes im-possible to combine the monitoring of different ap-plications working together. Examples of such sys-tems are Microsoft Operations Manager (Microsoft,2007). Third party companies will only risk devel-oping application specific systems for applicationsthat can guaranty a large user base. Quest Softwareprovides versions of the Spotlight (Quest-Software,2007) monitoring system for BEA WebLogic Server,Oracle, etc. General purpose monitoring systems aimto provide monitoring services to a range of applica-tions as wide as possible. For this they will implementsupport for standard monitoring technologies and pro-tocols, and for proprietary protocols used in applica-tions with a large user base. The option taken by mostcompanies for adding monitoring capabilities to theirapplications is to implement a standard managementtechnology such as SNMP, JMX or WMI, for integra-tion with a general purpose monitoring system. Oneexample is ManageEngine (AdventNet, 2007) fromAdventNet.

In terms of architecture, most monitoring systemsemploy a Manager-Agent architecture, there is usu-ally a central management system and several agentsdeployed on the monitored system and relay the ob-tained data to the central management system usinga protocol such as SNMP (Task-Force, 2007). Thisis the architecture that emerged from the first net-work management systems and is currently used byall monitoring systems based on the SNMP protocol.It is especially adequate for monitoring of vast num-bers of distributed resources such as computer net-works like Openview (Hewlett-Packard, 2007), Na-gios (Galstad, 2007) or grid computing systems likeMonALISA (Newman et al., 2003).

As more applications evolved from centralized todistributed and from raw processing to providing ser-vices, a new monitoring architecture became possi-ble, Agentless Monitoring. In this case there is onlya central management system, and all data gather-ing is accomplished by remote access to the inter-faces provided by the Monitored Application. Agent-less Monitoring is usually less intrusive, easier to de-ploy and does not require continuous development,but with limited depth of data gathering. AgentlessMonitoring is especially adequate for services based

BLACKBIRD MONITORING SYSTEM - Performance Analysis and Monitoring in Information Systems

47

applications and is the only option for proprietary andclosed source applications. One example of agentlessmonitoring system is Longitude (Heroix, 2007) fromHeroix.

Since the first monitoring systems one key ele-ment of the monitoring process has been applicationinstrumentation, which consists in modifying the ex-isting applications in order to collect additional dataduring run-time. The importance of application in-strumentation for the management of complex dis-tributed application resulted in several technologylevel standards like JMX for Java and Java2 PlatformEnterprise Edition (J2EE) and WMI for MicrosoftProducts and the Microsoft .NET framework.

1.2 Future Trends

Extensive research has been aimed at improving themonitoring support of Enterprise Applications, withthe main focus on improving the monitoring exten-sions provided by the monitored application. Most ofthis work is focused on the JMX (Kreger, 2001) tech-nology which is part of J2SE platform. Although it isonly applicable to Java and the J2SE platform, it hasa close resemblance to the WMI for the .NET frame-work, and the main concepts are applicable to otherprogramming languages and architectures, includinglegacy applications (Diakov et al., 2000). The use ofstandard management architectures and instrumenta-tion techniques on enterprise applications opens theway to automated application management and selfmanaged applications (Diaconescu et al., 2004). Allthis research presents a common characteristic, it tar-gets component based applications, and as a result itis possible to obtain a good detail of application mon-itoring.

Whatever the purpose or technology used thecomplexity of Enterprise Applications dictates thatthe full task to be executed must be split betweensimpler tasks, that will be performed by different pro-gram modules. By considering a definition of Com-ponent less restrictive than the one usually associatedwith Component Based Software it should be possibleto model any application as component based, whereeach component may have a number of parametersthat can be used as indicators of the general applica-tion health and performance. And, by modeling theapplication, it should be possible to capture a moreabstract level of application functionality, which iscloser to the business logic and to the main applica-tion goals of quality of service.

2 REQUIREMENTS

The BlackBird monitoring system must provide fivekey features: i) monitor a wide range of applications,being adaptable to the architecture and technologiesof the Monitored Application; ii) provide in depth ap-plication level monitoring, component based; iii) eas-ily adaptable to the evolution of the Monitored Ap-plication; iv) low impact on the monitored system,agentless and without additional application instru-mentation; v) graphic interface for data visualizationand configuration.

The Monitoring Systems referenced in the previ-ous section are some of the most widely used andpresent a representative sample of the existing mon-itoring solutions. The table 1 summarizes the mainfeatures of these systems. Most of these systems al-low user defined data gathering, however, the user isusually required to supply an extensive set scripts forobtaining data. The BlackBird System requires onlya minimum of information for executing the samecommand, handles all data validation and conversion.Also, the BlackBird System allows the simultaneousexecution of any commands regarded as necessary.Although most of these system provide some formof support to user defined Metrics, Metrics based ondifferent data sources are usually limited to reportingpurposes. The BlackBird System is able to providereal time Metrics based on any combination of datasources. From these systems, the ones that provide aComponent Based Monitoring, support only applica-tions developed using the frameworks J2EE or .NET.The BlackBird System introduces a simplified Com-ponent definition for extending the concept of com-ponent based monitoring to applications that were notdeveloped as Component Base Applications.

The Blackbird System aims to provide a monitor-ing service to an application without imposing anylimitations on the target architecture, therefore, theBlackbird System provides a simplified set of Mon-itoring Operations that allow a user with detailedknowledge of the Monitored Application to specifythe required monitoring service: Commands to beexecuted where the result will be stored and used tocalculate Metrics; Metrics defined by a formula to beexecuted on the stored data to produce a result thatis related to the application’s performance indicators;Alerts for evaluating thresholds on Metrics and sendnotifications; Graphics that use the Metric as a datasource and plot the data according to the type and for-mat; Pages for containing graphics and structuring themonitoring interface.

WEBIST 2008 - International Conference on Web Information Systems and Technologies

48

Table 1: Feature Comparison Table.

Targeted Supported Agent Interface User Defined User Defined ComponentSystems Protocols Based Data Gathering Metrics Based

HP Openview General SNMP Y Thin Y Y NOperations Purpose Client

Nagios Network Network services Optional Web Y Y Nand Host

Microsoft Microsoft WMI Y Web Y Y YOperations Manager Applications and Client

Spotlight Dedicated JMX Y Thin N N YClient

Manage Engine General Multiple Optional Web Y N YPurpose

Longitude General Multiple N Web Y Y YPurpose

BlackBird General Multiple N Web Y Y YPurpose

3 ARCHITECTURE

As proposed in section 1.2, by modeling the Moni-tored Application as a set of interacting componentsit should be possible to obtain a more de detailedview of the application status and performance, also,it should be easier to obtain Metrics that relate tothe application performance goals. For the purposesof this work, we consider a simplified definition ofApplication Component: i) executes a well definedtask within the application; ii) can be univocally ref-erenced; iii) has a set of working parameters whichcan be obtained using the application interfaces. Fig-ure 2 represents the Monitored Application accordingto this definition.

Figure 2: Application Model.

Figure 3 presents the high level Domain Modelfor the BlackBird System. Each of the requiredMonitoring Operations is implemented by a ded-icated class, except for Commands that is splitbetween the Module class and subclasses dedi-cated to specific technologies, DatabaseScript andWebserviceRequest . The dataStore attribute ofModule stores all data produced by the associatedcommand and provide support to Metrics calculation.

The Module sub classes provide an adaptationlayer that isolates the BlackBird architecture from anytechnology details, they handle all technology spe-

class Domain Model

Module

+ dataStore: DataStore

Protocol

Metric

+ formula: char

Page

+ layout: char+ title: char

Graphics

+ datasource: Metric+ format: char+ link: char+ type: int

Alert

+ condition: char+ message: char+ state: int

DtabaseScript

+ login: char+ SQL_statement: char

WebServ iceRequest

+ arguments: char+ login: char+ method: char+ WSDL_location: char

+Evaluates *1

1..*

+Contains 1

+DataSource1

1

+DataSource

1..* 1

Figure 3: Domain Model.

cific logic like establishing a connection, authentica-tion, formatting the command, obtaining and validat-ing the response. Finally, they convert the commandresult to a normalized Extensible Markup Language(XML) document and deliver that document to thedataStore . For simplifying Metrics definition andcalculation, the dataStore is designed to be accessedas a relational entity. The Metric class provides thedata processing and aggregation functionalities of theBlackBird System by computing the formula spec-ified in the Monitoring Requirement. The Graphicclass provides the visual presentation to the Metricobjects, it will use the output of the Metric as a datasource and apply the type of graphic and the formatrequest in the Monitoring Requirement. A Graphicmay be a table of values or various chart formats. ThePage class provides the base for generating the mon-


49

itoring interface that will be accessed and navigatedby the operators. By combining the information fromthe Page and Graphic objects, the BlackBird systemgenerates the required interface as an Web Applica-ton containing the requested charts and tables. TheAlert class provide automatic notification of perfor-mance problems. It is defined as boolean condition tobe evaluated.

3.1 BlackBird Components

The Blackbird system uses an agentless architecture,it is composed of an Adaptation Layer itself com-posed by a variable number local Interface Modulesdesigned for specific protocols, an Aggregation Layerthat handles data storage and Metrics calculation, anda Presentation Layer for generating the monitoringpages and graphics. The Component Diagram of theBlackBird System is represented in fig. 4.

cmp Architecture

Adaptation LayerPresentation Layer

DatabaseScriptModule

WEBSercicesModule

Database

Database::DataStore

ApplicationServer

ApplicationServer:Page

ApplicationServer::Graphic

Database::Metric

ModuleManagerMetricManagerPageManager

Monitored Application

Agregation Layer

Database::Alert

JDBC

WEBServiceCall

JDBC

JDBC

JDBC

Figure 4: BlackBird Components.

The Adaptation Layer provides one of the mainfeatures of the BlackBird System, adaptation to thetechnologies of the Monitored Application. All tech-nology and protocol specific processing is performedby interface modules, where each type of inter-face module handles a specific technology or proto-col, performs all the tasks necessary for executingthe requested command, converts the command out-put to the normalized format and delivers it to thedataStore . The Adaptation Layer also performs thefirst step for providing a monitoring service adapted tothe architecture of the monitored system. By allowingmultiple Module to execute independently it becomespossible to specify as many data sources as requiredfor compiling a complete repository of performancedata that will allow the calculation of any relevantMetrics. The ModuleManager component controlsexecution of the Interface Modules and manages thedataStore of those objects. For this work only twomodules where developed, an SQL module that usesJDBC to connect to the monitored Database for exe-cuting an SQL statement and an Web Services mod-ule that implements the Apache Axis Framework for

dynamically invoking Web services. These technolo-gies are base to most of Vodafone applications, andallow the demonstration of the BlackBird capabilitiesin combining data obtained using different technolo-gies.

The BlackBird architecture is expandable to othertypes of protocols. Adding support for aditional pro-tocols to the BlackBird system requires no changesto existing components only the development of: i) anew interface module to handle the required protocol;ii) two new configuration tables and associated views;iii) new stored procedures for the edit operations onthe new type of module.

The Aggregation Layer stores all performancedata in an organized an easily accessible form, per-forms Metrics calculation and alerts verification. TheAggregation Layer contains the dataStore attributesfrom all existing Module objects implemented asdatabase tables and views, the combination of allthese objects constitutes a complete repository of allperformance data gathered from the Monitored Appli-cation. And, since all this data is accessible throughrelational queries, it should be possible to implementany Metric required by the Application Owner. TheAggregation Layer is responsible for other of themain features of the BlackBird System, ComponentBased Monitoring. Since all performance data gath-ered from the application can be used as input forthe Metric objects, it is possible aggregate the datacollected by different Modules using the componentidentifier to produce Metrics that provide a completeview of all aspects for that Application Component.The MetricManager component is responsible for cre-ating and updating the implementation of Metric for-mulas, and actively evaluating alert conditions.

The Presentation Layer is responsible for the finaloutput of the BlackBird Monitoring System, whichare monitoring pages containing visualizations of thestatus and performance of the Monitored Applica-tion. The monitoring pages are generated from theGraphic and Page objects and deployed on an Appli-cation Server as a Web Application. However, the au-tomated generation of the Web Application was con-sidered to be out the scope of this work, so the We-bcockpit (Klauser, 2007) application is used to gen-erate the monitoring pages based on a configurationfile created by the BlackBird System from the existingPage and Graphic objects. The PageManager com-ponent is responsible for managing the generation anddeployment of the Web Application that provides themonitoring interface.


50

4 IMPLEMENTATION ANDDESIGN

In order to isolate the BlackBird architecture from anytechnology details it is necessary to define a normal-ized format for the results data, this format must be: i)capable of containing any result; ii) easily convertibleto a relational format. According to the applicationmodel defined in the previous section (fig. 2), and tosupport component level monitoring, the output of acommand executed on the monitored system may bedefined as a tree of components each containing anynumber of working parameters, including a compo-nent identifier. According to this definition, any re-sult may be formatted as a XML document with theschema from fig. 5.

Figure 5: XML Schema.

Because the BlackBird System is intended for anenterprise environment where security and role sepa-ration is always a major concern, for each MonitoredApplication there is a dedicated database schema forcontain the implementations of dataStore Metricand Alert for that application. The BlackBird mainschema presented in fig. 6 is used for controlling themonitoring process, contains Monitoring Objects def-initions, and additional entities for implementing ap-plication separation and access control. The Moni-tored Application Schema contains the database ob-jects that implements the dataStore and Metric .Figure 7 presents an example schema containing theimplementation of two dataStore and two Metric ob-jects.

The final implementation of dataStore is com-posed of a Results Table and a Translation View, re-sult data is stored XML format in the Results Table,then based on the format of the XML result the Black-Bird System creates a view for selecting all XML el-ements as columns of result set. The complex imple-mentation of the dataStore allows the implementa-tion of the Metric object simply as a database viewcreated from the SQL formula defined in the Metric .

The final output of the monitoring process is anWeb application containing charts and tables that pro-

class dbo

alert

application

graphic

metric

module

page

SQL_command SQL_login

application_owner error_messagesoperator

WEBServ_commandWEBServ _login

1..*

1..*

11..*

1

0..1

*

0..1

*

0..1

1

0..1

1

1..*

0..*

1

0..1

1

1

*

1

1..*

1

1..*1

1..*

1

1..*

1

1..*

*

1..*

Figure 6: Main Schema.

Figure 7: Monitored Application Schema.

vide a visual representation of Metrics. From the ex-isting Page and Graphic objects, the PageManagergenerates the Webcockpit configuration file that spec-ifies the contents of the required Web application.Then, the Webcockpit application is used to gener-ate the Java Server Pages (JSP) that implement therequired charts and tables.

5 CASE STUDY

The BlackBird System is currently integrated in theproduction environment of some of the main appli-


51

cations of Vodafone Portugal DTSI, Mediation De-vice (MD), Pre Paid Billing System (PPB), ARBORBilling System, Provisioning Agent (PA) and AC-TIVIS Number Portability. All are Mission Crit-ical applications that require 24 hour support andmonitoring. The previously existing monitoring sys-tems, HP Openview Operations and dedicated moni-toring systems, provide complete network and devicemonitoring and a small degree of application levelmonitoring. The BlackBird System is used to com-plement these systems providing detailed applicationlevel monitoring.

5.0.1 Pre Paid Billing System

The PPB application is responsible for lifecycle man-agement of all pre-paid clients, it was developed atVodafone, it contains an Oracle database and was de-veloped using C and PL/SQL. This example will addmonitoring of recent functionalities of the PPB appli-cation that have not yet been included in the dedicatedmonitoring system.

Recent Business Requirements have introducednew types of requests, a database table acts as pro-cessing queue and the processing of each request re-quires a call to a WebService for confirming currentbalance. The main performance criteria for this taskare the average processing time and maximum pro-cessing time for these requests. For monitoring thisaspect of the PPB application, problems diagnosis andperformance tuning, it would be extremely helpful tohave a line chart presenting the average request exe-cution time, the average Web Service response time,and the difference between the two times, which rep-resents the contribution of the PPB System to the to-tal processing time. This is a relatively simple ex-ample, however, the presented parameters have directrelation to the application performance goal. Further-more, by splitting the application goal into sub param-eters, it becomes easier to identify the contribution ofeach application component to the final result.

Once the required output is defined, the Applica-tion Owner must elaborate the Monitoring Require-ment in order to specify the Configuration Objectsnecessary for producing the desired monitoring out-put. The Monitoring Requirement for this examplecontains the following Configuration Objects: i) Lo-gin details for the PPB database; ii) Login details forWeb Service calls; iii) SQL command for obtainingthe average processing time; iv) Web command forobtaining the Web Service response time; v) Metricfor combining data from both commands; vi) A linechart that uses that Metric ; vii) A page for containingthe the Graphic .

The final output of the PPB monitoring example is

shown in fig. 8. This chart shows the evolution of the

Seconds

Figure 8: PPB Queue.

requests processing time and the contributions of theWeb Service call and PPB processing.

In this example the BlackBird System was usedfor fast implementation of a detailed monitoring ser-vice, and to generate Metrics that aggregate data ob-tained using different technologies, Database accessand Web Services call.

5.0.2 Mediation Device

The Mediation Device processes billing records of alltraffic types, GSM, 3G, GPRS, SMS, etc. It collectsbilling records from all Network Elements, validatesand rates billable records. The Mediation Device wasalso developed at Vodafone, it contains an Sybasedatabase and was developed using C and TransactSQL. The Mediation Device processes various typesof records refereed to as Data Stream, and for eachData Stream, there are various Network Elements,this architecture can easily be modeled into compo-nents. The basic components are the Network El-ements that have parameters such as current delayand records processing rate. These elements can begrouped into the main application components theData Streams. The most critical SLA defined for theMediation Device are all related to the delay betweenrecord generation on the network element and its de-livery to the destination billing systems, and the fun-damental indicator of performance is the number ofrecords processed per unit of time.

The first chart from fig. 9 shows the evolution ofthe delay in Call Records processing for the NetworkElement components of the MD application. The sec-ond char on fig. 9 also shows the evolution of num-ber of Call Records processed per hour for the Streamcomponents of the MD.

In this example the BlackBird System was usedto improve the existing monitoring to detail the infor-mation on the component level, and provide real timevisualization of Metrics directly related to the SLA anperformance indicators.


52

Figure 9: Mediation Device Streams.

6 CONCLUSIONS AND FUTUREWORK

The BlackBird System aims to provide detailed ap-plication level monitoring to a range of applicationsas wide as possible, requiring a minimum of con-figuration work. The use of a simplified applicationmodel creates a language for describing most appli-cations as a component hierarchy. Also, it allows thedefinition of common data format for containing per-formance information independently of any technol-ogy details. Data gathering is performed by technol-ogy specific Interface Modules, where the Applica-tion Owner may specify any number of modules forvarious protocols. This way, it becomes possible togather any relevant data independently of the targetarchitecture or the employed technologies. The per-formance data is stored in DataStore units accessibleas relational entities. Data can be referenced by spec-ifying the Module that collected the data and the ap-plication component it refers to. Based on this datarepository it is possible to create complex metrics thatmerge data from multiple sources, and metrics that re-late to the performance goals for the application.

Although the data repository provides an easilyaccessible source of data, it can become very complexas the number Interface Modules increases. Also, thetask of defining Metrics in the BlackBird system re-quires a good knowledge of the BlackBird architec-ture and SQL. This task can be greatly simplified bythe addition of a graphical query building interface, asan applet on the Configuration Pages or as a separateapplication.

The BlackBird was designed as an expandableSystem, therefore some of the main improvementswill be new Interface Modules for additional proto-cols: Remote Shell for access to unix systems, JMXfor J2EE applications, WMI for .NET application.

REFERENCES

AdventNet (2007). Applications Manager.http://manageengine.adventnet.com/products/index.html.

Diaconescu, A., Mos, A., and Murphy, J. (2004). Auto-matic performance management in component basedsoftware systems. Proceedings of the InternationalConference on Autonomic Computing.

Diakov, N. K., van Sinderen, M., and Quartel, D.(2000). Monitoring extensions for component-baseddistributed software. Fifth International Conferenceon Protocols for Multimedia Systems.

Galstad, E. (2007). Nagios 3.x documentation.http://nagios.sourceforge.net/docs/nagios-3.pdf.

Heroix (2007). Longitude. http://www.heroix.com/ agent-less/agentless performance monitoring main.htm.

Hewlett-Packard (2007). HP Operations Dashboard soft-ware. http://www.managementsoftware.hp.com/products/ovd/ds/ovd ds.pdf.

Klauser, P. J. (2007). Webcockpit Open Source Project.http://webcockpit.sourceforge.net/schema/schema.html.

Kreger, H. (2001). Java management extensions for appli-cation management. IBM SYSTEMS JOURNAL, 40.

Microsoft (2007). System Center OperationsManager. http://technet.microsoft.com/en-us/opsmgr/default.aspx.

Newman, H., I.C.Legrand, Galvez, P., Voicu, R., andCirstoiu, C. (2003). Monalisa : A distributed moni-toring service architecture. CHEP03.

Quest-Software (2007). Spotlight for BEA WebLogicServer. http://www.quest.com/spotlight-for-bea-weblogic-server/.

Task-Force (2007). Simple network management pro-tocol (snmp) applications. ftp://ftp.rfc-editor.org/in-notes/rfc3413.txt.


53

Documents

BLACKBIRD MONITORING SYSTEM - Performance Analysis and