Future Generation Computer Systems 47 (2015) 1630Contents lists available at ScienceDirectFuture Generation Computer Systemsjournal homepage: www.elsevier.com/locate/fgcsComparative analysis of architectures for monitoringcloud computing infrastructuresJose M. Alcaraz Caleroa,, Juan Gutirrez AguadobaUniversity of the West of Scotland, School of Engineering and Computing, High Street, Paisley Campus, Paisley, PA1 2BE, Scotland, United KingdombUniversity of Valencia, Departamento de Informatica, Avda. Universidad s/n, 46100 Burjassot, Valencia, Spainh i g h l i g h t sFirst paper with empirical comparison of monitoring architectures for cloud computing.5 novel monitoring architectures validated against a real cloud computing infrastructure.Free and Open Source Implementation released to the scientific community.1000 Virtual Machines executed for more than 2 months in order to come with these results.Recommendations about the best monitoring choice according to your necessities.a r t i c l e i n f oArticle history:Received 13 January 2014Received in revised form20 December 2014Accepted 27 December 2014Available online 5 January 2015Keywords:Cloud computingArchitecture evaluationInfrastructure monitoringService monitoringInfrastructure-as-a-ServiceMonitoring-as-a-Servicea b s t r a c tThelackof control overthecloudresourcesisoneof themaindisadvantagesassociatedtocloudcomputing. The design of efficient architectures for monitoring such resources can help to overcomethis problem. This contribution describes a complete set of architectures for monitoring cloud computinginfrastructures, and provides a taxonomy of them. The architectures are described in detail, comparedamongthem, andanalysedinterms of performance, scalability, usageof resources, andsecuritycapabilities. The architectures have been implemented in real world settings and empirically validatedagainst a real cloud computing infrastructure based on OpenStack. More than 1000 virtual machines(VMs) have been executed for more than 2 months in scenarios ranging between 18 and 24 simultaneousVMs in order to achieve the empirical comparison provided in this contribution. The implementation ofall the monitoring architectures has been released to the community as MonPaaS, a public open sourceproject for OpenStack. Also, some recommendations about the best architecture in terms of performanceand security have been covered in this contribution as part of the analysis carried out.Crown Copyright 2015 Published by Elsevier B.V. All rights reserved.1. IntroductionThe Infrastructure-as-a-Service (IaaS) associated to cloud com-putingischangingthewayinwhichbusinessesarefacingtheacquisition of new hardware.Now,businesses can take the ad-vantages of cloud computing in order to reduce significantly thecosts associated to up-front acquisition of hardware by means ofthe renting of a portion of the hardware requirements in a pay-as-you-go model. This hybrid scenario in which the owned and rentedresources are used to create an elastic infrastructure that is con-stantly changing according to the business requirements entailsCorresponding author.E-mail addresses: JoseMaria.AlcarazCalero@uws.ac.uk (J.M. Alcaraz Calero),juan.gutierrez@uv.es (J. Gutirrez Aguado).several challenges that may be addressed to make cloud comput-ing really attractive for the markets.One of the challenges associated to the usage of *aaS is the lackof architectures to allow an adaptive and secure monitoring of theresources both for the provider of the infrastructure and the con-sumer of the resources. This should be analysed carefully becauseenabling the consumer to configure the monitoring of the resourcecan entail a security risk as she could incidentally monitor the re-sources of other consumers or the physical infrastructure.Also,the provider should obtain from the rented resources the infor-mation in a transparent way but should monitor management ofVMs or physical machines in a more exhaustive way using a richset of metrics. This lack of control may be addressed by means ofmonitoring services to collect the status of the cloud computinginfrastructure. Thesemonitoringservicesenablebothcloudin-frastructure provider and cloud infrastructure consumer to get ahttp://dx.doi.org/10.1016/j.future.2014.12.0080167-739X/Crown Copyright 2015 Published by Elsevier B.V. All rights reserved.J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 17complete overview about the status of the cloud resources. Themainproblemwhencomingwiththesemonitoringservicesistwofold. On the one hand, the monitoring services currently avail-ableinthemarkethavebeenspecificallydesignedtomonitorphysical resources and for this reason they do not fit well whenmonitoring virtual resources that have a completely different lifecycle, being the resources constantly created and destroyed. No-tice that physical resources are not created and destroyed but vir-tual resources are constantly being created and destroyed. This factcauses serious problems in many of the current monitoring solu-tions due to the fact that they do not have beendesignedto forgetresources. In consequence, they assume a 1-by-1 relationship be-tween IP and monitored node. However, the reality in cloud infras-tructures is that in a very short period of time the IP address beingassigned to one VM is now being reused in another VM and evenit can be assigned to a different consumer. This change will notbe identified in traditional monitoring architectures and in conse-quence they will end up monitoring a complete different resource.Obviously, this is not acceptable for real deployments. So, the mon-itoring software should deal with frequent changes in the topol-ogy and with the newlife-cycle of the virtual infrastructures wherecreation and destruction are part of the cycle. On the other hand,the number of monitoring services specifically designed to fit ininfrastructures of cloud computing is really scarce (almost inexis-tent if we focus on open source solution). For this reason, it is re-ally difficult to find comparisons betweendifferent architectures ofmonitoring services for cloud computing. In fact, to the best of ourknowledge, this is the first attempt of providing such comparison.Themaincontributionof thispaperisthedescriptionof acomplete set of innovative and novel architectures of monitoringservices suitable for cloud computing infrastructures. All the ar-chitectures described have been designed, analysed, implemented,and released to the community as open-source software. The ar-chitectures have also been empirically validated in a real cloudcomputing infrastructure. A comprehensive analysis in terms ofperformanceandsecurityhasbeencarriedout comparingthedifferent architectures provided. A novel taxonomy and some rec-ommendations of the best choice have been also provided.Thecontribution pretend to provide to the scientific community a setof open source implementations of monitoring solutions and alsoto help any academic or practitioner to determine what is the bestmonitoring alternative according to her particular necessities. Themonitoring tools have been validated over different versions ofOpenStack including Folsom, Havana, Ice House and Juno and withdifferent networking modes including Nova-Network and Neutron.This contribution has been organized as follows: Sections 2 and3 introduce the basic background on infrastructures of cloud com-puting and in distributed monitoring services, respectively. Afterthat, Section 4 describes in detail the architectures of monitoringservices for cloud computing infrastructures. Then, Section 5 de-scribes the comparative analysis between all the architectures pro-viding as well a recommendation of the most suitable architecturefor different scenarios. Section 6 describes the prototypical imple-mentation of the architectures. An empirical evaluation of differ-ent architectures is presented in Section 7 in order to analyse theperformance of each of the architectures. Section 8 provides a com-plete state-of-the-art on monitoring services for cloud comput-ing and shows the mapping between the architectures provided inthis contribution and the solutions available in the market. Finally,Section 9 provides some conclusions and outlines future works.2. Architecture of an IaaS stackThis section provides an introduction to the components avail-able inside of a cloud computing infrastructure. Fig. 1 shows anFig. 1. Architectural overview of a cloud computing infrastructure.overview of the different components available therein. It is im-portant to remark that it is assumed a public cloud computing sce-nario in which cloud consumers, i.e. the organizations using thecloud services, and the cloud provider, i.e. the organization rentingthe cloud resources, are different organizations. This scenario en-tails more challenges than private clouds where both belong to thesame administrative domain.AccordingtoNISTcloudcomputingisamodel forenablingubiquitous, convenient, on-demandnetworkaccesstoasharedpool of configurable computing resources (e.g., networks, servers,storage, applications, and services) that can be rapidly provisionedand released with minimal management effort or service providerinteraction [1]. There are some essential services required to pro-vide the IaaS to the cloud consumers. These services are summa-rized as follows: (i) Web UI. The cloud consumers use the Web UIto rent cloud resources using a Web browser; (ii) API is used tointeract programmatically with the cloud computing infrastruc-ture; (iii) Authentication and Authorization are in charge of control-ling the actions allowed for the cloud consumer inside the cloudprovider; (iv) Scheduler is in charge of deciding the physical ma-chines (PM) in which the rented resources will be allocated; (v) VMImages manage the images of different operating systems availableto the cloud consumer; These images are lately used in the virtualmachines (VMs) rented by the cloud consumer; (vi) Storage is incharge of managing the storage devices rented to the cloud con-sumer; (vii) Billing controls the usage and resources billing of theresources; (viii) Networking is in charge of controlling the config-uration of networking associated to the VMs. Currently, this Net-working module can provide basic connectivity between VMs orcan also provide a complete Software-Defined Network solutionenabling cloud consumer to create their own network infrastruc-tures; (ix) Certificate service controls the cryptographic informa-tion used into the VMs.The previous services compose the Cloud Controller. They canbe seen in the upper part of Fig. 1. These services are deployed inonly one physical machine or even in only one VM, or in severalphysical machines or VMs depending on size and purpose of the in-frastructure deployed. Sometimes these deployment decisions aredetermined by the implementation of the cloud computing stackutilized. So, OpenStack,1a well-known open source IaaS stack, fos-ter the usage of physical machines for such deployment whereas1OpenStack is available for download at http://www.openstack.org/.18 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630Fig. 2. Architectural overview of a monitoring service.Apache CloudStack2foster the usage of VMs for such purpose. Allthe services are communicated by means of a communication mid-dleware. Usually, a message bus is used as a communication mid-dleware due to its innate advantages.There is also a set of computers which compose the computingmachines of the infrastructure. These machines are the computa-tional resources rented to the cloud consumers. These machineshave usually a virtualization layer installed to enable the manage-ment of VMs, virtual partitions, etc. Generally, these machines havealso installed a Computing service used to connect the machines tothe communication middleware. This connection allows the recep-tion of messages fromthe Cloud Controller to performactions in thevirtualization layer. These machines can be seen in the centre ofFig. 1. When a cloud consumer creates a VM, this VM is isolatedfrom the rest of VMs belonging to other consumers for securitypurposes. The cloud consumer can decide if her VMs are publicallyvisible inInternet or only internally accessible. These VMs are com-posing the virtual infrastructures created by each cloud consumer.They can be seen in the lower part of Fig. 1.Thevalidationof thedifferentmonitoringarchitecturesde-scribed in this paper has been successfully done using OpenStackas softwarefor managingthecloudcomputinginfrastructure.Concretely,the architectures described in this paper have beenvalidatedinOpenStackFolsom, Havana, andIceHouseusingtwodifferent networking architectures: Nova Network that only pro-vides basic networking connectivity and the new OpenStack Neu-tron3providingacompletenetworkingsolutionforbothcloudconsumer and provider. In all the cases, all the architectures de-scribed in this paper have been correctly validated. Technical de-tails will be provided lately in Section 6.3. Architecture of a distributed monitoring systemFig. 2 shows an overviewof the components available in a mon-itoring service. The monitoring core is in charge of performing andcontrolling all the monitoring tasks. This component uses configu-ration files to obtain the resources and services to be monitored.2Apache CloudStack is available for download at http://cloudstack.apache.org/.3Information about OpenStack neutron is available at https://wiki.openstack.org/wiki/Neutron.The monitoring logicimplemented inside the monitoring coreuses such configuration files to perform the monitoring of the dif-ferent metrics. These metrics are gathered using different exten-sions (see the right part of Fig. 2). This designis flexible as it enablesto extend further monitoring approaches in the future. Currently,the following ones are usually provided in any of the enterprise-level monitoring software available in the market:(i)Local monitoring. The monitoring logic uses plugins executedlocally to extract metrics directly from such local machine.This local machine is where the monitoring core is running.(ii)Agent-less remote monitoring. The monitoring logic uses plug-ins to extract metrics from remote resources in a transparentapproach, i.e. without the necessity of using any software in-stalled in the remote resource. Port scanning, ICMP requestand TCP connections are some examples of this type of moni-toring.(iii)Agent-basedactiveremotemonitoring. Themonitoringlogicuses plugins to interrogate a software agent running in theremote resources. When the remote software agent receivesrequests, it executes the plugins to extract locally the metricswhich are lately sent back to the monitoring core.(iv)Agent-basedpassiveremotemonitoring. Themonitoring logicuses a plugin which opens a port. Then, the software agentrunning in the remote resource sends information to such portwhen necessary, for example, periodically or using an event-based notification approach.The monitoring core is also composed by other functionalities. TheEvent Logic is in charge of registering event handlers which per-form actions when such events occur in the system. These actionsare usually the execution of commands. The Notification Logicisin charge of informing the administrators about events. When themonitoring service is running, the log files and status files are beingcontinuously updated with the metrics gathered. Then, the mon-itoring graphical interface shows graphically such information tothe user. This interface only enables the user to see the informa-tion. However, if the user wants to change the configuration files,she must use the monitoring management interface for suchpurposedefining what the resources to be monitored are, what the metricsto be monitoredover suchresources, are andhowthese metrics arebeing collected. Finally, the monitoring service may scale in a dis-tributed monitoring platform. The performance logic implementedinthe monitoring core is incharge of suchpurpose. This performancelogic enables an optional balancing of the workload of monitoringJ.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 19Fig. 3. Architecture overview of the internal monitoring architecture (IMA).tasks between different machines. This feature enables to scale themonitoring service to large deployments. The performance logic di-vides the tasks along all the registered monitoring instances (seethe middle part of Fig. 2). It enables to create a hierarchy of moni-toring instances to perform a smart load balancing of the tasks.4. Architectures for monitoring infrastructures in cloud com-putingThis section describes all the architectures of monitoringservices for cloud computing infrastructures. This section has beendivided into five subsections to make the reading easier.4.1. Monitoring physical machines in the internal networkIt is highly probable that the cloud provider wants to monitorhis physical machines to get an overview of the status of his in-frastructure. The monitoring of the physical machines requires theinclusion of the monitoring service inside of the internal network(usually referred to as management network as well) where theinfrastructure of cloud computing is running due to the fact that itis required a direct connectivity with the physical machines. Gen-erally, the physical machines are neither visible in Internet nor bythe virtualization layer. This isolation of the physical machines isdone for security concerns, minimizing external and internal se-curity threats. These architectures are referred henceforth as In-ternal Monitoring Architecture (IMA) due to its internal placement.Fig. 3 shows an overview of a Traditional Internal Monitoring Archi-tecture, the first kind of these architectures. As can be seen in Fig. 3,bothagent-less and agent-based monitoring are allowed due to thefact that the cloud provider has the complete control over all themachines involved. IMA does not entail important challenges withrespect to existent monitoring solutions in the market. In fact, ex-isting enterprise-class open-source monitoring solutions like Na-gios4and Ganglia5can be used for this purpose. Both monitoringand management graphical interfaces may never be accessible pub-lically in this architecture. If anyone gains access to such interfaces,she can act as an administrator which is simply unacceptable in areal deployment. Only the cloud provider may have access to suchinterfaces due to the intrinsic risks associated.It would be an arguable scenario in which the cloud consumerscould access to a limited set of metrics about the physical machinesinwhichtheirVMsarerunning. Thisinformationmaybewel-come for some consumers. They may use such metrics to performsmartdecisions. However, thesameinformationmaybedetri-mental for other consumers. They may use such information to4Nagios is available at http://www.nagios.org/.5Ganglia is available at http://ganglia.sourceforge.net/.Fig. 4. Architectural overview of the extended internal monitoring service (EIMA).identify collocated VMs inside of the same physical machine, het-erogeneous hardware, bandwidth consumption, etc. This informa-tion may cause that some cloud consumers may not be happy withthe results found in the analysis of such metrics, even taking thedecision to shift to a different infrastructure provider. This is prob-ably the main reason for which none of the current public vendorsprovide such information to the consumers. So, we have decidedto discard such option not from the technical side but for the busi-ness side. So, Traditional IMA is the only architecture described inthis paper suitable for the cloud provider but not suitable for thecloud consumer.4.2. Internal approaches for monitoring physical and virtual machinesIt is highly probable that the cloud provider wants to monitorboth physical and VMs. In case the cloud provider wants to mon-itor the VMs belonging to all the cloud consumers, there is a clearrequirement to be fulfilled. The consumers VMs have to be moni-tored using a non-intrusive approach, i.e. the monitoring approachhas to be hidden fromthe point of viewof the cloud consumer. Thisrequirement makes the installation of a monitoring agent inside ofthe VMs not a real option in production scenarios. The monitoringof VMs may be implemented extending the architecture previouslydescribed in Section 4.1. This extension consists in the usage of anewplugin, named Hypervisor Plugin, for local monitoring installedin the physical machines.This plugin extracts the metrics fromthe hypervisor in the virtualization layer, and is invoked by thesoftware agents installed in the physical machines when required.Currently, only a small and closed set of metrics can be gatheredfrom hypervisors. Concretely, bandwidth I/O, file I/O, CPU I/O andMemory I/Oare the provided metrics in the vast majority of hyper-visors. This plugingathers suchmetrics using transparent monitor-ing fromthe point of viewof the cloud consumer. This architectureis henceforth referred as Extended Internal Monitoring Architecture(Extended IMA), and is depicted in Fig. 4 (see the Hypervisor Plugin).Although Extended IMA is a good architecture, it does not coverthe case in which the cloud provider wants a wider range of met-rics. So, a new architecture can be provided in which the Hyper-visor Plugin is also complemented with other metrics gathered bymeans of agent-less remote monitoring of the VMs. These metricsenable the cloud provider to get more information like the status ofany TCP/UDP port, ICMP responsiveness, etc. This is a transparentand non-intrusive monitoring approach, but requires a direct com-munication with all the VMs. This communication is really pow-erful but at the same time is very challenging. On the one hand,the communication from physical to virtual machines can be en-abled in infrastructures of cloud computing but not the other wayaround usually blocked for security purposes. This fact needs to be20 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630Fig. 5. Architecture overviewof the extended adaptive internal monitoring service.carefully addressed, for example, enabling only active monitoringapproaches where the connection is originated from the physi-cal machine. On the other hand, even with the incredible numberof traditional monitoring solutions providing agent-less monitor-ing capabilities (IBM Tivoli Network Manager, OpenNMS, LiveAction,OpsView, OpenKBM, Pandora FMS to name a few), none of them fitin infrastructures of cloud computing. The main reason is the in-credible difference in the life cycle of the VMs (now being moni-tored in this architecture) with respect to the physical machines.The physical infrastructures rarely change their topology, mainlydue to hardware failures or replacements. In fact, the destructionof physical resources is conceptually impossible. Moreover, phys-ical infrastructure usually keeps constant the number of servicesprovided. These features have been the foundation during the de-velopment of all the services around the traditional monitoringecosystem: monitoringagents, hardwareauto-discoveryproto-cols, service auto-discovery protocols, etc. However, virtual infras-tructures expose a completely different set of requirements. First,services have to be monitored in a transparent way. Second, VMscan be destroyed causing frequent changes in the topology and cur-rent discovery protocols do not react properly under this destruc-tion of resources. Third, IP addresses are reused constantly and itcauses incoherent mixes of monitoring information due to colli-sions of IP addresses. So, traditional monitoring approaches do notreally fit well inthe monitoring of cloudcomputing infrastructures.This problem can be addressed by means of the creation of anewmonitoringmodule. Thisnewmoduleisconnectedtothecommunication middleware of the cloud computing infrastructurein order to monitor topological changes in the virtual infrastruc-ture. These changes are notified to the monitoring service as soonas they are detected enabling a self-adaptive monitoring service.So, this newmodule performs automatically the setupfor anagent-lessmonitoringofnewVMsandcanremoveautomaticallyre-sources from the monitoring service when they are destroyed. Thearchitecture including this newmodule is depictedinFig. 5 (see theMonitoring Module) and will be henceforth referred as Extendedand Adaptive Internal Monitoring Architecture (Extended and Adap-tive IMA).There are other approaches to achieve the monitoring of VMs.They are explained in the following subsection.4.3. External approaches for monitoring virtual machinesAnExternal MonitoringArchitecture (EMA) is definedas anarchitecture in which the monitoring services are located out ofthe internal network. This definition includes monitoring servicesdeployed either inside of VMs or in an external location outside ofthe cloud computing infrastructure. However, the deployment ofmonitoring services in an external location is not a suitable optiondue to the fact that only publically available VMs can be monitoredandnot internal VMswherenopublicIPs areavailable. Thisfact makes the cloud computing infrastructure the only suitablelocationtodeployanexternal monitoringarchitecture. Inthiscasethemonitoringserviceswill bedeployedinthesocalledVMdatanetwork. Theusageof VMsfor monitoringpurposesenables the setup of a distributed monitoring architecture whichscales following two different approaches: static and dynamic. Thestatic approach scales the number of VMs manually by means ofthe administrator policies whereas the dynamic approach scalesthenumberofVMsautomaticallyaccordingtoagivencriteriaproviding anelastic monitoring architecture. It is worthy toempathize that VMs do not have access to the physical machinesof the cloud computing infrastructure. For this reason, this kindof architectures is suitable for the cloud consumer. They are alsosuitableforthecloudprovider, however, onlyinthecasethathe does not require the monitoring of physical machines. In fact,notice how a static External Monitoring Architecture for providingmonitoring services to the cloud provider is architecturally similarto an Extended and Adaptive IMA where the monitoring of physicalmachines has been disabled. The main difference is the shifting ofthe deployment of the monitoring services fromphysical machinesto VMs which provides less performance than physical machinesand there is no reason for isolating the monitoring services insideof a VM so far than security concerns. So, it has been decided todiscard a static EMA for the cloud provider in favour of EAIMA.Moreover, static EMAs for providing monitoring services to cloudconsumersmaybeapotential bottleneckduetothefactthatit does not scales accordingtothesizeof theinfrastructure.This lack makes static EMAs an unsuitable architecture for largedeployments. So, allthestaticapproacheshavebeendiscardedandit has beenconsideredonlythedynamic approaches. Indynamic EMA, it is logical to consider the elasticity factor of theinfrastructureasonemonitoringVMpereachcloudconsumerthat has at least one VM running in their virtual infrastructure.Thisapproachscalesreasonablythenumber of VMsusedformonitoring purposes according to the size of the infrastructure.There are other options for defining the elasticity factor like thenumber of total VMs runningintheinfrastructure. However,theusageofaVMpertenantnetworkhasbenefitsduetothefact that these monitoring VMs can be used to provide differentservices to the cloud consumer (others rather than monitoring).In fact,extra VMs can be optionally instantiated to balance themonitoringworkloadofsuchcloudconsumeraccordingtothenumber of VMs running, if necessary. This is optionality used tocoverscenarioswherethereareonlyafewconsumerswithalotof VMsrunningbyeachconsumer. Anotherbenefitof thiselasticityfactoristhatitenablestheimplementationofmulti-tenancy support in the monitoring services by means of a multi-instantiation of monitoring services, one VM per each consumer.The main disadvantage of the External Monitoring Architectureswithrespect to Internal Monitoring Architectures is the usage of VMswhichentails a significant usage of virtual resources for monitoringpurposesratherthanbeingrentedtothecloudcustomers. Ascounterpart, theusageof VMs enables areal distributedandelastic architecture suitable for high scalability. Moreover, EMAsare also more extensible in terms of the services which may beprovided due to the fact that the VMs can be used not only formonitoring services but also for a myriad of services.EMAs arecomposed mainly of two different components. A self-containedVM image with all the monitoring services preloaded and readyto be instantiated, and a monitoring module connected internallyto themessage queue. Thismonitoringmodule is analogoustothe introduced in EAIMA but now it is in charge of starting newmonitoring VMs,and of notifying to the monitoring VMs abouttopological changes in the infrastructure. These VMs are referredJ.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 21Fig. 6. Architecture overview of the external monitoring architecture.asMonitoringVMs(MVMs). Fig. 6depictsanoverviewof thefirst external architecture proposed, the Sparse External MonitoringArchitecture (SEMA). The term Sparse is used to refer to the factthateachoftheMVMsisinstantiatedinthevirtualdomainofthe cloud consumer, i.e. the monitoring virtual machine belongsto the customer. This decision implies that SEMA is only suitablefor providing services to the cloud consumer and not to the cloudprovider. Otherwise, thecloudconsumercouldseeinherownvirtual domain VMs (used by the cloud provider for monitoringpurposes) that are unknown for her. This Sparse EMA Architecturemakes the cloud consumer responsible of the management of thelife cycle of such MVM. She can stop, start and even destroy eitherintentionallyorbyaccidenttheMVMwithitsriskassociated.This access may be a potential source of security threats. It alsorequires that the monitoring module must impersonate to eachof the cloud consumers to create the MVM on their behalf withthe risks associated to the execution of this privileged action. Forall these security issues, Sparse EMA is only welcome when it isexplicitly wanted to provide the control of the life cycle over theMVM machines to the cloud consumer.To overcome the security issues associated to Sparse EMA theConcentrated External Monitoring Architecture (Concentrated EMA) isproposed. This architecture imposes the creation of a special cloudconsumer called MonPaaS that is the only owner of all the MVMs.So, the monitoring module only needs to impersonate MonPaaSand also only needs access to the virtual domain of MonPaaS. Infact, only the monitoring module (who is directly managed by thecloud provider) cancontrol the life cycle of the MVMs and thus suchMVMs can be used for providing other services in the infrastruc-ture. In Concentrated EMA, all the MVMs have a public IP which isused by the cloud consumers to access her monitoring and man-agement interfaces.From the point of view of the performance, there are not sig-nificant differences between these two architectures, ConcentratedEMA and Sparse EMA. Thus, Concentrated EMA may be a best alter-native to Sparse EMA in almost all the scenarios. The only exceptionare the scenarios in which the management of the MVM life cycleis explicitly given to the cloud consumers.4.4. Monitoring VMs from the point of view of the cloud consumerThis subsection explains the modifications required over thearchitecturesdescribedintheprevioussubsectioninordertomakethemsuitableforprovidingmonitoringservicestocloudconsumer.Both Extended IMA and Extended and Adaptive IMA are suitablefor providing the monitoring services to the cloud consumer overher VMs. The only extra requirement over these architectures isthat the monitoring service must provide multi-tenant support toenable all cloud consumers to access their own overview of theinfrastructure. In fact, Extended and Adaptive IMA may be optionallyextended with the usage of a software agent installed in the cloudconsumersVMinorder toprovidemoredetailedmonitoringinformation to the consumer. Moreover, it is important to remarkthe fact that both architectures may have disabled the monitoringof the physical machines. Sparse EMA and Concentrated EMA aredirectly suitable for providing the monitoring services to the cloudconsumer over her VMs. The main architectural decision of thearchitectures oriented to the cloud consumer is not related to themonitoringgraphical interfacewhichmaybealwaysexposedtothe consumer but related to the management graphical interface.So, regardingthemanagementgraphical interface, thefollowingscenarios can occur:(i)NomanagementInterfaceisexposedtotheCloudConsumer.Thereareaclosedset of (predefined) metrics whicharetransparentlygatheredbythemonitoringservicefromtheconsumers VMs. These metrics are shown in the monitoringgraphical interface. This option is implemented by a numberof publiccouldproviderslikeAmazonEC2CloudWatchorRackSpaces, amongothers. Also, thisoptionisprovidedbyOpenStack Ceilometer.(ii)Basic Management Interface is exposed to the Cloud Consumer.This interface enables the cloud consumer to customize themetrics and services to be monitored inside of her VMs. Thisinterface can be also used to customize the metrics relatedwith the use of agent-based monitoring approaches which areonly available if the agent is optionally installed in the cloudconsumers VMs.(iii)Complete Management Interface is exposed to the Cloud Con-sumer. This interface extends the basic management interfaceto enable the definition and customization of new physicaland virtual resources and the services to be monitored, even ifthey are external resources to the cloud computing infrastruc-ture. This interface canbe seenas a smart Monitoring Platform-as-a-Servicefor the cloud consumer.Fromthe previous scenarios, it can be shown that the keyarchitectural decision is to expose or not to expose a managementinterface to the cloud consumer. This decision will be lately usedto identify different monitoring scenarios.4.5. Taxonomy of monitoring architectures for cloud computingThis section describes the taxonomy of the monitoring archi-tectures previously introduced. In the top level of the hierarchy,shown in Fig. 8, it can be seen the Internal and External Architec-tures grouping the five architectures described in the previous sec-tions. Note that eacharchitecture offers different choices accordingto the user using them, being CP (Cloud Provider), CC (Cloud Con-sumer) and CP +CC (both of them). For each of these alternatives,it may be considered the possibility to monitor either physical ma-chines (PM), virtual machines (VM) or both of them (VM + PM).As a result of these alternatives, 12 architectures may be suitablecombinations. They have been numbered just to enable other re-searchers and practitioners to easily refer them trying to create acommon vocabulary when referring and comparing monitoring in-frastructures. Each architecture can be deployed with or withoutthe usage of a monitoring agent. In fact, for the case in which the22 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630Fig. 7. Architecture overview of the concentrated external monitoring architec-ture.architecture is being used by both CP + CC three different combi-nations canappear for the usage of agents: (Agent/Agent), (Agent/NoAgent), (No Agent/No Agent). The first term is referred to the usageof agents in the cloud provider whereas the second one is referredto cloud consumer.Moreover, only for the case of external architectures, for eachof theabovecombinationsit canbechosenaswell differentcombinations depending if the management interface is exposedto the cloud consumer or not. These combinations are representedwith discontinuous lines in Fig. 8. In total, there are 53 differentcombinations.5. Analysis of the different architectures for monitoringservicesThis sectiondescribes a set of realistic scenarios that canhappenwhen a monitoring service needs to be deployed. These scenariosare analysed against all the monitoring architectures presented inorder to provide the suitable alternatives for each scenario. Thefollowinglistofmonitoringscenariosareconsidered:(i)CloudProvider wants to monitor PMPM(p); (ii) Cloud Provider wantsto monitor VMsVM(p); (iii) Cloud Consumer wants to see themonitoring information of her VMsVM(c); (iv) Cloud Consumerwantstocustomizeandconfigurethemonitoringinformationgathered from her VMsMgmt(c).All the possible combinations of these 4 assumptions representthescenarios that areshownintheleft sideof Table1. Allthemonitoringarchitectures describedinthis paper arealsoshown in the central side of Table 1. Each monitoring architectureencloses in parentheses the final user of this architecture, i.e. p =cloud provider, c = cloud consumer. It is important to remark thatdespite the users indicated in parentheses,if the scenario doesnot consider such users, they will be ignored for obvious reasons.Forexample, inscenarios3and4, SpareEMA(c +p)isSpareEMA(c) because such scenarios do not require monitoring servicesbythecloudprovider. Finally, therightsideofTable1showssuitable combinations of hybrid architectures, where a monitoringarchitecture is used for the cloud provider whereas another one isused for the cloud consumer. These combinations can be seen as asecurity improvement due to the isolation done between differentusers. Only the combinations that make sense have been included.Table 1 shows whether the architecture is suitable or not for thedifferent scenarios. If the architecture is not suitable, it has associ-ated a number which represents the reason for which the architec-ture has been discarded. Table 2 shows the list of possible reasons.Analysing Table 1 fromup to down, 5 architectures are unveiled asthe best candidates for a significant number of scenarios:(i)ExtendedandAdaptive IMAfor bothcloudconsumer andprovider.(ii)Concentrated EMA for both consumer and provider.(iii)Extended and Adaptive IMAfor cloudprovider andConcentratedEMA for cloud consumer.(iv)Traditional IMA for cloud provider and Concentrated EMA forboth consumer and provider.(v)Traditional IMAis the best choice whenthe cloudprovider onlywants to monitor physical machines.There are other architectures that even being suitable for somescenarioshavebeendiscardedindetriment of the5previousoptions.The reasons that have motivated such discards are thefollowing ones:Extended IMA (c +p) only makes sense in scenarios where only asmall set of predefined metrics, gathered from the hypervisor, arerequired. Besides, Extended and Adaptive IMA is an extension ofExtended IMA and can be configured to work only in ExtendedIMA mode with similar security and performance capabilities.Fig. 8. Proposed taxonomy of the monitoring architectures for cloud computing.J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 23Table 1Analysis of the suitability of the architecture proposed for different monitoring scenarios.Table 2List of possible reasons for which architectures are discarded as suitable for a given scenario.ID Reason1 No need for monitoring services. Neither the cloud consumer nor the cloud provider needs monitoring services. (strong reason)2 A scenario in which the cloud consumer can access the management interface but not the monitoring interface does not make sense. It is not realistic. (strongreason)3 Hybrid architectures only make sense when both consumer and provider use monitoring services (strong reason)4 IMA implies the monitoring of physical interfaces otherwise it makes no sense. (strong reason)5 The architecture does not provide the required capabilities. (strong reason)6 The public exposition of the management interface to the cloud consumer requires the customization of the metrics to be gathered and this feature is not providedby this architecture. (strong reason)7 The architecture has been discarded due to the lack in the customization of the metrics gathered for the cloud provider. He requires a complete control over theresources. There is another architecture which provides this functionality under similar security and performance capabilities. (weak reason)8 The architecture has been discarded due to the fact that there is another architecture which provides the required capabilities with a minimal overhead (weakreason)9 SEMA implies MVM managed by the cloud consumer and this makes the architecture not suitable when the cloud provider uses the MVM for monitoring purposes.(strong reason)10 External Monitoring Architectures cannot monitor physical machines. (strong reason)11 There is other architecture with similar capabilities and performance and with better security features. (weak reason)SpareEMA(c +p)isonlysuitableundertheparticularcir-cumstance in which the cloud consumer has the capability tostop and start her own MVM. In any other case, Condensed EMA(c + p) is a similar architecture in terms of performance andcapabilities whereas the security is really improved.Traditional IMA(p) + Sparse EMA(c + p) is only suitable un-der the particular circumstance in which the cloud consumerhas the capability to stop and start her own MVM. TraditionalIMA(p) +Condensed EMA (c +p) is the preferred choice whichis a similar architecture with better security capabilities.If the reader discards all architectures non suitable for any of thescenarios analysed and also discards the list previously indicated,the result set is composed by the 5 architectures indicated pre-viously. They have been emphasized in bold in Table 1. Table 1provides also ID of the monitoring architecture available in thetaxonomy (see Fig. 8) for the architecture indicated as suitable in24 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630order to make easier to match scenario and monitoring architec-ture.Table 2 indicates the reasons for which the monitoring archi-tectures have been discarded as suitable for the scenario analysed.We have classified them into two different kinds of reasons: weakreasons and strong reasons. On the one hand, a strong reason in-dicates that the architecture is directly not suitable under any cir-cumstance for a given scenario. On the other hand, a weak reasonindicates that the architecture may be suitable under certain cir-cumstances but it has beendiscardeddue to optimizationpurposesin terms of security, performance, capabilities or overhead.Let us analyse the 5 best monitoring architectures in detail. Todo so, notice that these 5 monitoring architectures are composedonlyby3monitoringarchitecturessincetheother2onesarehybrid architectures composed by combinations of these 3 ones.Traditional IMA(p) is the simplest architecture andthus the mostefficient one when the cloud provides only need to monitorphysical machines. This is a traditional monitoring architectureand it will be only suitable when the cloud provider does notneed to monitor VMs.Extended and Adaptive IMA (c + p) reduces significantly to theminimum the resources used for monitoring purposes which isprobably its main added value. This reduction is achieved by thereal multi-tenancy support of the monitoring service togetherwith the fact that it is deployed in PMs and thus, it has to followa static approach for elasticity. This makes this architecture lesssuitable for large deployments. Probably, Extended and AdaptiveIMA (c +p) is the less secure option. Any security threat in themonitoring software may result in an escalation of privilegesenabling the attacker to see the complete status of the infras-tructure, i.e. the information of the cloud provider.Concentrated EMA (c +p) imposed an important overhead in theusage of VMs for monitoring purposes. However, ConcentratedEMA (c+p) provides a complete isolation of the monitoring ser-vices between different cloud consumers. So, it may be a betteroption than Extended and Adaptive IMA (c +p) fromthe point ofview of the security. Moreover, Concentrated EMA(c + p) pro-vides a really good scalability due to the automatic elasticity ofthe infrastructure according to the number of tenants. The dis-tribution of the workload among all the MVMs makes Concen-trated EMA(c + p) good in terms of performance. Despite thesecurity advantages of Concentrated EMA (c + p) with respectto Extended and Adaptive IMA (c +p), it requires that both cloudprovider and cloud consumer use the same monitoring service.This fact can be another source of security threats. So, in scenar-ios in which strong isolation between provider and consumer isrequired, it may be worthy to use hybrid architectures even de-spite the consequent overhead.Extended and Adaptive IMA(p) can be used for the cloud providerwhereas Concentrated EMA(c) can be used for the consumers.This combination leads to a high scalable and very secure archi-tecture. The usage of MVMs is efficiently used to provide a goodperformance.Finally, the combination Traditional IMA(p) +Concentrated EMA(c+p) is usedto enable the cloudprovider to monitor PMs usinga separated monitoring service.A summary of the previous rationale is shown in Table 3. Note thatsecurity, scalability and usage of resources may be inferred fromthe analysis of the monitoring architecture; however performancerequires an empirical evaluation. Thus, all the information shownin the following tables take into account the empirical validation ofthe performance of the architectures lately described in Section 7.Joining the information shown in Table 1 with the informationshowninTable3, wewouldliketoshowTable4withourrecommendation of the best architecture for monitoring servicesfor each of the scenarios identified. The recommendation has beendone according to two different criteria: security and performance.The aim is to provide to the academics and practitioners a suitablearchitecture to be deployed according to their particular scenario.6. ImplementationAlmost all the monitoring architectures described in this contri-bution have been designed, implemented and tested in a real cloudcomputing infrastructure. The implementation has been releasedto the community as an open-source project under Apache 2.0 li-cense called MonPaaS.6MonPaaS has been designed to be pluggedto RabbitMQ, the message queue of OpenStack. OpenStack is a well-known open source IaaS stack which follows exactly the archi-tecture depicted in Fig. 1. We have validated MonPaaS and thusexecuted all the monitoring architecture analysed using three dif-ferent versions and configurations of OpenStack in order to vali-date the suitability of the proposed software. In all the cases all themonitoring architectures were working perfectly:OpenStack Folsom 2012 2.4-stable with single-host networkingmode. Only nova components are used in this experiment. So,we used Nova-Network flat mode for this installation. Noticethat flat mode enables a direct communication from physicalmachines and VMs (but not vice versa) and thus all the IMAarchitectures can work perfectly with this networking mode. Inthis scenario all IMAarchitectures are installed in the ControllerNode.OpenStackHavana2013withsingle-host networkingmode.Only nova components are used in this experiment. So, we usedNova-Network flat mode for this installation. In this scenario allIMA architectures are installed in the Controller Node.OpenStack IceHouse 2014. In this case, we used a dedicated Net-working Node. This computer provided a complete installationof OpenStack Neutron configured with OpenVSwitch in flat net-working mode providing a complete Software Defined Networksolutionfor OpenStack. Inthis scenario all IMAarchitectures areinstalled in the Networking Node.MonPaaS uses Nagios as monitoring service which is a well-known,mature, and enterprise-class monitoring software. The architec-ture of Nagios matches with the one depicted in Fig. 2. Two Nagiosextensions are usedfor carrying out active andpassive agent-basedmonitoring, respectively, NRPE7andNSCA.8The extensionfor gath-ering the metrics fromthe hypervisor is Nagios-Virt.9Moreover, themanagement interface for Nagios is NConf,10a web-based inter-face for performing the configuration of Nagios. The scalable op-tion is implemented by means of DNX,11another Nagios extensionfor enabling the distribution of the monitoring platform. All thesesoftware pieces are already available and they have been used toimplementthedifferentmodulesdepictedinthearchitecturesavailable in Figs. 37.Apart fromOpenStack, Nagios and Nagios plugins, MonPaaS soft-wareis composedbydifferent components whichhavebeendesigned and implemented by us to provide support for all the ar-chitectures described in this contribution. These new componentsdeveloped are our main contribution to the community and theyare:(i)The monitoring module available in almost all the moni-toring architectures with all the different business logics de-scribed. This module is in charge among others of: Manag-ing different Nagios instances; Reconfiguring Nagios instancesdynamicallyagainsttopologychanges;Deployingdifferentmonitoring VMs for each cloud consumers; Synchronize se-curity information; etc.6MonPaaS is available at http://sourceforge.net/projects/monpaas/.7NRPE is available at http://exchange.nagios.org/directory/Addons/Monitoring-Agents/NRPE2D-Nagios-Remote-Plugin-Executor/details.8NSCA is available at http://exchange.nagios.org/directory/Addons/Passive-Checks/NSCA2D-Nagios-Service-Check-Acceptor/details.9Nagios-Virt is available at http://people.redhat.com/~rjones/nagios-virt/.10NConf is available at http://www.nconf.org/.11DNX is available at http://dnx.sourceforge.net/.J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 25Table 3Summary of the features exposed by the different architectures analysed.Architecture Performance Usage of resources Security ScalabilityExtended and Adaptive IMA(c +p) Very high Low Low MediumConcentrated EMA(c +p) High High Medium Very highTraditional IMA(p) +Concentrated EMA(c +p) High Very high Medium Very highExtended and Adaptive IMA(p) +Concentrated EMA(c) High Very high Very high Very highTraditional IMA(p) Very high Very low Low MediumTable 4Recommended architectures.ID Scenario Maximizing performance while minimizing the usage ofresourcesMaximizing security and isolationPM(p) VM(p) VM(c) Mgmt(c)1 N N N N N/A N/A2 N N N Y N/A N/A3 N N Y N Extended and Adaptive IMA(c) Concentrated EMA(c)4 N N Y Y Extended and Adaptive IMA (c) Concentrated EMA(c)5 N Y N N Extended and Adaptive IMA (p) Extended and Adaptive IMA (p)6 N Y N Y N/A N/A7 N Y Y N Extended and Adaptive IMA (c +p) Extended and Adaptive IMA (p) +Concentrated EMA(c)8 N Y Y Y Extended and Adaptive IMA (c +p) Extended and Adaptive IMA (p) +Concentrated EMA(c)9 Y N N N Traditional IMA(p) Traditional IMA(p)10 Y N N Y N/A N/A11 Y N Y N Extended and Adaptive IMA (c +p) Traditional IMA (p) +Concentrated EMA(c)12 Y N Y Y Extended and Adaptive IMA (c +p) Traditional IMA (p) +Concentrated EMA(c)13 Y Y N N Extended and Adaptive IMA (p) Extended and Adaptive IMA (p)14 Y Y N Y N/A N/A15 Y Y Y N Extended and Adaptive IMA (c +p) Extended and Adaptive IMA (p) +Concentrated EMA(c)16 Y Y Y Y Extended and Adaptive IMA (c +p) Extended and Adaptive IMA (p) +Concentrated EMA(c)(ii)The VM image used in the External Monitoring Architectures.This image is ready to be usedto populate the Nagiosmonitoring services for the cloud consumers.(iii)The web interfaces required by some architecture likeCondensed EMA in order to share security information withthe consumers required to access the monitoring graphicalinterfaces.(iv)The scripts for performing the installation, configuration anddeployment of the different architectures.As a result, MonPaaS is a ready-to-use software package that en-ables the installation of any of the architectures described in thiscontribution. MonPaaS istheevolutionof aprevioussoftwarecontributioncalledIaaSMon12whichprovidesthefollowingar-chitectures: Traditional IMA(p), Extended IMA(p) and Extended andAdaptive IMA(p),. This paper is the result of a significant improve-ment in MonPaaS enabling us to compare different architecturesempirically. Currently, MonPaaS provides support for the follow-ingarchitectures:Traditional IMA(p), ExtendedIMA(p), Extendedand Adaptive IMA(p), Concentrated EMA(c), Concentrated EMA (c +p), Concentrated EMA(p), Extended and Adaptive IMA(p) + Concen-trated EMA(c), Traditional IMA(p) + Concentrated EMA(c), ExtendedIMA(p) +Concentrated EMA(c).The installation of MonPaaS automatically installs Nagios 3.4.4,NConf 1.3.0, NRPE 2.13, and DNX 0.20.1 client and server. Moreover,the installation also creates the newMonPaaS cloud consumer anduploads the MVM image to be used if required by the architectureselected.Along the fast development cycle of OpenStack, we have identi-fied two different stages of OpenStack. Until Havana release therewas a traditional encapsulationdone inthe messages. After Havanarelease including Ice House andJuno, OpenStack is using Oslo, a newframework for exchanging messages between OpenStack compo-nents. We have released two versions of MonPaaS, one per pre-Havana releases and another one for post-Havana new releases.12IaaSMon is available at http://sourceforge.net/projects/iaasmon/.The main intention is to demonstrate that MonPaaS will work forfuture releases of OpenStack even with Neutron and advanced net-working capabilities. The only requirement for MonPaaS to work infuture releases is that there are no significant changes inthe way bywhich the messages are exchanged between components and alsoin the format of the current messages. Both are very reasonableassumptions at this development stage of OpenStack, now becom-ing mature enough to keep at least the foundation stable enoughto maintain the compatibility with MonPaaS. Concretely, MonPaaSisinspectingthefollowinginternalmessages:run_instance, ob-ject_action, terminate_instance, service_update, report_state. So thatif there are no significant changes in these messages MonPaaS willsupport future releases of OpenStack.It is worthy to mention the way in which we have addressed thecommunication between physical machines and virtual machineswhen using Neutron since it provides a complete isolation layerbetweenVMs andphysical machines whichdiffers withthelegacy direct communication enabled in Nova-Network.Neutronimplements the isolation between tenants and between physicalmachinesusingtheso-calledIPNamespaces. Thus, wehaveadapted Nagios and NConf to execute all the Nagios plug-ins togather metrics inside of the namespace associated to the VMbeingmonitored. For example, the plug-in to./check_ssh is executed inthe following way: ip netns exec $NET./check_ssh $IPVM where $NETis the IP namespace where the VM is running and $IPVM is the IPof such VM. These values are intercepted by MonPaaS and passesto NConf in order to configure Nagios properly. This mechanismenables Nagios to communicate withVMs evenwhenusingNeutron and ensure compatibilities with future releases of Neutronsince this is the standard way to interact with IP namespaces.7. Empirical comparison of the monitoring architecturesInorder to compare the different monitoring architectures, a setof experiments has been executed in a real cloud computing archi-tecture. This section has been divided into different subsections inorder to describe the cloud computing infrastructure over whichthe tests have been executed, the design of the test bed and theachieved empirical results.26 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 16307.1. Cloud computing infrastructureThe cloud computing infrastructure used to analyse empiricallythe performance of the monitoring architectures is composed byone computer acting as cloud controller and seven computers act-ingascomputenodes. Thecomputenodesare8Bull R424-E3blades, each one equipped by 2 processors Xeon ES-2650 2 GHz20 Mb/cache (8 cores each processor16 threads), 1 TB SATA III,32 GB @ 1600 MHz. The cloud controller is a SunFire X2100 nodeequipped by 1 Dual Core AMD Opteron 18 GHz 1 Mb/cache (2cores), 2 256 SATA II, 8 GB @ 1333 MHz. For the validation ofthe architecture with Neutron we used another networking nodewithexactly the same specifications of the controller node, i.e. Sun-Fire X2100. This hardware is wired with two gigabit networks, onefor management purposes and the other one for connectivity be-tween the VMs. Both networks may be used along the experimentsdepending on the architecture being used. The management net-work is connected by means of a switch D-Link DGS-3024 whereasthe virtual connectivity network is connected by means of a switchDELL PowerConnect 5448. The cloud controller has an extra gigabitinterface to connect to the Internet. We are using a clean installa-tion of Linux CentOS 6.3 as base OS for all the nodes. It is not theintention of this paper to evaluate the performance of the differentversions of OpenStack. Thus, it has been used only one OpenStackinstallation, concretely Folsom release for more than 2 months fulltime in order to achieve the empirical results provided in the fol-lowing subsections.7.2. Design of the test bedThis test bed has not been designed to stress the different archi-tectures in order to see how they work under stressing conditions.Incase the reader is interestedina complete anddeepevaluationofstressing and scalability of both Extended and Adaptive IMA(p) andExtended and Adaptive IMA(p) + Concentrated EMA(c), we encour-age the reader to read our previous contributions, Alcaraz et al. [2]andGutierrez et al. [3] inwhichwe have analysedthemindetail us-ing respectively, IaaSMon and a previous version of MonPaaS. Thistest bed fixes the number of VMs to be created by means of a con-stant rate of VM creation requests arrival to the cloud computinginfrastructure. Each experiment has been executed on a differentmonitoring architecture which is the only variable parameter be-tween experiments. These conditions enable us to compare the be-haviour of the different monitoring architectures under the sameconditions. The number of VMs has been fixed to 16 due to the factthat it is not pursued the saturation of the physical resources of theinfrastructure which may create interferences in the performanceresults of the architectures. This number is big enough to calculatesignificant average times whereas it is small enough to not pro-vide important overheads in the infrastructure. The VMs createdare clean installationof Linux Ubuntu 12.10 inQCOW2 image formatwith 512 MB, and 1 Core. The interval between arrivals of requestshas been empirical adjusted for this cloud computing infrastruc-ture. Concretely, the architecture fails in the creation of at least 1VMwhen this interval is less than 14 s, even without any monitor-ing architecture running. This scenario represents a really stressingscenario for our cloud computing infrastructure. So, this intervalhas been fixed to 17 s which may represent a normal working situ-ation which is suitable for comparing the architectures. Regardingthe number of services to be monitored, they are also fixed to 10services (HDD, Ping Response Time, File I/O, Free Mem, etc.) for eachphysical or virtual resources registered in the monitoring service.The fixed number of VMs can be created by a variable numberof cloud consumers. The experiments have been run under twodifferent scenarios, a scenario with 8 consumers, creating 2 VMseach one (referred in the figures as 8C-2VM) and another scenariowith 2 consumers creating 8 VMs each one (referred as 2C-8VM).This enables us to see how the elasticity factor of the EMAs affectsthe performance of the monitoring service. These scenarios createin reality 16 VMs + 8 MVMs (24 VMs) and 16 VM + 2 MVMs (18VMs), respectively. Note the difference in the total number of VMscreated.For EMAs, it is also important the sequential order in which theVMcreation requests are received in the infrastructure. Notice thatthe creation of the MVMs is done during the arrival of the first VMcreation request for a given consumer. Thus, two different scenar-ios have been identified. In the first scenario, referred henceforthas ALLVMS, all the VM creation requests for a given consumers aresent before starting sending the VM creation requests for the nextconsumer, and so on. In the second scenario, referred henceforthas ONEVM, the first VM is created for all the consumers and thenthe second VM is created for all the consumers, and so on. Noticethe important difference of these two scenarios because the regis-tering of the first VM inside of the monitoring service requires animportant delay due to the automatic elasticity performed in themonitoring architecture for dynamic elastic approach, raising thecreation of a new MVM.All the architectures recommended in Table 4 have been runagainst this test bed. The only exceptions are Extended and Adap-tive IMA(c + p) and Extended and Adaptive IMA(c). These archi-tectures are not supported in MonPaaS due to the simple fact thatNagios does not support natively multi-tenancy support which isrequired for providing services to the cloud consumer. However,they may expose similar performance results than that providedby EAIMA(p) because they are exactly the same monitoring archi-tecture. The results presented in the following subsection are theaverage over the execution of each test bed 5 times in order to getresults statistically relevant.7.3. Empirical resultsFig. 9 shows the experiment time for all the monitoringarchitectures analysed. The experiment time is defined as the timeelapsed between the arrival of the first VM creation request to thecloud computing infrastructure and the time in which the last VMis responding with the first ICMP Echo Reply, i.e. ping. Moreover, ithas been also included the experiment time for the case in whichthereisnoanymonitoringarchitecturerunninginthesystem.This time is used to determine the overhead due to the usage ofa monitoring architecture in the cloud computing infrastructure.It is worthy to remark that all the architectures expose similarexperiment times andthus fromtheblackboxpoint of view,all ofthemaresimilarintermsoftheoverheadforthecloudcomputinginfrastructure. Concretely, themaximumdifferencei.e. the worst case corresponds to 6 s for Extended and AdaptiveIMA(p) +Concentrated EMA(c) andthis is only anoverheadaround 2.8%. This is almost a negligible time. In fact, this time issignificantly reduced for the rest of architectures to less than 1.5%.For the analysis of performance associated to the monitoringarchitectures, it has been gathered the time until a VM respond tothe first ICMP Echo Request (VM Ping Response). It has also beengathered, if available, the time until a VMis registered in the Nagiosdeployed for the cloud provider, and also if available, the time untila VM is registered in the Nagios deployed in the cloud consumer.All these times are calculated with respect to the time in whichthe creation request for such VM arrived to the cloud computinginfrastructure. These times enable us to see the real performanceof themonitoringarchitectures. It isimportant toremarkthedifference between the two scenarios designed for the completeanalysis of EMAs. So, the first scenario is a scenario in which thereis not any MVMrunning. This scenario is referred henceforth ascold initialization. So, the results of the experiment in this scenarioJ.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 27Fig. 9. Overhead of the cloud computing infrastructure due to the inclusion of a monitoring architecture. The experiment assumes an ALLVMS order in the arrivals ofrequests. Moreover, it assumes a cold initialization.Fig. 10. Performance comparison of the different monitoring architectures in a cold initialization scenario.include as well the overheadimposeddue to the auto-scaling of themonitoring architecture. The other scenario, referred henceforthas hot state, has already running all the MVMs before of the startof the experiment and thus there is not any auto-scaling action.The former may be good to analyse the response against scalabilitychanges in the monitoring infrastructure whereas the latter maybebettertoreallymeasuretheperformanceofthemonitoringinfrastructure. So, Figs. 10 and 11 showthe average times gatheredfrom all the 16 VMs created in the cold initializationand the hotstate scenarios, respectively.As can be seen in Figs. 10 and 11, the VM Ping Response time isfluctuating between 21 and 35 s according to the monitoring archi-tecture used, the order in the arrivals of requests and the scenariochosen. These numbers can be used to compare the overhead timeimposed to the VMs in average due to the fact that the monitoringarchitecture is being executed. The baseline over which comparedis the VM Ping Response time where there is not any monitoringarchitecture (21.00 s in average).Also, it can be seen how there are some missing values in bothfigures.This is due to the fact that these values cannot be cal-culated for the given architecture. For example, for ConcentratedEMA(c) cannot be calculated the time related to the registration inthe cloud provider due to the fact that this architecture only pro-vides services to the cloud consumer.Thus, it only makes sense to compare architectures which arecomparable. Regarding IMAs analysed, i.e. Traditional IMA(p) andExtended and Adaptive IMA(p), all of them follow a constant be-haviour in all the experiments depicted in both figures. Moreover,IMAs perform better than EMAs in all the cases depicted in Fig. 10where the auto-scaling times are also included. Regarding EMAs, itcan be seen in Fig. 10 how EMAs expose a more significant over-head for the 8C-2VM scenario than for the 2C-8VM scenario. Thisis due to the simple fact of the number of MVMs created in eachscenario is different introducing an important delay in the perfor-mance of such architectures due to the imposed delay associatedto the auto-scaling time.Moreover, it can be seen how EMAs perform better in ONEVMscenarios than in ALLVMS scenarios. In ONEVM the MVMs are cre-ated at the very beginning of the experiment and then the over-head related to the instantiation of such MVMs is absorbed by theinfrastructure at the very beginning of the experiment so that thesescenarios shift quickly to a hot state. However, in ALLVMS the MVMsare created at periodical intervals along the experiment, thus theoverhead is available during almost all the experiments.In Fig. 11, it does not make sense to do a distinction between AL-LVMS and ONEVM scenarios because all the MVMs are already run-ning. In hot state scenario the performance of both IMA and EMAis almost similar for all the cases. In fact, it is worthy to mentionthat EMAs acts in some scenarios better than IMAs, probably dueto the innate scalability done over the monitoring tasks in EMAs.Concretely, it can be seen how Concentrated EMA(c) performs bet-ter than Extended and Adaptive IMA(p) in the Extended and AdaptiveIMA(p) +Concentrated EMA(c)entry available in Fig. 11. To under-stand these results, it is important to remark the fact that EMAshave associated as extra overheads the network latency and the vir-tualization layer and despite this issue, it provides better results. It28 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630Fig. 11. Performance comparison of the different monitoring architectures in a hot state scenario.is important to mention that if we compare the overhead timeswith respect to the VM Ping Response times for all the architec-tures, the monitoring overheadis around7%for the worst cases andaround 1% for the best cases. These are really good numbers due tothe fact that a VM which respond to the first ping needs to do a lotof extra tasks before of being ready for production. This time maybe much more than this 7% and thus the monitoring architecturemay be monitoring such VM even before it starts in production.8. Related worksSurprisingly, the number of real downloadable monitoring solu-tions for cloud computing infrastructures is really scarce. Tovarnakand Pitner [4] provide an architecture based on the installation ofsoftware agents inside of the VMs. This fact makes it only suitablewhen providing monitoring services to the cloud consumer but notwhen providing monitoring services only to the cloud provider.Rather than providing a complete architecture, they only focus onthe design of the software agent. This solution is similar to the soft-ware agents used in Nagios, i.e. NRPE and NSCA. This contributionis also similar to the one provided by Huang and Wang [5] whichis based on a software agent combining active and passive agent-based monitoring, i.e. the usage of both NRPE and NSCA. Dhingraet al. [6] and Ma et al. [7] provide analogous monitoring architec-ture based on the installation of software agents inside of the VMs,thus only suitable when providing services to the cloud consumer.This architecture gathers also monitoring metrics directly fromthehypervisor. Thus, this architecture frames perfectly in Extended andAdaptive IMA(c). Selvi and Govindarajan [8] provide also a moni-toring architecture based on software agents installed in the VMs.They install gmon which is the software agent of Ganglia, otherwell-known enterprise-class monitoring software. This fact makesthis monitoring architecture only suitable when cloud consumersare aware of the monitoring architecture. In fact, gmon has beendesigned for PMs and it does not fit well in cloud computing in-frastructures, especially due to the fact that it is based on multicastand it does not adapt well against of VMs destroy requests. So, thisarchitecture may not be suitable for cloud computing and in conse-quence it does not fit in any of the architectures proposed. Katsaoset al. [9] provide an architecture matching with Extended and Adap-tive IMA + Sparse EMA or with Extended and Adaptive IMA + Con-centrated EMA. This architecture monitors metrics directly fromthephysical infrastructure and also uses VMs to perform the moni-toring of information. The architecture requires the usage of soft-ware agents inside of the VMs. This information is provided to bothcloud provider and cloud consumer. However, this architecture isdesigned in a private cloud scenario and thus this is not any infor-mation about the users who are using the infrastructure. This is thereason for which we are not able to put the users in parentheses orto know if the EMA is Sparse EMA or Concentrated EMA. Andreoliniet al. [10] describe a distributed monitoring architecture designedspecifically for large information systems. This architecture usessoftware agents, thus it is only suitable when cloud consumers areaware of this architecture. This architecture uses multiple nodesfor doing the workloadbalancing of the monitoring tasks. So, it mayfit in Extended and Adaptive IMA(c) with the optional scalability fea-tures configured.VMDriver[11] is focused on implementing a novel transpar-ent monitoring approach in which agents are not required and theinformation is gathered by means of the hypervisor. This featureenables the cloud provider to get basic information about the re-sources. This contribution is similar to the nagios-virt plugin usedin Extended IMA to gather the metrics directly fromthe hypervisor.VMDriverfocus on providing more metrics from the hypervisor.Sandoval et al. [12] analyse a number of existent traditional moni-toring tools to determine what is the best choice to be adapted tothe monitoring of cloudcomputing infrastructures. As a result, theyindicate Nagios as the best alternative. In consonance with theseauthors, Nagios has been used for all the deployments done in thiscontribution. Aceto et al. [13] have recently provided a completesurvey about monitoring architectures for cloud computing. Thissurvey describes a lot of commercial solutions like Amazon Cloud-Watch,13AzureWatch14and CloudKick15now renamed as RackSpaceCloud Monitoring,16to name a few.However,these commercialvendors have not published any information about the monitor-ing architecture describing how they implement the monitoringsolutions internally. However, we could describe their capabilities.So, AmazonCloudWatchdoes provide a basic management interfaceenabling not only a predefined and closed set of metrics but alsothe configuration of customized metrics to the cloud consumer.AzureWatch is based on agent-based monitoring and it is only fo-cused on the cloud consumer. RackSpace Cloud Monitoring enablesthe complete customization of the monitoring service includingeven the possibility of monitoring external resources. In all thesecases, a good candidate for implementing these services in produc-tion is CEMA(c) or CEMA(c + p) which foster security and isola-tion over any other feature. Aceto et al. [13] also describe severalopen-source and commercial downloadable monitoring architec-tures like OpenNebula Monitoring Subsystem [14], CloudStack Zen-Pack,17PCMONS [15], Sensu18and Dargos [16], among others. TheOpenNebula monitoring software is very limited providing only in-formation about the PMs (and not about the virtual ones). Thus,it matches perfectly with Traditional IMA(p). CloudStack ZenPackand OpenStack ZenPack are plugins for Zenoss other well-knownenterprise-class monitoring software. They perform the monitor-ing of CloudStack and OpenStack cloud computing infrastructures,13Amazon EC2 Cloud Watch is accessible athttp://aws.amazon.com/es/cloudwatch/.14Azure Watch is accessible at http://www.paraleap.com/azurewatch.15RackSpace CloudKick is accessible at http://www.rackspace.com/cloudkick/.16RackSpace Cloud Monitoring is accessible at http://www.rackspace.com/cloud/monitoring/.17CloudStack ZenPack is available at http://wiki.zenoss.org/ZenPack:CloudStack.18Sonian Sensu is available at http://www.sonian.com/about/sensu/.J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630 29Table 5Differentiation between the current state-of-the-art and all the monitoring architectures proposed.Optional usage of software agent(required for fitting in cloudscenarios)ArchitecturepublishedMetric customization(provider/consumer)Monitoring ofvirtualinfrastructure(provider/consumer)Tovarnak and Pitner [4] No Yes No/No Yes/NoHuang and Wang [5] No Yes Yes/No Yes/NoDhingra et al. [6] No Yes Yes/No Yes/NoMa et al. [7] No Yes Yes/No Yes/NoSelvi and Govindarajan [8] No Yes Yes/No Yes/NoKatsaos et al. [9] No Yes Yes/No Yes/NoAndreolini et al. [10] No Yes Yes/No Yes/NoVMDriver [11] Yes N/A No/No Yes/YesAmazon CloudWatch Yes No ?/No ?/YesAzureWatch Yes No ?/No ?/YesCloudKick Yes No ?/No ?/YesOpenNebula Monitoring Subsystem Yes No Yes/No No/NoCloudStack ZenPack Yes No Yes/No No/NoOpenStack ZenPack Yes No Yes/No No/NoSensu Yes No No/No No/NoPCMONS [15] No Yes No NoDargos[16] Yes Yes Yes/No Yes/YesGMonE [17] No Yes No NoHazmi et al. [18] No Yes No NoKoenig et al. [19] No Yes Yes/No Yes/YesOpenStack MONaaS Yes Yes No/Yes No/YesMonPaaS IMA Yes Yes (here) Yes/No Yes/NoMonPaaS Extended IMA Yes Yes (here) Yes/Yes Yes/YesMonPaaS Extended and Adaptive IMA Yes Yes (here) Yes/Yes Yes/YesMonPaaS Sparse EMA Yes Yes (here) Yes/Yes Yes/YesMonPaaS Concentrated EMA Yes Yes (here) Yes/Yes Yes/Yesrespectively.The CloudStack plugin retrieves data about the IPs,Memory, CPU and HDD, providing in all the cases the total, avail-able and used amounts whereas the OpenStack plugin retrievesinformation about the Exceptions,VM Flavours,OS Images,Secu-rity Groups and Servers available in the infrastructure. These plug-ins do not perform the monitoring of VMs. They match with IMAin both cases. Sensu is a monitoring architecture focused on in-specting the communication bus used by the cloud computing in-frastructure. Sensu follows the approach presented in many of thearchitecture described herein for which a monitoring module isattached to the message queue. So, rather than being a concretearchitecture, Sensu may be seen as this architectural componentavailable in all of them. PCMONS [15] provide a complete monitor-ing service based on Nagios. PCMONS is designed only for privateclouds and it is based on the usage of agents. PCMONS does not useany VM for monitoring purposes and it also gathers informationfrom the PMs. Thus, it matches with Extended and Adaptive IMA.GMonE [17] provides monitoring services to both cloud providerand cloud consumer. GMonE is based on the explicit use of agentsinside of the VMs, thus this architecture is suitable in scenariosin which both users want to get monitoring services. Moreover,two different monitoring services are provided to isolate the con-sumers fromcloud provider. These architectures for both users aresimilar with the only difference that the architecture for the cloudconsumer does not gather any information about the PMs. Thus,this architecture may fit with the hybrid Extended and AdaptiveIMA(p) + Extended and Adaptive IMA(c). This combination makessense, fosteringtheisolationbetweenconsumersandprovider.Concretely, this combination has not been included in this contri-bution for the simple fact that it is very similar to Extended andAdaptive IMA(p) +Concentrated EMA(c) following a static approachfor the elasticity of the infrastructure inwhichonly 1 VMis used forproviding monitoring services to all the consumers. Dargos [16]19alsoprovidesmonitoringservicestobothcloudprovidersandconsumers. It performs transparent monitoring for gathering all19Dargos is available at http://tl.ugr.es/dargos/.the information from the VMs. Dargos does not provide any man-agement interface for the customization of the metrics gatheredneither for the cloud consumer nor for the cloud provider. In fact,it uses the same monitoring architecture to provide the service toboth users. Thus, it may fit well with Extended IMA (c +p). Al-Hazmiet al. [18] provide a monitoring architecture focused on scenarioswhere different cloud providers are federated. This architecture isbased on the explicit use of agents and thus it is only suitable whenthe monitoring services are also provided to the cloud consumer.This architecture also gathers information about the physical re-sources so that it fit in Extended and Adaptive IMA (c +p).Koenig et al. [19] provides a radically new approach focused onthe elasticity of the monitoring platform. This architecture matcheswith Extended and Adaptive IMA in a private cloud scenario. Thisapproach is based on the complexity of the query inserted to ex-press the monitoring information wanted. The complexity of thequery is used to determine the elastic factor of the monitoring ser-vice. Thus, different VMs are used to perform the load balancingof such monitoring query rather than using the number of cloudconsumers or of VMs available. Ceilometer20is the metering toolprovided by OpenStack from Havana release.It is very efficientcollecting all the information about the status of OpenStack andthe assignment of resources done in OpenStack in order to latelyusethisinformationformeteringpurposes. Fromanarchitec-tural point of view, it could be similar to an Extended and Adap-tive IMA infrastructure. However, Ceilometer is not a traditionalmonitoring solution providing real-time key performance metricsabout the different physical and virtual resources available in thecloud infrastructure. It is just a monitoring solution for providinginformation gathered by the different components of OpenStackin order to be lately used for metering and billing purposes.Infact, recently there is a new OpenStack project entitled MONaaS2120CeiloMeter is available at https://wiki.openstack.org/wiki/Ceilometer.21MONaaS is available at https://wiki.openstack.org/wiki/MONaaS.30 J.M. Alcaraz Calero, J. Gutirrez Aguado / Future Generation Computer Systems 47 (2015) 1630Monitoring-as-a-Service which tries to provide a solution like ourproposed Sparse EMA architecture.Table 5 shows a comparative analysis of all the monitoring solu-tions designed for cloud computing analysed in this section againstall the novel monitoring architectures proposed in this contribu-tion. This table can help the reader to really see the differentiatingfeatures of our contribution with respect to the current state of theart.9. ConclusionsAsignificant number of architectures for monitoring cloudcom-puting infrastructures have been provided in this contribution. Anoveltaxonomyformonitoringarchitecturesincloudcomput-ing has been provided. These architectures have been analysed bymeans of an exhaustive comparative analysis.This analysis hasled to a recommendation of the best monitoring alternatives fora significant number of realistic scenarios. Concretely, it is recom-mended the usage Extended and Adaptive IMA (c + p) for scenar-ios in which is pursued the maximization of performance and theusage of Extended and Adaptive IMA(p) + Concentrated EMA(c) forscenarios in which is pursued the maximization of security andisolation. Acompleteempirical evaluationhasbeenalsodoneagainst a real cloud computing infrastructure in order to validateempirically the performance of the architectures by means of theexecution of more than 1000 VMs. These results have shown howboth IMAs and EMAs provide similar performance capabilities in ahot state. However, EMAs are much slower when they have to auto-scale the monitoring architecture. On the other hand, EMAs offerbetter security and scalabilities alternatives in comparison withIMAs.As future work, we want to shift from Nagios to another moni-toring tool which provides multi-tenancy support. We would alsolike to explore newmonitoring architectures with support for per-forming an auto-scaling of the architecture when overhead.AcknowledgementsThis research work has partially been supported and funded bythe grant A case on Cloud-based Mobile Network InfrastructureSharing in UAE funded by Zayed University, United Arab Emiratesand by the grant Research on Key Technologies of Energy-savingResource Integration and Task Scheduling for Green Cloud Com-puting with reference 61472192 sponsored by the National Natu-ral Science Foundation of China. The project has also been activelysupported by Wuxi Chigoo Interactive Technology, Co. Ltd (China)under the research agreement Service-Oriented Cloud Informa-tion Centre for Airport Services. Finally, authors wish also to ac-knowledge the help provided by Enrique Chirivella Perez duringthe deployment of OpenStack IceHouse used to validate the archi-tecture proposed herein.References[1] P. Mell, T. Grance, The NIST Definitionof Cloud Computing, REF 800-145. Available at: http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf.[2] J.M.Alcaraz Calero,J.Gutierrez Aguado,MonPaaS: an adaptive monitoringplatform as a service for cloud computing infrastructures and services, IEEETrans. Serv. Comput. (2015). http://dx.doi.org/10.1109/TSC.2014.2302810. inpress.[3] J. Gutierrez Aguado, J.M. Alcaraz Calero, Wladimiro Diaz Villanueva, IaaSMon:frameworkfor monitoringcloudcomputingdatacenters, J. GridComput.(2014) submitted for publication.[4] D. Tovarnak, T. Pitner, Towards multi-tenant and interoperable monitoringof virtual machinesincloud, in: de201214thInternational SymposiumonSymbolicandNumericAlgorithmsforScientificComputing, Timisoara,Romania, 2012.[5] H. Huang, L. Wang, P&P: a combined push-pull model for resource monitoringin cloud computing environment, in: de 2010 IEEE 3rd InternationalConference on Cloud Computing, Miami, USA, 2010.[6] M. Dhingra, J. Lakshmi, S.K. Nandy, Resource usage monitoring in clouds, in: deACM/IEEE 13th International Conference on Grid Computing, Beijing, China,2012.[7] K. Ma, R. Su, A. Abraham, Toward a lightweight framework for monitoringpublic clouds, in: de FourthInternational Conference onComputationalAspects of Social Networks, CASoN, San Carlos, Brazil, 2012.[8] T. Selvi, K. Govindarajan, Cloud monitoring and discovery service (CMDS) forIaaS resources, de Cloud monitoring and discovery service (CMDS) for IaaSresources, Chrometet, Chennai, 2011.[9] G. Katsaros, G. Kousiouris, S.V. Gogouvitis, D. Kyriazis, A. Menychtas,T. Varvarigou, A self-adaptive hierarchical monitoring mechanism for clouds,J. Syst. Softw. 85 (2012) 10291041.[10] M. Andreolini, M. Colajanni, M. Piet, Ascalablearchitectureforreal-timemonitoringof large informationsystems, in: de SecondSymposiumonNetwork Cloud Computing and Applications, London, UK, 2012.[11] G. Xiang, H. Jin, D. Zou, X. Zhang, VMDriver: adriver-basedmonitoringmechanism for virtualization, in: de 29th IEEE International Symposium onReliable Distributed Systems, Delhi, India, 2010.[12] Y.Sandoval,G.Gallizo,M.Curiel,Evaluation of monitoring tools for cloudcomputing, in: de XXXVIII Conferencia Latinoamericana en Informatica, CLEI,Medellin, Colombia, 2012.[13] G. Aceto, A. Botta, W. deDonato, A. Pescap, Cloudmonitoring:asurvey,Comput. Netw. 57 (2013) 20932115.[14] D. Milojii, I. Llorente, R.S. Montero, OpenNebula: a cloud management tool,IEEE Internet Comput. 15 (2) (2011) 1114.[15] S. Aparecida de Chaves, R. Brundo Uriarte, C. Becker Westphall, Toward anarchitecture for monitoring private clouds, IEEE Commun. Mag. 49 (12) (2009)130137.[16] J. Povedano-Molina, J.M. Lopez-Vega, J.M. Lopez-Soler, A. Corradi, L. Foschini,DARGOS: a highly adaptable and scalable monitoring architecture for multi-tenant clouds, Future Gener. Comput. Syst. 29 (8) (2013) 20412056.[17] J. Montesa, A. Snchez, B. Memishic, M.S. Prez, G. Antoniu, GMonE: a completeapproachtocloudmonitoring, FutureGener. Comput. Syst. 29(8)(2013)20262040.[18] Y. Al-Hazmi, K. Campowsky, M. Thomas, A monitoring system for federatedclouds, in: de 1st International Conference on Cloud Networking, Paris, France,2012.[19] B. Koenig, J.M. Alcaraz Calero, J. Kirchnick, Elastic monitoring framework forcloud infrastructures, IET Commun. 6 (10) (2011) 13061315.Jose M. Alcaraz Calero is a Lecturer on Networks at theUniversity of the West of Scotland, United Kingdom. Heholds an M.Sc. and a Ph.D. in computer engineering fromthe University of Murcia. He has published more than 70contributions in international journals and conferences.He is involved in more than 20 different editorship andsteeringcommitteeactivitiesforinternational journalsand conferences.His research interests include securityand management in distributed and heterogeneous sys-tems, cloud computing and big data.Juan Gutirrez Aguado obtained his Ph.D.degree fromthe University of Valencia (Spain) where he is anAssistant Professor. He has coauthored some journal andconferences papers mainly in the field of Computer VisionandImageProcessing. HiscurrentresearchfocusesonDistributed and Cloud Computing.