Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Managing the Virtualized Enterprise The Importance of Consolidation, Correlation, and Detection – Enterprise Security Series
White Paper
White Paper
Managing The Virtualized Enterprise
Abstract The benefits of employing virtualization in the corporate data center are compelling – lower operating
costs, better resource utilization, increased availability of critical infrastructure to name just a few. It is an
apparent “no brainer’ which explains why so many organizations are jumping on the bandwagon. Industry
analysts estimate that between 60 and 80 percent of IT departments are actively working on server
consolidation projects using virtualization. But what are the challenges for operations and security staff
when it comes to management and ensuring the security of the new virtual enterprise? With new
technology, complexity and invariably new management challenges generally follow.
Over the last 18 months, Prism Microsystems, a leading security information and event management
(SIEM) vendor, working closely with a set of early adopter customers and prospects, has been working on
extending the capability of EventTracker to provide deep support for virtualization, enabling our
customers to get the same level of security for the virtualized enterprise as they have for their non-
virtualized enterprise. This White Paper examines the technology and management challenges that result
from virtualization, and how EventTracker addresses them.
The information contained in this document represents the current view of EventTracker on the
issues discussed as of the date of publication. Because EventTracker must respond to changing
market conditions, it should not be interpreted to be a commitment on the part of EventTracker,
and EventTracker cannot guarantee the accuracy of any information presented after the date of
publication.
This document is for informational purposes only. EventTracker MAKES NO WARRANTIES, EXPRESS
OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the
rights under copyright, this paper may be freely distributed without permission from EventTracker,
if its content is unaltered, nothing is added to the content and credit to EventTracker is provided.
EventTracker may have patents, patent applications, trademarks, copyrights, or other intellectual
property rights covering subject matter in this document. Except as expressly provided in any
written license agreement from EventTracker, the furnishing of this document does not give you
any license to these patents, trademarks, copyrights, or other intellectual property.
The example companies, organizations, products, people and events depicted herein are fictitious.
No association with any real company, organization, product, person or event is intended or should
be inferred.
© 2017 EventTracker Security LLC. All rights reserved. The names of actual companies and products
mentioned herein may be the trademarks of their respective owners.
White Paper
Managing The Virtualized Enterprise
New Complexity, New Challenges
The introduction of virtualization has changed the playing field when it comes to managing the security
and operations of the corporate enterprise.
Until virtualization there had always existed a fairly close relationship between the hardware and software
layers of a computing infrastructure. A server machine was typically a “box’, .i.e. a self-contained machine
consisting of a chassis, CPU’s, an operating system (UNIX, Linux or Windows typically) with some
applications installed and some disk spaced mapped. Network equipment were other “boxes’ that
managed the network traffic between servers and desktops. Once provisioned the server and the network
equipment became fairly static and straightforward to manage.
Over the last ten years this relationship at least on the server side has been complicated by the move to
specialized storage devices and rack and blade systems. Despite this growth in complexity, it was still
relatively manageable overall. To provide visibility into the workings of the server you monitored the
Operating System and by doing this you got limited, but adequate, visibility into the underlying hardware
layer, as well as the application layer. The network produced management information that provided
visibility into the information flowing between machines. From a management standpoint you had a set
of trusted users or administrators that were responsible for the machines, a different network team and
in bigger companies, occasionally some storage specialists and a security group. Everyone had distinct and
fairly well defined duties. It was not perfect, but the complexity could be managed.
Virtualization
With the mainstream arrival of virtualization, the close relationship between the physical and the software
layer is now completely severed. Now at best there is a loose coupling of the OS instance with the platform
it runs on and there is an entirely new, virtualized layer that separates the two as well. The close
relationship of OS to physical infrastructure has been replaced by the virtualization layer – the hypervisors
and management tools that manage the setup and deployment of the virtual machines. The host OS still
has control over the application layer, but the hardware is allocated through the VM management layer.
The Hypervisors also support network communication between virtual machines which side-steps the
classic network group that traditionally controlled traffic on the wire. Further complicating the equation
is that with virtual networking, network traffic sometimes never gets onto the wire which renders most
network security tools ineffective.
Systems Management
Many organizations are also deploying systems management applications in the form of Dell OpenManage
or HP Insight Manager to manage large scale server farms. These have become important as enterprises
move to “rack and stack’, where virtual servers are often dependent on shared infrastructure to operate.
With potentially many servers dependant on shared infrastructure it becomes important to monitor the
White Paper
Managing The Virtualized Enterprise
hardware state, as a small hardware failure can have a catastrophic impact on service. These management
applications can help manage at the hardware layer and at the OS or software layer, but typically do not
provide the richness of a specialty virtualization management product and most experts in the field
caution against using such solutions for the virtual layer.
The addition of this new virtualization layer compounds the complexity of management and monitoring.
There are different and sometimes more critical points of failure, and there are entirely new systems and
applications that need to be monitored. Prior to racks and virtualization, if a machine failed it would take
out a couple (at most) of critical applications. Today if a rack fails it might take out 10 physical servers. A
single physical server could be running 8-10 guest Operating Systems, with each of those running critical
applications and services, so even a single physical server machine failure can be catastrophic. In addition,
if the management application for the virtual infrastructure is successfully attacked or hijacked there is
potential for operational carnage. Server sprawl was messy and inefficient to manage, with lots of points
of failure, but there were few points of failure that could literally take an entire company off-line. In the
new Virtualized enterprise there are more, different and even more critical services to monitor.
Organizational Change
For separation of duties and operational efficiency in many organizations that have adopted virtualization
there is now an admin team that is responsible for management of the virtual layer – the provisioning and
creation of virtual machines. But the clear separation of duties that existed pre-virtualization has blurred
– the virtual team might, for instance, have to worry about networks if they are using virtualization for
communication between guest machines.
Imagine the simplest of examples from this new paradigm – prior to virtualization you turned a machine
on and an OS typically booted up. Done. Now you switch on a machine and the virtualization layer takes
over. It then manages the creation of potentially multiple virtual machines running different Operating
Systems with distinct network configurations. Virtual Machines start and stop, they can move dynamically
from physical machine to physical machine. Even the disk space and often the network are mapped in the
virtual world.
This discussion is not to imply that virtualization is inherently insecure in any way, it is simply changing the
way businesses need to operate and think about their security. There are new and different critical
applications and infrastructure that need to be monitored and brand new threats – and consequently the
approaches to monitoring and prevention must adapt.
SIEM in the Virtual Enterprise
Security Information and Event Management solutions have three real purposes in life. First to help
prevent attacks and security breaches from either internal or external bad actors. With virtualization the
attack service changes. Before virtualization you could attack at the hardware layer or hijack a machine
during the boot process. The other option was to attack at the OS/software layer. Now a hacker can attack
White Paper
Managing The Virtualized Enterprise
the VM layer as well. Once in the VM layer, the hacker can reconfigure machines and potentially traverse
into a guest OS. Since VMs can all be running on the same physical machine the hacker can then traverse
from machine to machine in the host without the network traffic ever being visible on the wire.
The second purpose of SIEM solutions is to help companies meet compliance by tracking user and
administrator activity and access. With virtualization there is an entirely new set of power users that are
acting in the enterprise - administrators that manage the virtual layer. They need to be audited as well.
One of the best ways to secure a VM infrastructure is by enforcing strict separation of duties – for example
the persons responsible for the virtual infrastructure (provisioning etc.) and the virtual machine instances
themselves (OS and applications) should not be the same if at all possible, and the network, server and
virtual management teams should have policy-based segregation of duties.
Finally, the third purpose is to ensure smooth continuing operations. Having a consolidated view of all the
events happening in the enterprise increases the overall availability of IT service. In an increasingly
complex infrastructure automating these tasks with a SIEM solution is the only way to detect the small
signs of impending problems in advance.
In order to ensure security and smooth operations, enterprise visibility must be maintained and collection
of logs from all distinct layers must be performed. In the next pages we look at several important
technologies that need to be monitored as they have become important layers of the system
infrastructure in a virtualized enterprise, and offer a hacker new attack vectors. In order to keep this
manageable, we have focused on the “machines’ – the racks, the servers, the storage devices and the
software that controls them. We will look at the types of events generated by Dell OpenManage and both
VMware and Microsoft Hyper-V events. The ability to manage the network, application and OS elements
are an assumption, and are already supported with existing SIEM solutions.
White Paper
Managing The Virtualized Enterprise
Unified Server Management
Unified Server Management offerings or what HP refers to as “Unified Infrastructure Management’ are a
series of management products that are designed to manage the entire IT infrastructure – from the
Chassis to the Network Attached Storage and from the OS level down to the bare-bones hypervisor. These
applications can collect IPMI information that provides rich, low level information on the state of the
hardware, and as they are provided by the server vendors (Dell OpenManage, HP Insight Manager, IBM),
they provide a great deal of information on the state of the SAN devices if a company has standardized on
a single vendor for both storage and systems. These systems also provide a rich set of commands to
configure, patch and operate the hardware, OS and storage of the infrastructure.
With Blades and Racks and shared resource pools of hardware components it is advisable to collect and
monitor logs coming from these applications. In large scale virtualized enterprises these applications are
often used side by side with a Vcenter.
Pre-OS Events
Once, it was safe to assume that when you powered a machine off, it became unreachable. Now with a
combination of UPS, networks and IPMI, even machines that are powered off are still potentially
accessible.
The Intelligent Platform Management Interface (IPMI) standard has existed since 1998 with the majority
of the major chip set vendors such as Intel and AMD, and Server Vendors such as Dell and HP, supporting
the Standard. IPMI runs on the Baseboard Management Controller and allows Administrators to remotely
manage a system before an OS is even booted or the power switched on. This powerful combination of
capabilities enables an IT organization to substantially reduce the cost of server maintenance, however it
also opens a potential path for hackers to get in and cause damage. In IPMI 2.0, for example, a person
remotely accessing the interface is able to discover all the commands available to them and perform
inventory on the underlying platform, as well as change hardware settings on the machine. In addition,
once the OS has been booted, the BMC and IPMI can continue to run if provided a power source enabling
another entry point into the device, outside of the operating system.
With this capability, monitoring access through IPMI is a must. Unfortunately a single standard for IPMI
trap generation does not exist and the platform vendors have integrated the IPMI functionality into the
Server Management Systems. Information can be generated from various sources including the BIOS, OS
Bootstrap Loader, Network Interface Card, System Alert ASIC, System Management Micro-controller,
System Management Software and the Alert Proxy Software. A great deal of useful operational data with
regards to the state of the system hardware, memory and disks becomes available. In addition important
security and audit events are generated for IPMI user-logon failures, system reconfiguration or the turning
off of logging in IPMI.
White Paper
Managing The Virtualized Enterprise
OpenManage Events
Array Disk Events
2106 Smart FPT (predictive failure) exceeded. The disk is likely to fail in the near future.
2107 Smart configuration change. The disk is likely to fail in the near future.
2108 Smart warning. The disk is likely to fail in the near future.
2109 SMART warning temperature. The disk is likely to fail in the near future.
2110 SMART warning degraded. The disk is likely to fail in the near future.
2111 Failure prediction threshold exceeded due to test - No action needed.
2094 Predictive Failure reported. The disk is likely to fail in the near future.
2095 SCSI sense data. A SCSI device experienced an error, but may have recovered.
Automatic System Recovery
1006 Automatic System Recovery (ASR) action was performed. The Operating System was hung.
Battery Sensor Events
1700 Battery sensor has failed.
1701 Battery sensor value unknown.
1702 Battery sensor returned to a normal value.
1703 Battery sensor detected a warning value.
1704 Battery sensor detected a failure value.
1705 Battery sensor detected a non-recoverable value.
2104 Controller battery is reconditioning.
2105 Controller battery recondition is completed.
2169 The controller battery needs to be replaced.
2170 The controller battery charge level is normal.
White Paper
Managing The Virtualized Enterprise
Battery Sensor Events
2171 The controller battery temperature is above normal.
2172 The controller battery temperature is normal.
2174 The controller battery has been removed.
2175 The controller battery has been replaced.
2176 The controller battery Learn cycle has started.
2177 The controller battery Learn cycle has completed.
2178 The controller battery Learn cycle has timed out.
2179 The controller battery Learn cycle has been postponed.
2180 The controller battery learn cycle will start in %1 days.
2181 The controller battery Learn cycle will start in %1 hours.
2215 Battery charge process interrupted
2216 The battery learn mode has changed to auto.
2217 The battery learn mode has changed to warn.
BIOS Update Schedule Events
1002 A system BIOS update has been scheduled for the next reboot.
1003 A previously scheduled system BIOS update has been canceled.
Chassis Intrusion
1250 Chassis intrusion sensor has failed
1251 Chassis intrusion sensor value unknown
1252 Chassis intrusion returned to normal
1253 Chassis intrusion in progress
1254 Chassis intrusion detected
1255 Chassis intrusion sensor detected a non-recoverable value
White Paper
Managing The Virtualized Enterprise
Chassis Management Controller (CMC) Events
2000 CMC generated a test trap
2002 CMC reported a return-to-normal or informational
2003 CMC reported a warning
2004 CMC reported a critical event
2005 CMC reported a non-recoverable event
Cooling Device Events
1100 Fan sensor has failed
1101 Fan sensor value unknown
1102 Fan sensor returned to a normal value
1103 Fan sensor detected a warning value
1104 Fan sensor detected a failure value
1105 Fan sensor detected a non-recoverable value
Current Sensor Events
1200 Current sensor has failed
1201 Current sensor value unknown
1202 Current sensor returned to a normal value
1203 Current sensor detected a warning value
1204 Current sensor detected a failure value
1205 Current sensor detected a non-recoverable value
White Paper
Managing The Virtualized Enterprise
Disk Error
2273 A block on the physical disk has been punctured by the controller
2306 Bad block table is 80% full.
2307 Bad block table is full. Unable to log block
2331 A bad disk block has been reassigned.
2340 The BGI completed with uncorrectable errors.
2349 A bad disk block could not be reassigned during a write operation.
Enclosure Events
2138 Enclosure alarm enabled
2139 Enclosure alarm disabled
2151 Asset tag changed
2152 Asset name changed
2153 Service tag changed
2162 Communication with enclosure regained
2173 Unsupported configuration n detected. The SCSI rate of the enclosure management modules (EMMs) is not the same.
2190 The controller has detected a hot plugged enclosure.
2191 Multiple enclosures are attached to the controller. Unsupported configuration.
Firmware
2120 Enclosure firmware mismatch
2128 BGI cancelled
2131 Firmware version mismatch
2165 The RAID controller firmware and driver validation was not performed. The configuration file cannot be opened.
2166 The RAID controller firmware and driver validation was not performed. The configuration file is out of date or corrupted.
2311 The firmware on the EMMs is not the same version.
White Paper
Managing The Virtualized Enterprise
Hardware Log Sensor
1550 Log monitoring has been disabled
1551 Log status is unknown
1552 Log size is no longer near or at capacity
1553 Log size is near or at capacity
1554 Log size is full
1555 Log sensor has failed
Log Backup Clear
0000 Log was cleared
0001 Log backup created
Memory Device
1403 Memory device status warning. Correction rate exceeded acceptable value.
1404 Memory device status warning. A memory device correction rate exceeded an acceptable value, a memory spare bank was activated, or a multibit ECC error occurred.
Physical Disk
2049 Physical disk removed
2050 Physical disk offline
2051 Physical disk degraded
2052 Physical disk inserted
2060 Copy of data started on physical disk %1 from physical disk %2.
2062 Physical disk initialization started
2065 Physical disk rebuilds started
2074 Physical disk rebuilds cancelled
White Paper
Managing The Virtualized Enterprise
Physical Disk
2075 Copy of data completed on physical disk %2 from physical disk %1
2080 Physical disk initializes failed
2083 Physical disk rebuilds failed
2087 Copy of data resumed from physical disk %2 to physical disk %1
2089 Physical disk initializes completed
2092 Physical disk rebuilds completed
2141 Physical disk dead segments recovered
2146 Bad block replacement error. A portion of a physical disk is damaged.
2147 Bad block sense error. A portion of a physical disk is damaged.
2148 Bad block medium error. A portion of a physical disk is damaged.
2149 Bad block extended sense error. A portion of a physical disk is damaged.
2150 Bad block extended medium error. A portion of a physical disk is damaged.
2158 Physical disk online
2195 Dedicated hot spare assigned. Physical disk %1
2196 Dedicated hot spare unassigned. Physical disk %1
2198 The physical disk is too small to be used for Replace member operation
2211 The physical disk is not supported.
2183 Replace member operation failed on physical disk %1
2184 Replace member operation cancelled on physical disk
2185 Replace member operation stopped for rebuild of hot spare on physical disk
1650 Unknown device plug event type received.
1651 Device added to system
1652 Device removed from system
1653 Device configuration error detected
1500 AC power cord sensor has failed
1501 AC power cord is not being monitored
White Paper
Managing The Virtualized Enterprise
Physical Disk
1502 AC power has been restored
1503 AC power has been lost
1504 AC power has been lost
1505 AC power has been lost
1350 Power supply sensor has failed
1351 Power supply sensor value unknown
1352 Power supply returned to normal
1353 Power supply detected a warning
1354 Power supply detected a failure
1355 Power supply sensor detected a non-recoverable value
1600 Processor sensor has failed
1601 Processor sensor value unknown
1602 Processor sensor returned to a normal value
1603 Processor sensor detected a warning value
1604 Processor sensor detected a failure value
1605 Processor sensor detected a non-recoverable value
2048 Device failed
2056 Virtual disk failed
2076 Virtual disk check consistency failed
2077 Virtual disk format failed
2079 Virtual disk initialization failed
2080 Physical disk initializes failed
2081 Virtual disk reconfiguration failed
2082 Virtual disk rebuilds failed
2083 Physical disk rebuilds failed
2094 Predictive disk failure reported.
White Paper
Managing The Virtualized Enterprise
Physical Disk
2101 Temperature dropped below the minimum warning threshold
2102 Temperature exceeded the maximum failure threshold
2103 Temperature dropped below the minimum failure threshold
2106 Smart FPT (predictive failure) exceeded. The disk is likely to fail in the near future.
2107 Smart configuration change. The disk is likely to fail in the near future.
2108 Smart warning. The disk is likely to fail in the near future.
2109 SMART warning temperature. The disk is likely to fail in the near future.
2110 SMART warning degraded. The disk is likely to fail in the near future.
2112 Enclosure was shut down. The physical disk enclosure is either hotter or cooler than the maximum or minimum allowable temperature range.
2123 Redundancy lost
2125 Controller cache preserved for missing or offline virtual disk
2129 Virtual disk BGI failed
2131 Firmware version mismatch
2132 Driver version mismatch
2137 Communication timeout
2146 Bad block replacement error
2148 Bad block medium error
2149 Bad block extended sense error
2150 Bad block extended medium error
2163 Rebuild completed with errors
2165 The RAID controller firmware and driver validation was not performed. The configuration file cannot be opened.
2166 The RAID controller firmware and driver validation was not performed. The configuration file is out of date or corrupted.
2167 The current kernel version and the non- RAID SCSI driver version are older than the minimum required levels.
2168 The non- RAID SCSI driver version is older than the minimum required level.
White Paper
Managing The Virtualized Enterprise
Physical Disk
2169 The controller battery needs to be replaced.
2182 An invalid SAS configuration has been detected.
2183 Replace member operation failed on physical disk %1. The physical disk being replaced has failed.
2191 Multiple enclosures are attached to the controller. This is an unsupported configuration.
2201 A global hot spare failed.
2250 Redundant Path is broken
2264 A device is missing.
2265 A device is in an unknown state.
2268 Storage Management has lost communication with the controller.
2270 The physical disk clear operation failed.
2272 Patrol Read found an uncorrectable media error.
2282 Hot spare SMART polling failed.
2283 A redundant path is broken.
2289 Multi-bit ECC error.
2292 Communication with the enclosure has been lost.
2293 The EMM has failed.
2295 A device has been removed.
2297 An EMM has been removed.
2299 Bad physical connection
2300 The enclosure is unstable.
2301 The enclosure has a hardware error.
2302 The enclosure is not responding.
2307 Bad block table is full. Unable to log block
2310 A virtual disk is permanently degraded.
White Paper
Managing The Virtualized Enterprise
Physical Disk
2314 The initialization sequence of SAS components failed during system startup. SAS management and monitoring is not possible.
2316 Diagnostic test failed.
2319 Single-bit ECC error. The DIMM is degrading.
2320 Single-bit ECC error. The DIMM is critically degraded.
2321 Single-bit ECC error. The DIMM is critically degraded. There will be no further reporting.
2322 The DC power supply is switched off.
2336 Controller event log: %1. Controller generated event log while Storage Management was not running
2337 The controller is unable to recover cached data from the battery backup unit (BBU).
2340 The BGI completed with uncorrectable errors.
2346 Physical device error occurred.
2347 The rebuild failed due to errors on the source physical disk.
2348 The rebuild failed due to errors on the target physical disk.
2349 A bad disk block could not be reassigned during a write operation.
2350 There was an unrecoverable disk media error during the rebuild.
2356 SAS SMP communications error.
2357 SAS expander error.
2373 Attempted import of unsupported Virtual Disk type
Redundancy Unit
1300 Redundancy sensor has failed
1301 Redundancy sensor value unknown
1302 Redundancy not applicable
1303 Redundancy is offline
1304 Redundancy regained
White Paper
Managing The Virtualized Enterprise
Redundancy Unit
1305 Redundancy degraded
1306 Redundancy lost
2098 Global hot spare assigned
2099 Global hot spare unassigned
2122 Redundancy degraded
2123 Redundancy lost
2124 Redundancy normal
2163 Rebuild completed with errors
2166 The RAID controller firmware and driver validation was not performed. The configuration file is out of date or corrupted.
2167 The current kernel version and the non- RAID SCSI driver version are older than the minimum required levels.
2168 The non- RAID SCSI driver version is older than the minimum required level.
2197 Replace member operation has stopped for rebuild.
2200 Replace member operation is not possible as combination of SAS and SATA physical disks is not supported in the same virtual disk.
1000 Server Administrator starting
1001 Server Administrator startup complete
1050 Temperature sensor has failed
1051 Temperature sensor value unknown
1052 Temperature sensor returned to a normal value
1053 Temperature sensor detected a warning value
1054 Temperature sensor detected a failure value
1055 Temperature sensor detected a non-recoverable value
2100 Temperature exceeded the maximum warning threshold
2101 Temperature dropped below the minimum warning threshold
2102 Temperature exceeded the maximum failure threshold
White Paper
Managing The Virtualized Enterprise
Redundancy Unit
2103 Temperature dropped below the minimum failure threshold
2154 Maximum temperature probe warning threshold value changed
2155 Minimum temperature probe warning threshold value changed
Virtual Disk Events
2053 Virtual disk created
2054 Virtual disk deleted
2055 Virtual disk configuration changed
2056 Virtual disk failed
2057 Virtual disk degraded
2058 Virtual disk check consistency started
2059 Virtual disk format started
2061 Virtual disk initialization started
2063 Virtual disk reconfiguration started
2064 Virtual disk rebuilds started
2067 Virtual disk check consistency cancelled
2070 Virtual disk initialization cancelled
2076 Virtual disk Check Consistency failed
2077 Virtual disk format failed
2079 Virtual disk initialization failed
2081 Virtual disk reconfiguration failed
2082 Virtual disk rebuilds failed
2085 Virtual disk check consistency completed
2086 Virtual disk format completed
2088 Virtual disk initialization completed
2090 Virtual disk reconfiguration completed
White Paper
Managing The Virtualized Enterprise
Virtual Disk Events
2091 Virtual disk rebuilds completed
2114 A consistency check on a virtual disk has been paused (suspended)
2115 A consistency check on a virtual disk has been resumed
2116 A virtual disk and its mirror have been split
2117 A mirrored virtual disk has been un-mirrored
2118 The write policy change write policy
2125 Controller cache preserved for missing or offline virtual disk
2127 Background initialization (BGI) started
2129 BGI failed
2130 BGI completed
2136 Virtual disk initialization OK / Normal
2159 Virtual disk renamed
2192 The virtual disk Check Consistency has made corrections and completed.
2193 The virtual disk reconfiguration has resumed.
2194 The virtual disk read policy has changed.
2199 The virtual disk cache policy has changed.
Voltage Sensor Events
1150 Voltage sensor has failed
1151 Voltage sensor value unknown
1152 Voltage sensor returned to a normal value
1153 Voltage sensor detected a warning value
1154 Voltage sensor detected a failure value
1155 Voltage sensor detected a non-recoverable value
White Paper
Managing The Virtualized Enterprise
Virtualization Management
Virtualization technology comes in several different forms. There is virtualization running as a software
application running on a host Operating System such as Microsoft’s Virtual Server 2005 or the
virtualization support included in Windows Server 2008. This approach has perceived disadvantages from
a security perspective as the attack service of the virtualization layer is a general purpose OS. Microsoft
also offers Hyper-V Server 2008 that strips the host OS to Windows Server Core, but still the footprint and
the attack surface is larger than an embedded hypervisor and once into the host OS, the guest OS’s can
be compromised. For the Microsoft virtualization solutions, the logs are all stored in the Applications and
Service Logs in the EventViewer of the host OS. EventTracker is able to collect all these logs through the
standard windows collection methods.
In the case of VMware the 2 hypervisors available are ESX and ESXi ESX is similar to the Hyper-V Server
2008 model, and is a bootable hypervisor. The Operating environment in the ESX case is a stripped down
Linux kernel. It is argued that it is more secure than a general purpose OS installation such as Server 2008
or even Server Core as it is more stripped and it is Linux. ESXi on the other hand, represents the other
popular type of virtualization technique, and is usually embedded directly on the server hardware and
operates more like firmware than software like ESX or Hyper-V. ESXi is very small, and offers access only
through defined and limited APIs.
In larger installations, ESXi combined with a management application like Vcenter is emerging as the
preferred choice. As the hypervisors and the management application have been pared down, it is
expected that these are inherently more secure as the attack surface has been reduced. From a security
perspective this approach has a completely different management layer outside of the Operating System.
Both the Hypervisor and the management applications fortunately produce logs and these logs should be
collected and stored in the Log Management SIEM solution.
EventTracker is able to collect logs directly from the bare-bones hypervisors such as Vmware ESXi, or the
management application in the case of Vcenter, or from ESX. The following diagram shows the collection
architecture.
White Paper
Managing The Virtualized Enterprise
Hyper V Events Hyper-V is made up of distinctive services and each service generates an exhaustive list of events. The
events follow the general Microsoft approach to logging – log it all and log it in great detail. These events,
when normalized, provide a complete picture of what has occurred and when it occurred. Combine these
with login information provided by AD and you have a complete who/what/when picture of both manual
and automated changes in the virtual environment.
Hyper-V Hypervisor
1 Hyper-V successfully started.
5 Hyper-V launch aborted due to auto-launch being disabled in the registry.
6 Hyper-V failed Code Integrity check.
7 Hypervisor traces are corrupted
17 Hyper-V launch failed. The registry key could not be opened by the Hyper-V boot driver
18 Hyper-V launch failed. Registry value could not be read
19 Hyper-V launch failed; the registry value %2 of key %1 is not a string.
20 Hyper-V launch failed; sleep and hibernate could not be disabled (status %1).
26 Hyper-V launch failed. Hyper-V boot loader's internal logic failed
27 Hyper-V launch failed; the Hyper-V boot loader was unable to allocate sufficient resources to perform the launch.
28 Hyper-V launch failed. The Hyper-V boot loader does not support the vendor of at least one of the processors in the system.
29 Hyper-V launch failed. Processor does not appear to support the features required by Hyper-V
White Paper
Managing The Virtualized Enterprise
Hyper-V Hypervisor
30 Hyper-V launch failed. The system's combination of processors is not supported.
31 Hyper-V launch failed. The system does not appear to have a sufficient level of ACPI support to launch Hyper-V.
32 Hyper-V launch failed. At least one of the processors in the system does not appear to provide a virtualization platform supported by Hyper-V.
33 Hyper-V launch failed. the Hyper-V image could not be accessed
34 Hyper-V launch failed. Hyper-V image could not be loaded
35 Hyper-V launch failed. The Hyper-V image could not be read
36 Hyper-V launch failed; the Hyper-V image failed code integrity checks
37 Hyper-V launch failed. The Hyper-V image does not contain the Hyper-V image description data structures
38 Hyper-V launch failed. At least one of the processors in the system was unable to launch Hyper-V
40 Hyper-V launch failed The Hyper-V image is not the correct revision
41 Hyper-V launch failed. Either VMX not present or not enabled in BIOS.
42 Hyper-V launch failed. Either SVM not present or not enabled in BIOS.
Virtual Machine Management Service
2000 Could not register service connection point
2001 Could not unregister service connection point
10000 SID Mapping Error
10001 Failed to create NT VIRTUAL MACHINE security identifier mappings
10010 10011 The security identifier S-1-5-83 is already mapped to another domain.
10020 10021 Failed to create security identifier mapping
10030 10031 Failed to create security identifier mapping
10104 Failed to revert to VSS snapshot on one or more virtual hard disks of the VM
10107 Corrupt or invalid configuration files
11900 VM configuration section is corrupt
White Paper
Managing The Virtualized Enterprise
Virtual Machine Management Service
12242 Cannot mount the device read/write because the device is already mounted read-only
12243 Cannot mount the device
13000 User failed to create external configuration store
13001 Failed to create external configuration store at <location>
14030 14031 Failed to update the VM's saved state information
14040 14041 Failed to query domain information.
14050 Failed to register service principal name.
14060 14061 Failed to locate the default configuration store.
14062 14063 Failed to locate the default virtual hard disk directory
14072 Automatic restart has been disabled for VM because the VM stopped responding repeatedly
14073 VM stopped responding repeatedly.
14074 VM already running when the Hyper-V VM Management service started.
14080 14081 VM failed to automatically restart
14090 14091 Hyper-V VM Management service is shutting down while some VM's are running.
14092 14093 Service is shutting down.
14094 14095 Service started successfully.
14096 14097 Service failed to start.
14098 Required driver is not installed or is disabled.
14100 14101 Shutting down physical computer. Stopping/saving all VM’s .
14210 14211 Snapshot Operation failed to delete snapshot
14241 Cannot find the specified VM.
14270 VM unable to check user access rights
14330 14331 Failed to delete snapshot because it is specified as the automatic recovery snapshot for VM
15010 15011 Failed to create new VM
15040 15041 Failed to import VM
White Paper
Managing The Virtualized Enterprise
Virtual Machine Management Service
15050 15051 Failed to export VM
15070 15071 Service failed to remove snapshot
15080 A new VM was added in a different location and the creation process never completed
15110 15111 Failed to modify service settings.
15120 15121 VM failed to initialize
15140 15141 VM failed to turn off
15150 15151 VM Save Operation failed
15170 15171 VM failed to pause
15180 15181 VM failed to resume
15190 15191 Snapshot Operation failed
15220 15221 VM failed to reset
15240 15241 VM failed to begin delayed startup
15300 Failed to access configuration store
15310 Created configuration store
15320 Failed to create configuration store
15330 VM Bus (VMBus) cannot start because the physical computer's PCI chipset does not properly support Message Signaled Interrupts.
15340 The VM bus is not running.
15500 15501 VM failed to start worker process
16000 16001 VM Management service encountered an unexpected error
16020 VM encountered an unexpected error. The system cannot find the path specified.
16040 Cannot get information about available space for path
16060 16061 VM paused due to insufficient disk space
16090 16100 Worker Process validation failed
16110 An error occurred while waiting to start VM
16120 VM startup error
White Paper
Managing The Virtualized Enterprise
Virtual Machine Management Service
16140 VM cannot delete file
16150 Cannot delete directory
16160 Cannot delete snapshot file
16170 Cannot delete snapshot directory
16180 Service cannot update the snapshot list for deleted snapshot
16190 Service cannot update the parent for snapshot
16200 Service cannot update the instance of last applied snapshot
16330 Cannot load the snapshot configuration because it is corrupt
16370 Service cannot create the storage required for the snapshot
16371 Snapshot Operation failed
16430 Service timed out waiting for the worker process to exit
17010 Service assigned to an invalid authorization scope
17030 A VM is assigned to an authorization scope that is not defined in the policy store
17040 The authorization store could not be initialized
17050 Failed to initialize application in the current authorization store
17080 Updated the content of the authorization store successfully.
17090 Content of the authorization store could not be updated from the store persistent location
17100 Cannot open authorization store
18002 18003 Cannot take snapshot
18030 Import failed. Unable to create identifier while importing VM
18031 Import failed.
18080 18081 VM import failed
18160 Failed to get summary information for VM
18190 Worker process health is critical for VM
18200 Worker process health is now OK for VM
18240 18241 Unable to find virtual hard disk file
White Paper
Managing The Virtualized Enterprise
Virtual Machine Management Service
18540 VM was reset because the guest operating system requested an operation that is not supported by Hyper-V
19000 19010 WMI namespace is not registered in the CIM repository.
19020 WMI provider has started.
19030 WMI provider failed to start
19040 WMI provider has shut down.
19060 19061 Failed to get saved state information for VM. It is assumed that the VM is in a saved state
20100 20101 Failed to register the configuration for the VM
20102 20103 Failed to unregister the configuration for the VM
20104 20105 Failed to verify that the configuration is registered for the VM
20106 20107 Service did not find the VM
20108 20109 Failed to start the VM
20110 20111 Failed to shut down the VM
20112 20113 Service failed to forcibly shut down the VM
20114 20115 Service failed to verify the running state of the VM
20132 20133 Failed to delete the configuration for the VM
14250 14251 Cannot find the specified snapshot
14320 14321 Cannot delete snapshot
15060 15061 Failed to apply snapshot
15130 15131 VM failed to start
15510 15511 The worker process for VM failed to respond within the startup timeout period and was restarted
16010 16011 Operation failed
16050 16051 VM is about to run out of disk space
16360 16361 Cannot access the folder where snapshots are stored
18040 18041 Unable to rename file or directory
White Paper
Managing The Virtualized Enterprise
Virtual Machine Management Service
18050 18051 Failed to stop the rename of the file or directory
18060 18061 Import failed
18100 18101 Failed to create export directory.
18110 18111 Failed to copy file during export
18120 18121 An unknown device failed to import
18160 18161 Failed to get summary information for VM
18550 18560 VM was reset because an unrecoverable error occurred on a virtual processor
19050 19051 VM failed to perform operation. The VM is not in a valid state to perform the operation.
Virtual Hard Drive Management Service
12140 Failed to open attachment
12141 File extension is invalid
15050 The system successfully converted VHD
15051 The system successfully created VHD
15052 The Hyper-V Image Management Service started.
15053 The system is expanding VHD
15000 15001 Device mount failed. The device is already mounted read-only, and an attempt was made to mount it read/write
15100 Filename is invalid
15101 Failed to open attachment
15102 Invalid file extension
15103 The system is compacting VHD
15104 The system is merging VHD
15105 The system is converting VHD
15106 The system successfully compacted VHD
15107 The system successfully merged VHD
White Paper
Managing The Virtualized Enterprise
Virtual Hard Drive Management Service
15108 The system mounted VHD
15109 The system successfully expanded VHD
15110 Invalid VHD
15111 Invalid file name. You cannot use the following names (LPTn, COMn, PRN, AUX, NUL, CON) as they are reserved by Windows.
15200 The Hyper-V Image Management Service stopped.
15201 The Hyper-V Image Management Service failed to start
15202 The system successfully un-mounted VHD
12242 12243 The system is creating VHD
Hyper-V High-Availability Service
21100 Missing or invalid VM ID resource property
21101 Missing or invalid VmStoreRoot resource property
21102 21203 VM failed to register
21103 21104 21502 VM failed to unregister
21105 VM configuration update failed
21106 VM failed to initiate startup
21107 VM failed to initiate shutdown
21108 VM failed to start
21109 21110 VM failed to terminate
21117 Virtual network switch port settings creation failed.
21118 VM update settings failed
21119 VM successfully started
21120 VM successfully registered
21200 System not found
21201 Missing or invalid VM ID resource property
21202 Virtual network switch port already exists
White Paper
Managing The Virtualized Enterprise
Hyper-V Configuration
4096 Configuration no longer accessible. The system cannot find the path specified or configuration is deleted.
4097 Configuration no longer accessible.
4098 Configuration is now accessible.
Hyper-V SynthStore
12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write.
Hyper-V Network
14000 Switch created
14002 Switch deleted
14004 Switch port created.
14006 Switch port deleted
14008 Switch port connected
14010 Switch port disconnected
14012 Internal miniport created
14014 Internal miniport deleted
14016 External Ethernet port bound
14018 External Ethernet port unbound
14020 Switch set up
14022 Switch torn down
14050 Switch create failed
14052 Switch delete failed
14054 Switch port create failed
14056 Switch port delete failed
14058 Switch port connect failed
White Paper
Managing The Virtualized Enterprise
Hyper-V Network
14060 Switch port disconnect failed
14062 Switch port create failed
14064 Switch port delete failed
14066 Ethernet port bind failed
14068 Ethernet port unbind failed
14070 Switch set up failed
14072 Switch tear down failed
14108 Unable to open handle to switch driver
14110 Network WMI provider service started successfully
14112 Network WMI provider service failed to start
14116 Timed out trying to acquire network configuration lock
14118 Unable to initialize network configuration
Hyper-V Image Management Service
12140 12141 Failed to open attachment
12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write
15000 15001 Invalid virtual hard disk
15051 Invalid file extension
15052 Invalid file extension. You cannot use the following names (LPTn, COMn, PRN, AUX, NUL, CON) as they are reserved by Windows.
15053 Invalid file name
15100 System is compacting Image
15101 The system successfully compacted Image
15102 The system is merging Image
15103 The system successfully merged Image
15104 The system is expanding Image
White Paper
Managing The Virtualized Enterprise
Hyper-V Image Management Service
15105 The system successfully expanded Image
15106 The system is converting Image
15107 The system successfully converted Image
15108 The system mounted Image
15109 The system successfully un-mounted Image
15110 The system is creating Image
15111 The system successfully created Image
15200 Image Management service started.
15201 Image Management service stopped.
15202 Image Management service failed to start
Hyper-V Worker
3170 3171 Worker failed to initialize the virtual machine during reset
3200 3201 Worker failed to save, but ignored the error to allow the virtual machine to continue shutdown
3210 3211 Worker failed to save RAM contents during a snapshot operation
3220 3221 Unable to save RAM contents
3230 3231 Unable to restore RAM contents
3240 3241 Unable to save RAM block
3250 3251 Unable to restore RAM block because of an unexpected block data size.
3260 3261 Unable to restore RAM because some RAM blocks are missing.
3270 3271 Unable to restore RAM because some RAM block data is corrupt.
3280 3281 Failed to initiate a snapshot operation
3284 3285 VM was shutdown as a result of a failure to resume execution during a snapshot operation
3286 3287 VM was paused as a result of a failure to resume execution during a snapshot operation
White Paper
Managing The Virtualized Enterprise
Hyper-V Worker
3290 3291 Unable to restore RAM and unable to create a restore buffer.
3310 3311 Failed to initialize restore operation
3320 3321 Failed to create memory contents file
3330 3331 Failed to access the snapshot folder.
3350 3351 Failed to create auto virtual hard disk
3360 3361 Unable to stop the virtual processors.
3370 3371 Unable to reset the virtual hard disk path as a result of a failure to create a snapshot
3432 3433 Could not set the processor affinity for the worker process
5110 Failed to start the worker process using the correct security context
11901 Configuration section is corrupt
11902 RC Vista Ultimate SP1 x86 (Device 'Microsoft Synthetic Display Controller'): An unrecoverable internal error has occurred.
12010 VM' Microsoft Emulated IDE Controller failed to power on with Error 'Incorrect function.'
12070 RC Vista Ultimate SP1 x86 Microsoft Synthetic Video failed to pause with error 'Catastrophic failure'
12200 12201 Virtual machine Out of Memory Error
12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write
12440 12441 Error while opening file during ethernet device startup. The Hyper-V Networking Management service provider may not be installed
12540 RC Vista Ultimate SP1 x86 device Microsoft Synthetic Display Controller experienced a protocol error indicative of a deep system problem.
15160 15161 Failed to restore virtual machine state.
17010 Hyper-V Service is assigned to an unsupported authorization scope
17030 VM is assigned to an authorization scope that is currently not defined in the policy store. The VM will be reassigned to the default authorization scope
17040 The authorization store could not be initialized
17050 Failed to initialize application in the current authorization store
White Paper
Managing The Virtualized Enterprise
Hyper-V Worker
17080 The content of the authorization store has been updated
17090 The content of the authorization store could not be updated
18500 Virtual machine started successfully
18510 VM saved successfully
18520 Snapshot succeeded
VMware Events VMware generates far fewer raw events than Hyper-V but the events tend to focus on the types of
information that security personnel would need to know and less on general day to day health and status
messages. The following is a list of events emitted by VMware and included in the EventTracker Knowledge
Pack. Items marked "predefined alert" are included in the KP tested against VMware 3.x.
Virtual Center Events
Alarm created Datacenter renamed
Alarm removed High resource usage alarm (predefined alert)
Datacenter created Host added to datacenter
Datacenter removed Host removed from datacenter
Virtual Machine Management
Guest OS shutdown Virtual machine removed
VM resource allocation events Virtual machine renamed
Guest OS state changed Virtual machine reset
VM resource configuration updated Virtual machine relocated
Virtual machine cloned Virtual machine suspended
Virtual machine created Virtual machine switched off
Virtual machine powered on Virtual machine snapshot created
Virtual machine registered Virtual machine reverted
Virtual machine reconfigured
White Paper
Managing The Virtualized Enterprise
User Management
Successful user login User permission rule added
Failed user login (predefined alert) User permission rule removed
User logout Task failed or canceled by user (predefined alert)
User permission rule changed
User Management
Remote console connected Remote console disconnected
Summary At its most basic, security management is about first “seeing” everything that is happening, and then
applying processes, tools and solutions that can help you make sense of all the information and make you
more secure. In IT, with each new added technology comes complexity – distributed systems, remote
access, the internet, virtualization all create significant new challenges for security teams. Virtualization is
no different.
Also the real security requirements i.e. what is most critical to monitor, are generally driven by corporate
structure, infrastructure and policy. Businesses have different technology vendors, different
organizational structures, different compliance mandates and rarely, if ever, does one size fit all or even
more than one.
With EventTracker, the challenge of visibility is solved. EventTracker provides the most comprehensive
support for virtual environments of any vendor on the market. Having all the data collected dependably
in one place gives an organization the ability to become secure. This data is categorized and available for
advanced real-time analysis where events from all the different technology layers can be monitored. For
example, an enterprise critical application can be assigned to a virtual machine. Using Vmware’s Vmotion,
that virtual machine can be reassigned different hardware based on performance or availability measures.
It becomes critical to know that if a disk error is being received from OpenManage that that disk is mapped
to that VM, and that VM is running the critical service. With centralized visibility all that becomes possible.
Plus, descriptions on all events are available on the EventTracker Knowledgebase, so security personnel
don’t have to worry about understanding hundreds of new events.
From there, with an understanding of the organizational structure and policies, rules can be quickly setup
to alert on violations of policy. For compliance, auditing is easily facilitated and no trusted user is able to
effect change in the enterprise without at least a record being created. Security starts from visibility – not
only the simple ability to see it, but understand it and make sense of it.
White Paper
Managing The Virtualized Enterprise
The EventTracker Solution The EventTracker solution is a scalable, enterprise-class Security Information and Event Management
(SIEM) solution for Windows systems, Syslog/Syslog NG (UNIX and many networking devices), SNMP V1/2,
legacy systems, applications and databases. EventTracker enables “defense in depth”, where log data is
automatically collected, correlated and analyzed from the perimeter security devices down to the
applications and databases. To prevent security breaches, event log data becomes most useful when
interpreted in near real time and in context. Context is vitally important because often the critical
indications of impending problems and security violations can only be learned by watching patterns of
events across multiple systems. Complex rules can be run on the event stream to detect signs of such a
breach. EventTracker also provides real-time alerting capability in the form of an email, page or SNMP
message to proactively alert security personnel to an impending security breach.
The original event log data is also securely stored in a highly compressed event repository for compliance
purposes and later, forensic analysis. For compliance, EventTracker provides a powerful reporting
interface, scheduled or on-demand report generation, automated compliance workflows that prove to
auditors that reports are being reviewed and many other features. With pre-built auditor grade reports
included for most of the compliance standards (FISMA, HIPAA, SOX, GLBA, and others); EventTracker
represents a compliance solution that is second to none. EventTracker also provides advanced forensic
capability where all the stored logs can be quickly searched through a powerful Google-like search
interface to perform quick problem determination.
EventTracker lets users completely meet the logging requirements specified in NIST SP 800-92 Guide to
Computer Security Log Management, and additionally provides Host Based Intrusion Detection, Change
Monitoring and USB activity tracking on Windows systems, all in a turnkey, off the shelf, affordable,
software solution.
EventTracker provides the following benefits
A highly scalable, component-based architecture that consolidates all Windows, SNMP V1/V2,
legacy platforms, Syslog received from routers, switches, firewalls, critical UNIX servers (Red Hat
Linux, Solaris, AIX etc), Solaris BSM, workstations and various other SYSLOG generating devices.
Automated archival mechanism that stores activities over an extended period to meet auditing
requirements. The complete log is stored in a highly compressed (>90%), secured (Sealed with
SHA-1 checksum) archive that is limited only by the amount of available disk storage.
Real-time monitoring and parsing of all logs to analyze user activities such as logon failures and
failed attempts to access restricted information.
Full support for monitoring of virtualized enterprises.
Alerting interface that generates custom alert actions via email, pager, console message, etc.
White Paper
Managing The Virtualized Enterprise
Event correlation modules to constantly monitor for malicious hacking activity. In conjunction with
alerts, this is used to inform network security officers and security administrators in real time. This
helps minimize the impact of breaches.
Various types of network activity reports, which can be scheduled or generated as required for any
investigation or meeting audit compliances.
Host-based Intrusion Detection (HIDS).
Role-based, secure event and reporting console for data analysis.
Change Monitoring on Windows machines
USB Tracking, including restricted use, insert/removal recording, and a complete audit trail of all
files copied to the removable device.
Built-in compliance workflows to allow inspection and annotation of the generated reports.
About EventTracker EventTracker’s advanced security solutions protect enterprises and small businesses from data breaches
and insider fraud, and streamline regulatory compliance. The company’s EventTracker platform comprises
SIEM, vulnerability scanning, intrusion detection, behavior analytics, a honeynet deception network and
other defense in-depth capabilities within a single management platform. The company complements its
state-of-the-art technology with 24/7 managed services from its global security operations center (SOC)
to ensure its customers achieve desired outcomes—safer networks, better endpoint security, earlier
detection of intrusion, and relevant and specific threat intelligence. The company serves the retail,
hospitality, healthcare, legal, banking and financial services, utilities and government sectors.
EventTracker is a division of Netsurion, a leader in remotely-managed IT security services that protect
multi-location businesses’ information, payment systems and on-premise public and private Wi-Fi
networks. www.eventtracker.com.