Derived from VMware Education Services course ‘Virtual Infrastructure…download3.vmware.com/vmworld/2005/pac103.pdf · 2006. 3. 17. · Service console authentication Securing

Derived from VMware Education Services course ‘Virtual Infrastructure’See http://www.vmware.com/services/education.html for course schedule


PAC103Best Practices for Securing VMware ESX Server

Steven Landherr, VMwareMukund Gunti, Carnegie Mellon

Mike Sutton, VMware


This presentation may contain VMware confidential information.

Copyright © 2005 VMware, Inc. All rights reserved. All other marks and names mentioned herein may be trademarks of their respective

companies.


Agenda

Keeping things patchedService console authenticationSecuring service consolenetwork servicesVirtual machine access control


Why Bother Securing the Service Console?

Data security: The service console has access to each virtual machine's virtual disks and mapped LUNsControl security: All running virtual machines can be powered off from the service console


Keeping ESX Server Patched“In case any vulnerability is discovered in a supported configuration, VMware will proactively notify all customers with a valid support and subscription contract and also provide all necessary patches”Patches for all supported release lines can be found at http://www.vmware.com/download/esx/Any add-on software installed in the service console is the responsibility of the ESX Server administrator to maintain

This includes software on the ESX Server CD-ROM that is not installed by default


Keeping Dormant Virtual Machines Patched

Dormant virtual machines, like powered off physical computers, must be kept patched

High risk of compromise when powered on after a long dormant period

Infected virtual machines rob other virtual machines of shared resources


Identifying Dormant Virtual Machines1. Use ‘vmware-cmd –l’ to list registered

virtual machines2. Check the modification time of ‘vmware.log’

in the same directory with the .vmx file3. Check the modification time of all the .vmdk files

under /vmfs4. Any old virtual machines should be removed or

quarantined until they can be patched


Include Virtual Machines in Patch Process

Power on virtual machines during a maintenance period using ‘vmware-cmd vmxfile start’ from the Service ConsolePush patches to virtual machines via network or ISOs

Alternately, have guest OS check with patch server and pull patches

Power-off virtual machines when done, using‘vmware-cmd vmxfile stop’ from the Service Console

Alternately, shutdown from the guest OS


Authentication InfrastructureAll accounts other than root should use authentication infrastructure when available

Fewer passwords to rememberCentralized enforcement of password agingCentralized auditing of authentication events

Service Console supports Active Directory, LDAP, Kerberos, NTLM, and NIS


Service Console AccountsRequired for management interface, remote console and shell accessVirtual Machine administrators should be given non-shell accountsAdditionally, give ESX Server administrators remote shell access via sshUse tools like sudo to provide controlled, auditable access to privileged commandsRestrict root login to the server console and vmware-authd – no ssh!


Securing Service Console Network Services


SSL CertificatesESX Server provides Self Signed CertificatesThey are “self signed” in the true sense

Hey! I am Bill!Trust Me! Yeah Right!

These certificates are used in the SSL handshake when we use the Management Interface (MUI) or the remote console


SSL Certificates, cont.Yes, the data is encrypted!

How does Alice know that she’s interacting with Bob?!

E(“Hello Server”)

E(“Hello Client”)AliceBob (ESX Server)

But what about Authentication?


A ‘Man In The Middle Attack’ is possible!

Alice believes that she is talking to Bob and Bob believes that he is talking to AliceBUT….the “evil” guy has the last laugh!

AliceBob (ESX Server)

SSL Certificates, cont.

Start virtual machineDelete virtual machine

Done Done


SSL Certificates, cont.If you are vulnerable to such an attack:

Use SSL certificates signed by a global Certificate Authority (CA)Or, you could be your own CA and issue certificates across your ESX Server Farm and bootstrap trust from this CAThe bottom line is that using self signed certificates over an untrusted network (like the Internet) is a bad idea.


Open Network Port AuditingAny open port is a window of opportunity for an attackerPorts used by ESX Server:

Management interface: 80, 443, 8222, 8333(Service: Apache)

Remote Console: 902 (Service: VMware Authentication Daemon)

All management interface requests are redirected to port 443 when SSL is used


Not securevmware-authd902

Avoidrpcbind111

Not securehttp80, 443, 8222, 8333

Not securetelnet23

Securessh22

Not secureftp21

CommentsServiceOpen Port

Low Security Configuration

Not Secure!!


Medium Security Configuration

Securevmware-authd902

Avoidrpcbind111

Securehttps80, 443, 8222, 8333

Not securetelnet23

Securessh22

Not secureftp21


Not Secure!!


High Security Configuration

Using the high security configuration is an excellent idea ☺This configuration is recommended!

Securevmware-authd902

Securehttps80, 443, 8222, 8333

Securessh22



Open Network Port Auditing

Do not use the Service Console as a general purpose Linux serverAvoid running unnecessary network services like NFS on the Service ConsoleUse ssh for remote access. Telnet is a bad ideassh can be hardened by disabling root logins


Additional Network SecurityConnect the Service Console NIC to an isolated management networkDo not run a software firewall in the Service Console (ESX Server 2.x)

Use an external firewall/IDS applianceSoftware firewall inside a virtual machine is OK

Do not allow virtual NICs to enter promiscuous mode


Log File AuditingLog files capture important eventsInvaluable resources for forensicsMonitoring log files for unsuccessful login attempts is a good ideaHowever, log files are not the final authority as they can be altered if the root account gets compromised or the storage device is manipulated


Log Files to Watch/var/log/secure(Login Information from sshd and VMware Authentication Daemon). Invalid login attempts are logged here!!/var/log/vmware/event-vmxfile.log(Events associated with a virtual machine)/var/log/vmware-mui/access_log/var/log/vmware-mui/ssl_request_log(Apache Log files)


Example of Suspicious ActivityObserved in /var/log/secure

Oct 2 02:29:14 xxxx sshd[21186]: Illegal user a from 192.168.1.47Oct 2 02:29:18 xxxx sshd[21186]: Failed password for illegal user a

from 192.168.1.47 port 32798 ssh2Oct 2 02:29:32 xxxx sshd[21194]: Illegal user b from 192.168.1.47Oct 2 02:29:36 xxxx sshd[21194]: Failed password for illegal user b

from 192.168.1.47 port 32799 ssh2Oct 2 02:29:49 xxxx sshd[21196]: Illegal user c from 192.168.1.47Oct 2 02:29:53 xxxx sshd[21196]: Failed password for illegal user c

from 192.168.1.47 port 32800 ssh2

Looks like an automated attempt to break in!


A Final Note on Logs

Suspicious log file activity must always be investigated. This could be a sign of impending dangerTools like logwatch are available to automate this anomaly detection


Virtual Machine Access Control


Objectives

Understand the levels at which access control mechanisms are availableUnderstand the user roles offered by ESX Server and VirtualCenterUse these controls to set access policies for virtual machines


Authentication and Control Systems


VirtualCenter Roles

Add/remove user permissionsX

Add/change licensesX

Change VirtualCenter Settings including performance intervals, template upload directory, and SNMP and SMTP settings

XX

Add/remove templates, tasksXX

Add/remove farms, groupsXX

Add/remove/move/modify virtual machines, including deploy templates and clone existing virtual machines

XX

X

X

X

Virtual Virtual Machine Machine UserUser

X

X

X

X

Virtual Machine Virtual Machine AdministratorAdministrator

X

X

X

X

VirtualCenterVirtualCenterAdministratorAdministrator

Add/remove hosts

Access virtual machine console

Power operations on virtual machines

View farms, hosts, virtual machinesX

PrivilegesPrivilegesRead Read Only Only UserUser


VirtualCenter Permission Inheritance

Permission = user/group + VirtualCenter role

Applied to a level in theinventory hierarchyPermissions areinherited downwards

Server farmsPhoenix

ESX Server 01ESX Server 02

ESX Server 03

Finance

Web serversDatabase serversApplication servers

Human Resources

Web serversDatabase serversApplication servers

Josh:Read-only user

Phil:VM Admin

Phil:VM Admin


ESX Server Roles

Add/remove user permissionsX

Add/change licensesX

Change ESX Server settings including VMkernel parameters and SNMP settings

X

Add/remove/move/modify virtual machines, including deploy templates and clone existing virtual machines

XX

X

X

X

User with User with rxrxpermissionpermission

X

X

X

Virtual machine Virtual machine owner, or user owner, or user with with rwxrwxpermissionpermission

X

X

X

Root userRoot user

Access Remote Console console

Power operations on virtual machines

View virtual machinesX

PrivilegesPrivilegesRead Read only only useruser


You may power cycle thevirtual machine and use

remote console to it

Authentication Models With ESX Server

You may modify thevirtual machine’s

hardware

wWrite

XExecute

Virtual machineappears in your MUI

overview pageMeaning

rReadPermission

Everyone else

Persons other than the ownerwho are members of the group stamped on the file

The file’s owner

Meaning

Others

Group

User

Role R W XFull access, actions and modification privileges

R XStart, stop, reset, suspendvirtual machine via MUI

Remote console, APIAccess files read only

Only RCan see virtual machine in MUI


Setting a Virtual Disk’s PermissionsDefault permissions for .vmx files:

R W XFull access, actions and modification privileges

R XStart, stop, reset, suspend using MUI, Remote Console, or API; Access files read-only

Check permissions on both the .vmx file

and the parent directories


Default permissions for .vmdk files:

Owner of .vmx file must have access to .vmdk file(s)Simplest case: Make ownership the same

Setting a Virtual Disk’s Permissions

R WFull access, actions, and modification privileges


SummaryVirtualCenter roles are finer-grained than ESX Server rolesWith VirtualCenter, virtual machine permissions can use the power of Windows groupsWith ESX Server alone, virtual machine permissions are stored using the attributes of the virtual machine configuration file


PAC879: The Next Phase of Virtual Infrastructure: Introducing ESX Server 3.0 and VirtualCenter 2.0

PAC177: Distributed Availability Services ArchitecturePAC484: Consolidated Backup with ESX Server:

In-Depth ReviewPAC485: Managing Data Center Resources Using the

VirtualCenter Distributed Resource SchedulerPAC532: iSCSI and NAS in ESX Server 3

This presentation covers the current versions of our products. Details about

future releases of our products are available in select sessions at VMworld, including:

Documents

Derived from VMware Education Services course ‘Virtual Infrastructure…download3.vmware.com/vmworld/2005/pac103.pdf · 2006. 3. 17. · Service console authentication Securing