Configuration of VIO on Power6

© 2008 IBM Corporation

Session Title: Configuration of Virtual I/O on POWER6

Speaker Name: Janel Barfield

Session ID: pVI08

IBM Power Systems Technical Universityfeaturing IBM AIX and Linux

September 8 – 12, 2008 – Chicago, IL

IBM UNIX Technical Support Educationemail: [email protected]

IBM Training


AgendaVirtual I/O configuration concepts

Describe and configure virtual Ethernet

Describe the new Integrated Virtual Ethernet (IVE) adapter– Architecture– Multi-Core Scaling

Configure external network access using IVE

Describe and configure external network access using Shared Ethernet adapter (SEA)

– With Link Aggregation– With SEA Failover

Describe and configure virtual SCSI– Configure new file-backed virtual devices

Conclusion

IBM Training


Virtual I/O Configuration Concepts

On POWER5 and POWER6, virtual I/O devices include virtual Ethernet, Integrated Virtual Ethernet Adapter, Shared Ethernet

Adapter, and virtual SCSI devices

Virtual SCSI and Shared Ethernet adapters require the Advanced POWER Virtualization feature and a configured

Virtual I/O Server

Integrated Virtual Ethernet is a new POWER6 feature

The Virtual I/O Server is a dedicated special appliance partition

IBM Training


Virtual Ethernet: POWER5 Hypervisor Ethernet SwitchThe POWER Hypervisor provides the Ethernet switch functionality for virtual Ethernet adapters on POWER5 and POWER6 systems:

– Implements a software Ethernet switch– Supports up to 4094 networks (VLAN IDs)– Copies packets between partitions (memory to memory)– Consistent with VLAN IEEE 802.1Q standard– The ports on this “switch” are configured via the HMC and are part of the

partition’s configuration

VLAN 1 VLAN 2

VLAN 3

LPAR 1 LPAR 2 LPAR 3 LPAR 4 LPAR 5

POWER Hypervisor (switch)

IBM Training


Virtual Ethernet AdaptersVirtual Ethernet adapters are configured using the HMC or

IVM, in the partition’s profile or dynamically with DLPAR

Select the virtual slot number and PVID

• Configures the port for this slot on the

Hypervisor switch

Select additional VLANs• Allows the adapter to

communicate on multiple VLANS (up to 20 plus

PVID)• Requires additional

configuration in the LPAR

Virtual Ethernet interfaces are configured in AIX just like physical adapter interfaces

IBM Training


Virtual Ethernet Demonstration

Configure and test a virtual network between two AIX LPARs

LPAR1 LPAR2

POWER Hypervisor

POWER6 System

ent0 ent0

en0 en0

VLAN 1

192.168.1.1 192.168.1.2

PVID=1ieee_virtual_eth=0


VIOS

IBM Training


Integrated Virtual Ethernet (IVE)

X2

External Layer 2 or Layer 3 switch

Port Group 110 Gb or

1 Gb physicalport

1 Gb physical port

LPARs

Logical Switch Logical Switch

Three IVE models:• Dual-port Gigabit: 1 port group, 2 physical 1 Gb ports• Dual-port 10 Gigabit: 2 port groups, 1 physical 10 Gb port per port group• Quad-port Gigabit: 2 port groups, 2 physical 1 Gb ports per port group

IBM Training


lheaent

OS

IVE PortsLogical ports are associated with a specific physical port

Port group:– Set of 16 logical ports on 1 or 2 physical ports:

• Logical ports can be split evenly between the 2 physical ports in a port group, or unevenly

– 1 or 2 port groups per Host Ethernet Adapter (HEA), depending on model– 1 or 2 physical ports per port group, depending on model, each with own Layer

2 switch

Port Group

External Switch

LogicalPort

PhysicalPort

Logical Switch HEA

IVE

Logical devices as they appear in AIX

LPAR

IBM Training


Multi-Core Scaling (MCS)The MCS value sets the maximum number of logical ports for a port group

– Set per port group (regardless of number of physical ports)Use MCS to specify number of ports you need to support LPARs

– Default value is 4, so only 4 logical ports per port group available– Best IVE adapter performance when MCS value equal to # of processors

Must power off/on managed system to change MCS

Port Group

Switch

PhysicalSwitch

Port Group

Switch

PhysicalSwitch

MCS = 1 MCS = 4

4 ports16 ports

IBM Training


Managed System IVE Configuration (1 of 2)1. Select the managed system, and choose:

– Hardware Information -> Adapters -> Host EthernetPhysical port

settings(actual)

LPARs which are configured for logical ports using this physical port

2. Choose which IVE adapter

3. Choose physical port

4. Click Configurebutton to configure

physical port

IBM Training


A Virtual I/O Server partition must be listed as the promiscuous LPAR if it is using a port on the IVE as part of a Shared Ethernet adapter device

Managed System IVE Configuration (2 of 2)4. button is used to configure physical port settings:

MCS

Set to None or to a partition that will take entire physical port

Enabled means HMC

will attempt to negotiate flow

control for physical port

IBM Training


LPAR IVE Configuration

Configure logical ports

from the LPAR profile or with

DLPAR

IBM Training


IVE DemonstrationConfigure an IVE logical port on the VIOS LPAR

1. Check the configuration of the physical adapter2. Add a logical port for the VIOS LPAR

LPAR1 LPAR2

POWER Hypervisor

POWER6 System

ent0 ent0

en0 en0

VLAN 1

192.168.1.1 192.168.1.2



VIOS

HEA

P0 P1

lhea0

ent0

EthernetSwitch

IBM Training


Shared Ethernet Adapter (SEA)Shared Ethernet adapter bridges external networks to internal VLANs:

– Forwards frames at OSI Layer 2 and is transparent to IP layer

Virtual I/O Server Partition

Layer 2 Bridge (Shared Ethernet adapter)

Device Driver Device Driver Device Driver

PhysicalAdapter

VirtualAdapter

VirtualAdapter

ExternalLANs

1-16 virtual adapters which connect to VLANs (and client

partitions)

IBM Training


SEA Example

Shared Ethernet adapter is used to extend VLAN 2 and 3 using the physical Ethernet in the Virtual I/O Server

SharedEthernet

VLAN 3VLAN 3

VLAN 2VLAN 2

Virtual I/O Server

Managed systemEthernet Switch

Standalone serversLPAR 1, 2, 3

IBM Training


Configure Virtual Ethernet Adapter for SEA

• Virtual adapters that will be associated with an SEA on the VIO Server must have the “Access external network”checkbox selected

• Specify a PVID and optionally any VLAN IDs that the SEA will extend to an external network

- PVIDs are stripped from frames in the SEA, so use a unique VLAN for PVID when supporting access to VLANs on an external network

Virtual Ethernet adapters and Hypervisor switch ports are configured here on the HMC/IVM

IBM Training


Configure SEA Device in VIO ServerCommand to create Shared Ethernet Adapter:

$ mkvdev –sea ent0 –vadapter ent1 \–default ent1 –defaultid 1

Creates device (ent2 in this case):

$ lsdev –virtualname status description

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

ent2 Available Shared Ethernet Adapter

PVID 1ent0 ent1

ent2 (SEA)

VIOSswitch

physicalHypervisor

virtual

IBM Training


SEA DemonstrationConfigure an SEA adapter on the VIOS

1. Create a virtual Ethernet adapter on the VIOS LPAR that can access external networks

2. Create the SEA on the VIOS LPAR using the lhea port and the virtual adapter

EthernetSwitch

LPAR1 LPAR2

POWER Hypervisor

ent0 ent0

en0 en0

VLAN 1

192.168.1.1 192.168.1.2


VIOS

HEA

P0 P1

lhea0

ent0

ent1


POWER6 System

SEA


IBM Training


SEA with Link AggregationCreate link aggregation Ethernet adapter:$ mkvdev –lnagg ent0 ent1

ent3 shows up as:$ lsdev –adapter | grep ent3ent3 Available EtherChannel / IEEE 802.3ad Link Aggregation

Create Shared Ethernet adapter:$ mkvdev –sea ent3 –vadapter ent2 \

–default ent2 –defaultid 1

ent2 PVID 1

(Physical)ent4 (SEA)

ent0

ent1

ent3

(Aggregate)

Virtual I/O Server

switch Hypervisor(Virtual)

IBM Training


Virtual SCSI Overview

Physical Storage

S S S

PHYVirtual I/O Server

C

Client

C

Client

C

Client

VTDVTD

Hypervisor

PHY

PHY

PHY Physical Adapter

SVSCSI ServerVirtual Adapter

CVSCSI ClientVirtual Adapter

VTD Virtual Target Device

C

VTD

S

VTD

The red connections show two clients accessing the same physical storage (A) via two different server adapters (B) and virtual target devices (D)

VTD

PHY

The blue connection shows multiple target devices (D) attached to a single server adapter (B)

IBM Training


Virtual SCSI Configuration (1 of 3)1) Define virtual SCSI server in VIO Server partition

and client adapter in AIX or Linux partition

2) Check availability of virtual SCSI server adapters on VIO Server:$ lsdev -virtualname status descriptionvasi0 Available Virtual Asynchronous Services

Interface (VASI)vhost0 Available Virtual SCSI Server Adaptervsa0 Available LPAR Virtual Serial Adapter

IBM Training


Create vSCSI Adapters DemonstrationCreate virtual SCSI adapters on the VIOS and AIX LPARs

LPAR1

POWER Hypervisor

POWER6 System

VIOS

vscsi0vhost0

FC cardSAN

fcs0 fcs1

hdisk6

hdisk7

sas0

…hdisk5

hdisk0

Internal storage Optical device

cd0

IBM Training


Virtual SCSI Configuration (2 of 3)3) On the VIO Server, define storage resources

To create a volume group:$ mkvg [ -f ][ -vg VolumeGroup ] PhysicalVolume ...

To create a logical volume:$ mklv [ -lv NewLogicalVolume | -prefix Prefix ] VolumeGroup Size [PhysicalVolume ... ]

To create a storage pool:$ mksp [-f] StoragePool PhysicalVolume ...

To create a backing device from available space in a storage pool:$ mkbdsp [-sp StoragePool] Size [-bd BackingDevice]

-vadapter ServerVirtualSCSIAdapter

IBM Training


Virtual SCSI Configuration (3 of 3)4) On the VIO Server, define virtual target devices

5) Boot the client or run cfgmgr to use new virtual devices

$ mkvdev -vdev TargetDevice -vadapter VirtualServerAdapter [ -dev DeviceName ]For example:$ mkvdev –vdev hdisk3 –vadapter vhost0vtscsi0 Available$ mkvdev –vdev lv10 –vadapter vhost0vtscsi1 Available$ mkvdev –vdev cd0 –vadapter vhost0vtopt0 Available

Check the target devices with lsdev:$ lsdev -virtualname status descriptionvtscsi0 Available Virtual Target Device - Diskvtscsi1 Available Virtual Target Device - Logical Volumevtopt0 Available Virtual Target Device – Optical Media

IBM Training


Use lsmap from the VIO Server to verify mapping of virtual targets:$ lsmap -vadapter vhost0

SVSA Physloc Client Partition ID

--------------- ----------------------------------- -----------------

vhost0 U9111.520.10F191F-V3-C6 0x00000003

VTD vtscsi0

LUN 0x8100000000000000

Backing device hdisk3

Physloc U787A.001.DNZ00G0-P1-T10-L8-L0

VTD vtscsi1

LUN 0x8200000000000000

Backing device lv10

Physloc

VTD vtopt0

LUN 0x8300000000000000

Backing device cd0

Physloc

View Configuration with lsmap

Client LPAR ID

Physical location code

Server slot ID

LUN ID

IBM Training


View Configuration with lshwresUse lshwres from the HMC to see system-wide view of virtual I/O configuration (or view from HMC GUI)hscroot@skylab-hmc:~> lshwres -r virtualio --rsubtype scsi -m skylab

lpar_name=VIOS,lpar_id=1,slot_num=7,state=1,is_required=1,adapter_type=server remote_lpar_id=4,remote_lpar_name=node3,remote_slot_num=6, "backing_devices=drc_name=U787F.001.DPM0ZFL-P1-T10-L4-L0/log_unit_num=0x8100000000000000/ device_name=hdisk1,drc_name=U787F.001.DPM0ZFL-P1-T10-L5-L0/log_unit_num=0x820000000000000/“lpar_name=node3,lpar_id=4,slot_num=6,state=1,is_required=1,adapter_type=client,remote_lpar_id=1,remote_lpar_name=VIOS,remote_slot_num=7,backing_devices=none

IBM Training


Virtual Target Device DemonstrationConfigure virtual target devices for the vSCSI adapter with different physical backing devices

clientVG

LPAR1

POWER Hypervisor

POWER6 System

VIOS

vscsi0vhost0

FC cardSAN

fcs0 fcs1

hdisk6

hdisk7

sas0

…

hdisk5

hdisk0

Internal storage Optical device

cd0

vtscsi1cl_lv

vtopt0

vtscsi0

cd0

hdisk0

hdisk1

IBM Training


File-Backed Virtual DevicesFile-back (FB) virtual device types:

– File-backed disk devices• Files created in storage pools can be used as hdisk on client

– File-backed optical media devices

• Create a Virtual Media Repository which can be stocked with DVD-ROM/RAM media

• Clients can use images stored in repository as cd0 devices with media

FB virtual device characteristics:– Read-only FB devices can be shared by multiple clients

– Bootable FB devices appear in SMS

– Reside in FB storage pools

• Mount Directory = /var/vio/storagepools/<FBSP_Name>• LV_NAME = <FBSP_Name>

– Granularity as small as 1MB or as large as parent Logical Volume

FB virtual devices are new as of Virtual I/O Server V1.5

IBM Training


Creating File-Backed Virtual DisksFiles on the virtual I/O Server can be used as backing storage:

1. Create a volume group (mkvg) or storage pool (mksp -f)

2. Create a FB disk storage pool (mksp -fb) inside volume group/storage pool

3. Create a device in the pool (mkbdsp) and map to a vadapter

4. The client associated with that vadapter sees new FB device as an hdisk

Volume Group/Storage Pool - contains hdisk(s)

FB Disk Storage Pool (contains FB virtual disks)

Target dev Target dev Target dev

IBM Training


Create FB Virtual Disks Example (1 of 2)Create new volume group/logical volume storage pool:$ mkvg -vg newvg hdisk1 OR mksp -f newvg hdisk1

Create new FB storage pool in the logical volume storage pool:$ mksp -fb fbpool -sp newvg -size 10g

fbpool

File system created successfully.

10444276 kilobytes total disk space.

New File System size is 20971520

Create new file device with a certain size, create the VTD, and map to vhost adapter:$ mkbdsp -sp fbpool 30m -bd fb_disk1 -vadapter vhost3Creating file "fb_disk1" in storage pool "fbpool".

Assigning file "fb_disk1" as a backing device.

vtscsi3 Available

fb_disk1

New storage pool (newvg)

New FB storage pool (fbpool) that is 10 GB

inside of newvg

Create new 30 MB file called fb_disk1

Resulting VTD is named vtscsi3and is mapped to vhost3

IBM Training


Create FB Virtual Disks Example (2 of 2)View mapping with new backing device:

$ lsmap -vadapter vhost3SVSA Physloc Client Partition

ID

--------------- ----------------------------- ------------------

vhost3 U8203.E4A.10CD1F1-V1-C15 0x00000000

VTD vtscsi3

Status Available

LUN 0x8100000000000000

Backing device /var/vio/storagepools/fbpool/fb_disk1

Physloc

IBM Training


Create FB Virtual Optical Device (1 of 2)Create volume group/logical volume storage pool:$ mkvg -vg medrep hdisk4 OR mksp -f medrep hdisk1

Create 10 GB Virtual Media Repository in the LV pool:$ mkrep -sp medrep -size 10G

Virtual Media Repository Created

Repository created within "VMLibrary_LV" logical volume

Create media (aixopt1) in repository from a file:– Media could be blank, loaded from cd# device, or a file$ mkvopt -name aixopt1 -file dvd.product.iso -ro

New storage pool (medrep)

IBM Training


Create FB Virtual Optical Device (2 of 2)View repository and its contents:

$ lsrepSize(mb) Free(mb) Parent Pool Parent Size Parent

Free

10198 6532 medrep 69888 59648

Name File Size Optical Access

aixopt1 3666 None ro

Create FB virtual optical device and map to vhost adapter:

$ mkvdev -fbo -vadapter vhost4vtopt0 Available

Load the image into the media device:– Use the unloadopt command to unload

$ loadopt -vtd vtopt0 -disk aixopt1

New VTD name

IBM Training


Viewing FB Configuration from the HMC

HMC command line example:hmc:~> lshwres -m hurston -r virtualio --rsubtype scsi

lpar_name=VIOS,lpar_id=1,slot_num=16,state=1,is_required=0,adapter_type=server,remote_lpar_id=any,remote_lpar_name=,remote_slot_num=any,"backing_devices=""0x8100000000000000//""""/var/vio/VMLibrary/aixopt1""""""". . .

IBM Training


FB Device Command Examples (1 of 2)List the repository and any contents:$ lsrep

Size(mb) Free(mb) Parent Pool Parent Size Parent Free

10198 6532 medrep 69888 59648

Name File Size Optical Access

aixopt1 3666 vtopt0 ro

List the storage pools:– Notice both LVPOOL and FBPOOL types:$ lssp

Pool Size(mb) Free(mb) Alloc Size(mb) BDs Type

rootvg 69888 44544 128 1 LVPOOL

NewVG 69888 59648 64 0 LVPOOL

medrep 69888 59648 64 0 LVPOOL

fbpool 10199 6072 64 2 FBPOOL

List out volume groups/storage pools (LVPOOL type only):$ lsvg

rootvg

NewVG

medrep

IBM Training


FB Device Command Examples (2 of 2)List LVPOOL details:$ lssp -detail -sp NewVGName PVID Size(mb)hdisk3 000cd1f195f987df 69888

List FBPOOL details:$ lssp -bd -sp fbpoolName Size(mb) VTD SVSAfb_disk1 30 vtscsi3 vhost3fb_disk2 4096 vtscsi4 vhost3

Show all mounts including FB devices:$ mountnode mounted mounted over vfs date options

-------- --------------- --------------- ------ ------------ ---------------/dev/hd4 / jfs2 Apr 18 13:01 rw,log=/dev/hd8/dev/hd2 /usr jfs2 Apr 18 13:01 rw,log=/dev/hd8/dev/hd9var /var jfs2 Apr 18 13:01 rw,log=/dev/hd8/dev/hd3 /tmp jfs2 Apr 18 13:01 rw,log=/dev/hd8/dev/hd1 /home jfs2 Apr 18 13:01 rw,log=/dev/hd8/proc /proc procfs Apr 18 13:01 rw/dev/hd10opt /opt jfs2 Apr 18 13:01 rw,log=/dev/hd8/dev/fbpool /var/vio/storagepools/fbpool jfs2 Apr 28 12:04 rw,log=INLINE/dev/VMLibrary_LV /var/vio/VMLibrary jfs2 Apr 28 14:36 rw,log=INLINE

IBM Training


File-Backed Virtual Devices DemonstrationConfigure a file-backed virtual disk and file-backed virtual optical device

VIOS LPAR1

vhost1 vscsi1

rootvg

hdisk0

medrep(Virtual Media Repository)

AIX53_isoAIX61_iso

cl_mksysb

hdisk1

stpool1(LV storage pool)

fbpool1(FB storagepool)

fb_disk1fb_disk2

vtscsi2

vtopt1

POWER Hypervisor

hdisk2

cd1

IBM Training


ConclusionVirtual I/O devices on POWER5 and POWER6 systems include virtual Ethernet, Integrated Virtual Ethernet (IVE), Shared Ethernet (SEA), and virtual SCSI

Virtual Ethernet and IVE, are part of the base feature and do not require PowerVM (or Advanced Power Virtualization for POWER5)

– Devices are defined in the HMC– Configuration in AIX is same as for physical devices

Virtual SCSI and SEA devices extend the use of physical devices to multiple clients using the Virtual I/O Server and require the PowerVM feature

– Create virtual devices with mkvdev on VIO Server command line to bridge virtual Ethernet adapters (mkvdev –sea) and to associate virtual SCSI adapters with physical storage

File-backed virtual disks and optical devices provide even more flexibility in storage allocation

IBM Training


39

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workloadprocessed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.

Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

Documents

Configuration of VIO on Power6