Upload
upul-edirisinghe
View
252
Download
7
Tags:
Embed Size (px)
Citation preview
© 2010 Cisco Systems, Inc. All rights reserved. FICON 1
FICON 101
CCIE SAN Study Group
By Ozden Karakok – May 2011
© 2010 Cisco Systems, Inc. All rights reserved. FICON 2
Objective & Agenda At the end of this session, attendees will be able to answer what is FICON is all about! FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 3
FICON Terminology
© 2010 Cisco Systems, Inc. All rights reserved. FICON 4
IBM Mainframe Systems
Current IBM mainframe systems: S/390 G5 S/390 G6 zSeries 800 zSeries 890 zSeries 900 zSeries 990 z9 Ent. Class z9 Business C. z10 Ent. Class z10 Business C.
OS/390 VM VSE/ESA MVS
z/OS z/VM Linux
© 2010 Cisco Systems, Inc. All rights reserved. FICON 5
Introduction to FICON FICON Protocol Layer
ULP (Upper Level Protocol)
FC - 4
FC - 3
FC - 2
FC - 1
FC - 0
SCSI – 3/4
Common Services
SCSI— Command Set Mapping (FCP)
Framing Protocol/Flow control
Encode/Decode (8/10)
Physical Variant
FC-SB2/FC-SB3 (FICON)
Single Byte Command Code
Sets
• FICON is Fibre Channel FC-4 layer protocol
• It is at the same layer as FCP protocol
• Fibre Channel –”Single Byte Command Code” mapping protocol is described in FC-SB3, FC-SB4
• FC-SB2 is an older version. FC-SB3-4 is superset.
Ref: www.t11.org
© 2010 Cisco Systems, Inc. All rights reserved. FICON 6
FICON Terminology
A Channel is a physical entity/element of the processor, acting as the interface between the CPU and the CU
– A Channel Image instantiates multiple, logical channels behind a physical channel
A Channel Path includes the interface adapter, physical fiber pair, and up to two intervening FICON switches/directors between the channel and the CU
A Channel Path Identifier (CHPID) is a single hexadecimal byte value identifying each channel path to the system
© 2010 Cisco Systems, Inc. All rights reserved. FICON 7
FICON Terminology
Outboard I/O processors called Control Units (CUs) – CUs can be stand-alone or integrated in peripheral devices – CUs are identified by a two-byte hexadecimal CUNUMBER – A Control Unit Image instantiates multiple, logical CUs behind a physical CU
A Device Address (aka Unit Address) is a decimal value that selects a specific device on a given CU image
© 2010 Cisco Systems, Inc. All rights reserved. FICON 8
FICON Terminology
Legend
M A I N F R A M E
IBM
N Port
N Port
A “Channel” is an entity typically mapped to an N-port in a host computer which performs access to I/O devices
Channel
Control Unit (CU)
A “Control Unit” is the interface to I/O devices such as printer, disks etc.
Control Unit Image
There can be multiple “Control Unit images” behind each “Control Unit”
Channel Images
There can be multiple “Channel Images” behind one N port, each of which represent a separate logical channel.
Physical
Logical
© 2010 Cisco Systems, Inc. All rights reserved. FICON 9
FICON Terminology
A Logical Partition (LPAR) is a subset of mainframe resources, dedicated to implementing a particular programming environment
– Each LPAR functions independently of other LPARs and is protected against the actions of poorly-behaved LPARs – Resources such as CHPIDs (Channel Path Identifier) may be shared between LPARs
© 2010 Cisco Systems, Inc. All rights reserved. FICON 10
LPARs can share channels Multiple Image Facility (MIF)
Channel
MIF
CU
CU
N_Port
N_Port
N_Port
MIF
LPAR 1
LPAR 2
LPAR n
© 2010 Cisco Systems, Inc. All rights reserved. FICON 11
FICON Terminology
An Output Port is the FICON director or switch port that connects to the CU
– In a switched point-point environment, it can be a single byte hexadecimal value identifying the director/switch port number connected to the CU – FICON port numbering schemes are proprietary to the switch vendor/model; typically different than what’s used for FC config – The port number may also be added with the switch domain ID to form a two-byte Output Port value if supported by the CU – In a cascaded environment, Output Port must be a two-byte value derived from the non-Entry Switch
© 2010 Cisco Systems, Inc. All rights reserved. FICON 12
FICON Cascaded Topology Output Ports
Non-entry switch Entry Switch
CH = Initiator
CU = Target
Switch ID = Domain ID Portaddress = int fc X/Y
© 2010 Cisco Systems, Inc. All rights reserved. FICON 13
FICON Terminology
The Input Output Control Program (IOCP) is a mainframe application that defines the Input Output Control Data Set (IOCDS), which the mainframe uses to define & identify control units, devices, channel paths, channels and their relationships to each other.
I/O definition file (IODF)
All of the terms we’ve just covered are required IOCP/IOCDS parameters
© 2010 Cisco Systems, Inc. All rights reserved. FICON 14
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 15
FICON Topologies
A Point-to-Point (direct connection) A Switched Point-Point Environment results when there is one
FICON director/switch in the channel path between the channel and CU
A Cascaded Environment results when two FICON directors/switches are connected in series along the channel path between the channel and CU
An Entry Switch is the FICON director or switch connected to the channel via the channel path – In a cascaded environment, the Entry Switch is first switch
encountered in the channel path from the channel to the CU – The Entry Switch is identified by a single hexadecimal byte equal
to it’s FC domain ID
© 2010 Cisco Systems, Inc. All rights reserved. FICON 16
FICON Topologies (Contd)
In Point to Point topology, channel and control unit are directly connected. There is no switch between them.
Point to Point
FICON Channel
FICON Control
Unit
© 2010 Cisco Systems, Inc. All rights reserved. FICON 17
FICON Topologies (Contd)
In Single Switch Fabric path configuration, all channels and control units are attached to a single switch
This is called ‘switched point-to-point’ configuration.
Single Switch Fabric Path Configuration FICON
Channel
FICON Control
Unit
FICON Control
Unit
FICON Control
Unit
FICON Channel
© 2010 Cisco Systems, Inc. All rights reserved. FICON 18
FICON Topologies (Contd)
In Two Switch Fabric path configuration there may be more than 2 switches in the fabric, but only one hop between any pair of channel and control units.
Multiple ISLs between connected switches are allowed. However, the fabric should perform load balancing on these ISLs.
Two Switch Fabric Path Configuration
FICON Control
Unit
FICON Channel
FICON Control
Unit
FICON Channel
FICON Control
Unit
© 2010 Cisco Systems, Inc. All rights reserved. FICON 19
FICON Addressing
FICON defines concept of link address.
Each channel and control unit have one unique link address.
In older versions, link address used to be single byte and could only be used in single switch environment
FC-SB3 defines two byte link addresses which allow more than one switch in FICON path.
Link addresses are mapped to FC-ID as depicted in figure.
Link Address
Port Area
16
Link address
Domain
Static in FICON fabric (usually 0)
Two Byte Link address to FCID mapping (for Cascaded environments)
Single Byte Link address to FCID mapping
Port Area
8
Link address
Domain
Static in FICON fabric (usually 0)
© 2010 Cisco Systems, Inc. All rights reserved. FICON 20
HCD – Point to Point
© 2010 Cisco Systems, Inc. All rights reserved. FICON 21
HCD – Cascaded
© 2010 Cisco Systems, Inc. All rights reserved. FICON 22
Using DM to identify Port Addresses
Notice FICON port numbers are displayed
9513 - 16 ports per slot 9509 - 32 ports per slot
© 2010 Cisco Systems, Inc. All rights reserved. FICON 23
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 24
FICON Channel Initialization
CH to Switch
CH to CU
Same Well Known Fibre Channel Addresses
© 2010 Cisco Systems, Inc. All rights reserved. FICON 25
FICON Control Unit Initialization
Defined In FC-FS
Defined In FC-GS
Can register both SCSI and FC-SB FC-4 types
port-wwn (vendor) :50:05:07:63:00:c2:04:2b (IBM) node-wwn :50:05:07:63:00:c0:04:2b class :2,3 node-ip-addr :0.0.0.0 ipa :ff ff ff ff ff ff ff ff fc4-types:fc4_features:scsi-fcp fcsb2-ch-cu fcsb2-cu-ch
[snip]
© 2010 Cisco Systems, Inc. All rights reserved. FICON 26
QSA – Query Security Attributes
Used by Channel to verify Fabric Binding and Insistent Domain IDs is configured on MDS
Only used on Cascade environment
• QSA = 0x7E 00 00 00 • Revision field = 0x00 00 00 01
© 2010 Cisco Systems, Inc. All rights reserved. FICON 27
QSA Accept QSA Accept sent if Security Features enabled
Bit Meaning 0 Fabric Binding 1 Static DomainID 2 Soft Zoning 3 Hard Zoning 4 SLAP 5 SRP 6 MS CT
Authentication 7-31 Reserved
0000 0011 Bit 0 & 1 are on
7654 3210 BIT numbers
© 2010 Cisco Systems, Inc. All rights reserved. FICON 28
ESA – Exchange Security Attributes
Used between Switches (ISL) to validate QSA information Passed during ISL link initialization Address by and to MDS Fabric Controller 0xFF FF FD ESA frame command code is 0x32 00 00 00
272.210534 ff.ff.fd -> ff.ff.fd 0xa 0xffff SW_ILS 0x32000000!
272.210594 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!
272.241828 ff.ff.fd -> ff.ff.fd 0xa 0xa SW_ILS SW_ACC (0x32000000)!
272.242168 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!
ESA
Class F Ack
ESA Accept
Class F Ack
© 2010 Cisco Systems, Inc. All rights reserved. FICON 29
RNID – Request Node Id Data
• Used to determine characteristics of node at other end
© 2010 Cisco Systems, Inc. All rights reserved. FICON 30
RNID Accept
Accept is broken into two sections:
Common ID Field - shows pWWN and nWWN. Ignored by FICON
Specific ID Data – shows SB-2, FCID, Name, Manufacturer, Model…..
© 2010 Cisco Systems, Inc. All rights reserved. FICON 31
LIRR, RLIR, DRLIR LIRR (Link Incident Record Registration) used to
register for RLIR (Registered Link Incident Record) - similar to SCR
RLIR used by N-Ports to send link incidents to registered ports - similar to RSCN
DRLIR (Distribute Registered Link Incident Records) used to send incidents to entire fabric
Switch receives DRLIR and send RLIR to registered N-Ports
LIRR sent to all Switches in path and Control Unit
Sent to 0xFF FF FA (MDS Management Server)
© 2010 Cisco Systems, Inc. All rights reserved. FICON 32
ELP – Establish Logical Path LPE – Logical Path Established
• ELP - Sent from Channel to Control Unit to verify optional parameters and verify logical path • LPE – Sent from Control Unit to Channel as verification of logical path and pass optional parameters • LACK – Close exchange
© 2010 Cisco Systems, Inc. All rights reserved. FICON 33
Idles FC Header
4 24 0 - 8160 4 4 Bytes:
Frame Format
FICON Header
32
Data Payload CRC
FC-SB-3 Frame Format
Reserved CH_ID Reserved CU_ID Device Address Reserved
IUI DH_FLGS CCW Number Reserved Token
Command / Data / Status / Control / Link Header
Reserved IU Count DIB Data Count LRC
SB-3 Header
IU Header
DIB Header
Bit 0 7 8 15 16 31 23 24
S O F
E O F
Idles
© 2010 Cisco Systems, Inc. All rights reserved. FICON 34
FICON Read Operation
Command DIB (READ)
Command DIB (RSP)
Data DIB
Data DIB
Status DIB
FICON Read Operation
FCP_READ
FCP_DATA
FCP_DATA
FCP_RSP
FCP Read Operation
FICON Channel
FICON CU
FCP Initiator
FCP Target
© 2010 Cisco Systems, Inc. All rights reserved. FICON 35
FICON Write Operation
Command DIB (WRITE)
Command DIB (RSP)
Data DIB
Data DIB
Status DIB
FICON Write Operation
FCP_WRITE
FCP_DATA
FCP_DATA
FCP_RSP
FCP Write Operation
FICON Channel
FICON CU
FCP Initiator
FCP Target
FCP_XFER_RDY
© 2010 Cisco Systems, Inc. All rights reserved. FICON 36
Flow Control
FICON uses standard FC-2 BB_Credit flow control FICON also uses IU Pacing:
Similar to Fibre Channel EE_Credits IU Pacing credit is sent to the channel by the CU Indicates the maximum number of IUs a channel can send before a command response IU is expected
CU
N_Port
Channel
N_Port F_Port E_Port E_Port F_Port Frames
R_RDYs
Frames
R_RDYs
FC-2 BB_Credit FC-2 BB_Credit
CMR CRR
IU Pacing
© 2010 Cisco Systems, Inc. All rights reserved. FICON 37
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 38
Linux Support
IBM zSeries supports Linux LPARs: Allows Linux to access ESCON/FICON storage Linux also supports open systems FCP:
Same adapters used for FICON and FCP Different firmware versions
Applications: CPU-intensive workloads Server/storage consolidation NPIV (like MIF for FCP channels)
zSeries server
FC FICON
LINUX FCP
FICON
z/OS FICON FICON
FC
FC FC FC
FC
FC FC
© 2010 Cisco Systems, Inc. All rights reserved. FICON 39
Disaster Recovery Applications
Data replication over FICON: Backup, snapshot, and disaster tolerance Local and remote replication Globally Dispersed Parallel Sysplex (GDPS) w/ IBM XRC IBM PtP VTS GDPS w/ IBM PPRC (runs over FCP) HDS HXRC
© 2010 Cisco Systems, Inc. All rights reserved. FICON 40
Disaster Recovery Applications XRC - eXtended Remote Copy
© 2010 Cisco Systems, Inc. All rights reserved. FICON 41
Channel Extension
Channel extension is different than distance extension: Distance extension simply means extending FICON over a WAN Channel extension involves protocol translation Channel extenders proxy some commands—e.g. writes—to increase performance over distance MDS does not do channel extension in SAN-OS 3.X
© 2010 Cisco Systems, Inc. All rights reserved. FICON 42
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 43
MDS FICON Requirements
Feature Explanation Predictable and persistent FCID allocation
Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be predictable and persistent.
FICON also requires uniform “port” byte (last byte in FC-ID) in FICON fabric.
Support for new LIRR, RLIR, and DRLIR ELS
Link Incident Record (LIR) ELS reports link level events in the fabric to the channel N-port.
Examples of link incidents are loss of signal or synchronization, primitive sequence timeout, Invalid primitive sequence, NOS received
Concept similar to RSCN
Report Node Identification (RNID) ELS
RNID is an ELS command, which can have common and specific Node data.
FICON defines a specific node data field format.
Large Buffer to Buffer Credit Allocation
FICON requires 60 B2B credit for 100 Km and 6 B2B credit for 10 Km.
MDS 9000 FICON supports up to 255 B2B Credits per Port + extended buffer credits
© 2010 Cisco Systems, Inc. All rights reserved. FICON 44
MDS FICON Requirements
Feature Explanation Insistent Domain ID Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be
predictable and hence Domain ID to be static in cascaded FICON switches.
In-order Delivery FICON requires in-order delivery for class 2 and class 3 frames.
Fabric Binding and EFMD ELS Fabric Binding feature prevents unauthorized switches joining the fabric based on switch WWNs.
Exchange Fabric Membership Data (EFMD) is used to distribute this information fabric wide.
Security Attributes (QSA and ESA)
FICON requires Fabric Binding and Insistent Domain ids to be enabled in the entire fabric.
Query Security Attribute (QSA) is used by Channels to ensure these features are turned on.
Exchange Security Attribute (ESA) is used by switches to validate these attributes during E port bringup.
© 2010 Cisco Systems, Inc. All rights reserved. FICON 45
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 46
What is FICON manager?
FICON Manager
Running process that manages functionality required to support FICON in a VSAN, doing the necessary checks for enabling FICON and holding all FICON specific configuration information.
© 2010 Cisco Systems, Inc. All rights reserved. FICON 47
FICON Manager
FICON Manager is responsible for: - Providing CMI interface for configuration - Serialization of FICON config - Maintain and apply FICON configuration files - Port Number Allocation and assignment - Port Address Allocation and assignment - Consistency Checks during enabling of FICON - Maintaining port numbers in FICON VSANs - Interact with ACL Manager for necessary ACLs during port bringup - Exchanging new ELS During E-port bringup
© 2010 Cisco Systems, Inc. All rights reserved. FICON 48
FICON Manager Role
FCD (CUP)
FC2
Vsan Manager
Port Manager
Others
SNMP
CLI
FICON Manager
Mainframe GUI
© 2010 Cisco Systems, Inc. All rights reserved. FICON 49
FICON Configuration Files
Up to 16 configuration files per FICON VSAN - Restricted to 8 alphanumeric characters
Shared lock for file access
Proprietary IBM format IPL contains the following attributes for each implemented port:
- Port Address Name - Host Control - Block - Prohibit Dynamic Connectivity Mask
- ‘show ficon vsan <x>’ will show saved IPL files
© 2010 Cisco Systems, Inc. All rights reserved. FICON 50
Port Number and Port Address Implemented/Unimplemented Port Number
Refers to any available port in the chassis Reflected in the table to the right
Installed/Uninstalled Port Only ports participating in FICON VSANs are
considered ‘installed’ i.e. If port 4 is in a FICON VSAN on a 16 port line
card in a chassis, then ports 0-3, 5-31 are considered ‘uninstalled’
Port number is the physical location on the switch Port address is:
- Port name - Block/Unblock state - Connectivity attributes - Same as port number unless port “swapped”
256 possible port number/addresses for 9509 (8 bits, some unimplemented)
0xFE and 0xFF reserved An FCIP tunnel must be explicitly bound to a port
number for FICON - Available port numbers depends on the slot the
IPS blade is located in A Port Channel must be explicitly bound to a port
number for FICON - Available port numbers depends on chassis ‘show ficon first-available port-number’ reveals next port-number free for allocation
Slot 91xx 92xx 9506 9509 9513 1 0 - 39 0 - 31 0 - 31 0 - 31 0 - 15 2 32 - 63 32 - 63 32 - 63 16 - 31 3 64 - 95 64 - 95 32 - 47 4 96 - 127 96 - 127 48 - 63 5 -- -- 64 - 79
6 -- -- 80 - 95 7 128 - 159 -- 8 160 - 191 -- 9 192 - 223 96 - 111 10 112 - 127 11 128 - 143 12 144 - 159 13 160 - 175
Max Physical* 40 64 128 224 528
Port Channels 40-55 64-79 128-143 224-249 224 - 249
© 2010 Cisco Systems, Inc. All rights reserved. FICON 51
Using DM to identify Port Addresses
Notice FICON port numbers are displayed
9513 - 16 ports per slot 9509 - 32 ports per slot
© 2010 Cisco Systems, Inc. All rights reserved. FICON 52
FICON States
Various states for FICON
STATE Meaning
Enabled User has configured FICON on a given VSAN.
Disabled User configures a non-FICON vsan.
Up FICON enabled and all prerequisites met
Down FICON enabled, but one or more prerequisites not satisfied
© 2010 Cisco Systems, Inc. All rights reserved. FICON 53
FICON Prerequisites
Before FICON is ‘up’, FICON manager must check prerequisite conditions:
In-order delivery enabled Fabric Binding Insistent Domain ID No conflicting persistent FCIDs
- Does not allow user configuration for reserved range: D.0.0 thru D.MaxPa.0 where:
D is domain MaxPa is max Port Addr implemented in FICON
© 2010 Cisco Systems, Inc. All rights reserved. FICON 54
FICON Port Initialization
Blocked/Unblocked State - A blocked port is configured ‘down’ by Port manager, at request of FICON
manager - Blocked ports will continue to transmit an Off-Line State (OLS) primitive
sequence. - Similar to shut/no shut, as blocked port must be reinitialized
- Exception is TE port: - Removal of the VSAN from the VSAN allowed list - Port Channels - Members get port channel configuration (block/unblock)
Prohibit State - FICON manager programs ACLs to allow communication between certain
ports and prohibit others - Similar concept to zoning - Port remains ‘up’
© 2010 Cisco Systems, Inc. All rights reserved. FICON 55
FICON Port Initialization ELS
Replies to Query Security Attributes (QSA) - For Fx, Responsible for sending:
- Fabric Binding - Insistent Domain ID - In-order delivery
Sends Exchange Security Attributes (ESA) - Upon bring up of (T)E Ports
Sends Request Node Identification (RNID) - Sent after QSA reply
© 2010 Cisco Systems, Inc. All rights reserved. FICON 56
FICON Port Swap Support
Port Swap feature - Must be supported for FICON directors - Enables swapping configuration on two physical ports
- FICON manager swaps all config except Port # - Ports left in “shut” state following…’no shut’ required
FICON Configuration
Including Port Address,
Block State, etc.
FICON Configuration
Including Port Address,
Block State, etc.
Physical Port A Physical Port B
© 2010 Cisco Systems, Inc. All rights reserved. FICON 57
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 58
FICON Control Unit Port (CUP) CUP protocol provides inband
management of switch from mainframe
In order to be classified as “FICON director”, CUP protocol needs to supported.
CUP specification is proprietary to IBM and is available only through special license agreement with them.
CUP can configure a port to be prohibited from another port, blocked, etc.
Complimentary to SNMP, CLI, and is serialized through CMI
FICON Manager holds all configuration and FICON specific information; FCD acts as a client
SANOS 3.0(2) View of FICON IPL File
© 2010 Cisco Systems, Inc. All rights reserved. FICON 59
FICON Control Device (FCD)
FCD process is responsible for: - Acquiring persistent pWWN for Control Device
- Uses Key-Based Chassis Specific (KCS) WWNs - Uses VSAN based nWWN
- Acquiring FCID from FPort server - 0xFE is well defined address required - FCID will be <domain, 0xFE, 00> for Control Device
- Device allegiance based Serialization of FICON config - Support for both SB-3 configuration and query commands
© 2010 Cisco Systems, Inc. All rights reserved. FICON 60
FICON Port Configuration
Block Block all communication from a port. In this state port is down and will not send any information.
Prohibit Each port (identified by port address) can block any other port. This is done via ACLs.
Port Address Name
Each port (address) can have a 24 character port address name.
Port Swap Reassigns port address, connectivity attributes and port address name of ports by swapping.
Service Required
This is set when FICON Director determines that some service is required on the port. Port is kept offline. Cleared when required service performed.
Port Statistics The MDS needs to maintain detailed statistics on each port. The statistics are reported based on port numbers and not port addresses
© 2010 Cisco Systems, Inc. All rights reserved. FICON 61
HCD - CUP
© 2010 Cisco Systems, Inc. All rights reserved. FICON 62
FCD Control Device
An example of an FCD Control Device on a FICON vsan:
switch2# show ficon control-device sb3 vsan 1977
Control Unit Image:0x80b9df4 VSAN:1977 CU:0x4dfe00 CUI:0 CUD:0 CURLP:(nil) ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
Control Unit Image:0x80bcbec VSAN:1978 CU:0x4efe00 CUI:0 CUD:0 CURLP:(nil) ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
switch2# show flogi database vsan 1977 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- sup-fc0 1977 0x4dfe00 22:00:00:05:30:00:46:e0 27:b9:00:05:30:00:46:df
Total number of flogi = 1.
Domain ID = 77 (0x4d) Well known port = 0xFE
pWWN assigned
© 2010 Cisco Systems, Inc. All rights reserved. FICON 63
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 64
CLI Configuration of FICON An example of using the ‘setup ficon’ script: switch1# setup ficon --- Ficon Configuration Dialog --- This setup utility will guide you through basic Ficon Configuraton on the system. Press Enter if you want to skip any dialog. Use ctrl-c at anytime to skip all remaining dialogs. Would you like to enter the basic configuration dialog (yes/no)[yes]:yes Enter vsan [1-4093] :3 vsan 3 does not exist, create it? (yes/no)[yes]:yes Enable ficon on this vsan? (yes/no)[yes]:yes The following options required for ficon will be enabled: In-order-delivery Fabric binding Configure domain-id for this ficon vsan (1-239) :3 Would you like to configure ficon in cascaded mode: (yes/no)[no]:yes Configure peer wwn (hh:hh:hh:hh:hh:hh:hh:hh): 20:00:00:0d:ec:0e:5b:80 Configure peer domain (1-239) :4 Would you like to configure additional peers: (yes/no)[no]: Enable SNMP to modify port connectivity parameters? (yes/no)[yes]: Disable Host from modifying port connectivity parameters? (yes/no)[no]: Enable active=saved? (yes/no)[yes]:no
Would you like to configure additional ficon vsans (yes/no)[yes]:no The following configuration will be applied: vsan database vsan 3 in-order-guarantee vsan 3 fcdomain domain 3 static vsan 3 fabric-binding enable fabric-binding database vsan 3 swwn 20:00:00:0d:ec:0e:5b:80 domain 4 fabric-binding activate vsan 3 zone default-zone permit vsan 3 ficon vsan 3 no active equals saved
swwn from remote switch
# show wwn switch
© 2010 Cisco Systems, Inc. All rights reserved. FICON 65
CLI Configuration of FICON A FICON port-channel configuration example: switch1# conf t Enter configuration commands, one per line. End with CNTL/Z. MDS9506(config)# int port-channel 1
Remember that PortChannels need to be assigned FICON port numbers, so in the next command we are determining what the next available FICON port number is.
switch1(config-if)# do show ficon first-available port Port number 128(0x80) is available
Next step is to bind the FICON port number, set the port type to an ISL (E_Port), and set it up so that only VSANs 1 and 10 can trunk across the PortChannel.
switch1(config-if)# ficon portnumber 0x80 switch1(config-if)# switchport mode e switch1(config-if)# switchport trunk allowed vsan 1 switch1(config-if)# switchport trunk allowed vsan add 10
Next we move interfaces fc1/9 and fc2/1 into PortChannel 1
switch1(config-if)# int fc1/9 switch1(config-if)# channel-group 1 force fc1/9 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel,then do "no shutdown" at both ends to bring them up
switch1(config-if)# int fc2/1 switch1(config-if)# channel-group 1 force fc2/1 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel,then do "no shutdown" at both ends to bring them up
© 2010 Cisco Systems, Inc. All rights reserved. FICON 66
CLI Configuration of FICON A FICON port-channel configuration example continued:
Next we no shut interfaces fc1/9, fc2/1, and PortChannel 1.
switch1(config-if)# int fc1/9, fc2/1 switch1(config-if)# no shut switch1(config)# int port-channel 1 switch1(config-if)# no shut
Next we repeat the configuration steps on the switch2.
switch2(config)# int port-channel 1 switch2(config-if)# do show ficon first-available port-number Port number 64(0x40) is available swich2(config-if)# ficon portnumber 0x40 switch2(config-if)# switchport trunk allowed vsan 1 switch2(config-if)# switchport trunk allowed vsan add 10 swich2(config-if)# int fc1/1 switch2(config-if)# channel-group 1 force fc1/1 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring them up
switch2(config-if)# int fc1/9 switch2(config-if)# channel-group 1 force fc1/9 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring them up
switch2(config)# int fc1/1, fc1/9 switch2(config-if)# no shut switch2(config)# interface port-channel 1 switch2(config-if)# no shut switch2(config-if)# end
© 2010 Cisco Systems, Inc. All rights reserved. FICON 67
CLI RLIR MDS-1# show rlir ? erl Show Established Registration List history Show link incident history internal Show RLIR internal recent Show recent link incident statistics Show RLIR Statistics
MDS-1# show rlir internal vsan 3
Internals for VSAN: 3 ------------------------
VSAN state : Active Ficon State : Up Fabric Name : 20:03:00:05:30:00:51:1f R_A_TOV : 10000 ms Local Domain : 0x3(3) Remote Domains : None
MDS-1# show rlir statistics vsan 3
Statistics for VSAN: 3 -------------------------
Number of LIRR received = 0 Number of LIRR ACC sent = 0 Number of LIRR RJT sent = 0 Number of RLIR sent = 0 Number of RLIR ACC received = 0 Number of RLIR RJT received = 0 Number of DRLIR received = 0 Number of DRLIR ACC sent = 0 Number of DRLIR RJT sent = 0 Number of DRLIR sent = 0 Number of DRLIR ACC received = 0 Number of DRLIR RJT received = 0
© 2010 Cisco Systems, Inc. All rights reserved. FICON 68
CLI RNID and Port attributes
MDS-1# show ficon vsan 5 portaddress 0x21 Port Address 33(0x21) is up in vsan 5 Port number is 33(0x21), Interface is fc1/3 Port name is Port is not admin blocked Prohibited port addresses are 250-253,255(0xfa-0xfd,0xff) Admin port mode is FX Port mode is F Peer is type 002064 model 104 manufactured by IBM
MDS-1# show ficon vsan 5 portaddress brief installed ----------------------------------------------------------------------------- Port Port Interface Admin Status Oper FCID Address Number Blocked Mode ----------------------------------------------------------------------------- 0x40 0x40 port-channel 5 off up TE -- 0xfe 0xfe -- off up -- --
© 2010 Cisco Systems, Inc. All rights reserved. FICON 69
CLI FICON File Configuration switch1# show run
ficon vsan 2 file IPL
switch1(config)# ficon vsan 2 switch1(config-ficon)# ? FICON Configuration: active Configure automatic saving of ficon configuration code-page Ficon configuration for codepage on a VSAN do EXEC command end Exit from configure mode exit Exit from this submode file Ficon file configuration on a VSAN host Enable host control of ficon no Negate a command or set its defaults portaddress Ficon configuration for portaddress on a VSAN snmp Enable snmp control of ficon
switch1(config-ficon)# portaddress 1 switch1(config-ficon-portaddr)# ? FICON config for portaddress: block Block a range of portaddress do EXEC command end Exit from configure mode exit Exit from this submode name Give name for a portaddress no Negate a command or set its defaults prohibit Prohibit communication with a range of portaddress
© 2010 Cisco Systems, Inc. All rights reserved. FICON 70
Creating FICON VSAN from DM
© 2010 Cisco Systems, Inc. All rights reserved. FICON 71
Creating FICON VSAN from FM
© 2010 Cisco Systems, Inc. All rights reserved. FICON 72
Creating/Modifying FICON Files from DM
© 2010 Cisco Systems, Inc. All rights reserved. FICON 73
FICON Configuration File (IPL)
Edit IPL file, Block – Prohibit – Name Ports
© 2010 Cisco Systems, Inc. All rights reserved. FICON 74
Creating Port Channel through FM
Determine the ISLs to add to the Port Channel Now select the Port Channel Wizard
© 2010 Cisco Systems, Inc. All rights reserved. FICON 75
Creating Port Channel through FM (cont’d)
Fill in Port Channel Information
Select a Port number for the logical PC
Port Channel is now created
© 2010 Cisco Systems, Inc. All rights reserved. FICON 76
Showing Port Channel through DM
Select Interface->Port Channels
All Port channel information is displayed
© 2010 Cisco Systems, Inc. All rights reserved. FICON 77
Swapping FICON Ports through DM
Select Ports to swap Select FICON->Swap Selected Ports
© 2010 Cisco Systems, Inc. All rights reserved. FICON 78
Swapping FICON Ports through DM (contd)
Notice the swapped ports
© 2010 Cisco Systems, Inc. All rights reserved. FICON 79
FICON VSAN Options
• Offline SW = Enables ability for Host to disable FICON or offline MDS • Sync Time = Enables ability for Host to sync MDS time with Host
• Port Control – Host = Enables ability for Host to Name, Prohibit, Block MDS ports • Port Control – SNMP = Enables ability for Device Manager to Name, Prohibit, Block MDS ports • Active=Saved = Enables ability for IPL (ficon running configuration) to be saved automatically and immediately
© 2010 Cisco Systems, Inc. All rights reserved. FICON 80
FICON Load Balancing
Packets of a flow (SID,DID) or an exchange (SID, DID, OXID) are received in the order they were sent. Some applications can not handle Out Of Order exchanges
FICON is one such application. In fact FICON uses two unique exchanges, one for the entire outbound exchange (For example a write chain) with an OXID allocated by the CH. The CU sends back a command response with a different OXID representing the inbound exchange
© 2010 Cisco Systems, Inc. All rights reserved. FICON 81
FICON Load Balancing
The FICON load balancing tool has been enhanced in the Fabric Manager 3.0 release.
© 2010 Cisco Systems, Inc. All rights reserved. FICON 82
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 83
Configuring Devices on Mainframe Consider the following example and IOCP definition
© 2010 Cisco Systems, Inc. All rights reserved. FICON 84
Configuring Devices on Mainframe
Configuring CHPID 89 online HCD
SDSF TSO Command Input
© 2010 Cisco Systems, Inc. All rights reserved. FICON 85
Configuring Devices on Mainframe Varying devices online
HCD
SDSF TSO Command Input
Vary devices D40 through D74 online. I/O errors occur because the volumes at these addresses are not initialized.
V D40-D74,ONLINE IEF503I UNIT 0D50 I/O ERROR IEF503I UNIT 0D51 I/O ERROR ...
Following command automatically executed based on the range that was varied on in previous command.
D U,,,0D40,00052,L=OZI-Z VARY RANGE DISPLAY IEE457I 10.55.06 UNIT STATUS 198 UNIT TYPE STATUS VOLSER VOLSTATE 0D40 3390 O PPRD40 PRIV/RSDNT 0D41 3390 O PPRD41 PRIV/RSDNT … 0D50 3390 OFFLINE /RSDNT
© 2010 Cisco Systems, Inc. All rights reserved. FICON 86
Displaying Devices on Mainframe
Notice the online state for configured CHPID 86
Notice the availability of devices configured in HCD
HCD
TSO
HCD
HCD
TSO
© 2010 Cisco Systems, Inc. All rights reserved. FICON 87
Displaying Devices on Mainframe Displaying CHPID 86 and 89 Matrix
52 Devices Unit = 3390
Begins at D40
CUP Device Unit = 2032
at well-known 0xFE
© 2010 Cisco Systems, Inc. All rights reserved. FICON 88
Displaying Devices on Mainframe Displaying Path Matrix for Device D40
CHPID with corresponding Destination Link Address Entry Link Address
© 2010 Cisco Systems, Inc. All rights reserved. FICON 89
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 90
New FICON Port Numbering scheme Port-numbers are user-configurable
Duplicates allowed - Port-number 1 can be assigned to fc1/1 and fc2/1
Assigned on per-slot or logical basis ficon slot <x> assign port-numbers <range> ficon logical-port assign port-numbers <range>
Default configuration - No change for 1, 2, 6 and 9 slot chassis - 13 slot chassis will have 11 slots w/ 16 port-numbers each
HA compatibility
© 2010 Cisco Systems, Inc. All rights reserved. FICON 91
Configuring Port-Numbers
switch1# sh module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 2 24 1/2/4 Gbps FC Module DS-X9124 ok 3 8 IP Storage Services Module DS-X9308-SMIP ok 4 16 1/2 Gbps FC Module DS-X9016 ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 8 4 10 Gbps FC Module DS-X9704 ok
Consider the following 9509 with LC modules in slots 2, 3, 4, and 8
Finding out what is assigned – this example show the defaults for a 9509
switch1# show ficon port-numbers assign ficon slot 1 assign port-numbers 0-31 ficon slot 2 assign port-numbers 32-63 ficon slot 3 assign port-numbers 64-95 ficon slot 4 assign port-numbers 96-127 ficon slot 7 assign port-numbers 128-159 ficon slot 8 assign port-numbers 160-191 ficon slot 9 assign port-numbers 192-223 ficon logical-port assign port-numbers 224-249
© 2010 Cisco Systems, Inc. All rights reserved. FICON 92
Configuring Port-Numbers
switch1# show ficon port-numbers assign ficon slot 1 assign port-numbers 0-31 ficon slot 2 assign port-numbers 32-63 ficon slot 3 assign port-numbers 64-95 ficon slot 4 assign port-numbers 96-127 ficon slot 7 assign port-numbers 128-159 ficon slot 8 assign port-numbers 160-191 ficon slot 9 assign port-numbers 192-223 ficon logical-port assign port-numbers 224-249
switch(config)# no ficon slot 1 assign port-numbers 0-31 switch(config)# no ficon slot 7 assign port-numbers 128-159 switch(config)# no ficon slot 9 assign port-numbers 192-223
Unreserve FICON port-numbers from an unused slot, slots 1, 7, & 9 in this case
Reserve port-numbers for slot 2, a 24 port card
switch(config)# ficon slot 2 assign port-numbers 128-151
© 2010 Cisco Systems, Inc. All rights reserved. FICON 93
Configuring Port-Numbers
switch(config)# do show ficon vsan 2 portaddress brief | inc fc2
0x80 0x80 fc2/1 off not installed -- -- 0x81 0x81 fc2/2 off not installed -- -- 0x82 0x82 fc2/3 off not installed -- -- 0x83 0x83 fc2/4 off not installed -- -- 0x84 0x84 fc2/5 off not installed -- -- 0x85 0x85 fc2/6 off not installed -- -- 0x86 0x86 fc2/7 off not installed -- -- 0x87 0x87 fc2/8 off not installed -- -- 0x88 0x88 fc2/9 off not installed -- -- 0x89 0x89 fc2/10 off not installed -- -- 0x8a 0x8a fc2/11 off not installed -- -- 0x8b 0x8b fc2/12 off not installed -- -- 0x8c 0x8c fc2/13 off not installed -- -- 0x8d 0x8d fc2/14 off not installed -- -- 0x8e 0x8e fc2/15 off not installed -- -- 0x8f 0x8f fc2/16 off not installed -- -- 0x90 0x90 fc2/17 off not installed -- -- 0x91 0x91 fc2/18 off not installed -- -- 0x92 0x92 fc2/19 off not installed -- -- 0x93 0x93 fc2/20 off not installed -- -- 0x94 0x94 fc2/21 off not installed -- -- 0x95 0x95 fc2/22 off not installed -- -- 0x96 0x96 fc2/23 off not installed -- -- 0x97 0x97 fc2/24 off not installed -- --
The output should reflect the current status for all the ports
© 2010 Cisco Systems, Inc. All rights reserved. FICON 94
Configuring Port-Numbers
Switch-9509(config)# do sh run | incl assign no ficon slot 1 assign port-numbers ficon slot 2 assign port-numbers 128-151 force no ficon slot 7 assign port-numbers no ficon slot 9 assign port-numbers
Also the running config will show the newly reserved ports
© 2010 Cisco Systems, Inc. All rights reserved. FICON 95
Configuring Port-Numbers
switch(config)# ficon logical-port assign port-numbers 16-31,224-249
Assign logical port-numbers for FCIP interfaces and Port-channels
Logical ports are shown in a show running config
Switch-9509(config)# do sh run | incl assign no ficon logical-port assign port-numbers no ficon slot 1 assign port-numbers ficon slot 2 assign port-numbers 128-151 force no ficon slot 7 assign port-numbers no ficon slot 9 assign port-numbers ficon logical-port assign port-numbers 16-31, 224-249 force
© 2010 Cisco Systems, Inc. All rights reserved. FICON 96
Objective & Agenda FICON Concepts
• Terminology • Topologies • Addressing • Frames
FICON Applications
MDS Requirements • FICON Manager • FICON Control Unit Process (FCD)
Configuring FICON • SAN-OS • Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
© 2010 Cisco Systems, Inc. All rights reserved. FICON 97
Fabric Binding Troubleshooting First, verify that identical Fabric Binding Databases are active on both
switches Debug information:
debug fabric-binding all (collect from both switches) Activate the fabric binding database on both switchs while the debug is running. fabric-binding activate vsan X Flap the port-channel or ISL interface (Only if customer can take the outage)
Collect the following show fabric-binding statis vsan X (x=vsan 500) show fabric-binding efmd statistics vsan X show fabric-binding database vsan X show fabric-binding database act vsan X show fabric-binding internal global show fabric-binding internal info vsan X show fabric-binding internal event show fabric-binding internal efmd event interface fcx/x or port-channel x
© 2010 Cisco Systems, Inc. All rights reserved. FICON 98
CUP Troubleshooting Collect the following:
show tech-support details show ficon control-device sb3 show ficon control-device internal info show ficon control-device internal event-history vsan x (x=ficon vsan) show ficon control-device internal event-history errors show ficon control-device internal event-history msgs show ficon control-device internal event-history vsan <x> sh ficon internal event-history errors sh ficon internal event-history msgs sh ficon internal event-history vsan <x>
From the Mainframe: MVS SYSLOG before, during, and after the failure. Output from D IOS,FICON
© 2010 Cisco Systems, Inc. All rights reserved. FICON 99
Basic Troubleshooting
Troubleshoot port problems (performance, failing) like any other case. Assume the switch is the problem and rule it out!
attach mod x term len 0 show hardware internal debug-info interface fc x/x clear asic-cnt list-all-devices clear asic-cnt all clear asic-cnt device-id 3
From Supervisor show flogi internal event interface fcx/y (for both CH and CU interfaces) show port internal event-history interface fcx/y show ficon internal event-history vsan x interface fcx/y. show ficon internal event-history vsan 1 show ficon internal event-history msgs show ficon internal event-history errors show ficon internal info
© 2010 Cisco Systems, Inc. All rights reserved. FICON 100
Basic Troubleshooting
The list of commands that are useful from SAN-OS(later NX-OS) CLI: Setup ficon ficon slot X assign port-numbers 48-63 show ficon port-numbers assign slot 3 show ficon port-numbers assign logical-port show ficon vsan X portaddress brief show ficon vsan vsan-id file name filename portaddress show ficon vsan 20 portaddress 8 counters
ficon swap portnumber old-port-number new-port-number ficon swap interface old-interface new-interface
clear ficon vsan 1 allegiance
ficon vsan 2 apply file SampleFile ficon vsan vsan-id offline ficon vsan vsan-id online show ficon control-device sb3 vsan X show flogi database vsan X
© 2010 Cisco Systems, Inc. All rights reserved. FICON 101
Basic Troubleshooting continued
Gather the following Mainframe displays before and after any failures or recreates
D M=DEV(nnnn) D M=CHP(mmm) Collect the MVS SYSLOG before, during, and after the failure. Collect any messages that may appear on Hardware Console, for example, "LINK LEVEL ERROR - FICON PLOGI ELS ERROR",
© 2010 Cisco Systems, Inc. All rights reserved. FICON 102
Summary / Review
FICON uses the Single-Byte Command Code Sets-3 (SB-3) and
Single-Byte Command Code Sets-4 (SB-4)
Three types of FICON topologies: Point-to-point, Switched point-to-
point (via a FICON Director) ,Cascaded FICON Directors (through two FICON Directors)
For FICON, there is a maximum of 1 hop from CH to CU
Allow for fabric-binding between FICON fabrics
Utilize ‘setup ficon’ command
Utilize GUI (FM - DM) for basic troubleshooting / configuration
© 2010 Cisco Systems, Inc. All rights reserved. FICON 103
References FICON Planning and Implementation Guide:
http://www.redbooks.ibm.com/abstracts/sg246497.html?Open MDS FICON Configuration Guide:
http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/4_1/configuration/guides/cli_4_1/ficon.html
MDS Fabric Manager Configuration Guide http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/4_1/configuration/guides/fm_4_1/ficon.html
© 2010 Cisco Systems, Inc. All rights reserved. FICON 104
Question and Answer…
Thanks for attending this session and Good Luck in your CCIE Journey
© 2010 Cisco Systems, Inc. All rights reserved. FICON 105