Practical Linux Administration with IBM Wave

Sean P McBridez/VM, Linux, IBM Wave Specialist

zGrowth NA

Agenda

• Demo: Build three RHEL Guests via IBM Wave Cloning and Scripts

• Coffee Talk: 3270-based Linux Console 101• Demo: Use Linux Console to fix LVM• Coffee Talk: CLC Console 101• Demo: Use CLC Console to fix TCP/IP• Coffee Talk: ZIPL (grub for Linux on z)• Demo: Use Rescue System to fix ZIPL

Demo One: Build three RHEL Guests via IBM Wave Cloning and Scripts

• DBSERV: RHEL guest running MySQL database with Census 2010 Data

• APPSERV: RHEL guest running PHP web app that queries and displays data from DBSERV

• RESCULNX: RHEL Guest to be used as rescue environment

SSH Shell is the Sweet Spot

• The SSH shell is universal across platforms and offers familiarity, making SSH…

SSH is not always Possible

3270-based linux console 101Coffee Talk

Why use the Linux Console?

1. You want to watch the Linux guest boot up or shutdown completely to look for error messages.

2. You want to feed the Linux guest kernel parameters at boot.

3. SSH sessions are impossible to establish due to inability to fully boot errors in the guest’s TCP/IP configuration and I was unable to start a CLC session.

How do I launch the Linux Console?

• Right-click on guest• Hover on Access• Click on 3270 Linux

Console

How do I launch the Linux Console?

• Enter the guest’s z/VM password

• Click Login

I’m in… What is all this?

24 x 80 Display Area

Command Line and Go/Break

• All commands typed into Command Line• Hit Enter on Keyboard or Go button to

execute• Type ^C (equivalent of Ctrl+C) in Command

Line or click Break to kill Linux Process• You can also actually type Ctrl+C if the focus is

on the Command Line field

Status Line

• Keyword showing the Status of the Console

• Name of Linux Guest

The Linux Console shows RUNNING

• Normal Operation: This is the state that you want your Linux console to be in.

• Your console can receive messages from Linux• You can send messages to Linux

The Linux Console shows VM READ

• Root Cause: You hit enter without entering text in the command line. This causes the console to send an interrupt that the shell cannot resolve.

• Effect: Linux messages are queuing because the console cannot be reached.

• Solution: Hit the enter key again to return to RUNNING. All queued messages will appear on screen at once.

The Linux Console shows MORE…

• Root Cause: The terminal screen is full and does not have enough space to display one or more messages from Linux.

• Effect: Linux messages are queuing because the console cannot be reached.

• Solution: Hit the 3270 Clear key or wait for the 3270 terminal to automatically clear the screen (50 seconds by defaut)

The Linux Console shows HOLDING…

• Root Cause: The terminal screen is full and does not have enough space to display one or more messages from Linux. This caused MORE… to be displayed, and you hit the enter key, causing the status to change to HOLDING…

• Effect: Linux messages are queuing until the console is able to display them. The terminal is holding the current screen until manually cleared.

• Solution: Hit the 3270 Clear key

Key Pad Button and Panel

• The Key Pad button brings up the Key Pad panel.

• The Key Pad panel is filled with special 3270 terminal keys.

• Don’t worry! You really only need the Clear Key

Where is the Clear Key?

Where is the Clear Key?

3270 Juijitsu: Auto-Clear Super Fast

• If you are going to execute a long running command that produces a bunch of output, you can change how long your terminal waits before auto-clearing the MORE… state

• #CP TERM MORE 0 0 causes the screen to clear immediately instead of displaying MORE…

• #CP TERM 10 50

Why use the Linux Console?

1. You want to watch the Linux guest boot up or shutdown completely to look for error messages.

2. You want to feed the Linux guest kernel parameters at boot.

3. SSH sessions are impossible because Linux cannot boot or bring up the TCP/IP stack.

Demo: Use Console to fix LVM

• Scenario: When you arrive at work, your Inbox is filled with e-mails that the database for web application is down. You also have an automated notification from your storage administrator that a disk in your DS8000 system failed, and it impacted one of your guests.

Demo: Use Console to fix LVM

• Red Charm signifies that the IBM Wave server is unable to discover the guest’s IP address or establish TCP/IP connectivity.

Demo: Use Console to fix LVM• Once in Console: The

guest reports that an error occurred during the file system check.

• Either enter root password to login

• Or Enter ^D to continue• The String “^D” is the

3270 equivalent of Control-D on standard terminals

CLC Console 101Coffee Talk

Why use CLC?

• Full-screen editors like vim or emacs cannot be used via 3270 Linux Console

• Line-mode editors like ed and sed editors are unfamiliar and have complicated syntax

• CLC “screen scrapes” the 3270 Linux Console to construct an SSH-like experience with a GUI-based text editor.

• Perfect for fixing TCP/IP Issues

When should I use CLC?

1. You are unable to establish an SSH session with the guest.

2. Based on the 3270-based Linux Console, Linux booted cleanly to the login prompt.

3. You have determined that you need to correct TCP/IP configuration and reset the network interface to enable SSH sessions.

How do I launch CLC?

• Right-click on guest• Hover on Access• Click on CLC Access

SSH-like Navigation

Windowed CLC Editor

Demo: Use CLC to fix TCP/IP

• Scenario: After repairing the disk error on your Database Server, you attempt to do the same with your Application Server. Unfortunately, the root filesystem is unrecoverable. You restore a backup of the Application Server, but it turns out that the ifcfg-eth0 settings in the backup still point to a VLAN that you previously had to migrate off of.

Demo: Use CLC to fix TCP/IP

• Red Charm signifies that the IBM Wave server is unable to discover the guest’s IP address or establish TCP/IP connectivity.

Demo: Use CLC to fix TCP/IP

• Once in Console: The guest boots and presents a login prompt

• However, the APPSERV guest cannot be pinged

Zipl & linux rescue systemsCoffee Talk

zipl: Linux on z bootloader

• Equivalent of grub, the bootloader for x86• Allows definition of a menu of various sets of

kernels and kernel parameters• Configured by modifying zipl.conf file located

in /etc and running the zipl utility to read the configuration file and make the changes

• Also accepts par

Example zipl.conf file[defaultboot]

timeout=5default=2.6.32-431.11.2.el6.s390xtarget=/boot/

[2.6.32-431.11.2.el6.s390x]image=/boot/vmlinuz-2.6.32-431.11.2.el6.s390xramdisk=/boot/initramfs-2.6.32-431.11.2.el6.s390x.imgparameters="root=/dev/disk/by-path/ccw-0.0.0200-part1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD KEYTABLE=us

cio_ignore=all,!0.0.0009 rd_DASD=0.0.0901 rd_DASD=0.0.0900 crashkernel=auto SYSFONT=latarcyrheb- sun16 rd_DASD=0.0.0200 rd_NO_DM”[and other options…]

Example zipl menu at boot

Challenges with zipl and IBM Wave

• The zipl menu typically selects the default Linux kernel/parameter option after five seconds.

• The syntax for passing information to ZIPL is quite complex.

• IBM Wave currently lacks the ability to pass ZIPL parameters via the Activation task.

Use the IPL command in the 3270-based Linux Console

• If you are in a recovery situation, where Linux panics on boot, you can immediately pass the IPL command to the z/VM Control Program

• By using the LOADPARM and PARM parameters, you can pass arguments to zipl at boot.

• If you are not in a recovery situation, first issue #cp signal shutdown to cleanly bring down the Linux guest.

Example of using IPL command to pass ZIPL parameters

• #cp ipl 0200 loadparm 0 parm single• The loadparm keyword selects an option from

the zipl menu. 0 = Default. 1 = first item on list. 2 = second item on list. Etc.

• The parm keyword dynamically allows you to pass a string of kernel parameters up to 64 characters long.

Linux Rescue Systems

• If you can’t fix Linux via bootloader settings or kernel parameters, you need to mount the key filesystems of the damaged Linux guest to a separate Linux rescue system.

• In the x86 universe, this is often handled by bootable rescue USBs or CDs or ISOs. In the z universe, this is typically handled by a tiny Linux guest dedicated for this purpose.

Why can’t I just boot from an ISO?

• Linux on z doesn’t have a virtual CD drive• Linux on z had virtual SPOOL devices• The bootable media fed to the SPOOL devices

at boot lack most rescue utilities and need to pull these resources from a file server

• While it’s possible to build bootable rescue media that can just be fed into the reader, it can’t be easily built or bought.

How might I use a rescue system?

• Start SSH session to rescue system• Attach key storage devices of broken system• Mount partitions to /mnt/rescue• Mount proc to /mnt/rescue/proc• Chroot to /mnt/rescue• Perform Problem Determination and Repair• Exit Chroot, unmount filesystems, and detach

storage devices

Demo: Use Rescue System to fix ZIPL

• Scenario: After a hectic day repairing damage caused by storage failures, you go home for some much needed rest. The next day, you discover that your app server panics at boot because you accidently damaged the zipl.conf file late the night before.

Demo: Use Rescue System to fix ZIPL

Closing Thoughts

• Not everything matches up 1:1 with x86 Linux on ESXi

• SSH and that IBM Wave actions you experienced during the test drive handle 90% of day-to-day

• The 3270-based Linux console, the CLC tool, and the dedicated Linux rescue system are for the most advanced 10% of tasks. The tools are different from the x86 world, but they are fairly easy for Linux sysadmins to learn.

Questions???