Linux/390A tale of two A tale of two

PortsPorts(C) Neale Ferguson - CAI, published under the terms of GPL.

Agenda• Introduction to S/390• VM & Open Source• i370-ibm-linux• s390-ibm-linux• I/O layer & Virtualization• Applications• MVL - Virtual Penguin Power

Linux/390Background

Introduction to S/390• Descendent of S/360• Upwardly compatible architecture• EBCDIC character set• Big endian• 16 GPR, 4/16 FPR, 16 CR• 2 tier paging schema• Multiple address spaces

Introduction to S/390

• Huge range of processors supported:– Cards that fit in a lap top– Desktop and tower models– 16-way SMP configurations

• Emulators for your PC– Hercules– Flex-ES

Introduction to S/390

• Initially, batch oriented O/S– PCP, TOS, DOS & OS/360– OS/MFT OS/MVT – OS/VS1, OS/VS2– MVS, DOS/VS– OS/390, VSE/ESA

Introduction to S/390

• Time-share & Hypervisors– CP-40, CP-67, VM/370, VM/SP,

VM/ESA– Virtualize a real machine

•Padded cell for multiple users•“Guest” operating systems•Even run VM under VM

Previous UNIX Attempts

• True ASCII with major APIs– AIX/370 (defunct)– AIX/ESA (defunct)– UTS (alive and kicking)– OSIRIS

• $$$$

VM & Open Source

• VM/370 totally source maintained with significant user input

• VM/SP owned by IBM but source available

• VM/ESA most source but OCO and OCM pieces

• IBM now have “rediscovered” Open Source

Linux/390Two Ports in a

Storm

i370-ibm-linux• Started by Linas Vepstas• Developed port of GCC & GLIBC

based on work of David Pitts• Basic kernel• I joined in mid 1999• Developed architecture

– Memory management– Process– I/O fundamentals

i370-ibm-linux

• Developed under VM and Hercules• Trace facilities of VM indispensable• The LINUX-VM listserver was set up

to allow the world to contribute• We were aware of rumors and some

facts about an internal IBM project porting Linux to S/390 as well

s390-ibm-linux

• Skunk-work project within IBM• Kernel running by May 1999• Activity kept secret• Constant rumours• Published mods on December 18• Now actively working with

community• Rolling mods into 2.3.x

s390-ibm-linux

• ASCII• IEEE floating point• Runs:

– Bare iron– In an LPAR– In a virtual machine

Linux/390I/O Layer

I/O Layer

• S/390 I/O is “channelized”• Work performed by channels• Device eccentricities hidden from

O/S• Single mechanism of talking to

devices• DMA• Interrupt-based

I/O Layer

• New layer of abstraction• IRQ structure used• New major devices:

– /dev/ddxn - S/390 DASD (ECKD)– /dev/mndx - VM minidisks– /dev/ctcx - CTCA & LAN– /dev/iucv - Network (LAN) device– Dumb tty and hardware console

Linux/390

Address Spaces

Intel Address Spaces

S/390 Address Spaces

S/390 Address Spaces

• Kernel runs in Primary Space mode• User programs run in Home Space

mode• Copy to/from user just a MVC(L/E)

in Access Register mode with AR set for kernel/user address spaces

• Compare this to some of the other elaborate schemes used

Linux/390Applications

Applications

• Kernel, tools and file systems were available from Marist College

• “Porting” of packages commenced

• Porting generally consists of an update to config.sub/guess to recognize s390-ibm-linux

Applications

• THE 3.0• Regina 0.08h• OpenSSL• linuxconf 1.17r2• wxxt-1.67• Postgresql-6.5.3• Apache 1.3.9• Samba 2.0.6

• sendmail 8.9.3• bind 8.2.2-p5• sshd 1.2.27• hercules 1.46• BOCHS • Big Brother 1.3a• Lynx 2.8.2• Emacs 20.5a

Applications

• ncFTP 3.0b21• PGP 5.0i• cron 3.0.1• PHP 3.0.14• ytalk 3.1.1• perl 5.005_3• ltxml 1.1

• GIMP• mySQL (in

progress)• X11• vnc-3.3.3r2

Linux/390 in Action

Linux/390 in Action

Linux/390 in Action

Linux/390 in Action

Linux/390 in Action[root@penguinvm cpint]# more /proc/cpuinfovendor_id : IBM/S390# processors : 3bogomips per cpu: 52.42processor 0: version = FF, identification = 045226, machine = 9672processor 1: version = FF, identification = 145226, machine = 9672processor 2: version = FF, identification = 245226, machine = 9672[root@penguinvm cpint]# more /proc/interrupts CPU0 CPU1 CPU2 0: 2964 2883 3421 none ctc0 1: 5840 2941 1920 none ctc0 3: 2 0 0 none mnd 4: 2 0 0 none mnd 5: 2 0 0 none mnd 9: 50 34 28 none 3215 terminal driverNMI: 0IPI: 0

Linux/390 in Action[root@penguinvm cpint]# more /proc/devicesCharacter devices: 1 mem 2 pty 3 ttyp 4 ttyS 5 console 10 misc254 cpint

Block devices: 1 ramdisk 7 loop 64 mnd 80 dasd

Linux/390 in Action[root@penguinvm cpint]# hcp q all | moreSTORAGE = 0128MXSTORE = noneCPU 00 ID FF04522696720000 (BASE)CPU 01 ID FF14522696720000CPU 02 ID FF24522696720000DASD 0101 3380 VMM600 R/O 20 CYL ON DASD 0742 SUBCHANNEL = 000DDASD 0190 3380 VMM785 R/O 178 CYL ON DASD 0702 SUBCHANNEL = 000EDASD 0191 3380 LINRES R/W 110 CYL ON DASD 06FE SUBCHANNEL = 0003DASD 0193 3380 LINRES R/W 1131 CYL ON DASD 06FE SUBCHANNEL = 0004DASD 0194 3380 LINRES R/W 597 CYL ON DASD 06FE SUBCHANNEL = 0005DASD 019D 3380 230RES R/O 108 CYL ON DASD 07E6 SUBCHANNEL = 000FDASD 019E 3380 VMM628 R/O 100 CYL ON DASD 074F SUBCHANNEL = 0010DASD 0320 3380 VMM612 R/O 75 CYL ON DASD 06C9 SUBCHANNEL = 0002DASD 0991 3380 LINRES R/W 365 CYL ON DASD 06FE SUBCHANNEL = 0006DASD 0992 3380 LINRES R/W 276 CYL ON DASD 06FE SUBCHANNEL = 0007DASD 0993 3380 LINRES R/W 175 CYL ON DASD 06FE SUBCHANNEL = 0008CTCA 0C10 COUPLED TO TCPIP31 0C10 SUBCHANNEL = 0000CTCA 0C11 COUPLED TO TCPIP31 0C11 SUBCHANNEL = 0001

Linux/390Virtual Penguin

Power

Multiple Virtual Linux

• Instead of a box per server connected by metres of wire

• A virtual machine per server connected by storage sharing data at 500MB/s

• DASD may be shared• DASD may be located 43Km away• “Virtual Penguin Power”

VPP

• VM is another device Linux can talk to

• Huge data cache• Resource sharing and capping• Need a new server? Just spend 5

minutes and add a new virtual machine

• Error recovery and reporting

VPP in Action• Tests conducted by David Boyes• G5 processor with 2 real CPUs• 2750 Linux/390 virtual machines

– 1000 with 4 vCPUs– 500 with 2 vCPUs– 250 with 8 vCPUs

• Booted concurrently• Running Apache serving same

page• VM didn’t bat an eyelid

Some Final Words“Nice(?!) to know, you can even boot Win95 under Linux for S/390 using the BOCHS x86-Emulator.”

“Fine, you sick perverts. You leave me no choice. When I can get the disk space and Copious Free Time for it, I'll build Hercules on Linux under Bochs under L/390. Or maybe I'll just port Basilisk II so you can have your Mac apps too.”

Further Information

• http://penguinvm.princeton.edu• http://linux390.marist.edu• http://linas.org/linux/i370.html• http://www.ibm.com/s390/vm/linux/• http://www.ibm.com/developer/

opensource/• http://www.funsoft.com

Trademarks

• Linux is a trademark of Linus Torvalds• S/390, VM/ESA, OS/390, VSE/ESA, AIX are

trademarks of IBM• UTS is a trademark of Amdahl• OSIRIS is a trademark of HDS• Flex-ES is a trademark of Fundamental

Software Inc.