Euro mainframe report cover page

v6.8

Peerstone Research Paper

Benefits of Mainframe Competition for the European Economy

Jeff Gould

Peerstone Research

January 15, 2009


Jeff Gould January 15 2009 v6.8 1

Table of Contents

1. z/OS competition could save Europe $48 billion over 20 years ......... 2

Defining the scope of IBM’s mainframe monopoly................................................. 2 Table 1: IBM mainframe stack exit points for legacy applications .......................... 5 Estimating the cost to Europe of IBM’s mainframe monopoly ................................ 5 Table 2: Savings from European mainframe competition ...................................... 9 Mainframe savings from all sources could reach $100 billion over 20 years ........ 10

2. Legacy z/OS: migration is not an option .......................................... 11

Legacy z/OS applications are worth at least $5 trillion......................................... 11 Moving legacy applications: “migration” vs. “virtualization” .................................. 11 Why z/OS migration is so hard ........................................................................... 12 Successful z/OS migrations are extremely rare ................................................... 15

3. Virtualization is key to z/OS competition .......................................... 17

Replacement of legacy z/OS applications is costly and risky ............................... 17 z/OS virtualization would spur mainframe market competition ............................. 18 Table 3: IBM mainframe stack revisited .............................................................. 23

4. Appendix A: z/OS-compatible server prices .................................... 24

Identifying alternative z/OS server platforms ....................................................... 24 Table 4: Common enterprise server microprocessors ......................................... 25 Table 5: Enterprise Server Classes .................................................................... 25 Computing market price equivalents for IBM mainframes.................................... 26 Table 6: Enterprise server configurations and prices ........................................... 29 Intrinsic product features do not explain IBM mainframe prices ........................... 30 Table 7: Shared architectural features of enterprise servers ............................... 32

5. Appendix B: Portability layers and migration ................................... 33

Defining portability layers ................................................................................... 33 When migration works: the case of SAP R/3 ....................................................... 34 When migration doesn’t work: the case of IBM DB2 ............................................ 36

6. Appendix C: Mainframes and one monopoly rent theory ................. 39

One IBM mainframe monopoly or several? ......................................................... 39 Author’s Biography ............................................................................................. 43



1. z/OS competition could save Europe $48 billion over 20 years1

Defining the scope of IBM’s mainframe monopoly

The goal of this paper is to quantify the costs imposed on the European economy by the absence of competition in the market for computers capable of running IBM’s z/OS mainframe operating system. z/OS is the modern-day 64 bit incarnation of a line of mainframe operating systems that stretches back through variants such as OS/390 and MVS all the way to the 24 bit OS/360 first introduced in 1964. While IBM continues to sell other mainframe operating systems such as VM and VSE, z/OS is by far the most widely used IBM mainframe OS and for the purposes of this paper we will consider it to be representative of the entire family2.

From the late 1970s through the end of the 1990s the market for computer hardware compatible with IBM’s mainframe operating systems was competitive. Under pressure from the U.S. Department of Justice, IBM agreed to license its mainframe operating systems for use on so-called “plug compatible machines” manufactured by companies such as Amdahl (later acquired by Fujitsu) and Hitachi3. In 2000, however, both Amdahl and Hitachi abruptly withdrew from the market, citing the difficulty of following IBM in the development of mainframe hardware based on a 64 bit architecture4. Subsequent to the withdrawal of the plug compatible vendors, IBM has systematically refused to renew its long-standing policy of licensing z/OS for use on competing hardware offered by vendors such as PSI and T3 Technologies. The result, according to market researchers, has been a “dramatic change” in IBM’s pricing behavior. Instead of the steep annual declines in the price per MIPS which had long

1 We use dollar values instead of euros throughout this paper because the IDC server market sizing data we refer to is expressed in dollars. 2 For background, see “History of IBM Mainframe Operating Systems”, Wikipedia [http://en.wikipedia.org/wiki/History_of_IBM_mainframe_operating_systems]. VM, now rebaptized by IBM as z/VM, is a so-called hypervisor capable of hosting virtualized versions of other operating systems, including z/OS itself and Linux. VSE, also known as z/VSE, is a less modern OS generally used on smaller IBM mainframes. 3 See “The rise and fall of plug-compatible mainframes”, Takahashi, S., Annals of the History of Computing, IEEE, Vol. 27, Issue 1, Jan.-March 2005. 4 According to documents later published by PSI, Amdahl had worked on an Itanium-based microcode implementation of IBM’s 64 bit hardware architecture in the late 1990s, but apparently decided not to pursue it (see “Itanium as a Horizontal Microcode Engine for Legacy Architecture Enablement”, Ron Hilton, PSI Founder & CTO, 2007 [rogue.colorado.edu/EPIC6/EPIC-6-Keynote-PSI.pdf]). In the two year period following Hitachi’s withdrawal from the plug compatible market, IBM agreed to sell its disk drive business to the Japanese company and gave it a contract to manufacture certain components of IBM’s new z family of mainframes.

http://en.wikipedia.org/wiki/History_of_IBM_mainframe_operating_systems



been the norm in the IBM-compatible mainframe market, prices flattened significantly after 20005. There can be little doubt that, owing to IBM’s actions, the market for computers capable of running IBM’s mainframe operating systems is now accurately characterized as a monopoly.

A complete mainframe computing solution consists of several distinct but complementary components, the combination of which is commonly referred to as a stack (see Table 1). The controlling element in IBM’s mainframe monopoly is z/OS. Many large organizations have over a period of decades expended large sums of money to develop mission-critical business applications whose functionality is highly taylored to their needs and which require z/OS to operate. For reasons that we will discuss in detail later in this paper, the great majority of these extremely valuable legacy applications cannot migrate away from z/OS to other enterprise operating systems such as Linux, Unix or Windows Server (LUW)6. Consequently, these customer organizations are trapped in the z/OS market by the very scale of their own prior investments in that market. They have no choice but to pay monopoly prices to IBM for the use of its mainframe operating system and related mainframe middleware if they wish to continue using their legacy applications. IBM’s mainframe software prices, which are set as a function of the capacity of the underlying IBM hardware system, often range into the hundreds of thousands or even millions of dollar per month.

During the decades when IBM allowed competition in the mainframe hardware market, its monopoly extended only to z/OS and related IBM middleware. With the end of z/OS licensing to competitors, however, that monopoly now also extends to the underlying computer hardware, i.e. the family of IBM-brand server systems known as System z. In this paper we will interchangeably refer to IBM-brand computers compatible with z/OS as “servers”, “mainframes”, or “mainframe servers”. This is because a detailed examination of the architecture of System z mainframes reveals that these machines are fundamentally similar in both their design and constituent components to computers supplied by other vendors designed for use with operating systems such as Linux, Unix or Windows Server and which are commonly known as “servers”.

This usage, suggested by common sense and adopted in order to ease understanding, does not imply that LUW-compatible servers can in any way “substitute” for z/OS-compatible mainframe servers. As we shall see later in some detail, there are fundamental technical constraints that prevent most legacy z/OS applications from moving to other operating systems. It is precisely the existence of these obstacles to market exit that make possible the IBM mainframe monopoly. The obstacles to moving applications off z/OS exist regardless of the nature of the underlying server hardware. In particular, they exist whether this hardware is an IBM-brand System z machine or some other kind of server 5 See “Mainframe Processor Pricing History”, Graham, z Journal, Feb 2006 [www.zjournal.com/index.cfm?section=article&aid=346#]. 6 For convenience, we will sometimes refer to the Linux, Unix and Windows Server operating systems by the acronym “LUW”.

http://www.zjournal.com/index.cfm?section=article&aid=346#



that has been modified by a technology such as a software-based virtualization of the z/Architecture instruction set to make it compatible with z/OS.

Later in this paper we will review several ways in which the z instruction set can be implemented using virtualization techniques on suitable microprocessors of non-IBM design and manufacture, including Intel Itanium, Intel Xeon and AMD Opteron7. Although these implementations can support z/OS very effectively and therefore have the potential to spur the creation of a competitve market for z/OS-compatible servers, IBM’s restrictions on z/OS licensing do not currently allow such a business model to be executed. Therefore, while our use of the term “server” to describe both z/OS-compatible servers and LUW servers does not imply that LUW servers in any way “compete” with IBM System z mainframes, it is compatible with the view that System z “servers” offers no intrinsic differentiation in functionality or performance compared to other kinds of servers capable of supporting the z/Architecture instruction set and which may also be capable of supporting LUW operating systems.

Because z/OS is proprietary software available only from IBM, and because the company has elected to abandon its decades-old practice of licensing its mainframe operating systems for use on compatible computers from other vendors, customers who wish to use z/OS have no choice but to purchase IBM mainframes in order to run it. By this tying action IBM has extended its historic monopoly on the intellectual property embedded in its z/OS software to the adjacent market for z/OS-compatible server hardware8.

7 It would be perfectly possible to include IBM’s own Power6 chip to this list of possible z/Architecture instruction set virtualization platforms. As we detail in Appendix A, the current Power6 and z10 (mainframe) chips are closely related in both their design and manufacturing characteristics. It has been widely reported in the industry press, although never officially confirmed by IBM, that IBM ultimately intends to converge the Power and z chips into a single microprocessor platform. See for example “Project ECLipz Surfaces, But Not the Way You Think”, Morgan, IT Jungle, Nov. 2007 [www.itjungle.com/tfh/tfh110507-story04.html]. 8 For a discussion of whether IBM’s mainframe monopoly should be treated as indivisible or on the contrary as several related but distinct monopolies, see the discussion of “one monopoly rent” theory and recent criticisms of its applicability to antitrust in Appendix C.

http://www.itjungle.com/tfh/tfh110507



Table 1: IBM mainframe computing stack, showing possible exit points for legacy applications

Legacy Mainframe Application

(e.g. compiled COBOL code)

IBM Middleware

(e.g. CICS, DB2, IMS)

z/OS system level interfaces (APIs) Application Migration

IBM z/OS

System z Instruction Set (implementation of IBM z/Architecture Principles of Operation)

Instruction Set Virtualization

Server Hardware compatible with System z Instruction Set

Compatible Peripherals (e.g. storage,

networking, printing)

source: Peerstone

Estimating the cost to Europe of IBM’s mainframe monopoly

Our methodology for estimating the cost to Europe of IBM’s monopoly on z/OS-compatible server computers is straightforward. We compare current European expenditure on monopoly-priced IBM-brand System z servers with the cost of competitively-priced non-System z servers that are capable in principle of supporting the z/OS operating system with equivalent functionality and performance. The difference between these two values represents the annual cost to Europe of the IBM monopoly. We do not attempt to address the pragmatic question of how long it would take monopoly prices in the z/OS-compatible server market to fall to competitive market levels after the lifting of IBM’s restrictions on the licensing of z/OS to alternative hardware vendors. Instead, we simply measure the gap between today’s costs and competitive market pricing. Because the expected lifetime of existing legacy z/OS applications does not have a presently foreseeable end, we also provide an extrapolation of the annual cost of the IBM monopoly to Europe over a 20 year period.

We acknowledge that our estimate of the cost to Europe of IBM’s mainframe monopoly is a model, not reality itself. However, we have sought at every point to formulate the quantitative assumptions behind our model as precisely and as explicitly as possible. Objections to the conclusions implied by the model, if they are to be considered reasonable, should meet the same standards. In particular, they should explictly state which of our assumptions have been modified or rejected, with what justification, and what alternative quantitative values are proposed.



The foundation of our methodology is the claim that we can identify competitive market prices for alternatives to standard System z servers that are capable in principle of supporting z/OS with equivalent performance. We employ two distinct approaches for identifying such prices.

§ In the first approach, we compare System z servers priced by IBM for z/OS use with identical System z servers priced by IBM for Linux use. This comparison is made possible by the fact that the z10 (or previous z family) microprocessors sold by IBM for use with Linux are physically identical to those sold for use with z/OS except for minor modifications in the microcode which prevent the former from executing z/OS code. IBM z family microprocessors with this “Linux-only” limitation are known as IFLs (Integrated Facility for Linux)9.

§ In the second approach, we develop a method for comparing the performance of System z and non-System z servers in terms of the standard MIPS benchmark of mainframe computing performance. Using this MIPS-equivalence method we identify System z and non-System z server configurations of comparable performance and then contrast their published market prices. The price differential between between an IBM-brand System z server and a non-System z server capable of running z/OS with equivalent performance represents the cost of IBM’s monopoly on z/OS-compatible server hardware.

The first method is easier to execute, because it relies solely on the published price and performance characteristics of IBM’s own System z server products. In effect this method shows us IBM’s own public estimate of what System z pricing might look like in a competitive market. However, the drawback of this method is that it fails to reflect the likely intensity of price competition between System z and non-System z servers in a truly open market. The latter scenario is a more realistic model for the competitive market that would emerge if IBM agreed to 9 In addition to IFLs for Linux, IBM sells several other “specialty engines” for System z servers that are similarly restricted to run only certain types of applications. The z Integrated Information Processor (zIIP) offloads DB2 workloads, while the z Application Assist Processor (zAAP) executes Java code. A similar strategy of deliberately disabling certain built-in microprocessor functionality in order to exercise customer price discrimination was used by Intel with the early generation 486 processors introduced in 1989 (see Wikipedia, “Intel 80486” [http://en.wikipedia.org/wiki/Intel_80486]). 486 chips bearing the “DX” suffix contained both fixed and floating point arithmetic units. 486 chips bearing the “SX” suffix had their floating point units disabled and were sold at a lower price. In the Intel case the price discrimination is based purely on voluntary customer preference (i.e. willingness to pay more for the superior performance offered by a chip with a floating point unit). In the IBM case, however, the price discrimination reflects the fact that one group of customers, namely those who own valuable legacy z/OS applications, have no alternative but to pay the higher price IBM demands for fully functional z10 processors, since there is at present no other legal option for running z/OS.

http://en.wikipedia.org/wiki/Intel_80486



renew its previous policy of licensing z/OS to alternative hardware vendors. It is highly likely that these vendors would deploy z/OS on more cost-effective platforms than current generation System z mainframes, which distinctly evince a certain amount of the “gold-plating” (i.e. the addition of costly extra features not strictly justified by customer requirements) that is characteristic of monopoly suppliers.

We explain and justify the inner workings of these methods in Appendix A of this paper. Here we simply summarize the results of their application. Referring to approach (2) above, we find that IBM System z mainframes priced for z/OS cost from 16 to 38 times as much as z/OS-compatible servers of equivalent computing power that use industry standard Intel or AMD x64 microprocessors10. The lower ratio is based on a comparison between a z10 E26 mainframe priced for z/OS and a MIPS-equivalent IBM-brand multiprocessor server using Intel’s 6-core Xeon Dunnington chip (see Table 2). The higher ratio assumes a comparison with a Dell-brand multiprocessor server using AMD’s 4-core Opteron chip11. Referring to approach (1) above, we find that IBM System z mainframes priced for z/OS cost approximately three times as much as identical IBM System z mainframes priced for Linux.

Table 2 below presents the implications of these price differences for the total size of the European z/OS-compatible server market under conditions of monopoly and competition. The table shows that actual total European expenditure on IBM-brand System z mainframes in 2008 (including the purchase cost of newly installed hardware systems as well as maintenance fees on the existing installed base) is expected to reach approximately $2.6 billion. If all current and weighted historical System z prices for z/OS were set to IBM’s corresponding System z prices for Linux (e.g. IFL prices), total European expenditure in 2008 would amount to only $966 million, for a net savings to the European economy of approximately $1.7 billion this year and $33.5 billion over the next 20 years12. If all System z prices were set to the price of MIPS-equivalent high-end multiprocessor x64 servers (assumed to be rendered z/OS-compatible by relevant emulation technology, as discussed below), total European expenditure in 2008 would fall to just $218 million, for a one year net savings to European customers of approximately $2.4 billion and savings of $48.5 billion over the next 20 years.

10 For the data referred to in this paragraph see tables 2 and 6 in this report. 11 In order to make the comparison as fair to IBM as possible, in Table 2 we have deliberately taken as our x64 reference an extremely high-end IBM-brand clustered multiprocessor System x server using the latest Intel x64 processors available (6-core Dunnington Xeons as of December 2008). However, it is possible to define comparable MIPS-equivalent server configurations using less expensive x64 servers. Table 6 hereafter gives as an example a Dell-brand server that is 58% cheaper per MIPS than the IBM x64 configuration used as the reference in Table 2. 12 Our model assumes the IDC figure of $1.402 billion for European purchases of IBM mainframes in 2008 is a blended average of z/OS and specialty engine MIPS. According to Gartner, specialty engines now account for about 15% of the total installed base and 20% of current year sales. (See Gartner, Chuba, op. cit.)



These excess costs borne by European users as a result of the absence of mainframe competition in Europe could generate significant benefits to the European economy if invested in appropriate alternative uses. Some of the savings would be passed on to European consumers in the form of lower prices. Some of it would be retained by corporations and used to invest in new capital projects or new business undertakings. Finally, some of it would accrue to European governments in the form of greater tax revenues on increased corporate profits. Ultimately most of the savings would serve to drive employment in one form or another. If we assume that an average “good” job in the European Union economy, fully weighted for social benefits and taxes, represents on the order of $60,000 per year, a savings of $2.4 billion per year due to enhanced mainframe competition would translate into the creation of 40,000 new jobs in Europe.

Additional economic benefits would result from the lowering of the implicit “tax on IT innovation” that current monopoly pricing of IBM System z hardware imposes on European companies and government agencies. Investment in new IT applications is a key driver of productivity and competitive advantage. Organizations that pay less for competively price mainframe-compatible server hardware will have more funds available for more productive IT investments. The introduction of competition in the European market for mainframe hardware would thus be likely to have a cascade of additional benefits above and beyond the immediate savings created by lower prices for z/OS-compatible server hardware.



Table 2: Savings from European mainframe competition

European z/OS Mainframe Market

Market Size Potential Savings

At Current System z

Prices

At Linux IFL Prices

At x64 Prices

(High-End)

At Linux IFL Prices

At x64 Prices

Server Hardware Purchases

System z Mainframe Servers (2008 actual)13 $1.402 billion

MIPS-equivalent z/OS-compatible Servers (2008 hypothetical)14

$450 million $86 million $952 million $1.316 billion

MIPS

Price per System z MIPS or equivalent (2008 actual) $1,300 $417 $80

Installed System z Mainframe MIPS (2008 actual)15 5,000,000

Installed z/OS-compatible Mainframe MIPS (2008 hypothetical)

5,000,000 5,000,000

Maintenance Fees

Weighted Average Historical Price per Installed MIPS16 $3,000 $1,251 $320

Acquisition Price of System z Installed Base (2008 cumulative actual)

$15 billion

Acquisition Price of z/OS-compatible Installed Base (2008 cumulative hypothetical)

$6.26 billion $1.6 billion

8.25% Annual Maintenance on Installed Base (actual)17 $1.24 billion

8.25% Annual Maintenance on Installed Base (hypothetical)

$516 million $132 million $724 million $1.12 billion

Hardware + Maintenance Current (2008) $2.642 billion $966 million $218 million $1.68 billion $2.42 billion

Hardware + Maintenance Forecast (2008-2027)

$52.84 billion $19.32 billion $4.36 billion $33.52 billion $48.48 billion

Source: IDC and Peerstone

13 z/OS mainframe sales data from IDC, “Worldwide and Regional Server 2008-2012 Forecast”, March 2008. 14 Market prices for MIPS-equivalent z/OS-compatible servers estimated according to the methodologies described in section 3 of this report. 15 Assumes Europe is 33% of worldwide installed IBM MIPS, and incorporates Gartner estimate of 14 million total MIPS installed worldwide end 2007, adjusted to 15 million to represent the market in mid 2008. (Gartner, Chuba, op. cit.) 16 Weighted average historical price of installed IBM mainframe MIPS based on average of 1998-2008 yearly prices according to IDC (cited in PSI, “Itanium as a Horizontal Microcode Engine for Legacy Architecture Enablement”, op. cit.). Weighted average historical prices for x64 MIPS based on Peerstone estimates. 17 Annual maintenance calculated at 8.25% of initial hardware system price, per estimated IBM z10 prices published by Technology News [www.tech-news.com/publib/pl2097.html].



Mainframe savings from all sources could reach $100 billion over 20 years

In addition to the excess costs incurred by European purchasers of IBM System z hardware detailed in this paper, further excess costs to European users result from above-market pricing of mainframe software sold by IBM and other vendors. Because mainframe users are trapped on the z/OS platform, IBM is able to extract above-market prices not only for its System z hardware (necessary to run z/OS in the absence of competition) but also for all other types of software that run on z/OS and whose use is required for z/OS legacy applications to function properly. For example, the z/OS version of IBM’s relational database DB2 is far more expensive than the versions of DB2 it sells for Unix, Linux or Windows Server systems18. Even non-z/OS software is more expensive on the mainframe. A standard subscription to Red Hat Enterprise Linux for x64, Itanium or Power based servers costs $1,499 per year, while the System z version of the same open source operating system costs $15,000 per year19.

Mainframe users spent approximately $24.5 billion worldwide on all mainframe software in 2007, of which 40% for mainframe software supplied by IBM itself20. We estimate that the excess costs imposed on European users due to the tying of z/OS to other software are at least as high as the excess costs imposed by non-market pricing of System z hardware. We also believe that similar findings will hold true for the adjacent markets of labor and professional services relating to IBM System z mainframes.

It is thus likely that the total excess costs imposed on European users by non-market pricing of System z and related products approach or exceed $5 billion per year. Extrapolated over the estimated 20 year remaining lifetime of typical legacy z/OS applications, the total savings to Europe resulting from a competitve mainframe market could reach $100 billion. We observe here that the aggregate savings resulting from the elimination of a substantial portion of these excess costs, which would be the consequence of the development of a competitve market for mainframe-compatible server hardware in Europe, would result in the creation of several hundred thousand additional jobs in Europe. We will examine these adjacent software, labor and service markets in subsequent research.

18 See our discussion of these versions of DB2 in Appendix B. 19 See Red Hat Store, Server Operating Systems [www.redhat.com/apps/store/server/]. 20 See IDC document #213259, “Worldwide Software 2008-2012 Forecast”, July 2008.

http://www.redhat.com/apps/store/server/



2. Legacy z/OS: migration is not an option

Legacy z/OS applications are worth at least $5 trillion

The value of legacy z/OS applications is extremely high. According to estimates published by IBM, the cumulative value of the worldwide installed base of legacy software applications running on its mainframes is $5 trillion21. IBM evaluates the size of this legacy installed base at 180 to 200 billion lines of COBOL and an additional amount of PL/1 code variously estimated by different IBM sources at 50 to 200 billion lines22. Even if we take the low end of these estimates and neglect other programming languages, legacy IBM mainframe applications represent well over 200 billion lines of code. IBM executives further state that the typical asset value of a single line of legacy mainframe programming code is on the order of $25, which is in accord with their estimates of the aggregate dollar value and size of the installed base23.

The cost of replacing these legacy z/OS applications would be extraordinarily high, in fact astronomically high. IBM has published estimates stating that the cost of replacing the entire worldwide installed base of 200 billion or more lines of legacy mainframe code could range from $5 trillion to as high as $20 trillion24. This represents a cost of $25 to $100 per line of legacy code replaced, which is in line with conventional industry estimates. IBM also estimates that “it costs five times more to rewrite a business function than to reuse existing code”25. It is worth noting that the higher range of the IBM estimate for the aggregate z/OS legacy replacement cost exceeds the combined gross national product of the member states of the European Union, which make up the largest economy in the world.

Moving legacy applications: “migration” vs. “virtualization”

Since the value of z/OS applications is so high, and since IBM System z hardware prices are by our most conservative estimate more than 15 times greater than those of MIPS-equivalent z/OS-compatible servers using x64 processors, it is natural to ask what if anything prevents z/OS users from moving their applications to cheaper platforms. To answer this

21 See IBM, “Enterprise collaboration with SOA” [www-03.ibm.com/systems/z/advantages/soa/values.html]. 22 See IBM, “Rational Software Development Conference”, 2008 [www-07.ibm.com/in/events/rsdc2008/presentation/day2no4.pdf]. 23 See IBM, “ISPF productivity to WebSphere Developer for System Z”, 2007 [ftp.software.ibm.com/software/systemz/pdf/cicsseminars/handouts/1115am_ISPF_productivity_to_WDz.pdf]. 24 See IBM Rational Software, “The Future of Software Delivery”, 2007 [www-05.ibm.com/il/news/events/ruc/pdf/the_future_of_software_delivery.pdf]. 25 IBM, “Rational Software Development Conference”, op. cit.

ftp://ftp.software.ibm.com/software/systemz/pdf/cicsseminars/handouts/1115am_ISP



question, it is helpful to refer again to the depiction of the mainframe solution stack in Table 1 above. The diagram shows the stack as a vertical sandwich ranging from the application layer at the top to the hardware layer at the bottom. The z/OS operating system itself occupies the middle of the stack, surrounded above and below by two interface layers in gray. These interface layers represent the only two points where it is theoretically possible to detach a complex legacy application from its original hardware platform. The highest of these two layers is the z/OS system interface level (or API level, as explained below). This is the layer where the application code interacts with the operating system code. The lower gray layer represents the interface between the operating system and the hardware. This is the layer where the hardware, using technical means that may vary considerably from one implementation to another, exposes a set of instructions that the operating system uses to tell the hardware what to do.

These two possible points of separation between a legacy application and its underlying platform have quite different cost and risk characteristics in practice. Detaching an application from the system level interface of one operating system in order to move it to another operating system (possibly executing on a different hardware platform) is known as application migration. Decades of painful customer experience have shown that migration is technically very difficult to execute successfully for applications of any significant scale or complexity. Entirely distinct from migration is the technique of detaching an operating system from the underlying hardware instruction set of one microprocessor in order to move it to another. This technique is known as operating system virtualization (or sometimes “emulation”). Unlike migration, virtualization, although technically far from trivial, has proven to be feasible and cost-effective in many circumstances. It is probably for this reason that IBM has frequently sought to leverage intellectual property issues and other forms of “lock-in” to prevent customers from using virtualization to move legacy z/OS applications to cheaper hardware platforms. We will discuss the different technical and market aspects of migration and virtualization in considerable detail in the remainder of this paper.

Why z/OS migration is so hard

To see why the “migration” of a complex application away from its underlying operating system is so difficult, recall that the purpose of an operating system is to perform certain low-level but vital housekeeping tasks on behalf of the applications that use it. This system level functionality is presented as a collection of “application programming interfaces”. These APIs, as they are known26, are operating system commands that application developers invoke to instruct the operating 26 z/OS concepts and terminology evolved before the expressions “application programming interfaces” or “APIs” became current as descriptions for operating system interfaces. Comparable z/OS and MVS terms may include “system services”, “functions”, “operating system commands”, “macro-commands”, “control blocks”, and “run-time libraries”, among others.



system to perform certain essential tasks that it alone can perform. The operating system APIs are in effect incorporated into the program code of the higher-level applications when the latter are executed.

Historically the tasks performed by operating system APIs involved mostly the management of low-level hardware functions such as scheduling time on the processor for applications, allocating real and virtual memory to them, and providing access to external resources such as hard disks, printers and networks, as well as certain somewhat higher levels tasks such as managing a file system for storing data persistently on disks. However, developers of high-value transaction-oriented business applications soon discovered that higher level functions beyond the basic APIs were necessary, the most crucial of which involved the management of reliable communications with the all-important database software that evolved as a more efficient way of storing transaction data than basic file systems. These additional transaction APIs were bundled together into what came to be known as transaction monitors in the mainframe world. Today the vast majority of legacy mainframe applications running on z/OS also use one of IBM’s proprietary transaction monitors, the most important of which are CICS, IMS and TPF27.

Other enterprise server operating systems such as Linux, Unix and Windows Server provide APIs and transaction-oriented middleware28 that are broadly similar in purpose to those of z/OS. However, for the overwhelming majority of z/OS applications, migration to alien operating systems such as those belonging to the Unix or Windows families is technically infeasible29. By the same token, even beyond the mainframe world non-trivial business applications generally do not migrate readily between Unix-like and Windows-derived operating systems, unless they were written to a so-called portability layer rather than to native operating system APIs30. Most “migrations” are in fact expensive manual reconstructions of the original application on a new platform.

The reason for the near-impossibility of operating system migration is that virtually all complex business applications derive crucial portions of their functionality by incorporating large chunks of the underlying operating system and transaction-oriented middleware directly into their program code by virtue of their invocations of the system-level APIs. A fully functional application combines its business logic and its embedded system-level APIs into an inseparable whole. To detach the application from the APIs is analogous to removing the external structure of a large building from its internal framework. Both elements have been carefully designed and assembled to fit with each other to exacting tolerances, and 27 IBM, “Introduction to the New Mainframe: z/OS Basics”, July 2006. 28 Although the equivalence is not exact, mainframe transaction monitors are roughly comparable to the middleware programs commonly known as “application servers” on Linux and Unix platforms. 29 Defined broadly, the Unix family of operating systems includes not only HP-UX, IBM AIX, Sun Solaris, but also Unix-like operating systems such as Linux and BSD. 30 See Appendix B for a detailed discussion of portability layers.



neither can retain its functionality in isolation. Similarly, it is not possible to replace the embedded APIs from one operating system by those from another OS, because the intricate code-level design of the original application will have been carefully taylored to the unique formats of the original APIs31.

The code of any non-trivial business application is at every point a mixture of business logic written by the application programmer and operating system or middleware commands embedded by the programmer into the application. The embedded operating system APIs are referenced at many thousands of locations throughout the program code of a large application and cannot be isolated from the business logic, no more than the internal steel skeleton of a skyscraper can be extracted from its glass and marble outer shell.

Furthermore, each operating system uses a different syntax and vocabulary to express its APIs. Fundamental differences exist between the ways in which different operating systems define even such basic entities as the elementary units of work performed by programs. For example, Unix and Windows Server divide work into program threads executing within operating system processes, while z/OS employs task control blocks (TCBs) and service request blocks (SRBs)32. Similarly, Unix and Windows Server store data as 8-bit bytes of data organized into files, while z/OS typically organizes data into 80-byte records that reflect the format of the paper punched cards used in early mainframes. It is because of these differences that attempting to translate the operating system commands of a legacy application while leaving its business logic unchanged is generally not feasible33.

The above scenario describes only the difficulties of migrating from one operating system to another while retaining the same application programming language (e.g. COBOL). These difficulties are greatly compounded when attempting to migrate simultaneously from one API set to another and from one programming language to another (e.g. from COBOL to Java). The differences between older mainframe programming languages such as COBOL and modern object-oriented languages such as Java and C# are even more significant than those between operating system APIs. In addition to differences in syntax and vocabulary these languages embody fundamentally incompatible concepts of how a program should be organized. A discussion of the highly technical nature of these differences is beyond the scope of this paper.

31 For an academic review of operating system APIs focusing on the technical differences between IBM mainframe, Unix and Windows APIs, see Bradfield, “Operating Systems”, 2007 [www.inf.ed.ac.uk/teaching/courses/os/slides/notes-forviewing-2007.pdf]. 32 IBM, “Introduction to the New Mainframe: z/OS Basics”, July 2006. 33 For discussion see Appendix B.

http://www.inf.ed.ac.uk/teaching/courses/os/slides/notes



Successful z/OS migrations are extremely rare

The existence of very high technical barriers to migration between the major enterprise server operating systems finds strong confirmation in both the shape of the operating system market as it exists today and in its historical development. None of the major operating systems such as Unix, Linux, or Windows Server that grew up after the mainframe have ever been able to capture any significant share of applications from other operating systems. On the contrary, each new operating system that rose to prominence in the market did so by co-evolving with an entirely new set of applications34. For Unix it was graphics-based Computer-assisted Design and Engineering (CAD/CAE) applications on scientific workstations in the 1980s, followed a few years later by relational databases such as Oracle, Informix or Sybase and packaged ERP applications such as SAP R/3 on multiprocessor Unix servers. For Windows NT (the precursor of Windows Server) it was the network file and printer server, followed by the Exchange corporate e-mail server and the SQL Server relational database. For Linux (which is ultimately just a variant of Unix from an API standpoint), it was the web server and the network utility server (e.g. firewalls), followed by Unix-friendly applications such as the Oracle database. While all of these new operating systems were evolving along with their associated new applications, legacy mainframe applications running on z/OS and its ancestors (e.g. OS/360, OS/370, and MVS) failed to benefit from more cost-efficient server hardware because they remained – and remain to this day – indissolubly wedded to their original operating system APIs.

The majority of legacy mainframe applications are high-value business applications operated by organizations in a wide array of sectors such as banking, manufacturing, government, defense, retail distribution, health care, transportation and public utilities. These organizations range from Fortune 500 corporations to small and medium businesses (SMBs). Their demonstrated willingness to invest in sophisticated software and hardware makes them some of the most desirable IT buyers in the world. As such, they constitute natural target customers for leading rivals of IBM such as Hewlett-Packard, Sun Microsystems, Microsoft and Oracle. It is logical to suppose that these rival vendors would make aggressive efforts to persuade IBM mainframe customers to migrate z/OS applications to their own brands of server operating systems, transaction-oriented middleware, database management software and server hardware. And 34 This view is supported by an in-depth survey of several hundred ERP customer organizations that we conducted in 2004. One particularly striking finding was that while both Linux and Windows Servers have captured significant shares of the installed base of packaged applications such as SAP at the expense of Unix, there is essentially no migration between Linux and Windows Server. Migration from Unix to Linux motivated by cheaper server hardware is common, but this movement is facilitated by the fact that Linux is a variant of Unix with a substantially identical set of APIs. By contrast, the remarkable rise of Windows Server’s share of the SAP installed base is largely due to new SAP installations rather than displacements of Unix. See Peerstone, “Linux and ERP: A White Paper”, 2004.



indeed, beginning in the 1990s, these vendors have made many such efforts. Many software tools have been developed to aid the task of extracting business rules from COBOL code and automating their translation into more modern programming languages such as Java, C++ or C#.

However, despite these efforts, only a very small number of large scale mainframe applications have been successfully transplanted to other operating systems using these tools or any other method. An examination of the published financial results of the vendors promoting mainframe migration confirms that they have never derived any significant revenue from these activities. In fact, the marketing resources devoted to this area have declined significantly in recent years. One major systems vendor (Sun) which made a substantial investment in this area in 2001 decided to exit the market entirely in 2006, while the Hewlett-Packard, Microsoft and Oracle mainframe migration groups appear to have engaged in very little activity in recent years.35

Based on the number of case studies published by vendors engaged in mainframe migration efforts36, we estimate that successful migrations from z/OS to other operating systems represent no more than a few tens of cases per year out of a worldwide installed base of approximately 10,000 IBM mainframe systems currently in operation. Given the impossibility of translating directly between different operating system API sets described above, the migrations that do go forward invariably require extensive and costly manual adaptation of key portions of the source application code. In fact, the handful of vendors who remain active in this area primarily make their living by selling professional services, confirming our view that most so-called mainframe “migrations” are in fact very labor-intensive reconstructions of their source applications on other platforms37.

35 Software vendors face a different set of OS migration issues than legacy application owners in end-user organizations. See Appendix B for discussion. 36 See the Mainframe Migration Alliance web site (mainframemigration.org). 37 Several software vendors have attempted to facilitate mainframe migrations by developing software “clones” of IBM’s CICS transaction monitor that are able to run on other operating systems such as Unix or Windows Server. However, these CICS clones have achieved only very limited success in the market due to significant gaps in functionality and are regarded by Gartner as unsuited for complex legacy applications. The most widely deployed CICS clone is Unikix, now owned by Clerity, a 50 employee systems integrator in Chicago. This product was originally developed by French computer vendor Bull in the late 1980s but has changed owners six times since then. According to Clerity, which acquired Unikix from Sun in 2006, this product has garnered only 250 users during its 20 year history, some of which are universities and software developers. Of those that represent actual mainframe migrations, Gartner estimates that the majority were low-end mainframe users with fewer than 500 MIPS installed and relatively uncomplicated applications [see Gartner, “Can Software Clones Enable Mainframe Migrations?”, January 2006]. Even in these cases the use of Unikix to migrate legacy applications is very labor intensive and typically requires a costly professional services engagement.



3. Virtualization is key to z/OS competition

Replacement of legacy z/OS applications is costly and risky

Legacy z/OS applications constitute a $5 trillion software asset trapped on System z mainframe hardware which is priced by IBM at levels far above those that would prevail in a competitive market. The extremely elevated prices associated with System z hardware provide z/OS application owners with a powerful incentive to seek more cost-effective alternatives. Although as we have seen it is not possible in the vast majority of cases to move z/OS applications to other operating systems, z/OS users do have at least two other technically feasible alternatives to System z hardware which we will discuss in turn, namely replacement and virtualization.

Replacement, as its name suggests, consists in replacing legacy z/OS applications entirely with brand new applications of equivalent functionality that run on less expensive server platforms such as those designed for Linux, Unix or Windows Server. This “forklift upgrade” approach is by definition extremely costly, since it requires the user to throw away a still-functioning software asset and invest large sums of money in a technically risky endeavor to duplicate the asset’s functionality in a new software project. This approach is nevertheless occasionally resorted to by System z customers who determine that their legacy z/OS applications no longer meet critical new business requirements and cannot feasibly be extended to do so.

A recent example of this approach is the State of California’s decision to replace a COBOL-based IBM mainframe human resources and payroll application with an ERP software package from software vendor SAP deployed on Unix servers. The primary motivation for the change was a high-level political decision in California to decentralize HR processes to the dozens of individual departments and agencies that make up the state’s government. An immediate result of the decision was a major new functional requirement that these autonomous units be able to make frequent changes to the business rules embedded in the HR application and be able to obtain locally customized management information reports. The state’s IT managers and outside consultants determined that the new requirements could not be met by the legacy mainframe application and recommended adoption of the SAP solution.

However, despite several years of careful planning, the California legacy replacement project has not gone smoothly. At the time of writing (November 2008) the SAP implementation project is more than one year behind schedule and has experienced huge cost overruns, resulting in a doubling of its budget to $177 million38. It is highly likely that the SAP 38 See California State Controller’s Office, “Transforming Human Resources Management in the State of California” (www.21stcentury.ca.gov/overview).

http://www.21stcentury.ca.gov/overview)



application will incur additional delays and cost overruns before its effective deployment. The full application functionality initially planned may never be delivered. As this example shows, the costs and risks of legacy replacement are high. As a result of these high costs and risks, which are well known in the industry, few mainframe users willingly choose to go down the path of wholesale replacement of their legacy z/OS applications.

z/OS virtualization would spur mainframe market competition

The second escape option that z/OS application owners have is to employ the operating system virtualization technique we mentioned in the previous section. In computer science the term “virtualization” is used with a variety of meanings. But the core concept involves abstracting a software object such as an operating system away from the actual physical instruction set implemented by the underlying execution hardware and substituting in the latter’s place a virtual instance of the same instruction set. When the operating system code (which has been compiled into the machine language of the target processor) attempts to invoke the hardware’s physical instructions, the virtualizing layer intercepts these requests and decides what to do with them.

A virtualizing hypervisor generally hosts several higher-level operating system instances simultaneously on a processor that implements the same instruction set the guests were originally compiled for. It relays each guest’s requests to the physical hardware, but it coordinates all requests and prevents interference or collisions between them, maintaining each guest in the illusion that it is the sole user of the hardware. In much the same way a virtualizing emulator intercepts requests to execute physical hardware instructions, but is designed to operate on a processor with a different instruction set than the original target. An emulator usually hosts only one instance of a higher-level operating system (unless the guest is itself a hypervisor) and translates the guest’s requests into the instruction set of the new hardware target, while maintaining the guest in the illusion that it is executing on its intended hardware platform39. In this sense a virtualizing emulator is a more complex piece of software than a virtualizing hypervisor, because the emulator must not only create a virtual instance of the target instruction set but do so on a different microprocessor architecture (i.e. one that uses a different native instruction set). In this paper we will use the term “emulator” informally to refer to this specialized type of virtualization software.

Although virtualization solutions are technically complex and require high levels of programming skill to create (they are in no sense mere “clones” 39 There is an extensive technical and popular literature on virtualization. For brief tutorials and discussion of the various types of virtualizing hypervisors and emulators, see “The Virtualization Reality”, ACM, Crosby and Brown, 2006 [queue.acm.org/detail.cfm?id=1189289]; Wikipedia, “Virtual Machine” [en.wikipedia.org/wiki/Virtual_machine]; and “An Introduction to Virtualization”, Singh 2006 [www.kernelthread.com/publications/virtualization].

http://www.kernelthread.com/publications/virtualization



of the instruction sets they implement), a number of such solutions have been developed over the course of the past four decades. The wide and successful deployment of operating system virtualization solutions on a variety of hardware platforms stands in sharp contrast to the much poorer record of the legacy application migration approaches described in the previous section. Examples of virtualization include the following:

§ IBM’s VM mainframe operating system, the first major example of a successful virtualizing hypervisor, originally developed in the 1970s and still in widespread use40;

§ VMware’s ESX virtualizing hypervisor, which is the market leader on Intel and AMD-based hardware, hosting primarily Windows and Linux operating systems41;

§ Microsoft’s Hyper-V and the open source Xen (distributed by Citrix), virtualizing hypervisors that compete with VMware42;

§ Virtualizing emulators of the IBM z/Architecture instruction set developed by IBM iself and by several independent developers using a variety of distinct technologies, as discussed below;

§ A wide variety of virtualizing emulators targeting other operating systems and hardware platforms43.

As mentioned, several successful implementations of the IBM System z instruction set on non-System z hardware exist which run z/OS with good to excellent performance, although their use is currently barred by IBM’s restrictive z/OS licensing policy. Virtualization of the z/Architecture instruction set on alternative processors accomplishes the same goal as migrating legacy z/OS applications to other operating systems. However, it does so with far less risk and cost, because it enables both the z/OS applications and z/OS itself to run on the alternative hardware without any changes to their underlying software code.

Broadly speaking, we can distinguish three types of z/Architecture instruction set virtualization or “emulation”, each with different performance and cost characteristics:

§ Virtualization in the microcode of another processor, as in the Itanium-based technology developed by the now defunct Platform

40 See Wikipedia, “VM” [en.wikipedia.org/wiki/VM_(operating_system)] and IBM, “Introduction to the New Mainframe: z/VM Basics”, 2007 [www.redbooks.ibm.com/redbooks/pdfs/sg247316.pdf]. 41 See Wikipedia, “VMware ESX Server” [en.wikipedia.org/wiki/VMware_ESX_Server] and VMware, “VMware ESX Server” [www.vmware.com/pdf/esx_datasheet.pdf]. 42 See Wikipedia, “Hyper-V” [en.wikipedia.org/wiki/Hyper-V]; Wikipedia, “Xen” [en.wikipedia.org/wiki/Xen], and references therein. 43 See Wikipedia, “Virtual Machine” [en.wikipedia.org/wiki/Virtual_machine], Wikipedia, “Emulator” [en.wikipedia.org/wiki/Emulator] and references therein.

http://www.redbooks.ibm.com/redbooks/pdfs/sg247316.pdf

http://www.vmware.com/pdf/esx_datasheet.pdf



Solutions and incorporated by T3 Technologies into its z/OS-compatible Liberty Server line44;

§ Virtualization in the assembly language of another processor, as in Fundamental Software’s FLEX-ES x86-based emulator, also sold by T3 Technologies in a bundle with Intel-based servers as part of its MVS and z/OS-compatible tServer product line;

§ Virtualization in a high-level language such as C, e.g. as in the Hercules C language open source z/Architecture emulator45.

In each of the above three cases, a specialized software layer, characterized as a virtualizing emulator, is installed on a non-System z microprocessor and creates on that processor a virtual instance of the native z/Architecture instruction set (as defined by IBM’s published Principles of Operation documents46). The emulator operates at program run-time to make this virtual instruction set available to an instance of the z/OS binary executable code and to the binaries of whatever z/OS-compatible middleware and applications the user chooses to install, all of which then execute as if they were running on a native System z processor. An important difference between the three types of virtualizing emulator is the software level at which the emulator code is executed. Microcode47 (also known as firmware) is closest to the target physical microprocessor and yields the best performance. High-level languages such as C are easier for programmers to work with and are much more 44 For a detailed technical discussion of the microcode approach to emulation, see “Itanium as a Horizontal Microcode Engine for Legacy Architecture Enablement”, Ron Hilton, PSI Founder & CTO, 2007 (op. cit.). Notice that in this document PSI does not employ the informal term “emulator” to characterize its microcode technology, using instead the more precise description “firmware-based hardware virtualization” (p. 11). 45 See www.hercules-390.org. In 2007 IBM released its own PC-based z/Architecture software emulator called zPDT (z Personal Development Tool). Although zPDT appears to be similar to Hercules, very little information about it has been released by IBM, and the product is said to be available only to IBM employees. According to Roger Bowler, zPDT performance is about 50% of that of Hercules. Personal communication, Roger Bowler, September 2008. 46 See IBM, “z/Architecture Principles of Operation” [publibfp.boulder.ibm.com/cgi-bin/bookmgr/BOOKS/dz9zr002/CCONTENTS]. 47 Microcode refers to a method that chip designers use to implement microprocessor instructions by decomposing them into smaller building blocks known as “microinstructions”. The alternative is to implement the higher level instructions directly in silicon, i.e. directly in the permanent physical architecture of the chip. With microcode, unlike physical implementation, each higher level instruction is a “microprogram” written in microcode. The microprograms are stored in a fast-access read-only memory (ROM) section on the chip. One advantage of microcode is that it is often simpler and cheaper to implement the simpler microinstructions in silicon than the more complex “macroinstructions” they are used to express. Microcode also makes it easier to change instruction sets or correct design mistakes. Its utility for implementing virtualized (or “emulated”) instruction sets has long been recognized by chip designers. See Wikipedia, “Microcode” [en.wikipedia.org/wiki/Microcode].

http://www.hercules-390.org



readily portable to other microprocesssor architectures (providing suitable compilers are available), but have slower performance. Assembly language solutions fall in between microcode and high-level languages in terms both of performance and degree of programming difficulty.

A detailed discussion of the technical aspects of virtualizing emulators capable of executing z/OS on non-System z processors is beyond the scope of this paper. However, it is worth noting that the frontier between a “native” and an “emulated” instance of the z/Architecture instruction set is not as clear in practice as our simplified discussion here may suggest. Indeed, there is a sense in which all microprocessors that use microcode in their designs are emulators. In effect, even the “native” instruction set of a microcode-based chip is an emulation. It is interesting to note in this context that IBM has used microcode for decades to implement the instruction sets in its own mainframe processors, going back at least to the 1970s. Similarly, mainframe PCM manufacturer Amdahl made extensive use of microcode to produce its IBM-compatible mainframe servers in the 1980s and 1990s48. According to IBM’s published descriptions of its most recent z/Architecture microprocessor, this processor – known as the z10 – implements some 25% of its instruction set in “millicode”, which is IBM’s term for microcode49. IBM’s refusal to license z/OS for use on non-System z microprocessors that implement the z/Architecture instruction set by means of a virtualizing emulator is thus somewhat paradoxical, given that its mainframe chip designers have long practiced instruction set emulation in their own designs.

It has long been observed in the industry that the IBM System 370 hardware architecture and the accompanying MVS operating system (the ancestor of z/OS), both introduced in the early 1970s, were expressly designed to facilitate instruction set virtualization50. It is thus no

48 See Hilton, op. cit., p. 6. 49 See IBM, “IBM z6 –The Next-Generation Mainframe Microprocessor”, Webb, 2007 [speleotrove.com/decimal/IBM-z6-mainframe-microprocessor-Webb.pdf]. Note that the z10 chip was originally designated z6 by IBM, reflecting the chip’s close similarity to IBM’s Power6 chip, with which it shares core architectural and semiconductor technology features. See Wikipedia, “IBM z10 (microprocessor)” [en.wikipedia.org/wiki/IBM_z10]. The proportion of microcoded – in effect, “emulated” – instructions in IBM’s processors was significantly higher in previous generation chips. The increasing proportion of instructions implemented in native silicon rather than ROM-based microcode is probably due to the greater performance potential of native implementations. For discussion of the history of microcode in IBM processors up through the z10 generation, see “Green Machines: The System z10 Enterprise Class”, Smith, in Mainframe Executive June 2008 [www.mainframe-exec.com/articles/?p=32]. See also “Millicode in an IBM zSeries processor”, Heller and Farrell, in IBM Journal of Research and Development, Volume 48, Number 3/4, 2004. 50 For a technical discussion of the design features of the IBM System 370 architecture that make it particularly well-suited for virtualization and a comparison with the much less virtualization-friendly architecture of Intel’s x86 family, see “A Comparison of Software and Hardware Techniques for x86 Virtualization”, Adams and Agesen, paper presented at ASPLOS’06, October 2006.

http://www.mainframe-exec.com/articles/?p=32



coincidence that a long line of programmers both inside and outside IBM have taken advantage of this design feature, beginning with IBM’s own VM virtualizing hypervisor and continuing through many subsequent virtualizing emulators, the most prominent of which were mentioned above. The author of the widely used Hercules IBM mainframe emulator, Roger Bowler, has stated that the design of the IBM z/Architecture instruction set makes its emulation on alternative microprocessors conceptually very straightforward, although of course far from trivial to execute in practice51. As we noted previously, this is in marked contrast to the far greater technical difficulty of migrating applications that use z/OS system-level APIs to the incompatible API sets of other operating systems.

In sum, it is on the one hand technically very difficult to detach z/OS applications from z/OS, because among other reasons the z/OS system level interfaces (APIs) were never designed to facilitate this kind of migration. On the other hand, it is eminently feasible to detach z/OS itself from System z hardware (and thereby detach legacy z/OS applications at the same time), because the z/Architecture was specifically designed to make this separation easy to accomplish using virtualization techniques. These findings are summarized in Table 3 below.

It is highly likely that the renewal of z/OS licensing for use on non-System z hardware would lead to the reemergence of a competitive market for z/OS-compatible servers such as existed prior to the withdrawal of Fujitsu-Amdahl and Hitachi from the market. In addition to renewed activity by established market participants such as T3 Technologies and FSI, we would expect to see significant activity from new market entrants. For example, it is likely that new entrants would launch z/OS-compatible products based on the Hercules open source mainframe emulator. Some entrants would be likely to adopt the business model pioneered by Red Hat in the Linux market of bundling commercially packaged open source software with paid support services. Others might follow T3 and choose a hardware appliance model bundling mainframe emulation software with commercially available Intel or AMD servers. It is also likely that one or more of IBM’s large server system rivals such as Hewlett-Packard, Sun, Fujitsu, Bull or Unisys would choose to enter the market with midrange or high-end multiprocessor systems based on x64 or other comparable processors. We might even see a commodity server vendor such as Dell enter the market by offering Intel or AMD systems preloaded with Hercules or FLEX-ES, much as it already does with Linux. But perhaps the most intriguing possibility of all is that ambitious new market entrants would develop additional virtualizations of the z/Architecture instruction set designed for commercially available microprocessors, perhaps adding value with innovative new features or very high performance.

The return of competition in the IBM-compatible mainframe market would rapidly result in large savings for European and other owners of high-value legacy z/OS business applications which are currently restricted to

51 Personal communication, Roger Bowler, September 2008.



running only on IBM-brand System z hardware. In Appendix A of this paper we present a simple model that quantifies the savings that European users could expect to achieve in such a competitve market.

Table 3: IBM mainframe stack, showing possible exit points for legacy applications

Legacy Mainframe Application

(e.g. compiled COBOL code)

z/OS system level interfaces (APIs) Application Migration (high cost, high risk)

IBM z/OS

System z Instruction Set (IBM z/Architecture Principles of Operation)

Instruction Set Virtualization (low cost, low risk)

Processor compatible with System z Instruction Set

source: Peerstone



4. Appendix A: z/OS-compatible server prices

Identifying alternative z/OS server platforms

In order to quantify the excess costs borne by European users as a result of the restriction of z/OS to IBM-brand System z servers, it is necessary to compare System z prices to those of other servers that would be capable of supporting z/OS with equivalent performance and functionality. We have already summarized the results of this comparison in section 1 above. Here we present the details of the comparison. We examine the pricing, features and estimated MIPS-equivalent performance of a number of commercially available enterprise servers which, although designed to run Linux, Unix or Windows Server (LUW) operating systems, are also capable of supporting the z/Architecture virtualizing emulators described previously. It is important to understand that LUW servers cannot be regarded as “substitutes” for System z servers running z/OS, because – for the reasons we have discussed at length in section 2 – legacy z/OS applications generally cannot migrate to LUW systems. Thus, these LUW servers, because they are “z/OS capable” by virtue of their ability to execute z/OS emulation, are ideal price proxies for our analysis of the costs of IBM’s mainframe monopoly.

Most, but not all, of the servers we use as price proxies contain so-called x64 (or x86-64) microprocessors supplied by Intel or AMD. Some of the servers we consider, specifically those designed for use with vendor-specific variants of the Unix operating system such as IBM AIX, HP-UX or Sun Solaris, contain non-x64 microprocessors such as IBM’s Power6, Intel’s Itanium or Sun’s UltraSPARC. The basic features of these microprocessors are summarized in Table 4. In the case of servers built with Intel or AMD x64 processors and commonly used with Linux or Windows Server, there is a robust and highly competitive hardware market that is independent from the adjacent operating system markets. In fact, the microprocessors, the operating systems and the finished server hardware systems are all supplied by different vendors engaged in intense competition in their own upstream markets. In the case of servers designed for one of the vendor-specific Unix variants, although the server vendors offer their own customized versions of the Unix operating system to accompany their own brand servers built with non-x64 processors, the market remains highly competitive, and users as well as independent software vendors (ISVs) face no significant technical barriers to migration from one version of Unix to another or to Linux52. We therefore characterize the LUW server prices as competitive, in contrast to the 52 Recall that although we commonly think of Linux as a “different” operating system than Unix, it is in fact an offshoot of the Unix family, with a substantially identical set of APIs. The relative ease of migration of applications between Linux and the different varieties of Unix (AIX, Solaris, HP-UX, etc.) is in striking contrast to the much greater difficulty of application migration from z/OS to either the Unix or the Windows families.



monopoly prices of IBM System z servers configured for z/OS. We characterize IBM System z mainframes configured and priced for Linux (i.e. by the use of specialty engines, discussed in section 1) as semi-competitive. The pricing levels of these three classes of servers are indicated in Table 5.

Table 4: Common enterprise server microprocessors

Processor Max Cores

Fab Process Transistors

Max Clock GHz53

Typical OS Estimated Total MIPS (1-way)54

IBM z10 4 65 nm 991 million 4.4 z/OS 920

Intel Xeon Dunnington 6 45 nm 1.9 billion 2.66 Windows Server, Linux 834

Sun UltraSPARC T2 8 65 nm 500 million 1.4 Solaris, Linux 585

IBM Power6 2 65 nm 790 million 5 AIX, IBM i (i5/OS) 523

AMD Opteron Barcelona 4 65 nm 463 million 2.5 Windows Server, Linux 523

Intel Itanium55 4 65 nm 2 billion 2+ HP-UX, Windows Server 418+

Table 5: Enterprise Server Classes

Server Class Processor Type Market Status $ per MIPS56

z/OS Mainframe IBM z10 Monopoly $1,000 - $1,500

Linux Mainframe IBM z10 Semi-competitive $415 - $670

Unix IBM Power Sun UltraSPARC Intel Itanium (HP)

Competitive $400 - $750

Linux & Windows Server

Intel Xeon x64 AMD Opteron x64 Competitive $30 - $80

53 Maximum clock rates current as of Q4 2008 (manufacturers may increase clock rates over the life cycle of a chip product). 54 Millions of instructions per second (MIPS) equivalents for non-IBM processors derived by assigning value of 52 MIPS per core per GHz (52 MIPS per core per GHz = IBM published value for IBM z10 1-way system); estimate assumes 1-way system (i.e. with no reduction due to SMP scaling factor, compare Table 7 below). 55 Itanium quad-core chip expected late 2008 or early 2009. 56 $ per MIPS is an estimate of the price/performance ratio of an overall server system, not of an isolated microprocessor.



Computing market price equivalents for IBM mainframes

In this subsection we present the empirical pricing data used in our calculation of the excess costs to European users caused by IBM’s mainframe monopoly. The results were summarized in Table 2 in section 1 of this paper.

The modern enterprise server market consists of three distinct classes of machines which have similar intrinsic technology and performance characteristics but radically different pricing levels for equivalent configurations. These classes were described in Table 5 above.

Most enterprise servers share a common set of core features (see Table 7 below). However, the great variety of possible configurations as well as systematic vendor exaggeration of small packaging and feature variations for purposes of marketing differentiation make direct comparisons challenging. Nevertheless, in response to intense user demand, a number of industry groups have developed a range of useful metrics and benchmarks. Among the most widely used are the following:

§ MIPS57, or millions of instructions per second, a traditional measure of performance IBM mainframes, sometimes extended to non-mainframe systems;

§ the TPC58 family of benchmarks developed by the Transaction Processing Performance Council, such as the well-known TPC-C benchmark for on-line transaction processing;

§ the Standard Performance Evaluation Corporation’s SPEC59 benchmarks, commonly used to measure raw CPU performance independent of particular business applications such as transaction processing;

§ SAP’s Standard Application60 benchmarks developed for its well-known line of Enterprise Resource Planning (ERP) packaged business applications, of which the SD benchmark (Sales and Distribution) is the most widely used.

For purposes of comparing the relative performance of enterprise servers used for mission-critical business applications, the TPC and SAP benchmarks are probably the most useful, since they correspond to the most common large-scale business applications (OLTP and ERP, respectively). Most leading server vendors submit their products for independent third-party testing against these rigorous and well-designed standard benchmarks. Results are publicly available at the TPC and SAP web sites for dozens of systems including many servers from IBM,

57 See Gartner, “MIPS Rating for IBM's System z10 EC Processors”, April 2008. 58 See www.tpc.org. 59 See www.spec.org. 60 See http://www.sap.com/solutions/benchmark/index.epx.

http://www.tpc.org

http://www.spec.org

http://www.sap.com/solutions/benchmark/index.epx



HP, Sun, Dell, Bull and other vendors, and are often the subject of colorful competitive advertising in industry periodicals.

Unfortunately, while IBM aggressively touts the impressive test results of its PowerPC and Intel Xeon based systems, it systematically refuses to allow its System z mainframes to be subjected to these public benchmark tests, on the pretext that the tests are not “reflective of what mainframe customers do with mainframes”61 and do not “properly reflect the value of the Z platform”62. Indeed, IBM mainframes were last submitted for public SAP benchmarking in the 1990s. However, SAP customers have recently stated63 that they have received private System z benchmarking data under non-disclosure agreements.

IBM’s refusal to allow public benchmarking of System z servers is surprising, since these results are often studied by customers before major system purchases. It may be that IBM does not wish owners of legacy z/OS applications to able to easily measure the extent of the savings they would realize if allowed to use their z/OS licenses on alternative mainframe-compatible hardeware. But while very few new native z/OS applications are being developed today, IBM does aggressively market the speciality engine versions of its System z hardware processors as platforms for Linux, Java and database consolidation. Since IBM frequently uses third-party benchmark results in marketing its System p (AIX) and System x (Windows Server and Linux) servers, one would expect it to do the same when marketing its System z specialty engines. The fact that it systematically declines to do so suggests that the benchmark data would reveal dramatically inferior performance to cost ratios for System z servers compared non-System z servers of equivalent performance.

In the absence of public benchmark data for System z, we have developed an alternative methodology for comparing System z performance and price/performance with other commodity and non-commodity servers using the more traditional MIPS (millions of instructions per second) metric. IBM employees have publicly stated that it is feasible to establish equivalences between standard IBM System z MIPS values and server systems using x64 class microprocessors such as AMD’s Opteron chip. Writing on the IBM web site, IBM Senior Consultant Tony Pearson gives this equivalent as follows:

§ “6.866 z10 EC MIPS per GHz of dual-core AMD Opteron for I/O-intensive workloads running only 10 percent average CPU utilization…”64

61 IBM employee cited by Jeff Savit, a former IBM employee now employed by Sun (http://blogs.sun.com/jsavit/entry/the_ten_percent_solution). 62 IBM employee cited by SAP customer (https://www.sdn.sap.com/irj/sdn/thread?threadID=774368) 63 SAP customer (https://www.sdn.sap.com/irj/sdn/thread?threadID=774368). 64 See Pearson, “Yes, Jon, there is a mainframe that can help replace 1500 x86 servers”, June 11 2008

http://blogs.sun.com/jsavit/entry/the_ten_percent_solution)

https://www.sdn.sap.com/irj/sdn/thread?threadID=774368)

https://www.sdn.sap.com/irj/sdn/thread?threadID=774368)



Indeed, IBM published a comparison65 of z10 and x86-64 performance using this equivalence at the time of the launch of the first z10 mainframes. However, the IBM comparison is marred by the implausible (but convenient) assumption that x64-based servers used in a mission-critical business application will run at only 10% utilization while the z10 mainframe run at 90% in similar circumstances. Former long-time IBM employee Jeff Savit, who during his many years of benchmarking mainframe performance while at IBM gained first-hand knowledge of the performance characteristics and typical usage patterns of both IBM System z and Intel or AMD based x64 servers, has stated that this assumption is not valid, and that transaction-intensive database applications in real-world customer usage routinely achieve similarly high utilization rates on both server platforms66.

It is therefore reasonable to adjust the equivalence stated above to 61.8 (=9 x 6.866) MIPS per GHz of dual-core AMD Opteron for input/output intensive workloads such as database transaction processing. This corresponds to about 31 MIPS per core per GHz for an Opteron processor. According to the previously cited Gartner report, the comparable value for IBM’s quad-core 4.4 GHz z10 processor is 52 MIPS per core per GHz in a 1-way server configuration. However, according to IBM’s Pearson and Sun’s Savit in the documents already cited, a significantly higher value on the order of 100 MIPS per core per GHz should be attributed to an Opteron performing compute-intensive tasks such as numerical calculations. We therefore feel justified in adjusting the Opteron’s MIPS rating per core per GHz up to the same 52 MIPS value as the IBM z10 processor.

Based on these observations, and given that there is no empirical evidence to support the idea of radical differences in performance on a MIPS per core per GHz basis among the various processors described in Table 4 above, we conclude that a reasonable informal method for attributing a System z MIPS equivalent to other non-z processors is to use the 52 MIPS value of the z10 processor. It is certainly possible that accurate laboratory tests would reveal somewhat different actual values. However it seems highly unlikely that these values could differ from our baseline by more than 20% or 30%. We would of course welcome any published lab results from IBM or other sources on this point.

Table 6 below presents the results of our methodology applied to two variants of IBM’s z10 mainframe, two leading proprietary Unix servers, and two commodity servers based on industry-standard Intel Xeon or AMD Opteron chips.

(http://www.ibm.com/developerworks/blogs/page/InsideSystemStorage?entry=yes_jon_there_is_a). 65 See IBM, “IBM launches New ‘System z10’ Mainframe”, February 2008 (http://www-03.ibm.com/press/us/en/pressrelease/23592.wss) 66 Op. cit.

http://www.ibm.com/developerworks/blogs/page/InsideSystemStorage?entry=ye

http://www-03.ibm.com/press/us/en/pressrelease/23592.wss)



Table 6: Enterprise server configurations and prices

Server IBM z10

E26 (z/OS)67

IBM z10 E26 (Linux

IFL)68

HP Superdome69

IBM Power 59570

5 x IBM System x385071

8 x Dell PowerEdge

M90572

Processor IBM z10 4-core

IBM z10 4-core

Intel Itanium 2-core

IBM Power6 2-core

Intel Xeon 6-

core

AMD Opteron 4-core

CPUs 26 26 64 32 20 32

Cores 104 104 128 64 120 128

GHz 4.4 4.4 1.6 5 2.66 2.3

MIPS/Core/GHz 1-way

52 52 52 52 52 52

SMP Factor 57% 57% 57% 57% 57% 57%

Estimated Total MIPS

13,634 13,634 6,102 9,535 9,511 8,772

RAM GB 768 768 512 1024 1280 768

Estimated System Price

$17,724,720 $5,680,540 $4,400,212 $5,443,610 $758,365 $297,072

$/MIPS $1,300 $417 $721 $571 $80 $34

The estimated prices are for server hardware only, excluding external storage, operating systems, and maintenance. MIPS equivalents for non-IBM servers were derived by assigning a value of 52 MIPS per core per GHz for a 1-way system, then reducing by SMP scaling factor (52 MIPS per core per GHz = IBM published value for IBM z10 1-way system).

67 z10 E26 total MIPS and SMP factor are Gartner estimates (see Gartner doc # G00156987, April 2008). System price based on published estimates of IBM z10 EC mainframe prices (see www.tech-news.com/publib/pl2097.html) plus Gartner estimate of $6,000 per GB of z10 RAM. The estimate of $1,300 per MIPS matches published industry estimates of $1,000 to $1,500 per z10 MIPS (see Morgan, “IBM Launches 64-Way z10”, IT Jungle, March 2008). 68 IFL = Integrated Facility for Linux. System price based on IBM published prices of $125,000 per z10 IFL plus Gartner October 2008 estimate of $2,200 per GB of RAM for specialty engines plus estimated 15% for chassis & system components. 69 HP Superdome configuration and price from TPC.org March 2008. 70 IBM Power 595 configuration and price from TPC.org June 2008. Price adjusted down to reflect 1024 GB RAM instead of 4096 GB. 71 Assumes 5 separate servers linked by Infiniband in shared memory cluster (e.g. for OLTP application using Oracle RAC). Configuration and price from TPC.org September 2008 + $47K for Infiniband switches & HBAs (based on Panta cluster component price list, TPC.org, October 2006). 72 Assumes 8 separate servers linked by Infiniband (e.g. for Oracle RAC). Configuration and price from Dell.com October 2008 + $47K for Infiniband.

http://www.tech-news.com/publib/pl2097.html)



Intrinsic product features do not explain IBM mainframe prices

In competitive markets significant price differences between products of the same general category are due to significant differences in performance or functionality. When such differences are lacking the pressures of competition tend to drive the prices of similar products form different vendors toward a common level. In non-competitive or monopoly markets, by contrast, price differences even in the absence of significant product differentiation can arise when unnatural barriers to competitor entry or customer exit exist. In the case of IBM-brand z/OS-compatible mainframes both barriers exist: on the one hand, IBM prevents competitors from entering the market for z/OS compatible mainframes by refusing to license z/OS to them or their customers; on the other hand, z/OS application owners cannot exit the market because in the vast majority of cases it is not possible to migrate applications designed for z/OS to other enterprise operating systems such as Linux, Unix, or Windows Server.

As we shall document below, IBM goes to great lengths to prevent the publication of independent data comparing System z performance to that of other servers. Nevertheless, IBM sales and marketing representatives commonly assert that System z hardware possesses distinctive features and performance characteristics that justify its extraordinarily high prices compared to other servers. But a comparison of System z servers with servers based on other server architectures reveals fundamental similarities. The performance characteristics of the microprocessors most commonly used in such servers were described in Table 3 above. The shared architectural features of System z mainframes and leading non-System z enterprise servers are described in Table 7 below. The very large differences in price that exist between System z mainframe servers and other kinds of servers are therefore due not to fundamental technical differences between them, but to the monopoly pricing leverage that IBM gains over z/OS application owners by restricting the use of z/OS on alternative servers.

Virtually all the hardware components in the standard enterprise server architecture described in Table 7 are produced using industry-standard design principles and manufacturing processes. Although some critical components such as microprocessors and SMP interconnects do embed significant amounts of vendor-exclusive intellectual property, the actual scope of the differentiation is limited. The underlying technological and engineering knowledge that goes into these components is available to all industry participants and is widely discussed at trade conferences and in academic journals. While implementation details may vary as a function of vendor business objectives and marketing strategies, there are no fundamental scientific secrets between server vendors.

Most of these components are either manufactured by third parties and procured by the server vendor on the open market or, if they are manufactured by the server vendor itself, made available to external customers on standard commercial terms. One notable exception to this



rule is the z10 microprocessor used in IBM’s z10 mainframes, which to our knowledge is not available for purchase by external users even though its “silicon sister” the Power6 chip is. IBM actively promotes the entire Power chip family to other manufacturers and system vendors through its Power.org assocation73. The Power6 chip, which is the most recent member of the family, is now common both to IBM’s System i servers (running the proprietary IBM i operating system) and its System p servers (running IBM’s proprietary Unix variant AIX). We refer to the z10 microprocessor as Power6’s “silicon sister” because, according to industry sources, although the z10 implements a very different instruction set and overall processor design, it was:

§ “co-developed with and shares many design traits with the Power6 processor, such as fabrication technology, logic design, execution unit, floating-point units, bus technology and pipeline design style…”74

IBM’s decision not to make the z10 chip available to other customers most likely reflects a strategic desire to restrict mainframe competition. Whether such a restriction was the intention or not, it is certainly a natural consequence of that decision. But as the commonality with the Power6 shows there is no reason to believe that the z10 represents any unique technological or scientific breakthroughs resulting in performance or functionality that cannot in principle be found in other server systems. Therefore it is reasonable to conclude that the very large price differences observed between servers equipped with z10 processors and similarly configured servers using equivalent chips sold on the open market by semiconductor manufacturers such as Intel and AMD are due to IBM’s anti-competitive marketing practices rather than to intrinsic product differentiation.

73 See www.power.org. 74 See Wikipedia, “IBM z10 (microprocessor)”, October 2008.

http://www.power.org



Table 7: Shared architectural features of enterprise servers

Feature Description

Processors

Mass-produced 64 bit microprocessors with very high transistor counts, built with 45 to 65 nanometer semiconductor manufacturing processes, possessing clock speeds in the multiple Gigahertz range and multiple processor cores (from 2 to 8 depending on the model). Examples of such microprocessors include: • IBM’s z10 and Power6 chips; • Intel’s Xeon x64 and Itanium processor families; • AMD’s x64 Opteron family; • Sun Microsystem’s UltraSPARC T1 and T2 chips.

SMP Architecture Sockets for 2 to 64 of these microprocessors, mounted on multiple motherboards or multi-chip modules and connected by high bandwidth, low latency circuits in a Symmetric Multiprocessing (SMP) architecture in order to enable memory sharing and the loading of a single operating system “image” onto all of the processors.

Memory Large amounts of high speed random access memory (RAM), ranging from tens to hundreds of Gigabytes.

Internal Bus High speed internal bus architectures linking processors, memory, input/output subsystems, and internal disk storage.

Input/Output High speed input/output (i/o) subsystems using standardized interfaces and protocols such as Infiniband, Gigabit Ethernet, Fiber Channel, etc, used to connect the server to external storage devices and external networks such as local corporate networks, private wide area networks or the Internet;

External Storage High capacity external disk arrays presented in Storage Area Network (SAN) and/or Network Attached Storage (NAS) architectures, with data capacities ranging from hundreds of gigabytes well into the terabyte range or beyond;

Operating System

A modern multi-user, multi-tasking enterprise server operating system; examples include: • Vendor-exclusive OSs such as z/OS for IBM System z mainframes, IBM i

(formerly i5/OS) for IBM System i minicomputers; • Vendor proprietary Unix variants such as IBM AIX, HP-UX or Sun Solaris; • Enterprise Linux variants such as Red Hat Enterprise Linux (RHEL) or SUSE

Linux Enterprise Server (SLES); • Microsoft Windows Server 2008.

Virtualization

Optionally, a software virtualization subsystem or “hypervisor” functioning below the operating system level that allows the creation of multiple “virtual” machines, each running a copy of the installed operating system; examples include:

• IBM’s VM (IBM mainframes only); • VMware’s ESX (Intel and AMD processors); • open source Xen hypervisor (Intel, AMD, and PowerPC processors); • Microsoft’s Hyper-V (Intel and AMD processors).



5. Appendix B: Portability layers and migration

Defining portability layers

As discussed in section 2, successful migration of non-trivial business applications from z/OS to other operating systems such as Unix, Linux or Windows Server is both technically very difficult and rare in actual practice. It is useful in this context to clarify certain issues posed by the use of so-called “portability layers” on modern post-mainframe operating systems. Beginning in the late 1980s and early 1990s, a number of software vendors, observing the increasing diversification of operating systems in use among their customers, sought to increase the ease of migration of application code between operating systems. The vendors had an immediate and powerful financial interest in doing so, because improved OS portability promised to enlarge the addressable markets served by their products, thereby allowing them to reap higher returns on their large investments in application code development. Fostering greater portability was therefore worth substantial additional investment by software vendors in middleware, over and above the cost of developing the applications themselves. Individual user organizations that built custom applications for their own internal use, on the other hand, generally lacked such incentives to invest extra money in portability layers, and tended to spend only what was necessary to deploy applications effectively on whatever hardware and OS platform they preferred.

While implementation details vary, all portability layers adopt the same basic approach, which is to place a standardized layer of “middleware” code between the application and the operating system. Such layers, variously known as “application servers”, “virtual machines” or simply “containers”, typically share all or most of the following features. In particular, they:

§ Assume certain tasks normally assigned to the operating system, such as scheduling, memory management and security, as well as providing higher-level services such as database access;

§ Are implemented in an intermediate-level compiled language that combines good performance with (relative) ease of source code portability, most commonly C/C++;

§ Allow (or require) the use of a different high-level programming language (such as Java, SAP’s ABAP, or PHP) for the writing of actual application code on top of the portability layer, this language



being interpreted rather than compiled to provide greater ease-of-use to the end developers75;

§ Rely on underlying “portable APIs” such as the Portable Operating System Interface (POSIX), which is a standardized set of Unix APIs that has been fully implemented on most variants of Unix and fully or partially implemented on certain non-Unix operating systems such as Windows and z/OS76;

§ Exact a performance penalty for the execution of application code written in the higher-level language compared to “native” code (the penalty can range from less than 5% to more than 100%, depending on the nature of the applications and systems involved);

§ Are not themselves automatically “portable”, requiring on the contrary a significant manual programming effort as well as recompilation in order to be implemented on different operating systems, with the cost and degree of difficulty rising as the target OS diverges in basic architecture from the original source OS;

When migration works: the case of SAP R/3

Putting aside the special case of Java application servers, one of the best-known and most important portability layers of relevance to our discussion of OS migration is the SAP application server, historically known as SAP Basis but now integrated into the broader SAP Netweaver product family. SAP Basis was developed in the early 1990s as part of SAP’s R/3 ERP application. The previous generation of SAP applications, known as R/2, was originally implemented in the 1970s in mainframe assembler and ran on IBM’s MVS (later OS/390) mainframe operating system. R/3 was not a “port” of R/2 but an entirely new application, built from scratch to run on the leading versions of Unix and

75 By our definition, a Java Virtual Machine (JVM), implemented in C/C++ and interpreting programs written in Java, is a portability layer, while a “Java application server” such as IBM WebSphere (WAS) is a collection of Java runtime libraries integrated with the JVM. 76 The POSIX standards are defined and certified by the IEEE (see standards.ieee.org/regauth/posix/). All leading commercial versions of Unix and most versions of Linux are POSIX compliant. Certain non-Unix-like operating systems can be made fully or partially POSIX compliant by the addition of a compatibility layer above the OS kernel. Microsoft’s Windows Server is made fully POSIX compliant with Services for UNIX. IBM’s z/OS can be made partially POSIX compliant with z/OS UNIX System Services. POSIX requires C/C++ and cannot be natively implemented in legacy mainframe languages such as COBOL or PL/1. However, POSIX functions can be called from traditional z/OS COBOL programs using a feature known as the CALL literal statement. If porting from z/OS to a POSIX-compliant OS such as Unix or Windows Server is attempted, legacy COBOL programs must be rewritten to use the POSIX interfaces.



to use the then-emerging generation of relational database products available on Unix, such as Oracle, Informix and Sybase77.

The SAP Basis layer incorporated into R/3 was specifically designed to enable portability between modern operating systems and between databases. It was implemented in C/C++, required a POSIX layer on the target OS, and provided OS-independent services to higher-level business applications written in SAP’s own ABAP language (which was interpreted by Basis). Although SAP variously referred to Basis as a portability layer, an application server or simply as the R/3 “kernel”, we can also think of it as the combination of an ABAP virtual machine and an ABAP application server in exactly the same sense that IBM’s WebSphere contains both a JVM and a Java application server. Indeed, the more recent versions of Basis, now known as SAP Netweaver Web Application Server, incorporate both ABAP and Java virtual machines and application servers in a dual stack arrangement.

While SAP R/3 and SAP Basis were originally designed for Unix, by the late 1990s SAP had ported Basis both to Microsoft Windows NT (now Windows Server) and to IBM OS/390 (z/OS’s predecessor), leveraging C/C++ and the POSIX APIs to do so. Subsequent market acceptance of the Windows version of R/3 was extremely strong, especially after the performance and scalability improvements that accompanied the release of Windows 2000 and Microsoft’s SQL Server 2000 database. Indeed, the period since 2001 has seen a dramatic shift in the SAP installed base from Unix to Windows Server as the primary server operating system both for the SAP application server and for the database server.

However, although approximately 1,000 IBM customers chose the mainframe version of DB2 as their SAP database server between 1997 and 200578, the mainframe version of the SAP application server (i.e. Basis) was not well received by SAP customers and was in fact withdrawn from the SAP product line in 2005 when SAP desupported 32

77 While R/2 was a monolithic architecture that implemented all major tiers of the application (application server, database and user interface) on a single platform (the mainframe), R/3, in a radical departure from the traditional model, implemented a three tier architecture that placed each of the major components on its own platform. As such, R/3 and post-R/3 versions of SAP do not require the application server and the database server to reside on the same machine or even to use the same operating system, and it is standard procedure among most SAP customers to implement these components as physically distinct tiers. Thus, while the vast majority of SAP customers today implement their application severs on Windows Server or Linux, the database server tier of the SAP installed base, although dominated by Windows Server and Unix, has some z/OS implementations as well. According to a survey of several hundred SAP customers by Peerstone Research, roughly 5% of the SAP R/3 customer base was using z/OS as a database platform in 2003. See Peerstone, “Linux and ERP”, November 2004 and Peerstone, “The Future of the ERP Suite”, February 2004. 78 See Heininger and Pfister, SAP Technology Paper, “SAP Solutions on zSeries”, May 2005.



bit versions79. We believe that the primary reason for the lack of customer demand was the poor performance of Basis on z/OS, probably due to inefficiencies introduced by the POSIX layer and to other fundamental differences between z/OS and Unix discussed previously. SAP’s decision not to pursue a 64 bit version of its application server for z/OS despite the existence of a fairly sizeable installed base of SAP database servers on z/OS strongly suggests that it considered the mainframe version of Basis to be a technical dead end. Additional support for this inference comes from IBM’s long-standing refusal (discussed previously) to allow the publication of benchmarking data concerning SAP application server performance on z/OS or its predecessors.

Based on the facts we have just summarized, it is reasonable to conclude that OS portability layers of the type exemplified by SAP Basis, while permitting successful migration among the POSIX-compliant LUW operating systems (i.e. Linux, Unix and Windows), do not facilitate migration between LUW systems and z/OS. In fact, the failure of the z/OS version SAP application server and its subsequent withdrawal by SAP is a conspicuous example of the non-portability of complex business applications on z/OS.

When migration doesn’t work: the case of IBM DB2

We will conclude our discussion of portability issues by briefly considering another highly significant example of non-portability between z/OS and modern operating systems such as Unix and Windows, namely IBM’s own flagship database product DB2.

It is well known that IBM’s DB2 is not a single product but in fact a marketing label that unites several quite different products under a single umbrellla brand. The most significant of these products are DB2 for z/OS (formerly known as DB2 Universal Database) and DB2 for Linux, Unix and Windows (also known as DB2 LUW). IBM’s own web site acknowledges that the DB2 branding, particularly the now deprecated “Universal Database” label used for the z/OS version, has often conveyed the erroneous idea that DB2 for LUW represented a migration of the original mainframe DB2 product, which was not the case. As of November 2008, IBM’s web site states that:

§ “The most common misconception about DB2 Universal Database branding is that it infers that a common code base is implemented on all supported platforms and operating systems. In fact, each DB2 UDB brand member's code version is unique and developed by different IBM laboratories, but there is a tremendous amount of technology sharing at different levels that takes place across the DB2 Universal Database brand. The different code bases allow us

79 See SAP, “Application Server Strategy for SAP Solutions on IBM zSeries”, June 2005.



to exploit the hardware, microcode and operating system of each of the platforms; thus at the lowest level each DB2 is tightly integrated into, and can thoroughly exploit, its operating environment.”80

Although IBM’s official explanation for the confusing branding of the LUW and z/OS versions of DB2 is couched in vague terms, it is important in the context of our discussion of mainframe migration issues to examine the real reasons that led IBM to the costly expedient of creating and maintaining two separate code bases for its flagship database brand. These reasons are quite simple. The mainframe version of DB2 was originally developed in the late 1970s using IBM mainframe assembler (much like SAP’s R/2 from the same era). In the late 1980s IBM recognized market demand for a relational database product on what it referred to as “distributed” systems, first on the OS/2 operating system jointly developed with Microsoft, and subsequently on Unix and Windows NT (now the LUW family of POSIX compliant systems). For IBM’s developers it was out of the question to attempt a migration of the mainframe assembler version of DB2 to the distributed platforms, because it was obvious to them that this was technically impossible. The following published statement by an IBM product manager is quite illuminating in this regard:

§ “I'll be quite blunt and say that DB2 on z/OS is a different code base from DB2 on Linux, UNIX, Windows. Why? Well back in the late 80s early 90s when DB2 on the distributed platform was being developed, it made no sense to try to ‘port’ DB2 for z/OS. That code base is written primarily in assembler language which as you can imagine is great on z/OS and delivers outstanding performance due to the tight integration with the OS. But assembler does not port and when you build a database that is going to run on distributed systems (AIX, HP-UX, Solaris, Linux, Windows) you need a code base that is portable (like C or C++).”81

It is true that over the years IBM has attempted to ensure that the functionality and features of the distinct DB2-branded products are similar from the standpoint of application developers and database administrators. This is particularly so in the areas of SQL (Structured Query Language) compatibility and query optimization. In effect, IBM has over a period of nearly 20 years evolved the DB2 family into the equivalent of a high-level portability layer, situated this time not at the interface between applications and operating systems but between applications and databases. However, fundamental differences remain between the z/OS and LUW versions of DB2 in areas such as database clustering. Mainframe z/OS clustering uses Parallel Sysplex to implement shared memory, much like Oracle’s Real Applications Clusters (RAC) feature on Linux, Unix and Windows. DB2 on LUW, by

80 See Milligan, “DB2 UDB Family On Common Ground”, April 2002 [www.ibm.com/developerworks/db2/library/techarticle/0204milligan/0204milligan.html}. 81 See Eaton, “An Expert's Guide to DB2 Technology”, October 19, 2005 [it.toolbox.com/blogs/db2luw/db2-on-linux-unix-windows-for-a-zos-dba-6198].

http://www.ibm.com/developerworks/db2/library/techarticle/0204milligan/0204milligan



contrast, implements shared-nothing clustering, also known as database partitioning, which is generally recognized to be a less sophisticated feature. An application that relied on the z/OS version of DB2 clustering would not be portable to the LUW version without a major redesign and significant loss of performance.

The example of DB2, like that of SAP, demonstrates once again that complex business applications which have been carefully crafted to exploit the features of a mainframe operating system such as z/OS or its predecessors generally cannot be transplanted to modern operating systems such as Linux, Unix or Windows Server. In fact, most so-called “migrations” of legacy mainframe applications are really “reconstructions from scratch”. In other words, as in the case of DB2 LUW or SAP R/3, they are largely or entirely new applications, developed with incompatible programming languages, using different APIs and middleware, that seek at great cost and considerable technical risk to emulate certain aspects of their mainframe predecessors. Despite its high cost and risk, this migration-by-reconstruction approach makes sense for certain software vendors who have established widely recognized brands and can look forward to a large increase in the size of their addressable markets in return for their additional investment. But for the vast majority of owners of legacy z/OS applications there can of course be no such expectation. For these users a far less costly and less risky form of migration is to move the entire z/OS operating environment, together with its attached applications, to an alternative mainframe-compatible hardware platform using one of the mainframe instruction set emulation techniques discussed above.



6. Appendix C: Mainframes and one monopoly rent theory

One IBM mainframe monopoly or several?

For the sake of simplicity, in this paper we have adopted the assumption that the z/OS mainframe operating system and System z mainframe servers constitue separate monopolies, according to the scheme described in section 1 where z/OS is the controlling monopoly and System z hardware is an adjacent monopoly that IBM has obtained by tying the two products together through restrictive licensing. This assumption allows us to compute the cost of the System z hardware monopoly separately from the cost of the z/OS operating system monopoly. It should be noted in passing however that there is an extensive academic literature devoted to so-called “one monopoly rent” theory82 which would seem to suggest that, contrary to our assumption, z/OS software and System z hardware could in fact constitute a single monopoly. The implication of this theory is that if IBM were compelled to license z/OS in order to promote competition for mainframe hardware, it could respond by simply raising its price for z/OS alone, thereby eliminating the customer savings gained from hardware competition. However, there appears to be signifcant disagreement among the relevant academic and regulatory communities concerning the validity and applicability of one monopoly rent theory83.

In our view there are several reasons why “one monopoly rent” theory does not apply in the real world of high-value IT systems procurement by large and midsize corporations. First, as a matter of empirical observation, IBM itself does not appear to adhere to “one monopoly rent” theory in its own commercial practices. On the contrary, it very conscientiously distributes the aggregate cost of its mainframe monopoly over all the major components of the stack, including the operating system and related middleware, the server hardware (i.e. System z), mainframe-compatible storage and networking systems, hardware maintenance fees, and professional services. Second, if the entire pricing burden of the IBM mainframe monopoly were to be carried by z/OS alone, the resulting extreme price differential between the (monopoly priced) operating system and the other (competitively priced) components of the mainframe stack would be highly visible to customers and would be likely to provoke retaliation. For example, although users of z/OS legacy applications would still have no choice but to pay IBM’s price for z/OS, they could retaliate by refusing to buy other unrelated IBM products such as software and servers designed for LUW operating systems or related 82 See “Competition and Intellectual Property Law and Policy in the Knowledge-Based Economy”, FTC, 2003 [www.ftc.gov/opp/intellect/index.shtm]. 83 As noted by Pate in “Antitrust and Intellectual Property”, Department of Justice, 2003 [www.usdoj.gov/atr/public/speeches/200701.htm].

http://www.ftc.gov/opp/intellect/index.shtm

http://www.usdoj.gov/atr/public/speeches/200701.htm



professional services. Third, results from behavioral economics as developed by Kahneman and Tversky suggest that IT buyers like other consumers are subject to the constraints of “bounded rationality” and in particular are likely to reject offers that they deem “unfair” even when it is in their apparent self-interest to accept them. In other words, some mainframe customers might choose to forego their legacy applications and purchase new applications on alternative platforms in order to “punish” what they perceived as unfair behavior by IBM, even in the absence of an expected positive return on such investments84.

Finally, the antitrust applicability of one monopoly rent theory to z/OS and System z hardware is questionable, because IBM is not the sole supplier of server systems capable in principle of supporting the z instruction set with good performance characteristics85. On the contrary, as we argued in Appendix A above, the fundamental architecture of IBM-brand z/OS-compatible servers (i.e. System z) has substantially converged over the past decade with the architectures of both IBM and non-IBM servers designed for LUW systems. Since these latter servers are, with the addition of appropriate firmware or software implementing the z/Architecture instruction set, perfectly capable of serving as effective platforms for z/OS, and since the technical innovations in recent generations of System z hardware are not intrinsically different from or superior to those observed in LUW servers based on non-IBM microprocessors, there is no benefit to customers in IBM’s unilateral extension of its z/OS monopoly to encompass all “legitimately licensable” z/OS-compatible hardware86. The tying is merely an abitrary exercise of market power by a monopoly incumbent vendor and does not result in increased technical innovation, greater efficiencies, lower aggregate prices or any other benefit for mainframe customers.

84 For an accessible review of behavioral economics, see “Behavioral Economics: Past, Present, Future”, Camerer and Loewenstein, 2002 [www.hss.caltech.edu/~camerer/ribe239.pdf]. Recently some scholars have used behavioral economics to challenge the traditional unquestioning acceptance of Chicago School rational choice theory by antitrust authorities. For an extended discussion see “Behavioral Economists at the Gate: Antitrust in the 21st Century”, Stucke, 2007 in University of Tennessee College of Law Legal Studies Research Paper Series #12 [ssrn.com/abstract=981530]. 85 For a discussion of one monopoly rent and related theories in the context of complementary products in the high technology sector, see “Innovation, Rent Extraction, and Integration in Systems Markets”, Farrell and Katz, 2000 [oz.stern.nyu.edu/phd04/farrell.pdf]. 86 In other words, the extension of the z/OS monopoly to System z does not result in the “internalization of complementary efficiencies”, to use economist Joseph Farrell’s characterization of the primary justification for avoiding antitrust action against a “single monopolist”. See Farrell’s comments in “Federal Trade Commission and Department of Justice Antitrust Division Roundtables: Competition and Intellectual Property Law and Policy in the Knowledge-based Economy”, Nov 2002, p. 132 [www.ftc.gov/opp/intellect/021106ftctrans.pdf]. For additional arguments of a more technical nature against one monopoly rent in this context, see “Tying, Foreclosure, and Exclusion”, Whinston 1989 [papers.ssrn.com/sol3/papers.cfm?abstract_id=227224].

http://www.hss.caltech.edu/~camerer/ribe239.pdf

http://www.ftc.gov/opp/intellect/021106ftctrans.pdf



In sum, we believe it is highly unlikely that IBM would be able to respond to a competitive market for z/OS-compatible server hardware by transferring the above-market profit margins it currently obtains from System z mainframes to the monopoly license cost of z/OS itself. If it attempted to do so, its customers would revolt. An IT vendor such as IBM with a broad portfolio of products that obtains the lion’s share of its revenue from a few thousand large IT buying organizations can not afford to alienate its customer base in this way87. Rather than raising z/OS prices to compensate for the loss of monopoly profit margins on System z hardware, it is far more likely that IBM would attempt to preserve share in a competitive z/OS-compatible server market by cutting hardware prices and by attempting to gain competitive advantage through increased technical innovation, both of which actions would bring substantial benefits to z/OS customers.

We have therefore based our calculation of the costs of IBM’s mainframe monopoly in this paper on the assumption that the excess costs due specifically to IBM’s monopoly over z/OS-compatible servers can be isolated from the costs of the z/OS monopoly itself. However, this assumption is not fundamental to the methodology we use to make these calculations. If the one monopoly rent theory were to be retained, we could readily apply the same methodology to calculate the total excess cost imposed on Europe by IBM mainframe pricing taken as an indivisible single monopoly. Since it would include z/OS, this cost would be considerably higher than the figures we give in section 188. Furthermore, even with the retention of monopoly rent theory, all of the “behavioral economic” aspects of customer responses to IBM pricing described above would still hold true. In particular, if IBM were required to license z/OS for use on alternative server hardware and thereby compelled to shift the burden of its monopoly rent to z/OS pricing alone, the resulting negative customer reactions would weaken the z/OS monopoly and perhaps ultimately threaten its existence. In other words, even under the conditions of one monopoly rent theory, IBM’s extension of its z/OS operating system monopoly to the adjacent market of servers capable of running z/OS must be seen as a defensive measure on its part to preserve the viability of the z/OS monopoly. In sum, although we have

87 In its September 2008 quarter IBM reported that 81% of its total revenue derived from large IT buying organizations vs. 19% from Small & Medium Businesses. See IBM, “3Q 2008 Earnings Presentation”, p. 6 [http://www.ibm.com/investor/3q08/presentation/3q08.pdf]. 88 An indication of just how much higher the cost would be if we were to include z/OS is given by IBM’s New Application License Charges (NALC), which grant large discounts on z/OS license pricing to users who run qualified new workloads (e.g. Java, WebSphere, or certain ERP packages such as SAP). According to Gartner, NALC charges can be as low as 5% of standard IBM charges for running z/OS with legacy applications. See Gartner, “The IBM Mainframe Platform: Ongoing Challenges, New Opportunities”, Mike Chuba, December 2008, and IBM, “System z New Application License Charges” [www-03.ibm.com/servers/eserver/zseries/swprice/znalc/]. In other words, IBM itself values the monopoly rent it derives from legacy z/OS licenses at 95% of the price it charges for these licenses.

http://www.ibm.com/investor/3q08/presentation/3q08.pdf



rejected the application of one monopoly rent theory to this case for the reasons given in the previous paragraphs, our overall position on this theory remains neutral.



Author’s Biography

Jeff Gould is the founder and CEO of Peerstone Research, a San Francisco-based IT industry consulting and market research firm. He is the author of numerous publications about enterprise software and hardware. Prior to founding Peerstone in 2001, he served as Editorial Director for the European operations of American IT publisher CMP Media, where he oversaw the European editions of InformationWeek and Computer Reseller News. Prior to joining CMP Europe in 1994, he was the founder and CEO of Datastrategies SA, a Paris-based IT market research firm specializing in surveys of European corporate IT users. He has extensive experience in all aspects of enterprise computing in both Europe and the United States, and is a UK-US dual national. He can be contacted at [email protected].

mailto:[email protected]

Documents

Euro mainframe report cover page