Rod Johnson Cathedral

The Cathedral, the Bazaar and the Commissar

Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited.

The Evolution of Innovation in Enterprise Java

Topics

• The Sources of Innovation

• A History of Innovation in Enterprise Java

• Where next?

Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 2

• Where next?

Assumption:

Innovation matters and is essential to


Innovation matters and is essential to the flourishing of a technology platform

What Drives Innovation?

• There has been much research into the drivers of innovation

• Key themes identified include

– Creativity


– Creativity

– Experimentation

– Competition

– Economic motivation

Sources of Creativity

• Source: National Center of Education and the Economy (2006)

• Creative Thinking

– Comfort in disagreeing with others and trying


– Comfort in disagreeing with others and trying solutions that depart from the status quo

– Combining knowledge from previously disparate fields

– Ability to persevere through difficult problems and dry spells

– Ability to step away from an effort and return later with a fresh perspective (“incubation”)

Value of Experimentation in Innovation

• Stefan Thomke of Harvard Business School argues that every company’s ability to innovate depends on a series of experiments [successful or not], that help create new products and services or improve old ones. That period between the earliest point in the design cycle and the final release should be filled with experimentation,


final release should be filled with experimentation, failure, analysis, and yet another round of experimentation.

• Experimentation can only work if failure is punished and failed ideas are killed off

Innovation and competition

• The Department of Justice and the Federal Trade Commission have frequently raised innovation concerns as reasons to challenge mergers


• Harvard Business School – “Competition and Innovation” paper

– Empirical work…has pointed to a positive correlation between product market competition and innovative output

The ages of Enterprise Java

• Before J2EE

• The promise of J2EE

• The decline of J2EE

• The rise of open source


• The rise of open source

Before J2EE

• Mid 1990s

– Java gradually moves to the server side

• Largely unregulated

• Many competing products in different areas

– NetDynamics


– NetDynamics

– TopLink

– Silverstream

– Persistence PowerTier

– Apple WebObjects

Before J2EE…

• Good and Bad– Innovation and choice of approaches

–Fragmented server-side market

–Real danger of vendor lock-in


–Real danger of vendor lock-in

–Many solutions very expensive,• No impact from open source

The Glory Days of J2EE

• 1999-2003

– The JCP becomes dominant in the space

• TopLink and other “non-standard” technologies cannot compete with J2EE


technologies cannot compete with J2EE standards

– ORM versus EJB entity beans

– Velocity vs JSF

– WebObjects vs web tier

The Glory Days of J2EE

• Good and bad

– A market is created

– Vendor lock-in is reduced, but not eliminated


– Increasing thought control strangles innovation

– Flaws in the model take years to be resolved, while many projects fail

The decline of J2EE

• 2003-

• Move away from traditional application server towards lighter-weight

• Proportion of enterprise Java users using Tomcat

40

50

60

70


lighter-weight solutions such as Tomcat

– Tomcat now clear leader in enterprise Java deployments

0

10

20

30

40

WAS JBoss WLS Tomcat

Springframework.orgBZ Research

The rise of open source

• Fewer and fewer organizations develop enterprise Java applications without using open source

– Those that do face increasing competitive


– Those that do face increasing competitive disadvantage

• Numerous open source projects help to shape the future

– Eclipse

– Spring

– AspectJ

The three key sources of innovation in Enterprise Java

• The Cathedral

• The Bazaar

• The Commissar


• Let’s understand each

• Understand why none alone is sufficient

The Cathedral or Bazaar?

• Cathedral model has software built by relatively few people, with centralized design

• Bazaar has many deveolopers who lay out their wares


• Classic statement on open source by Eric Raymond, 1996– Linux is subversive. Who would have thought

even five years ago (1991) that a world-class operating system could coalesce as if by magic out of part-time hacking by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet?

Reality check

• As Linux has matured into enterprise use, it’s no longer predominantly developed this way


Case Study: Linux

• Dispelling the perception that Linux is cobbled together by a large cadre of lone hackers working in isolation, the individual in charge of managing the Linux kernel said that most Linux improvements now come from corporations.

“People’s stereotype [of the typical Linux developer] is of a male computer geek working in his basement writing code in his spare time, purely for the love of his craft. Such people were a significant force up until about five years ago,” said Andrew Morton, whose role is maintaining the Linux kernel in


Andrew Morton, whose role is maintaining the Linux kernel in its stable form.

Morton said contributions from such enthusiasts, “is waning.” Instead, most code is generated by programmers punching the corporate time clock.

About 1,000 developers contribute changes to Linux on a regular basis, Morton said. Of those 1,000 developers, about 100 are paid to work on Linux by their employers. And those 100 have contributed about 37,000 of the last 38,000 changes made to the operating system.

Not everyone wants to play

• As open source becomes mainstream, more and more users of open source won’t want to contribute and shouldn’t need to

• “The myth of open source software is the aura of freedom that surrounds it. Download the source code and play with it if you wish. And best of all, you won’t have to pay the freight.


Although that’s the public image of open source, the reason why open source software is growing popular within enterprises has nothing to do with open source itself. Few enterprises care about whether they can monkey around with source code because the relative minority that still have active internal software development staffs have more important things to do.”

Tony Baer, Datamonitor Computerwire

Myth: Community magically generates open source software

• All we need is lots of developers

• With all those fingers on all those keyboards, great software just happens


happens

Reality check

• Innovation is not a numbers game

• The bazaar model encourages competition in implementation but may not produce innovation


not produce innovation

• The cathedral model is more likely to produce innovation

• Neither is a complete solution

The Commissar

• Java has its own somewhat unique model

– The Commissar


• In this model, the politburo knows what’s best for the proletariat (you)

The Commissar Knows Best

• JCP expert groups talk largely in private

• Typically composed of software vendors


software vendors

• Relatively slow pace of change, like Soviet 5 year plans

Why standards are needed

• Standards can create markets

• Standards can provide a base on which competing open source and commercial alternatives can flourish

– JTA

– Servlet API


– Servlet API

– JMS

• Standards can protect customers from lock in to a proprietary technology

• To ensure interoperability

– Web Services

– IIOP

How much standardization is too much?

• In the Java world we have an unhealthy obsession with standards

• Desire to standardize everything

• Failure to critically evaluate standard


• Failure to critically evaluate standard technologies

Where Standards Don’t Work

• Jim Waldo (Sun Distinguished Engineer)

– Kowtowing to the god of standards is, I believe, doing great damage to our industry, or craft, and our science. It turns technical discussions into political debates. It misunderstands the role that standards have played in the past. Worst of all, it is leading us


have played in the past. Worst of all, it is leading us down absurd technological paths in the quest to follow standards which have never been implemented and aren’t the right thing for the problems at hand.

Where standards don’t work

• CORBA history (1990s)

– Death by committee

– Attempts to innovate by committee (distributed persistent objects)


• When they’re too slow

• When they’re divorced from reality

– Ivory castle

• When they are about politics, not technology

Politics

• Like the Soviet Union, the JCP has also seen its great purges

• JDO

– Arguably still the best technology choice for ORM


ORM

– JPA is promising, but JDO 2.x remains superior

– Yet JDO is dead in the market

• Database vendors didn’t like it and they are influential in the JCP

• No application server vendor liked it as it competed with EJB technology

• …so it was taken out and shot

Today’s Political Struggle: OSGi vs JSR-277

• OSGi: Proven technology for dynamic modularization

• From the OSGi Alliance

• Yet JSR-277 attempts to reinvent modularization for Java


modularization for Java

• Make or break issue for the JCP

• IBM, Oracle, BEA and many other vendors are building their middleware strategy on OSGi

– They are not going to walk away just because Sun doesn’t like it

Enterprise Java is no longer a one party state

• Not just the Party (the JCP)

• OASIS

– SCA


– SCA

– Web Services standards

• OSGi Alliance

– Dynamic modularization standards

– More enterprise standards

• Open source projects

The JCP must look at wider world and accept that it doesn’t need to reinvent everything

• History of failure when this has not happened

JCP technology Ignored existing technology

Negative consequences

Entity beans TopLink and all other ORM solutions

•Two complete failures (EJB 1.x and 2.x) •ORM in Java loses at least 6 years•Billions of dollars of wasted development effort from customers

Commons Log4J Added complexity of pointless abstraction


Commons Logging

Log4J Added complexity of pointless abstraction layers such as Commons Logging

EJB (DI) Spring, PicoContainer, Hivemind

Limited DI functionality in EJB 3 specification misses opportunity to match best practice

EJB3 (interception)

Spring, AOP Alliance, AspectJ, AspectWerkz

Lack of knowledge of AOP in the expert group produces fragile, clunky API missing central AOP concepts

JSR 277 (modularization)

OSGi •Ignoring input and experience from OSGi•May split JCP as many organizations are deeply committed to OSGi

The standards check list

1. Will the pace of change and innovation required by met in a standards process cycle

2. Do we benefit from competing implementations?

3. Does this affect wire protocols (in which standards are probably outside Java)

4. Is there an entrenched open source solution, in


4. Is there an entrenched open source solution, in which case competition may not occur?

5. Is the field mature and well understood

1. Has the proposal been tested in the market?

2. Do not want design by committee?

6. Is the standards committee representative of the users of the technology?

Does the standard exist to lock out newcomers?

• When standards become too complex, like J2EE, they effectively lock out new entrants and benefit existing franchises, not consumers

• Study of the NHS

– In open markets the threat of entry by newcomers


– In open markets the threat of entry by newcomers not only puts pressure on prices; it also acts as a pressure towards innovation because if, say, a hospital – or new provider – pioneered new techniques that provide higher quality, more cost-effective, care, it should both attract more patients and make greater profit

Competition is diminishing in Java EE

• Oracle purchase of BEA leaves IBM, Oracle dominant in Java EE


• Since Red Hat acquisition, JBoss momentum has stalled

Just the right amount of competition

• There must be a reward for innovation

• Bad solutions must die

• Standard economic theory predicts that


• Standard economic theory predicts that innovation should decline with competition, as more competition reduces the monopoly rents that reward successful innovators

The JCP and creativity

Creativity enabler JCP Open source

Cathedral

Comfort in disagreeing with others and trying solutions that depart from the status quo

Combing knowledge from previously disparate fields


disparate fields

Ability to persevere through difficult problems and dry spells

Ability to step away from an effort and return later with a fresh perspective

Competition and elimination of bad ideas through market pressure

Create market

JCP: Conclusion

• The JCP is unlikely to produce innovation but should focus on what it can succeed at

– Creating a market where innovators can


– Creating a market where innovators can compete above fundamental stanards

• Innovation by committee is a bad idea, and has traditionally produced poor results

Innovation needs Competition: External Forces

• Ruby on Rails

• .NET

• External competition is very important


• External competition is very important to Java

• Good to Great

– One of the companies profiled succeeded because they chose to compete with a highly efficient larger competitor, with the aim of lifting their own performance

The Commissar doesn’t like Competition

• Communism views competition as wasteful and the root of all evil

– The proletarian liberates himself by abolishing competition, private property, and all class differences


and all class differences

• Friedrich Engels - The Principles of Communism (1847)

The Reality

• Competition leads to the best performance, even among communists

• Ironically, the Soviet Union only ever succeeded when forced to compete to survive


But lack of inherent competition in the system brought it down

• What really destroyed the USSR was the rapid advancement of technology developed within the fires of cooperative competition among Western nations. The Soviets had brilliant scientists and researchers, but they couldn’t


scientists and researchers, but they couldn’t outmaneuver the efforts of thousands of individual companies, whose efforts were being selected by merit in a free market by millions of consumers.

Things need to become faster

• Competition and experimentation needs to occur rapidly

• As with the Soviet Union, technology change and the increasing pace of


change and the increasing pace of business leaves 2-3 years committee-driven cycles looking less and less relevant

Open Source implications: MySQL

• Open source produces fast experimentation/review cycles

• Biggest event in the future of the JCP is


• Biggest event in the future of the JCP is not connected to Java

• Sun is becoming an open source company

• The JCP is not a true standards body, so is inevitably shaped by Sun’s agenda

Evidence of positive change

• JCP sessions, chaired by Patrick Curran (JCP chair) show that Sun is opening up

• The JCP has gradually evolved over time to be more open and less bureaucratic


to be more open and less bureaucratic

• Needs to change much more and much faster, but there is real hope…

What does Tomorrow Look Like?


The standardization cycle

Breakdown of standards process

Breakdown of standards process

No standardsNo standards

Lack of innovationLack of innovation


StandardizationStandardization

Create coherent market

Create coherent market

Lock out new entrants /

Produce lowest common denominator

Lock out new entrants /

Produce lowest common denominator

The future

• No longer a one party state

– No one vendor or organization (even Sun) will control all the pieces

• OSGi is shaping up as a make-or-break test


• Three sources of innovation

– Cathedral (proprietary vendors)

– Bazaar (open source)

– The Commissar (JCP)

• All three are needed

The future

• Open source innovation will continue

– Apache

– Eclipse

– Spring


– Spring

• Change needs to be more rapid

– JCP needs to adapt to survive

– Likely to be run according to true open source

The future

• Remember that you can be more than a spectator

1. Evaluate technologies on merit, not necessarily because of where they come from


from

2. Choose the technologies you want to use

3. Take the opportunity to participate in the JCP if you want to help keep it relevant

1. It’s not just a question of Sun “fixing” the JCP

Q&A


Technology

Rod Johnson Cathedral