How to (Almost) Kill a Successful Project and Bring It

Back to Life Again:

Lessons Learned from the Xen Project

Russell Pavlicek

Xen Project Evangelist

Citrix Systems

Russell.Pavlicek@xen.org

So Who is the Old, Fat Geek Up Front?

A guy with a lot of experience and a really

big mouth

About the Speaker...

● Linux user since 1995; Linux desktop since 1997

● Linux advocate before I ever saw the software

● Early advocate in DEC, Compaq

● Former columnist for Infoworld, Processor

● Former panelist on The Linux Show

● Wrote book, Embracing Insanity: Open Source Software Development

● Speaker at 40+ conferences

● Currently Xen Project Evangelist employed by Citrix

About This Talk

● We will spend a couple minutes reviewing the

project

● We will spend a few minutes considering its history

● But we will spend the bulk of the time considering

lessons to take away

● We are not here for the project's history; we are here

for your future

What is the Xen Project?

● The premier Open Source Hypervisor

● Powering some of the biggest Clouds in the industry

(Amazon, Rackspace, Terremark)

● Celebrating its 10th Anniversary

● Now a Linux Foundation Collaborative Project

– Sponsoring organizations include: Google, AMD, Intel,

Samsung, Cisco, Oracle, Amazon, Verizon and more

What Does the Xen Project Produce?

● The Xen Hypervisor, including ARM servers

– Type 1 (Bare Metal)

● Xen Cloud Platform & XAPI

– Cloud readiness out of the box

● Other Projects

– Mirage

– ARM Hypervisor for mobile devices

The Xen Project Story (30 second

version)

● It was the first industrial-strength Open Source

Hypervisor

● It enjoyed a very high rate of adoption

● It employed excellent technology

● It had a FOSS-friendly corporation behind it

● And, yet, 2 years ago, it ran the risk of being

abandoned by the FOSS community before its 10th

birthday

How Did This Happen?

● The project, though viable, developed an inward

focus

– Reach out to the rest of the Open Source community was

limited

– Reach out to its users was minimal

– Code development continued, but the community became

insulated and stagnated

– No one stepped up to contradict the rumor that Xen was

dying technology, overcome by competitors

How Did This Happen? (continued)

● The project forgot the importance of working with

its ecosystem

– Upstream projects (Linux, QEMU) were branched rather

than engaged with patches

– The project decided that others in the ecosystem (i.e., the

distributions) would have to carry the burden of

maintaining and supporting those differences

– This went on too long, and the ecosystem got fed up

How Did This Happen? (continued)

● The corporation backing it (XenSource) was sold to

a company with a long closed source software

history (Citrix)

● The new corporation was interested in the

technology, but had no particular interest in the

project itself

Why Did This Happen?

● It was not about malice

● It was not about fear

● It was about disconnection

– The project became disconnected from the FOSS

community

– The project became disconnected from the users

– The new company became disconnected from the needs

of the project, because, in part, the project never really

explained what it needed from the company

About Two Years Ago: Battling for a

Future

● The prognosis was not good

● Xen Hypervisor had been overtaken by a

commercial offering in IT mindshare

● Xen Project had been overshadowed by another

Open Source Hypervisor in the community

● Distributions stopped facilitating Xen

● The FOSS Community began to forget Xen

A Conscious Reversal in Direction

● About 2 years ago, a new direction was plotted

● Citrix decided it wanted to understand FOSS and

reinvigorate the Xen community

● The company began to hire FOSS-savvy people to

reconnect with the community and with users

● Brought about efforts to birth Apache CloudStack,

OpenDaylight, and to move the Xen Project under

the Linux Foundation

Reality Two Years Ago: Xen Who?

● Common themes heard at FOSS events:

– What is Xen?

– Xen is dead, right?

– Isn't Xen closed source?

– No one uses Xen anymore

Reality Today: Xen is Back!

● Linux kernel 3.0 contains all that Xen needs to exist

by default

● Most Linux distributions are Xen-enabled

– CentOS has a project to give RHEL6 users a Xen option

● Xen Project now part of Linux Foundation

● Launch of a new user-friendlier website

(XenProject.org)

So What Did We Learn?

Lesson 1

● It is possible to die while you are winning

– Being first is not enough

– Great technology is not enough

– Having a FOSS-friendly corporation behind you is not

enough

● A project must stay vibrant as an Open Source

organism or it will perish

Lesson 2

● Disconnection from users can make you a “Dead

Project Walking”

– You can be adding functionality, issuing new releases,

and still be dying

– The connection between project and users is essential

– Focusing on software alone is not enough

● If you are not interacting with your users, you are at

serious risk

Lesson 2 (continued)

● Connecting with your developers != connecting with

your users

– You need information sources for both users and

developers

– If users have to dig through technical websites, wikis,

etc. to answer simple questions, you are in trouble

– Even Linux kernel development – arguably one of the

most insular projects – cannot thrive in a vacuum

Lesson 3

● Never ignore your project's Open Source root

structure

– Cut flowers are beautiful – until they die

– Living plants need their roots

– FOSS community is the root structure, and it must spread

wide

– The project team cannot stand alone

● Pay attention to your partners in FOSS: libraries,

kernel, packaging

Lesson 4

● Never ignore your support structure

– Xen needed cooperation from Distributions to be

properly supported

– The relationship with the distributions was allowed to

stagnate; it was not continuously cultivated

– When one distribution invested in another Open Source

virtualization solution, other distributions were swayed

● Your distribution route can be critical to success

Lesson 5

● Having corporate backing is not enough

– The corporation has its own set of goals, and they rarely

align exactly with the project's goals

– When the project and the company don't mesh, friction

can occur

– This isn’t about good versus evil; business and projects

are just two separate animals with different needs

Lesson 6

● Having a FOSS company backing you is no

guarantee

– Even FOSS-centric companies can be sold

– Sometimes they are sold to other FOSS companies (e.g.,

JBoss, Gluster)

– Sometimes they are sold to closed source companies

(e.g., MySQL, Xen, Cassatt)

● If a project won’t survive without FOSS company

backing, consider options (e.g., Linux Foundation)

Lesson 7

● In FOSS, there is no such thing as autopilot

– Intent is critical

– If you are not planning to succeed, you are planning to

fail

– Great software is not enough; you can have the best

technical solution, but if a “big dog” starts throwing its

weight around, you need to be able to respond

– If you're not looking at your whole ecosystem, you are

inviting failure

Lesson 8

● If it ain't growing, it's dying

– If your project team is seeing no new blood over time, be

concerned

– Open Source organisms must move and grow

– New folks are needed from time to time to add new ideas

and keep focus on what users need

Lesson 9

● Know where your project could fit in the world and

make a plan to get there

– Competition means you probably won't be best fit for

every situation

– It may not be possible to have every feature your

competitors have (especially if they have much corporate

backing)

– Figure out who your users are, what they need, and what

you need for them to use your code

Lesson 10

● Competition Increases Innovation

– Lack of competition can cause stagnation (consider Unix

CDE)

– Competing technologies keep the ball moving forward

continuously

– Xen's competition with KVM and VMware insured that

new virtualization capabilities would keep flowing

– A competing project has to stay on top of its game or it

won't make it

Lesson 11

● Major new features can keep your mindshare alive

in the community

– Large advances (e.g., ARM and Mirage) generate

attention from the FOSS ecosystem and the userbase

– If you aren’t making headway, your root and support

structures may stop working to give you what you need

– Periodic advances keeps the project growing

Lesson 12

● Sometimes, perception really is reality

– You can have the best code in the world, but if no one

cares, it’s useless

– If the rumor arises that you are dying, outmoded, or

outdone by some other project, you must fight that

perception

– The unchallenged lie will become fact for many people

Lesson 12 (continued)

● In contrast, KVM managed perceptions well

– It could have looked like a purely Red Hat/IBM business

play when Red Hat purchased Qumranet

– The relationship between Red Hat and the KVM project

was well-defined & appropriate; no disconnect occurred

– FOSS community embraced KVM project

– Clearly, Red Hat and IBM are still focusing major

business initiatives on KVM, but the community accepts

that because it was done correctly

Crash Course in Perception

Management

● Go to local FOSS events

– Submit talks

● Get used to rejection and learn from it

● Talk to the track chairmam

– “I don’t like speaking” – get over it; calculus was

way harder than this

– Get an ORG booth for cheap

● Stock it with flyers, CDs, business cards,

stickers

● Shoot your mouth off

– Blogs

– A usable website

– Podcasts (TLLTS)

– Social Media

● Twitter, Facebook, Google+, LinkedIn

● More mouthing off

– YouTube demos and tutorials

– Write for Linux.com Lxer, LWN.net

● Get a “kick-*aas” mascot!

– But our buddy Xen Fu is taken!

● Shout out and live, or shut up and die!

– Passion is your ally

– Let it leak over everyone

– Don’t imitate the suits; do what fits you

Lesson 13

● There's a new reality for FOSS projects: the

corporate connection

– Projects used to be primarily volunteers working nights

and weekends

– Today, corporations play a big role in development

● You need to have a good grip on what your

corporate sponsors want from you, and what you

need from them; disconnection can be fatal

Lesson 13 (continued)

● Manage the relationship between business and

project

– Prevent the loss of the project’s identity

– If the project appears “owned” by a business, the FOSS

community might become suspicious and back away

– In this case, perception is as dangerous as reality

– If you forget what the project is, every else will too

– Project ecosystem will wither away; only the business

remains

Lesson 13 (continued)

● Establish a symbiotic relationship

– Business provides user feedback, resources

– Project provides overall focus, goal, direction, labor

– Both sides need to color in the lines

– Otherwise, you get “fake Open Source:” the code is

open, but there is no community, no support, no

ecosystem

Final Thoughts

● Make sure your project addresses its entire

ecosystem:

– Is the code good?

– Are you reaching out to your users?

– Is your development community active, engaged, and

growing?

– Are reaching out to the FOSS community?

– Have you insured you have proper support (libraries,

distros, kernels, etc.)?

Final Thoughts (continued)

● If you are in a relationship with a corporation, is that

relationship healthy?

– Do you have freedom to do what you need to do?

– Are you getting user feedback to seed new growth in the

project?

– Is your project's identity getting lost in the corporate

identity? (if so, consider a foundation route)

● Whatever else, don't give up!

Questions?

Russell.Pavlicek@xen.org

XenProject.org