20
Who are ’we’? How and why we bootstrapped Functional Programming Evaluation Summary Functional Programming @ Ghent IT Valley R. Slootmaekers N. Trangez Incubaid Research Lab Commercial Users of Functional Programming, 2012 R. Slootmaekers, N. Trangez FP@GhentITV

Cufp 2012 talk

Embed Size (px)

DESCRIPTION

slides from the talk we gave in Copenhagen at cufp 2012.

Citation preview

Page 1: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Functional Programming @ Ghent IT Valley

R. Slootmaekers N. Trangez

Incubaid Research Lab

Commercial Users of Functional Programming, 2012

R. Slootmaekers, N. Trangez FP@GhentITV

Page 2: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 3: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Incubaid is a technology incubator, active in datacenterand cloud computing

Prior exits through Terremark, Telenet (Belgian telco),Veritas/Symantec, Sun Microsystems,. . .

some of our current involvements: Amplidata , Awingu ,CloudFounders , Dacentec , Racktivity

R. Slootmaekers, N. Trangez FP@GhentITV

Page 4: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 5: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Amplidata’s Dispersed Storage System20 000ft view

Client

encoder

decoder

provisioning

policies

m e t a d a t a

StorageNodeA

.... . . . .

S torageNodeZ

diskA0

diskA1

diskA...

client streams object dataencoder generates equations : Ax = b or x = A−1bpolicies determine how many disks | equationsprovisioning determines which diskssend equations to StorageNodesrecord metadata

R. Slootmaekers, N. Trangez FP@GhentITV

Page 6: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 7: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Pain

Pain: resource management

Pain: sockets

Pain: threads

Pain: data corruption

Pain: objects

development was grinding to a halt

R. Slootmaekers, N. Trangez FP@GhentITV

Page 8: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Pain

Pain: resource management

Pain: sockets

Pain: threads

Pain: data corruption

Pain: objects

development was grinding to a halt

R. Slootmaekers, N. Trangez FP@GhentITV

Page 9: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

The 1.6 Codebase (Core)

0

1

StIO MdIO Codec Rest

Time (Blue):Pareto applies60% is IO30% is encoding/decoding

size (Red):> 50 klocCodec ∼ 1 klocIO < 1 kloc

R. Slootmaekers, N. Trangez FP@GhentITV

Page 10: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

The Bet

I placed a bet to rewrite the StorageNode component in OCaml

Functionally identical

Same Performance

in 2 days. . .

1.5 day later the drop in replacement was demoed, and wewere allowed to Investigate alternatives to C++ for version 2.0

R. Slootmaekers, N. Trangez FP@GhentITV

Page 11: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

The Bet

I placed a bet to rewrite the StorageNode component in OCaml

Functionally identical

Same Performance

in 2 days. . .

1.5 day later the drop in replacement was demoed, and wewere allowed to Investigate alternatives to C++ for version 2.0

R. Slootmaekers, N. Trangez FP@GhentITV

Page 12: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

How is this possible?

component is very simple

perfect specs

existing component test suite

performance is mostly determined by network & disk speed

OCaml has object orientation.

R. Slootmaekers, N. Trangez FP@GhentITV

Page 13: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 14: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

2.0 Estimates & plan

// in IO ⇒ 30 → 6

recycle codec

do the rest in comfort ⇒ 10 → 20

maybe improve metadata IO (?)

⇒ (time) 100% → 86%⇒ (size) 100% → 40%

R. Slootmaekers, N. Trangez FP@GhentITV

Page 15: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Amplidata’s unbreakable storageour pain2.0 plan and execution

What happened?

We decided on Functional programming & picked OCaml.

use Light Weight Threads library.

Blinded by success.

rewrote codec (in C)

metadata → Arakoon

feature creep

⇒ (time) 100% → 40%⇒ (size) 100% → 100% (90% OCaml, 9% C, 1% Asm)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 16: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Were we Happy?(2010)Are we still happy?(2012)

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 17: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Were we Happy?(2010)Are we still happy?(2012)

Were we Happy? (2010)

Yes:

Type Inference rocks

Superfast compiler

Fast Enough Code

Nice C integration

Refactoring

No:

Tools? what tools?

32/64 bit issues

Multicore issues

OCaml(build) learning curve

Emacs only

OO does not fit well

Architecture?

R. Slootmaekers, N. Trangez FP@GhentITV

Page 18: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Were we Happy?(2010)Are we still happy?(2012)

Outline

1 Who are ’we’?

2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution

3 EvaluationWere we Happy?(2010)Are we still happy?(2012)

R. Slootmaekers, N. Trangez FP@GhentITV

Page 19: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Were we Happy?(2010)Are we still happy?(2012)

Are we still happy?(2012)

Yes:

FP strategy works.

No:

OCaml language evolution.

OCaml runtime evolution.

OCaml library evolution.

people & process mistakes.

R. Slootmaekers, N. Trangez FP@GhentITV

Page 20: Cufp 2012 talk

Who are ’we’?How and why we bootstrapped Functional Programming

EvaluationSummary

Summary

Doing everything in 1 programming language is plain silly.(But you need pain and some guerilla tactics to convincepeople)

Functional programming and distributed systems match

R. Slootmaekers, N. Trangez FP@GhentITV