Upload
toolslive
View
384
Download
2
Embed Size (px)
DESCRIPTION
slides from the talk we gave in Copenhagen at cufp 2012.
Citation preview
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Functional Programming @ Ghent IT Valley
R. Slootmaekers N. Trangez
Incubaid Research Lab
Commercial Users of Functional Programming, 2012
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Incubaid is a technology incubator, active in datacenterand cloud computing
Prior exits through Terremark, Telenet (Belgian telco),Veritas/Symantec, Sun Microsystems,. . .
some of our current involvements: Amplidata , Awingu ,CloudFounders , Dacentec , Racktivity
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Amplidata’s Dispersed Storage System20 000ft view
Client
encoder
decoder
provisioning
policies
m e t a d a t a
StorageNodeA
.... . . . .
S torageNodeZ
diskA0
diskA1
diskA...
client streams object dataencoder generates equations : Ax = b or x = A−1bpolicies determine how many disks | equationsprovisioning determines which diskssend equations to StorageNodesrecord metadata
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Pain
Pain: resource management
Pain: sockets
Pain: threads
Pain: data corruption
Pain: objects
development was grinding to a halt
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Pain
Pain: resource management
Pain: sockets
Pain: threads
Pain: data corruption
Pain: objects
development was grinding to a halt
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
The 1.6 Codebase (Core)
0
1
StIO MdIO Codec Rest
Time (Blue):Pareto applies60% is IO30% is encoding/decoding
size (Red):> 50 klocCodec ∼ 1 klocIO < 1 kloc
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
The Bet
I placed a bet to rewrite the StorageNode component in OCaml
Functionally identical
Same Performance
in 2 days. . .
1.5 day later the drop in replacement was demoed, and wewere allowed to Investigate alternatives to C++ for version 2.0
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
The Bet
I placed a bet to rewrite the StorageNode component in OCaml
Functionally identical
Same Performance
in 2 days. . .
1.5 day later the drop in replacement was demoed, and wewere allowed to Investigate alternatives to C++ for version 2.0
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
How is this possible?
component is very simple
perfect specs
existing component test suite
performance is mostly determined by network & disk speed
OCaml has object orientation.
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
2.0 Estimates & plan
// in IO ⇒ 30 → 6
recycle codec
do the rest in comfort ⇒ 10 → 20
maybe improve metadata IO (?)
⇒ (time) 100% → 86%⇒ (size) 100% → 40%
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Amplidata’s unbreakable storageour pain2.0 plan and execution
What happened?
We decided on Functional programming & picked OCaml.
use Light Weight Threads library.
Blinded by success.
rewrote codec (in C)
metadata → Arakoon
feature creep
⇒ (time) 100% → 40%⇒ (size) 100% → 100% (90% OCaml, 9% C, 1% Asm)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Were we Happy?(2010)Are we still happy?(2012)
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Were we Happy?(2010)Are we still happy?(2012)
Were we Happy? (2010)
Yes:
Type Inference rocks
Superfast compiler
Fast Enough Code
Nice C integration
Refactoring
No:
Tools? what tools?
32/64 bit issues
Multicore issues
OCaml(build) learning curve
Emacs only
OO does not fit well
Architecture?
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Were we Happy?(2010)Are we still happy?(2012)
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional ProgrammingAmplidata’s unbreakable storageour pain2.0 plan and execution
3 EvaluationWere we Happy?(2010)Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Were we Happy?(2010)Are we still happy?(2012)
Are we still happy?(2012)
Yes:
FP strategy works.
No:
OCaml language evolution.
OCaml runtime evolution.
OCaml library evolution.
people & process mistakes.
R. Slootmaekers, N. Trangez FP@GhentITV
Who are ’we’?How and why we bootstrapped Functional Programming
EvaluationSummary
Summary
Doing everything in 1 programming language is plain silly.(But you need pain and some guerilla tactics to convincepeople)
Functional programming and distributed systems match
R. Slootmaekers, N. Trangez FP@GhentITV