ERLANG EVOLVES FOR MULTI-CORE AND CLOUD ...gotocon.com/dl/goto-aar-2012/slides/TorbenHoffmann...Erlang Wednesday, 3 October 2012 W 9 Erlang was intended to deal with the control plane

ERLANG EVOLVES FOR MULTI-CORE AND CLOUD

ENVIRONMENTS

Torben HoffmannErlang Solutions Ltd.

@LeHoffhttp://musings-of-an-erlang-priest.blogspot.dk/

1Wednesday, 3 October 2012 W

http://musings-of-an-erlang-priest.blogspot.dk/

http://musings-of-an-erlang-priest.blogspot.dk/

© 1999-2012 Erlang Solutions Ltd.

Agenda

•Erlang fundamentals

•Challenges

2



Warning 1: The Truth


I will do a few simplifications in order to get the main points across.



3





3

Will you tell the truth?





3

Will you tell the truth? Yes





3


The whole truth?





3


The whole truth? No





3


The whole truth? No

So help you OTP?





3


The whole truth? No

So help you OTP? Yes




Warning 2: Serious Love Ahead

4

I love Erlang!


It was part of a major career shift and I have never looked back.Apologies if I get too intense.


Realities Of Software Development

•Time-to-market pressure

•Utilisation of computing resources

•Scaling successes

•Maintenance burden

5



What Could Be...

6


3x productivity over C++/JavaSeamless scaling on multicoreScaling nicely over machinesLess code per featureThe future is here today - it’s called Erlang!


What Could Be...

6




What Could Be...

6




What Could Be...

6




What Could Be...

6




What Could Be...

6

The future is here...




What Could Be...

6

The future is Erlang!The future is here...




Erlang’s Original Requirements

7




• Large scale concurrency

7





• Soft real-time

7





• Soft real-time

• Distributed systems

7





• Soft real-time


• Hardware interaction

7





• Soft real-time



• Very large software systems

7





• Soft real-time




• Complex functionality

7





• Soft real-time





• Continuous operation for many years

7





• Soft real-time






• Software maintenance on-the-fly

7





• Soft real-time







• High quality and reliability

7





• Soft real-time








• Fault tolerance 7





• Soft real-time









Soundsfamiliar?





• Soft real-time









Soundsfamiliar?

Soundsgood,right?



Small semantic gap

General vs Domain Specific

8

Telecom



Small semantic gap


8

Telecom

C++/Java



Small semantic gap


8

Telecom

C++/Java



Small semantic gap


8

Telecom

Erlang

C++/Java



Small semantic gap


8

Telecom

Erlang

C++/Java



Small semantic gap


8

Telecom

Erlang

C++/Java

Smaller gap =

benefits!



Erlang’s Sweet Spot

9

GUI

Drivers

MiddlewareCoordination

Control

Erlang


Erlang was intended to deal with the control plane in telecom, which is all about orchestration of what goes on.GUI and low-level things are not what Erlang was created for - hence Erlang has good support for integration with other languages.Read the wonderful doctor thesis by Bjarne Däcker if you want to learn more: http://www.erlang.se/publications/bjarnelic.pdf


Other Erlang Domains

10

•Messaging - XMPP et al

- ejabberd, MongooseIM

•Webservers

- Yaws, Chicago Boss

•Payment switches & soft switches

- Vocalink, OpenFlow/LINC

•Distributed Databases

- Riak, CouchDB, Scalaris



Other Erlang Domains

10

•Messaging - XMPP et al

- ejabberd, MongooseIM

•Webservers

- Yaws, Chicago Boss

•Payment switches & soft switches

- Vocalink, OpenFlow/LINC

•Distributed Databases

- Riak, CouchDB, Scalaris

If the tool fits,you must select!

Tech Mesh Conference4-5 December

London



To Share Or Not To Share

11


Death propagates in shared memory unless you do a ton of defensive programming.Due to the actor model with no shared memory it is custom in Erlang to do fail-fast programming.No shared memory allows you to fail fast when suitable.Erlang uses message passing between processes to exchange information.



11

Memory





11

Memory

P1





11

Memory

P1 P2





11

Memory

P2

Corrupt





11

MemoryCorrupt





11

Memory MemoryCorrupt





11

Memory Memory

P1

Corrupt





11

Memory Memory Memory

P1

Corrupt





11

Memory Memory Memory

P1 P2

Corrupt





11

Memory Memory

P2

Corrupt Corrupt





11

Memory Memory

P2

Corrupt




Dealing With Failures

12

P1 P2

P1 P2

link =die together

monitor =notification

of death


The ability to link processes and monitor them is the key to adopting fail-fast fully.Link & monitor works across machines!


Supervision Trees

13

worker

supervisor

workersupervisor

worker workerworkerThe OTP library is

built on this principle


Robust systems does not happen by accident - even in Erlang!You have to think about the consequences of a worker process that fails and let the supervisor take appropriate actions.Using the OTP library’s components makes it straightforward to implement the supervision tree, which has the added benefit that all things are started in the right order.OTP is lightweight enough to allow fast prototyping and the prototype can evolve into a high quality solution in a gradual way.


Distribution Over Cores

14

Core

Scheduler

Processes

Core

Scheduler

Processes


There is 1 scheduler per core.The VM tries to load balance across the available cores.Scales extremely well with the addition of extra cores - WITHOUT changing the programs!


Distribution Over Machines

15


Each instance of the Erlang runtime is called a node.There can be several nodes on one machine if you fancy that.Nodes detect when other nodes are not around any more - the programmer can then decide what to do.If you have the PID (Process Identifier) of a process you do not care which node it is on. You can decompose the PID to figure out if it is a local process, but you do not need to do so - locality is transparent.



15

Node





15

Node Node





15

Node

Node

Node





15

Node

NodeNode

Node




Learn New Moves!

Staying Alive...

16


As style changed a new group entered the top of the hip scale.For most it meant a serious restart or a stop altogether.


Learn New Moves!

Staying Alive...

16




Learn New Moves!

Staying Alive...

16




Learn New Moves!

Staying Alive...

16




Staying Alive Erlang Style

17


With Erlang you can survive upgrades without loosing service.Along with the code change signal you specify how the internal state of the process should be updated before continuing.



17

v1

Process running





17

Code loaded: v2

v1

Process running





17

Code loaded: v2

v1

Process running

Code change signal





17

Code loaded: v2

v1

Process running

Code change signal v2




Challenges

18



How Will I Know If It Really Scales?

19

???

Cores/Machines

Performance


A Scalability Benchmark Suite for Erlang/OTP.http://www.softlab.ntua.gr/release/bencherl/index.htmlA number of synthetic benchmarks plus real-world (dialyzer and scalaris).Extendable to test your own application.


How Will I Know If It Really Scales?

19

Cores/Machines

Performance

Bencherl


A Scalability Benchmark Suite for Erlang/OTP.http://www.softlab.ntua.gr/release/bencherl/index.htmlA number of synthetic benchmarks plus real-world (dialyzer and scalaris).Extendable to test your own application.


Who Is Doing What?

20

Core

Scheduler

Processes

Core

Scheduler

Processes



Who Is Doing What?

20

Core

Scheduler

Processes

Core

Scheduler

Processes

When is this process running?



Who Is Doing What?

20

Core

Scheduler

Processes

Core

Scheduler

Processes

When is this process running?

Who started this guy?



Percept2 To The Rescue

21

Active functions


Percept: Erlang Concurrency Profiling Tool, utilizes trace informations and profiler events to form a picture of the processes's and ports runnability.Percept2 is an extension of Percept (part of the OTP release).Extensions: # of schedulers active, active functions, process migration, message passing stats, inter-node communication


R12B-1

Memory Alloc Previously

22

Slide title minimum 32 pt

(32 pt makes 2 rows

Text and bullet level 1 minimum 24 pt

Bullets level 2-5minimum 20 pt

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™−≤≥fifl

ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨ¬ΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the footer area

Taking a Virtual Machine Towards Many-Core | Public | © Ericsson AB 2012 | Rickard Green & Patrik Nyblom | 2012-05-29

Memory Allocators R12B-1

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

mseg_allocS

cheduler 1

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

Scheduler 2

Scheduler N

Thread 1

Thread N

Tuesday, May 29, 2012


One central memory allocator for all schedulers on the same machine


R15B

Memory Alloc Now

23

Slide title minimum 32 pt

(32 pt makes 2 rows

Text and bullet level 1 minimum 24 pt

Bullets level 2-5minimum 20 pt

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™−≤≥fifl

ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨ¬ΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the footer area

Taking a Virtual Machine Towards Many-Core | Public | © Ericsson AB 2012 | Rickard Green & Patrik Nyblom | 2012-05-29

Memory Allocators R15B

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

mseg_allocS

cheduler 1

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

Scheduler 2

Scheduler N

Thread 1

Thread N

mseg_alloc mseg_alloc

sl_alloc

eheap_alloc

binary_alloc

ets_alloc

mseg_alloc

Tuesday, May 29, 2012 23Wednesday, 3 October 2012 W

One central memory allocator for all schedulers on the same machien


Upgrading Blocks

24

Current Code

Next Code

All schedulers blockedLoad code



Upgrade Without Blocking

25

Current Code

Next Code

Last Code

Each scheduler does it when needed

Load code


Coming in R16!


Fully Connected

26


Erlang connects all nodes fully. So you get a lot of connections.


Scaling Over Machines

27

Node Node

Node

Node

Node


A fully connected system might not be right for every problem.s_groups allows you to create clusters of nodes.Nodes inside a cluster are fully connected. Connections between clusters can be arbitrary.


Managing Erlang Systems

28


Basic Erlang has the ability to go in and monitor what is going on in any node you can attach yourself to.But no tool exists to manage a big number of nodes in a coherent fashion.This is no different from any other language/technology!



•Provision machines

28






•Deploy Erlang application

28







•Attach to node

28







•Attach to node

•Dig out metrics

28







•Attach to node

•Dig out metrics

28

Memory usageCPU loadProcess hierarchy




Managing Big Systems

29

CCL/sɪˈsɪlɪ/


Erlang Solutions are building a tool as part of the RELEASE project to manage and operate big Erlang systems.CCL = Cloud Computing Lace or Cloud Cuckoo Land depending on your mood.


Erlang And Parallelism

30




•Created for

30




•Created for

- explicit concurrency

30




•Created for


- fault tollerance

30




•Created for


- fault tollerance

- highly concurrent systems

30




•Created for


- fault tollerance


•No direct support for

30




•Created for


- fault tollerance



- matrix multiplication

30




•Created for


- fault tollerance




- ray tracing

30




•Created for


- fault tollerance




- ray tracing

- coarse grained parallel problems

30



Intensional Parallelism

31


We are taking the good things from what has been learnt in the Haskell & Data-flow language communities and building a DSL which helps us leverage these types of parallel optimisations"



•Lucid like: demand-driven data computation

•Find short comings in the Erlang VM

•Variables are infinite streams of values

31








31

running_avg where sum = first(input) fby sum + next(input); n = 1 fby n + 1; running_avg = sum / n; end;








31

running_avg where sum = first(input) fby sum + next(input); n = 1 fby n + 1; running_avg = sum / n; end;

Black magic




Going Forward

32



Going Forward

•Consider Erlang when the problem fits

32



Going Forward


•More focus on right tool for the job

32



Going Forward


•More focus on right tool for the job

32


Documents

ERLANG EVOLVES FOR MULTI-CORE AND CLOUD ...gotocon.com/dl/goto-aar-2012/slides/TorbenHoffmann...Erlang Wednesday, 3 October 2012 W 9 Erlang was intended to deal with the control plane