Clustered Serialization with Fuel

RMod

1

Mariano Martinez [email protected]

http://marianopeck.wordpress.com/

Monday, August 22, 2011

mailto:[email protected]




THANKS a lot!!!

2

SummerTalk 2011

Student: Martin DiasMentor: Mariano Martínez Peck


Object references

3


Object graph

4


5

An objet graph serializer.


Serialize

6

Input: an object graph

Output: stream of bytes


Materialize(deserialize)

7

Output: an object graph

Input: stream of bytes


Once serialized...

8

Database File Memory Socket

Stream of bytes


Fuel’s main goals

Provide fast object serialization and materialization.

Be flexible and easy to customize.

Have a good OO design, well tested and benchmarked.

No need of special support from the VM.

Be a general purpose serializer.

Allow tools to be built on top of Fuel.9


Key features Fast serialization and materialization.

Class reshape support.

Serialization of any kind of object.

Cycles support.

Global objects references.

Buffered writing.

Support for some “hook methods”.10


Key Characteristics

Pickle format.

Objects grouped in clusters.

Analysis phase before writing.

Stack over recursion.

Two phases for writing instances and references.

Iterative graph recreation.11


Pickle format

12

Invest more time in serialization so that objects can then be materialized much faster.


Grouping objects in clusters

13

“Similar” objects (they share writing/loading information) are grouped together in clusters. The most common case, yet not the

only one, takes place when a class is a cluster for its instances.


14


14

Each jar has a specific type of element


14


Jars are in order


14

Label: - What’s inside? - How much?Each jar has

a specific type of element

Jars are in order


14

Label: - What’s inside? - How much?

Different sizes and different amounts of elements


Jars are in order


14




Jars are in order

s/jar/cluster


14




Jars are in order

s/jar/cluster


14



Jars are in order

s/jar/cluster

Each cluster has a specific type of object


14



Jars are in order

s/jar/cluster



14



s/jar/cluster


Clusters are in order


14



s/jar/cluster




14


s/jar/cluster



Label: - Cluster ID - Amount of

objects


14


s/jar/cluster




objects


14

s/jar/cluster




objects

Different sizes and different amounts of

objects


Pickle format basic

15

Stream


Pickle format basic

15

Stream


Pickle format basic

15

Stream


Pickle format basic

15

Stream


16

Why the pickle format is so fast in materialization?

Standard serializers Fuel pickle format


16

Why the pickle format is so fast in materialization?

Standard serializers Fuel pickle format

Recursive

materialization Iterative

materialization


Pickle advantages

Batch/Bulk/Iterative materialization.

Efficient since types are stored and fetch only once.

Fast because at materialization we know the size of everything.

The generated stream is smaller.

More next....

17


There is no silver bullet...

18

Fast serialization

(without pickle)

Slow serialization(with pickle)


Fuel requires

Traversing the object graph.

Mapping each object to a specific cluster.

19

This is done in a phase before serialization called “Analysis”.


Analysis phase

20

cluster1 aPointcluster2 x y

......

... ...

key (a cluster) value (a set)

1) Traverse (#fuelAccept: aVisitor)2) Fill dictionary(#fuelSerializer)


Serialization

21

key (a cluster) value (a set)

serialize: anObject on: aStreammaterializeFrom: aStream

IDCluster

A cluster defines how its objects are serialized and materialized.

cluster1 aPointcluster2 x y

......

... ...


Stack over recursion

22

To traverse the object graph, Fuel uses a custom stack implementation rather than a recursion.


Basic stepsSerialization

1. Analyze.2. Serialize header.3. Serialize instances.4. Serialize references.5. Serialize root.

Materialization1. Materialize header.2. Materialize instances.3. Materialize references.4. Materialize root.

23


Fuel for software(so far)

Moose export utility.

SandstoneDB persistence.

Pier kernel persistence.

Newspeak language.

Marea (my own research project!).

24


Future Work

Continue efforts on performance optimization.

Create a tool for loading class and trait packages.

Support user-defined Singletons.

Fast statistics/brief info extraction of a stored graph.

Partial loading of a stored graph.25


Future work 2

Enable to deploy serialization and materialization behavior independently.

Support object replacement for serialization and materialization.

Allow cycle detections to be disabled.

Partial loading.

26


Serialization of primitive objects

27

Memory based stream



27

Memory based stream

StOMPSRPFuel



27

Memory based stream File based stream

StOMPSRPFuel



27

Memory based stream File based stream

StOMPSRPFuel

FuelSRPStOMP


28

Materialization of primitive objects


28

Fuel

ImageSegment

StOMP

Materialization of primitive objects


Non primitive objects

29



29

StOMPSRPFuel



29

StOMPSRPFuel



29

FuelImageSegmentStOMP

StOMPSRPFuel


Links

Website: http://rmod.lille.inria.fr/web/pier/software/Fuel

Issue tracker: http://code.google.com/p/fuel

Source repository: http://www.squeaksource.com/Fuel

Continuous integration server: https://pharo-ic.lille.inria.fr/hudson/job/Fuel/

30


http://rmod.lille.inria.fr/web/pier/software/Fuel




http://www.squeaksource.com/Fuel.html

http://www.squeaksource.com/Fuel.html

http://www.squeaksource.com/Fuel




https://pharo-ic.lille.inria.fr/hudson/job/Fuel/

https://pharo-ic.lille.inria.fr/hudson/job/Fuel/

Excellent performance without special support from VM and good OO design.

Conclusionfor us

31


Fuel is a vehicle. It is infrastructure. You can build cool stuff on top of it.

Conclusionfor YOU

32


Thanks!

RMod

Mariano Martinez [email protected]

http://marianopeck.wordpress.com/






34

Concrete example


35


Analysis phase

3620

cluster1 aRectanglecluster2 anOrigin aCornercluster3 10

key (a cluster) value (a IdentitySet)

20 30 40

Analysis phase


Serializationinstances

phase

cluster1 aRectangle

cluster2 anOrigin aCorner

cluster3 10 20 30 40



phase

cluster1 aRectangle


cluster3 10 20 30 40



phase

cluster1 aRectangle


cluster3 10 20 30 40

1 Rectangle 2 origi corne 1

instance variable names

instance count

instance variable count

class name

cluster ID



phase

cluster1 aRectangle


cluster3 10 20 30 40



instance count


class name

cluster ID



phase

cluster1 aRectangle


cluster3 10 20 30 40


1 Point 2 x y 2


instance count


class name

cluster ID



phase

cluster1 aRectangle


cluster3 10 20 30 40


1 Point 2 x y 2

54 10 20 30 40


instance count


class name

cluster ID


cluster1 aRectangle


cluster3 10 20 30 40

aRectangle anOrigiaCorner20 40 10 30instancesIndex

(IdentityDictionary)

1 5 7 3 2 4 6

54 10 20 30 40

2

Serializationreferences

phase

instances

references


cluster1 aRectangle


cluster3 10 20 30 40



1 5 7 3 2 4 6

54 10 20 30 40

2

2

3


phase

instances

references


cluster1 aRectangle


cluster3 10 20 30 40



1 5 7 3 2 4 6

54 10 20 30 40

2

2

3

4 5 6 7


phase

instances

references


cluster1 aRectangle


cluster3 10 20 30 40



1 5 7 3 2 4 6

54 10 20 30 40

2

2

3

4 5 6 7


phase

instances

references


2

2

3

4 5 6 7

final stream


1 Point 2 x y 2

54 10 20 30 40

{instances

{references


2

2

3

4 5 6 7

final stream


1 Point 2 x y 2

54 10 20 30 40

{instances

{references

1{trailer

7 3{header clustersCount

objectsCounts

rootMonday, August 22, 2011

2

Materialization

instances {2 3

4 5 6 7

1 Rectangle 2 origin corner 1

1 Point 2 x y 2

5 4 1 2 3 4

{references

1{trailer

7 3{header clustersCount

objectsCounts

root


41


Technology

Clustered Serialization with Fuel