1

IntroductionWilfred Springer

Table of Contents

1. Introduction ................................................................................................................. 12. Declarative Binding ....................................................................................................... 13. Convention over configuration ........................................................................................ 34. More than just numbers ................................................................................................ 35. Composite content ....................................................................................................... 46. Inheritance .................................................................................................................. 57. Lists ........................................................................................................................... 68. Lazy loading lists .......................................................................................................... 69. Expressions ................................................................................................................. 710. Limbo ........................................................................................................................ 711. Conditionals ................................................................................................................ 812. Complex references ..................................................................................................... 813. Documentation ........................................................................................................... 9

1 Introduction

Decoding a compressed data format in Java can be quite a mundane task. This chapter will hopefullyconvince you that it does not need to be all that complicated. Preon aims to provide a simple andpowerful solution.

This chapter will introduce the most important principles behind Preon. Don't expect this chapter tobe an exhaustive reference guide for everything Preon has to offer. Completeness is not consideredto be achieved if there is nothing else to add; it is considered to be achieved if there is nothing left tobe taken out. The sole objective of this chapter is to prevent surprises once you start to use Preon.

2 Declarative Binding

Preon is not just a library allowing you to access bits in a bitstream, or a library that has a number ofconvenience mechansisms allowing you read data from a compressed bitstream encoded format. Itis much more than that, and - just like IoC - it obeys the Holywood paradigm: "don't call us, we callyou."

If Preon would just have been a library of functions facilitating different compression mechanisms,then you would have been required to load the decoded data into your own data structures yourself.

Introduction

2

But Preon turns that arround: you just provide it the blueprints of the data structure, and Preon willmake sure the data gets loaded into the appropriate places.

Now, the big question is of course: how does Preon know how to load data from a sequence of bitsand reconstruct a data structure? What is the recipe? It turns out, the recipe is the data structureitself; it's just classes and attributes, period. And in case it the information provided by the datastructure is not enough, we will just augment it using annotations.

So, let's look at an example. Example 1, “First data structure” defines a data structure defining thecoordinates of the two points defining a rectangle, in some coordinate system.

Example 1. First data structure

class Rectangle {private int x1;private int y1;private int x2;private int y2;}

Let's just say that the data would be encoded on disk as four consecutive 32-bit integers. In that case,this is the way you would decode a Rectangle from a file:

byte[] buffer = new byte[] { 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4};Codec<Rectangle> codec = Codecs.create(Rectangle.class);Rectangle rect = Codecs.decode(codec, buffer);

That's how easy it is. Although... To be perfectly honest, the example is not entirely complete yet. Ifyou would use the code as-is, you would not get anything at all. (Well, an empty Rectangle, maybe.)Problem is, Preon does not assume anything about your decoding requirements. In order to tellPreon that it needs to decode the x1, y1, x2 and y2 fields, you will need to explicitly state it in thedefinition of the data structure, like this:

Example 2. First data structure annotated

class Rectangle { @Bound private int x1; @Bound private int y1; @Bound private int x2; @Bound private int y2;}

If you run the same code now, you will get a populated Rectangle.

Introduction

3

3 Convention over configuration

So, binding a Java data structure to its encoded representation seems to be fairly easy. However,Preon intends to support bitstream encoded data. That is, in many cases you want don't want toallocate the full 32 bits for decoding an integer, and you would just rely on a couple of bits instead.Or what if you don't want big endian byte order, but little endian byte order instead?

All of this turns out to be additional configuration only. If you just specify the @Bound annotation,you basically tell Preon to use its default mapping to decode an integer. And by default, it willassume big endian byte order, and decode an integer using 32 bits. If you want to change that, youjust need to tell Preon to do so, using other annotations, or (sometimes) other annotation attributes.

Example 3. Encoding integers using two bytes

class Rectangle { @BoundNumber(size="16") private int x1; @BoundNumber(size="16") private int y1; @BoundNumber(size="16") private int x2; @BoundNumber(size="16") private int y2;}

Example 3, “Encoding integers using two bytes” changes the example given before to make Preonallocate only two bytes (16 bits) per number. ??? illustrates how to change byte order.

Example 4. Encoding integers using little endian byte order

class Rectangle { @BoundNumber(byteOrder=LittleEndian) private int x1; @BoundNumber(byteOrder=LittleEndian) private int y1; @BoundNumber(byteOrder=LittleEndian) private int x2; @BoundNumber(byteOrder=LittleEndian) private int y2;}

4 More than just numbers

Preon is not just about decoding numbers. By default, it already supports much more than that. Hereare some examples:

// Will just read one bit, interpreting 1 as true, and 0 als false@Bound boolean visible;

// Reads a String from a fixed number of bytes@BoundString(size="10") String value;

// Reads a bit from the buffer, and interprets it as an enum value, // interpreting the number as its ordinal value.@BoundNumber(size="1") Type type;

Introduction

4

The first example is a good example of convention over configuration; an @Bound annoation on aboolean attribute will cause Preon to read one bit, and interpret it as true in case it's the value 1, andfalse otherwise.

The String example illustrates one way of decoding Strings. Note that (in this case) the size attributedenotes the number of bytes and not the number of bits. In this case, the bytes read will beinterpreted as US-ASCII.

The third example illustrates how an type-safe enum value is bound to the bit buffer. Again, itrelies on an existing annotation that we have already seen before: the @BoundNumber annotationspecifies how a certain numeric value needs to be read. That numeric value read needs tocorrespond to the ordinal value of one of the enum's values.

5 Composite content

The examples that we have seen so far were all pretty simple: in all cases, it was just about asequence of attributes. But what if - in your format - you have have some higher-level conceptualorganization of some elements? How would you need to deal with that?

Let's take our Rectangle from Example 2, “First data structure annotated” as an example. What if wewant it to have a fill and border color as well? Would this be the solution?

@Bound int fillRed;@Bound int fillGreen;@Bound int fillBlue;@Bound int borderRed;@Bound int borderGreen;@Bound int borderBlue;

... or would you rather have this?

@Bound RgbColor fillColor;@Bound RgbColor borderColor;

I hope you agree with me that the second option would be the preferred one. And it turns out thisis possible. If you define your RgbColor class as in Example 5, “RGB Color”, then you can used theRgbColor type as attributes in other classes.

Example 5. RGB Color

class RgbColor { @Bound int red; @Bound int green; @Bound int blue;}

It's imporant to emphasize that - from a processing expectations point of view - there isno difference between both options. Bot options will result in 6 numeric values being readconsecutively. However, the second allows you to stick with the conceptual organization of your file,results in less code and makes it easier to maintain.

Just to complete the example, Example 6, “Colored Rectangle” gives the source code of theRectangle supporting colors.

Introduction

5

Example 6. Colored Rectangle

class Rectangle { @Bound private RgbColor fillColor; @Bound private RgbColor borderColor; @Bound private int x1; @Bound private int y1; @Bound private int x2; @Bound private int y2;}

6 Inheritance

In the previous section (Section 5, “Composite content”), we saw one way of factoring outcommonalities. Preon allows you to factor out recurring snippets of content and use thosedefinitions at various places. However, it is not the only way to organize content. In this section, I willshow you another way.

Example 6, “Colored Rectangle” defines just one shape. And of course, that shape also defines somecolor attributes. But what if your application defines more shapes. And what if you want all of thoseshapes to define the same color attributes?

In order to facilitate scenarios like these, Preon supports inheritance. That is, subclasses will allwaysinherit the bindings defined by the superclass. This allows you to define color bindings on a Shapesuperclass, and have all of the other shape-specific attributes on the various Shape subclsses.

Example 7. Shape Inheritance

class Shape { @Bound fillColor; @Bound lineColor;}

class Rectangle extends Shape { @Bound private int x1; @Bound private int y1; @Bound private int x2; @Bound private int y2;}

class Circle extends Shape { @Bound int centerX; @Bound int centerY; @Bound int radius;}

When Preon is required to decode a subclass, it will always first process the inherited bindings.In case of a Rectangle, it will first decode the colors it inherited from Shape, and only then theattributes of the rectangle itself.

Introduction

6

7 Lists

The 'struct'-type of support highlighted in the previous sections is an important concept, but notsufficient. We also need a 'sequence' concept.

In Preon, the @BoundList annotation allows you to address all of these 'sequencing' concerns. Andthe annotation driven approach makes it fairly easy to use.

Let's just say you want to store a pseudo-Mondriaan type of painting, existing of colored rectanglesonly. In that case, you could use code like this:

class PseudoMondriaan { @BoundList(type=Rectangle.class, size="20") Rectangle[] rectangles;}

The code given above simply states that - whenever you want to decode a PseudoMondriaan, read20 Rectangle instances according to its specifications, and store them as them as the attribute'rectangles'.

Now, it is of course questionable if it is realistic to expect that every PseudoMondriaan containsexactly 20 rectangles. However, the @BoundList's size attribute also takes expressions that areevaluated at runtime. More on that in Section 9, “Expressions”.

8 Lazy loading lists

Arrays are not the only type of lists supported by default; Preon also supports Lists (the java.util.List)kind, as you might have expected. There is however a difference with the way it treats Lists.

Whenever Preon decodes a List of items from the bit buffer, there is a chance it will inject a lazy-loading List into the List type attribute instead of fully-populated eagerly loaded List instance. Isay there is 'a chance', since it really depends on a number of conditions if it will do so. The actualconditions are quite complex, but they can be summarized as "the principle of the least surprise."

class PseudoMondriaan { @BoundList(size="20", type=Rectangle.class) private List<Rectangle> rectangles;}

Preon has a preference for loading data lazily. In the above case, the size of each Rectangle instanceis known in advance. In fact, we know that the number of bits occupied by the entire List will be 20times the size of single Rectangle. And we also know that - in order to get the third element - we justneed to skip over the first two elements (the number of bits of a single Rectangle element, multipliedby two).

Using a lazy loading List is in the above case probably a fairly efficient thing to do. There are othercases in which this would be much harder, typically when the List item's size is determined at

Introduction

7

runtime. In cases like these, Preon will automatically pick an alternative Codec, most likely one thateagerly loads the data.

9 Expressions

By now, you probably wondered a couple of times why the size attribute on @BoundList and@BoundString accepts String values instead of numbers. It seems rather a wasteful, and the compileris not going to help you to make sure you enter numeric values in these situations.

The truth is, these attributes expect more than just numeric values: a lot of annotations in Preonallow you to pass in expressions, instead of numeric values. The size attribute on @BoundList and@BoundString are just examples of cases in which these expresions are accepted.

class PseudoMondriaan { @Bound int numberOfRectangles; @BoundList(size="numberOfRectangles", type=Rectangle.class) List<Rectangle> rectangles;}

The example above is still a fairly simple example. It refers to a variable called numberOfRectangles,and it's now hard to guess how this variable is getting resolved: by default, Preon will try to resolvevariables into bound attributes defined before. (There are other types of references, but let's stickwith this simple example for now.)

Now, you can obviously construct more complex expressions than this. And in order to do that, youcan use a range of arithmetic or logical operators. Here are some examples:

@BoundList(size="width * height") byte[] pixels;@BoundNumber(size="nrBits * 2") int value;// Clearly pointless, but you know...@BoundString(size="x * (y + z) / 23 ^ t");

10 Limbo

The expression language used in the examples of the previous language is Limbo. Limbo is a fairlysmall expression language, and is explicitly aiming at not being as feature rich as JSP's EL or OGNL,and for good reason.

First of all, JSP's EL and OGNL allow you to call methods on objects. Limbo does not allow you tocall methods, because it wants to capture the meaning of the expression explicitly. And it would beimpossible to capture the meaning of your expression if it would involve operations defined outsideof Limbo itself. So, Limbo works on data only, and the number of operators is finite.

Limbo is not just about evaluating expressions. It also wants to capture those expressions in sucha way that it will be able to generate human-readable expressions from it afterwards. So, if thesewould be the Limbo expressions:

width * heighta^2 + b^2 == c^2

Introduction

8

... then Limbo wants to be capable of turning this into this:

• width of the image times the height of the image• the sum of a to the power of 2 and b to the power of two equals c to the power of two

In a couple of sections, we will see why this is relevant.

11 Conditionals

The size attribute defined on the @BoundList annotation obviously acts as a type of controlmechanism. However, in order to be able to deal with the majority of encoding formats, we need tointroduce a couple of other control structures, in order to support with conditionals.

Now, the number of places in which you would be able to use a condition is again open-ended. Preonis extensible, so you can define your own annotations using conditions wherever you like. However,the framework supports some by default, and we are going to mention two here.

The first one is the @If annotations. By putting this annotation in front of an attribute, you basicallystate that - whatever attribute is following - it should only be decoded if the condition inside the @Ifholds.

Now, let's see what that means in our example. Let's just say that a Shape allows you to specify thatyou want a dashed border, and that it defines some additional attributes allowing you to control howthe dashes are drawn.

class Shape { @Bound Color fillColor; @Bound Color borderColor; @Bound boolean dashed; @If("dashed") @Bound int spaceBetweenDashes;}

12 Complex references

Section 9, “Expressions” already mentioned that Preon does not limit you to refer to other boundattributes on the same class only. It also supports the dot operator (to access attributes of objectsreferenced) and the item operator ( to access items in list-type attributes). Example 8, “Validreferences” lists some sample references.

Example 8. Valid references

a.b.ca.b[0].da.b[e].da.b[e * 21].d

Introduction

9

Another thing that Preon allows you to do is to refer to the outer context. Example 9, “Outer contextreferences” relies on this to refer to the bitsPerColorComponent variable in the Image, fromwithin the RgbColor object.

Example 9. Outer context references

public class Image { @Bound int bitsPerColorComponent; @Bound int nrColors; @BoundList(size="nrColors") RgbColor[] colors; ... public class RgbColor { @BoundNumber(size="outer.bitsPerColorComponent") red; @BoundNumber(size="outer.bitsPerColorComponent") green; @BoundNumber(size="outer.bitsPerColorComponent") blue; }}

13 Documentation

Various sections in this chapter already alluded to the fact that Preon aims at capturingdependencies between different pieces of data explicitly, in order to be able to generatedocumentation. If this is the first time you read about Preon, you might wonder what that was allabout.

Preon was born out of unease with an existing situation. In that existing situation, one guy wasresponsible for the software for decoding a complicated file format, while another was responsiblefor encoding data; both of them were responsible to keep documentation in sync. In the end,everything turned to be out of sync. Keeping the encoder in sync with the decoder was hard, and thedocumentation was always out of whack.

Preon aims to change all of that: it set of to provide a framework in which you specify the encodingformat once, and get decoders, encoders and documentation for free.

So, how does it work? Well, if you remember the first example in this chapter, we constructed aCodec and used a Codec using the Codecs class, like this:

Codec<Rectangle> codec = Codecs.create(Rectangle.class);Rectangle rect = Codecs.decode(codec, buffer);

Generating documentation just requires using another operation on Codecs:

Codec<Rectangle> codec = Codecs.create(Rectangle.class);Codecs.document(codec, DocumentType.Html, new File(...);

When you apply that to Example 7, “Shape Inheritance”, this is what you get:

Introduction

10

Figure 1. Sample report