JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit

Medallia © Copyright 2016. Confidential. 1

Code Generation with JavaCompilerfor Fun, Speed, and Business ProfitThorvald Natvig and Juan Cruz Nores

Medallia © Copyright 2015. 2

At Medallia, we collect, analyze, and display terabytes of structured & unstructured feedback for our multibillion dollar clients in real time.And what’s more: we have a lot of fun doing it.

Thorvald - Since 2010, growing from 70 to 1200 employeesJC - First team in Argentina office, 2011

Who we areHi, we’re JC & Thorvald @ Medallia


How to run an analytics engine at 30x speed1/30th the cost

Optimize for maximum performance with minimum engineering time

All code shown today is on GitHubIncludes bootstrap framework to do this yourself!

What’s this all about?

Fast Code in Short Time


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen

Problem and Java Solution

CodeGen

Learnings and Challenges

Demo!

“What’s next?”


Two ways to generate code:Method A: Use Java ASM and generate byte-code

You spend most of the time writing a compilerMethod B: Use Java Compiler to generate byte-code

Someone else wrote the compiler

Crash Course in Code Generation

Don’t write a compiler


Crash Course in Code Generation

Field f = Fields.get("age");int min = 20;int max = 99;long sum = records.stream() .filter(x -> x.getValue(field) > min) .filter(x -> x.getValue(field) < max) .map(x -> x.getValue(field)) .mapToInt(x -> x) .sum();


Autoboxing and generics in generalStream constructsFunction call to get valueSpend most of the time in “streaming”, not in “computing”

Problems!


// Assuming the field “age” is column 3 in the datastore...long generated(int [][] records) { long sum = 0; for (int i = 0; i < 88737273; ++i) { int fieldVar1 = records[3][i]; if (fieldVar1 > 20 && fieldVar1 < 99) sum += fieldVar1; } return sum;}

Transformed code


Generate Java code as a StringInvoke Java compiler in memoryLoad generated class and instantiate an objectCall function

How to guide

“Somewhat faster”


We live in the real worldEngineering time costs $$$; salary + lost opportunities

Often cheaper to buy hardware than spend time to optimizeOnly apply this level of optimization if the saved server cost

greatly outweighs the engineering cost.

When should you do this?

Optimize for Total Cost


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen


CodeGen


Demo!



An in-memory query engineData is a segmented 2D table, each segment an int[][]We want to be able to execute queries such as:

newQuery() .filter(field("a").in(1, 2, 3).or(field("b").is(3))) .aggregate(statsAggregate("ltr"))

This is roughly equivalent to: SELECT AVG(ltr) FROM data WHERE a IN (1, 2, 3) OR b = 3

Problem


Query Execution

b ltr1 103 7

Filter passes? Aggregate

no

yesa52


newQuery() .filter(field("a").in(1, 2, 3).or(field("b").is(3))) .aggregate(statsAggregate("ltr"))

Pure Java

OR

“a” in [1,2,3]

“b” in [3]


expression.visit(new ExprVisitor<Boolean>() { public Boolean visit(AndExpr andExpr) {

return andExpr.getLeft().visit(this) && andExpr.getRight().visit(this); } public Boolean visit(InExpr inExpr) { /* ommitted */ } public Boolean visit(OrExpr orExpr) {

return orExpr.getLeft().visit(this) || orExpr.getRight().visit(this); } /* other nodes */});

Pure Java


Simple query:newQuery() .filter(field("a").in(1, 2, 3).or(field("b").is(3))) .aggregate(statsAggregate("ltr"))

Complex query:newQuery() .filter(field("a").in(1, 2, 3, 5).or(field("b").is(3))) .filter(field("c").in(3, 5, 8, 1, 2).or(field("d").in(1,3,4))) .filter(not(field("e").in(2, 5, 8,10,12,13,14,15,17)).or(field("f").in(1,4,2))) .aggregate(statsAggregate("ltr"))

1M rows, split in 50k row segments (avoid large allocations)

About the benchmarks


We actually built thisAnd ran some benchmarks on it:

Benchmark Mode Cnt Score Error UnitsInterpretedQueryBenchmark.complexQuery thrpt 6 14.453 ± 5.557 ops/sInterpretedQueryBenchmark.simpleQuery thrpt 6 46.057 ± 2.227 ops/s

Pure Java

Benchmark early+often


Idea: Reduce unnecessary per-traversal operations by using lambdas and their closures

It also reduces overall number of method callsTake advantage of the massive optimizations implemented

in HotSpot to support Streams and lambdas (invokedynamic)

Optimized Pure Java


@FunctionalInterfaceinterface Filter { boolean eval(Segment segment, int row);}

public Filter visit(AndExpr andExpr) { Filter left = andExpr.getLeft().visit(this); Filter right = andExpr.getRight().visit(this); return (segment, row) -> left.eval(segment, row) && right.eval(segment, row); }

Optimized Pure Java


public Filter visit(OrExpr orExpr) { Filter left = orExpr.getLeft().visit(this); Filter right = orExpr.getRight().visit(this); return (segment, row) -> left.eval(segment, row) && right.eval(segment, row); } public Filter visit(ConstantExpr constantExpr) { boolean val = constantExpr.value; return (segment, row) -> val; }

Optimized Pure Java


public Filter visit(InExpr inExpr) { final int[] values = inExpr.getValues(); final int column = dataSet.getFieldByName(inExpr.getFieldName()).getColumn(); return (segment, row) -> { int fieldVal = segment.rawData[column][row]; for (int value : values) { if (value == fieldVal) return true; } return false; };}

Optimized Pure Java


So we ran some benchmarks on this too:

Benchmark Mode Cnt Score Error Units

StreamQueryBenchmark.complexQuery thrpt 6 33.046 ± 2.625 ops/s

StreamQueryBenchmark.simpleQuery thrpt 6 124.077 ± 7.449 ops/s

~3X improvement on simple queries~2X improvement on complex ones

Optimized Pure Java


Results


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen


CodeGen


Demo!



Query execution boils down to: “iterate over every record, if it matches, aggregate”

This implies a simple query structure:for (int i = 0; i < numRecords; i++) { if (/* record matches */) { /* aggregate */ }}

Only the filter and the aggregate depend on the query

CodeGen


cg.print(query.buildExpressionTree().visit(new ExprVisitor<String>() { public String visit(AndExpr andExpr) {

return "(" +andExpr.getLeft().visit(this) +" && "+ andExpr.getRight().visit(this) + ")"; } public String visit(InExpr inExpr) { FieldDefinition field = dataSet.getFieldByName(inExpr.getFieldName()); return "(" + IntStream.of(inExpr.getValues()) .mapToObj(v -> String.format("rawData[%d][row] == %dL", field.getColumn(), v)) .reduce((a,b) -> a + " || " + b) .orElseThrow(() -> new RuntimeException("empty filter")) + ")"; } public String visit(NotExpr notExpr) { "!" + notExpr.getTarget().visit(this); } public String visit(OrExpr orExpr) { return "(" +orExpr.getLeft().visit(this) + " || " + orExpr.getRight().visit(this) + ")"; } public String visit(ConstantExpr constantExpr) { return String.valueOf(constantExpr.value); }}));

Naive CodeGen


public void process(Segment segment) { int[][] rawData = segment.rawData; int nRows = rawData[0].length; FieldStats result$1 = (FieldStats)this.result; for (int row = 0; row < nRows; row++) { if ((rawData[0][row] == 1L || rawData[0][row] == 2L

|| rawData[0][row] == 3L) || (rawData[1][row] == 3L)) { result$1.count++; result$1.sum += rawData[6][row]; } }}

Naive CodeGen


Benchmark Mode Cnt Score Error UnitsCompiledQuery.complexQuery thrpt 6 72.509 ± 13.526 ops/sCompiledQuery.simpleQuery thrpt 6 208.281 ± 11.335 ops/s

~4X improvement on simple queries (over the simple interpreter)~5X improvement on complex ones

Results


Results


Higher level transformsextract common subexpressionsorder filters to maximize the chance of rejecting a filter...

many of these apply also to the interpreterLow level transforms

branch reductionperfect hashing...

Optimizing Your CodeGen


General strategy: instead of a set.contains() or set of ‘==’ comparisons

Less than 3 values: a or b or cValue-range <= 64: reverse inverted bitmask.Value-range <= 1024: array of reverse inverted bitmask.Otherwise: Generated Perfect Hash

Branch Reduction


Branch reduction1 0 0 0 0 0 0 0 >>> test-val

0 1 0 0 0 0 0 0

0 0 0 1 0 0 0 0

0 0 0 0 1 0 0 0

0 0 0 0 0 0 0 1

1:

3:

4:

7:

0 1 0 1 1 0 0 1mask:

bitwise orfield("a") .in(1, 3, 4, 7)


Evaluation

1:

2:

0 1 0 1 1 0 0 1(

field("a") .in(1, 3, 4, 7)

<< fieldVal ) < 0

01 0 1 1 0 0 1

00 1 1 0 0 1 0

Field Value

✅

❌

= -78

= 100


newQuery().filter(field("a").in(1, 2, 3, 9).or(field("b").is(3))).aggregate(statsAggregate("ltr"));

public void process(Segment segment) { int[][] rawData = segment.rawData; int nRows = rawData[0].length; FieldStats result$1 = (FieldStats)this.result; for (int row = 0; row < nRows; row++) { if (((0x7040000000000000L << rawData[0][row]) < 0 || (rawData[1][row] == 3L))) { result$1.count++; result$1.sum += rawData[2][row]; } }}

Variant Code Generation


protected String generateInExpr(JavaCodeGenerator cg, InExpr inExpr) { final FieldDefinition fieldDef = dataSet.getFieldByName(inExpr.getFieldName()); final FieldSpec fieldSpec = fieldDef.getFieldSpec(); final int numValues = fieldSpec.getBound() - fieldSpec.getOrigin(); final int[] inValues = inExpr.getValues(); if (inValues.length > 3 && numValues < 64) { long mask = 0; for (int value : inValues) { mask |= 0x8000_0000_0000_0000L >>> (value - fieldSpec.getOrigin()); } if (fieldSpec.getOrigin() == 0) { return String.format("(0x%xL << rawData[%d][row]) < 0", mask, fieldDef.getColumn()); } else { return String.format("(0x%xL << (rawData[%d][row]-%dL)) < 0", mask, fieldDef.getColumn(), fieldSpec.getOrigin()); } } // Other cases, fall back to the naive one return super.generateInExpr(cg, inExpr);}

Variant Code Generation


Benchmark Mode Cnt Score Error UnitsVariantQueryBenchmark.complexQuery thrpt 6 88.972 ± 2.206 ops/sVariantQueryBenchmark.simpleQuery thrpt 6 210.281 ± 8.007 ops/s

about the same on simple queries (over the naive compiler)~20% improvement on complex ones

Results


Results


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen


CodeGen


Demo!



public void process(com.medallia.data.Segment); Code: // Save Segment.rawData into a local variable (var 2) 0: aload_1 1: getfield #3 4: astore_2 // get rawData[0].length and store it // in a local variable (var 3) 5: aload_2 6: iconst_0 7: aaload 8: arraylength 9: istore_3 // cast this.result into a FieldStats object and store it in a variable (var 4) 10: aload_0 11: getfield #2 // Field result:Ljava/lang/Object; 14: checkcast #4 // class FieldStats 17: astore 4

Why not do bytecode?


// Begin the loop // set i = 0 (var 5) 19: iconst_0 20: istore 5 // if i >= (var 3) -- number of rows // jump to end the loop (jump to instr 113) 22: iload 5 24: iload_3 25: if_icmpge 113 // rawData[0][i] == 1 goto 78 28: aload_2 29: iconst_0 30: aaload 31: iload 5 33: laload 34: lconst_1 35: lcmp 36: ifeq 78 // rawData[0][i] == 2 goto 78… (more conditions)

Why not do bytecode?… (more conditions) // fieldStats (var 4).count ++ 78: aload 4 80: dup 81: getfield #9 // Field FieldStats.count:J 84: lconst_1 85: ladd 86: putfield #9 // Field FieldStats.count:J // fieldStats (var 4).sum += rawData[6][i] 89: aload 4 91: dup 92: getfield #10 // Field FieldStats.sum:D 95: aload_2 96: bipush 6 98: aaload 99: iload 5 101: laload 102: l2d 103: dadd 104: putfield #10 // Field FieldStats.sum:D // i++ (var 5), jump to 22 (loop start) 107: iinc 5, 1 110: goto 22 113: return}


Javac is designed as a command line toolbut… API in javax.tools!

For the basic case we have to deal with three entities:Source files (input)Class files (output)Classpath

File operations are abstracted behind JavaFileManager

Compiling in-memory


Be careful with multithreadingCache warmup is the most time consuming part of

compilationSimple strategy:

Keep a JavaCompiler and a StandardJavaFileManager in a thread local variable

Use a fixed size thread pool to constrain memory usage

Compiling in-memory


ClassloadingThe simplest strategy is to use one class per ClassLoader:public static <T> Class<? extends T> classFromBytes(

final Class<T> baseClass, final String name, final byte[] bytecode) { return new ClassLoader(baseClass.getClassLoader()) { Class<? extends T> c = defineClass(name, bytecode, 0, bytecode.length)

.asSubclass(baseClass); }.c;}


Pros:Easy to implementMemory reclamation is easy

(each class/classloader pair is independent)Cons:Metaspace chunks are tied to a classloader

→ wasted metaspaceConstrained to generating a single class

Classloading


Most Java programs are fairly staticJVM needs to warm up new code

InterpreterC1 (-client compiler): fast compilationC2 (-server compiler): 20%-30% faster code for some

workloadsDynamic class loading breaks some assumptions

code size increases over application run timecompiled code size does too: HotSpot CodeCache

Classloading


HotSpot stores native code in its code cacheYou can run out of itIf you do, the compilers are disabled

"CodeCache is full. Compiler has been disabled"Some JVM options to deal with it:-XX:InitialCodeCacheSize-XX:ReservedCodeCacheSize-XX:+UseCodeCacheFlushing

CodeCache


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen


CodeGen


Demo!



Demo


AGEN

DA

1

2

3

4

5

6

Basics of CodeGen


CodeGen


Demo!



As in x86-64 assembly.

And yes, YES WE ARE.

We’re just one step away from ASM


Clang as a libraryCompile in-memory and execute C++ codeUsing medallia/unsafe (open source!)

Java↔C++ argument marshallingImplement native methods on-the-fly!

Generating C++ on-the-fly


#include <jni.h>/* omitted - initialization of field constants */void process(JNIEnv* env, jobject self, jobject segment) { jint* rawData[cols]; /* omitted - get raw data arrays */ jdouble sum = 0; jlong count = 0; for (int row = 0; row < (int)nRows; row++) { if (((rawData[0][row] == 1L || rawData[0][row] == 2L || rawData[0][row] == 3L) || (rawData[1][row] == 3L))) { count++; sum += rawData[6][row]; } } /* omitted - release raw data arrays */ jobject result = env->GetObjectField(self, resultFld); env->SetLongField(result, countFld, env->GetLongField(result, countFld) + count); env->SetDoubleField(result, sumFld, env->GetDoubleField(result, sumFld) + sum);}

C++ query


Benchmark Mode Cnt Score Error UnitsUnsafe.complexQuery thrpt 6 97.135 ± 4.905 ops/sUnsafe.simpleQuery thrpt 6 251.735 ± 14.856 ops/s

~20% improvement on simple queries (over the variant compiler)~35% improvement on complex ones

C++ query


Results


Java can now segfault.Using codegen limits probability of “incident”

Not recommended.

But fun.

C++ violates everything Java


CodeGen is easy (and also fun)Focus on generating String code

Faster to writeJust as fast to execute (and often faster)Easy to debug

Spend your time wisely

Bottom Line


Code for this talk: https://github.com/medallia/javaone2016

Clang as a library: https://github.com/medallia/unsafe

Thank you!

https://github.com/medallia/javaone2016

https://github.com/medallia/unsafe

Software

JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit