TourGustavo Fernandes

gustavo@redhat.com

About Infinispan• Apache license in-memory Key/Value data store

• API: Map based (ConcurrentMap) and JCache (JSR-107) compliant

• Embedded or Remote: Hot Rod, Rest, Memcached, Websocket

• Main cache modes: Distributed, Replicated, Local

• XA Transactions

• Events

• Distributed processing/Map Reduce

• Client language (C#, C++, Java, Javascript)

• Query: indexed with Lucene or non indexed

• 8.0.0.Final Released Aug-2015 / 8.1.0.Final Dec-2015

What’s new tl;dr• First based on Java 8

• New APIs leveraging the best java 8 has to offer

• New Admin Console

• New Integrations: Apache Hadoop, Apache Spark

• Querying: continuous, grouping, aggregation, Lucene 5, hybrid

• Expiration and eviction enhancements

New APIs

• Experimental Functional Map API

• Distributed Streams

Functional Map API motivations

• Not a replacement for the current Map/JCache API

• Asynchronous

• Lazy

• Lambda based

Functional Map API entry points AdvancedCache<String, String> cache = cacheManager.getCache().getAdvancedCache();

FunctionalMapImpl<String, String> functionalMap = FunctionalMapImpl.create(cache);

ReadOnlyMap<String, String> readOnlyMap = ReadOnlyMapImpl.create(functionalMap);

WriteOnlyMap<String, String> writeOnlyMap = WriteOnlyMapImpl.create(functionalMap);

ReadWriteMap<String, String> readWriteMap = ReadWriteMapImpl.create(functionalMap);

Functional Map API - ReadOnlyMap

<R> CompletableFuture<R> eval(K key, Function<ReadEntryView<K, V>, R> f);

<R> Traversable<R> evalMany(Set<? extends K> keys, Function<ReadEntryView<K, V>, R> f);

Functional Map API - ReadOnlyMap example

public class Transaction implements Serializable { int size; String script; boolean processed; public String getScriptHash() { byte[] digest = MessageDigest.getInstance(“SHA-256”).digest(script.getBytes("UTF-8")); return Base64.getEncoder().encodeToString(digest); } . . . }

CompletableFuture<String> value = readOnlyMap.eval("key223", entryView -> entryView.get().getScriptHash());

Functional Map API - ReadOnlyMap example

// key1, key2, ..., key10Set<String> keys = IntStream.rangeClosed(1, 10).boxed() .map(i -> "key" + i) .collect(Collectors.toSet()); Integer sum = readOnlyMap .evalMany(keys, entryView -> entryView.get().getSize()) .reduce(0, (a, b) -> a + b);

Functional Map API - WriteOnlyMap

CompletableFuture<Void> eval(K key, V value, BiConsumer<V, WriteEntryView<V>> f);

CompletableFuture<Void> eval(K key, Consumer<WriteEntryView<V>> f);

CompletableFuture<Void> evalMany(Map<? extends K, ? extends V> entries, BiConsumer<V, WriteEntryView<V>> f);

Functional Map API - WriteOnlyMap examples

// key1, key2, ..., key10Set<String> keys = IntStream.rangeClosed(1, 10).boxed() .map(i -> "key" + i).collect(Collectors.toSet()); writeOnlyMap.evalMany(keys, EntryView.WriteEntryView::remove).get();

CompletableFuture<Void> future = writeOnlyMap .eval("newKey", v -> v.set(new Transaction(5000, "13E3401")));future.thenAccept(e -> System.out.println(cache.get("newKey")));

Remove values from a set of keys:

Insert new value and print when done:

Functional Map API - Read/Write

Traversable<CompletableFuture<Void>> futures = readWriteMap.evalAll(view -> { Transaction t = view.get(); if(t.isProcessed()) { view.remove(); } if (t.getSize() > 1000) { return writeOnlyMap.eval(view.key() + "_", v -> { int newSize = t.getSize * 0.90; Transaction newTx = new Transaction(newSize, t.getScript()); v.set(NewTx); }); } return null; });

Distributed Streams

• Distributed implementation of java.util.stream.Stream over cache data

• Topology Aware

• Parallel

• Lambdas and Collectors must be serializable

Distributed Streams - Usage

CacheStream<Entry<String, Transaction>> entryStream = cache.entrySet().stream();

CacheStream<Transaction> valueStream = cache.values().stream();

CacheStream<String> keyStream = cache.keySet().stream();

Distributed Streams - Usage

Stream<Transaction> valueStream = cache.values().stream();Double averageSize = valueStream .filter(Transaction::isProcessed) .collect(Collectors.averagingInt(Transaction::getSize));

Distributed Streams - example

Double averageSize = valueStream .filter((Serializable & Predicate<Transaction>) Transaction::isProcessed) .collect(CacheCollectors.serializableCollector(() -> Collectors.averagingInt( (Serializable & ToIntFunction<Transaction>) Transaction::getSize) )

);

Admin Console

• Angular.js, bootstrap.js and PatternFly based

• Server mode

• Lifecycle/Monitoring

Admin Console - Demo• Start Domain controller

•Start slaves

docker run -it --name master -h master -e "SLAVES=1" gustavonalle/infinispan-server-domain

docker run -it --link master:master -h slave1 gustavonalle/infinispan-server-domain

Integrations• Hadoop and Spark Connectors

• Read/Write

• Infinispan Server as a datasource

• Separated Clusters

• Server mode only

• Task distribution with best locality

ArchitectureMaster

Worker Worker Worker Worker

Infinispan Spark Connector - Demo

• Start Infinispan docker run -it --name master -h master gustavonalle/infinispan-server-domain

• Start Spark docker run --name spark-master -ti gustavonalle/spark

• Start Spark shell docker exec -it spark-master /usr/local/spark/bin/spark-shell --master spark://172.17.0.3:7077 —-packages org.infinispan:infinispan-spark_2.10:0.2

Integrations - Demo import org.infinispan.spark._ import org.infinispan.spark.rdd._ import scala.util.Random

val wordList = scala.io.Source.fromFile("/usr/share/dict/words").getLines.foldLeft(List[String]())( (s, w) => w :: s)

val phrases = (0 to 400).toStream.map(i=> Random.nextInt(wordList.size)).sliding(4,4).map(_.map(wordList).mkString(" ")).toSeq

val phraseRDD = sc.parallelize(phrases).zipWithIndex.map(_.swap)

val config = new java.util.Properties config.put(“infinispan.client.hotrod.server_list”,“172.17.0.2:11222“) phraseRDD.writeToInfinispan(config)

val infinispanRDD = new InfinispanRDD[Long,String](sc, config) infinispanRDD.values.flatMap(_.split(" ")) .map( word => (word.length, 1) ) .reduceByKey(_+_) .sortBy(_._1).toArray .foreach{ case (size, count) => println(s"$size chars words: $count occurrences") }

Query - Aggregations

Query size = qf.from(Transaction.class) .select(Expression.avg("size"), Expression.max("size"), Expression.min(“size")) .having(“processed").eq(true).toBuilder() .build();

Query - Grouping QueryFactory qf = Search.getQueryFactory(cache); Query q = qf.from(Transaction.class) .select( Expression.property("accountId"), Expression.avg(“amount")) .groupBy(“accountId") .having(Expression.avg(“amount”)).lt(130.0).toBuilder() .orderBy(“accountId") .build();

Query - ContinuousQueryFactory<?> qf = Search.getQueryFactory(cache);Query query = qf.from(Person.class) .having("salary").lte(3000) .toBuilder().build();ContinuousQueryListener<Object, Object> listener = new MyListener<>(); ContinuousQuery<Object, Object> cq = new ContinuousQuery<>(cache); cq.addContinuousQueryListener(query, listener);

Query - Continuous/** * Listener for continuous query events. * * @author anistor@redhat.com * @since 8.1 */public interface ContinuousQueryListener<K, V> { /** * Receives notification that a cache entry has joined the matching set. * * @param key the key of the joining entry * @param value the joining entry or the Object[] projection if specified */ void resultJoining(K key, V value); /** * Receives notification that a cache entry has left the matching set. * * @param key the key of the leaving entry */ void resultLeaving(K key);}

Core - Eviction• Eviction can be triggered by memory size (estimation):

ConfigurationBuilder cfg = new ConfigurationBuilder();cfg .eviction().strategy(EvictionStrategy.LRU).type(EvictionType.MEMORY).size(MAX_MEMORY) .build();

• Works for primitives key/value, byte arrays

• Pluggable estimator for custom classes planned

Core - Expiration• New clustered expiration events

@Listener(clustered = true) protected class ClusterListener { @CacheEntryExpired public void onCacheEvent(CacheEntryEvent event) { log.debugf(“Received expiration event %s", event); }}

Thankshttps://github.com/infinispan/infinispan

https://issues.jboss.org/browse/ISPN

#infinispan Freenode

https://twitter.com/infinispan

http://infinispan.org/download

https://github.com/infinispan/infinispan-spark