Fpx talk 2014

NameChris Marshall @oxbow_lakes

GSA Capital Partners LLPMarch 2014

Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority.

Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261

Wrapping an imperative API in a functional one

3

Mathematics Degree

Working in financial software since 1999

• Smalltalk for ~6 months

• Java thereafter

• Scala since Dec 2008

JP Morgan for 6 years

• ~200,000 employees

GSA Capital for 7 ¾ years

• Quant hedge fund

• ~130 employees

Backgroundwho am i

Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261

4

Low-latency links & feeds

• Up to 800,000 trades /day

• 108 market events / day

Historic / Current Market Data

• Listing changes (e.g. SUNW.O becomes JAVA.O becomes ORCL.O)

• News Events

Backtesting / Execution Framework

Everything Else

• This is where I come in

• What are our positions? What is our P&L?

• Are our trades reconciled?

• Reporting (brokers, regulators, administrators)

GSAWhat do we do? Roughly half the company are technologists


5

Embarrassingly basic stuff• Target audience (people who’ve not done this before)• Scala programmers working on the JVM

Scala• I’m going to assume you can read scala code• I’m going to assume familiarity with typeclasses and their formulation in scalaz• I’m going to assume you understand implicits

Overview• Outline of (simplified) imperative API• Wrapping in functional scala• A tangent on futures• An example• Extending the example

This talk


6

Furnace• Furnace is one of GSA’s central systems• It’s a trade capture system which contains all of GSA’s daily trading activity• Currently handles flow of up to 800k trades/day• Plus many more cash events • Including state-transitions (modifications, confirmations etc), it is growing by ~2m events per day• Here’s an extremely simplified view of what its (Java) client API looks like

Furnace: an imperative Java data API


interface FurnaceService { FurnaceSession createSession(String, String); //user, password }

interface FurnaceSession { Iterator<FurnaceEvent> replayEvents(Date); Future<Long> insertEvent(FurnaceEventBuilder); void close(); }

7

Java/Scala interop• Scala’s Java interop means this is pointless, right?

Rationale• furnace’s API provides a very low-level view of interacting with the system• ARM• Idiomatic property access• For the hell of it

Typical Query• Give me a Map[TxnId, Event] where the values are the latest event on each transaction, for those

transactions whose latest event is not in state DELETED

Why implement a wrapper at all?


8

The Basic Wrapper


A datatype “Furnace”• Representing the calculation of a value from the interaction• Obviously, this should be a functor and a monad• We should be able to put an arbitrary value inside our “program”• We need to be able to derive a value from a session

sealed trait Furnace[+A] { def unsafePerformIO(): A

def map[B](f: A => B): Furnace[B] def flatMap[B](f: A => Furnace[B]): Furnace[B] }

object Furnace { def point[A](a: => A): Furnace[A] def apply[A](f: FurnaceSession => A): Furnace[A] }

9

StackOverflowException


Trampolines• Go and look at how scalaz 7 defines IO• Ivory Towers!

import scalaz._; import Scalaz._; import Free._

sealed trait Furnace[+A] { ... private def apply(rw: FurnaceSession): Trampoline[(FurnaceSession, A)] }

10

(I don’t follow this either)


Now define some more stuff• A trampolining IO operation• Implement the module “constructors”

trait FurnaceFunctions { def furnace[A](f: FurnaceSession => Trampoline[(FurnaceSession, A)]) = new Furnace[A] { private def apply(rw: FurnaceSession) = Suspend(() => f(rw)) } }

object Furnace extends FurnaceFunctions { def point[A](a: => A) = apply(_ => a) def apply[A](f: FurnaceSession => A) = furnace(rw => return_(rw -> f(rw))) }

11

“rw” is Real World


Now for map/flatMap• They make use of the previously defined furnace and apply

sealed trait Furnace[+A] { ...

def map[B](f: A => B) = furnace(rw => apply(rw) map { case (nw, a) => (nw, f(a)) })

def flatMap[B](f: A => Furnace[B]) = furnace(rw => apply(rw) flatMap { case (nw, a) => f(a)(nw) }) }

12

Getting a value out


unsafePerformIO• Resource management

object Furnace { ... def unsafePerformIO[A](fa: Furnace[A])(implicit l: Login): A = { val s = new FurnaceService //simplifying val t = s.createSession(l.user, l.password) try fa.apply(t).run._2 finally t.close() } }

13

Typeclasses


object Furnace { module => ... implicit val FurnaceInstances = new Monad[Furnace] { def point[A](a: => A) = module.point(a) def bind[A, B](fa: Furnace[A])(f: A => Furnace[B]) = fa flatMap f } }

14

That was it?


We can now start defining some useful methods• Querying

package gsa.furnace

package object dsl { def dateQuery(date: Date): FurnaceSession => List[FurnaceEvent] = fs => { import collection.JavaConverters._ fs.replayEvents(date).asScala.toList } }

15

Useful wrappers


Decorate the basics

implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => list.foldLeft(Map.empty[Long, FurnaceEvent]) { (m, ev) => m + (ev.getTransactionId -> ev) } filter { case (_, v) => excludeDeleted implies (ev.getState =/=

DELETED) } }

16

Where have we got to?


A simple program• Report is cheating a bit – probably should interleave with the IO monad• The expression inside the unsafePerformIO is a Furnace[Unit]

def report(evs: Iterable[FurnaceEvent]): Unit = //create a file, send an email?

import gsa.furnace.dsl._ val x = Furnace(dateQuery(today)).latest().map(xs => report(xs.values)

implicit val login = Login(“cmarsha”, “CorrectBatteryHorseStaple”) Furnace.unsafePerformIO(x)

17

Updating


More useful methods• Inserting

package gsa.furnace

package object dsl { ...

def insert(ev: FurnaceEventBuilder): FurnaceSession => j.u.c.Future[j.l.Long] = fs => fs.insert(ev) }

18

Still vestiges of Java


We are still working with j.u.c.Future• It’s actually rather trivial to create a “listenable” future• If you control the mechanism of submission (Use CompletionService and inform the listeners on the

completion thread)

We want to use s.c.Future• Must implement a few methods, isCompleted, onComplete, ready, result, value• Easy to convert a Listenable future into a s.c.Future

def onComplete[U](f: Try[T] => U)(implicit e: ExecutionContext) = { val l = new FutureListener[T] { def onCancelled() = try f(Failure(new CancellationException)) finally

removeListener(this) def onResult(r: T) = try f(Success(r)) finally removeListener(this)

def onThrowable(t: Throwable) = try f(Failure(t)) finally removeListener(this) } addListener(l) }

19

Our victory is complete!


We modify our wrapper• We first persuade the Furnace library author to give us this!

• Then we modify our API to remove the last traces of the old order

package gsa.furnace package object dsl { ...

def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long] = fs => fs.insert(ev).asScala.map(_.longValue) }

interface FurnaceSession { ... ListenableFuture<j.l.Long> insertEvent(FurnaceEventBuilder); }

20

Aside: Scalaz s.c.Future instances


implicit def FutureInstances(implicit EC: ExecutionContext) = new Monad[Future] with Traverse[Future] { def point[A](a: => A) = Promise.successful(a).future

override def map[A, B](fa: Future[A])(f: A => B) = fa map f

override def ap[A, B](fa: => Future[A])(fab: => Future[A => B]) = (fa zip fab) map { case (a, f) => f(a) } //for (f <- fab; a <- fa) yield f(a)

def bind[A, B](fa: Future[A])(f: A => Future[B]) = fa flatMap f

def traverseImpl[G[_], A, B](fa: Future[A])(f: A => G[B]) (implicit A: Applicative[G]): G[Future[B]] = A.map(f(Await.result(fa, Duration.Inf)))(b => point(b))

}

21

Where were we again?• We’ve seen how we could query furnace in a more “functional” style• We were removing Java futures, which were returned by the insert method and replacing them with scala

futures

Why?• Well, we can now compose a “pipeline” which involves querying and updating (inserting) events• What does one of those look like?

Quick recap


22

A “pipeline” where we query/update


A look at the finished article

type CorrectedEvents = s.c.Future[Stream[Long]]val correctFxRates: Map[Currency, Double] = ???def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Set[Currency]

val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z

Await.result(Furnace.unsafePerformIO(pipeline)(login), atMost = 5.minutes)

23

Let’s home in on correct_


type CorrectedEvents = s.c.Future[Stream[Long]]

//recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long]

def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents]

= xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[Furnace[Future[Long]]]

= xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy)))).sequenceU //<error>: is a Furnace[Stream[Future[Long]]]

= xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy)))) //<error>: is a Furnace[Stream[Future[Long]]]

= _.map(_.sequenceU) //is a Furnace[Future[Stream[long]]] is a Furnace[CorrectedEvents]

24

But we can pretty this last bit up


Another implicit wrapper later

• We can now implement correct_ as follows

implicit class FurnaceFutureStreamW[A](f: Furnace[Stream[Future[A]]]) { def sequenced(implicit EC: ExecutionContext) = f map (Future.sequence(_))}

def correct_(xs: Iterable[FurnaceEvent] ], rates: Currency => Double) = xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))).sequenced

25

A “pipeline” where we query/update


val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z

26

Error Handling


Recall the type of our pipeline

• We “cheated” by using inserting an exception into our future• What if dealing with furnace throws exceptions?• What if computing our suspicious FX rates throws exceptions?

val pipeline: Furnace[CorrectedEvents]

def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Exception \/ Set[Currency]

27

Monad Transformers: EitherT


What’s one of those then?• Monads do not compose (an M[N[A]] is not a monad, even if both M and N are)

If Banana is a monad with the correct structure• We denote its transformer as BananaT• Then for any monad M, M[Banana[A]] gives us a BananaT[M, A]• BananaT is a monad• BananaT looks like a Banana

There are monad transformers for many monads in Scalaz• StreamT, ListT, OptionT, StateT, ReaderT, EitherT• But not all monads (no FutureT, IOT)

28

Error Handling


Adding error handling to the furnace wrapper• Pretty trivial

object Furnace { ... def eitherT[A](fa: FurnaceSession => A): EitherT[Furnace, Throwable, A] = EitherT(apply(fs => \/.fromTryCatch(fa(fs)))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Furnace[Throwable \/ A] }

29

A “pipeline” handling errors



type CorrectedEvents = s.c.Future[Stream[Long]]def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable \/ Set[Currency]

val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

Furnace.unsafePerformIO(pipeline.run)(login) // Exception \/ CorrectedEvents

30

Let’s home in on correct_ (again)


type CorrectedEvents = s.c.Future[Stream[Long]]

//recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long]

def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents]

= xs.toStream.map(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[EitherT[Furnace, Throwable, Future[Long]]]

= xs.toStream.traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a EitherT[Furnace, Throwable, Stream[Future[Long]]] // Furnace[Throwable \/ Stream[Future[Long]]]

= _.map(_.sequenceU) //is a EitherT[Furnace, Throwable, Future[Stream[long]]] //is a EitherT[Furnace, Throwable, CorrectedEvents]

31

What kind of wizardry is this?


//Before def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents] = xs.toStream .traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU) //After def correct_(xs: Iterable[FurnaceEvent] , rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents] = xs.toStream .traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU)

32

A “pipeline” handling errors



type CorrectedEvents = s.c.Future[Stream[Long]]def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable \/ Set[Currency]

val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

Furnace.unsafePerformIO(pipeline.run)(login) // Exception \/ CorrectedEvents

33

What I’ve shown you• We’ve scala-fied a Java API• We’ve removed resource management from a programmers’ concern• We can create blocks of reusable, composable code• We have been able to take advantage of the higher-order abstractions provided by libraries like scalaz

What I’ve not shown you• The library throttles the rate at which updates can be made, all transparently to the client• Using other monad transformers with the library (like StreamT)• Using the library with Monoids• Lots more wrappers• Value classes (unboxed typesafe primitives) for transaction id, event id, String properties etc

What you might do yourself• Implement ListenableFutureExecutorService which returns listenable futures• Given a listenable future, implement map and flatMap. It’s not as trivial as it might first appear.

So what was the point?


34

Extra – abstracting over “latest”


We couldn’t call latest on EitherT• Or OptionT, StreamT etc.• We’d need to declare multiple implicit conversions for each type, wouldn’t we?

• Wouldn’t we?

implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => ... } }

35

We wouldn’t


We can abstract over a functor of List of events• I’ll see your type-fu and raise you• Thanks, Kenji Yoshida

final class ListOfEventFunctorOpsW[F[_]: A](self: F[A], F: Functor[F]) { def latestF(excludeDeleted: Boolean = true) (implicit A: A =:= List[FurnaceEvent]): F[Map[Long, FurnaceEvent]] = F.map(self) { a => A(a) ... } }

implicit def listOfEventFunctorOps[FA](fa: FA)(implicit F: Unapply[Functor, FA])

= new ListOfEventFunctorOps[F.M, F.A](F(fa), F.TC) ^^^ ^^^ dependent types

36

And now, eternity


val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { x <- Furnace.eitherT(dateQuery(d)).latestF() y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

x <- Furnace(dateQuery(d)).latest() y <- suspiciousFxRates(x.values).point[Furnace] z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace]

37

StackOverflows - approaches


final class SequencedFunctorOps[F[_], A](self: F[A], F: Functor[F]) { def sequencedF[B, M[_] <: TraversableOnce[_]] (implicit EC: ExecutionContext, A: A => M[Future[B]], cbf: CanBuildFrom[M[Future[B]], B, M[B]]): F[Future[M[B]]] = F.map(self)(a => Future.sequence(A(a))) }

object fewchaz { object Implicits { object sequential { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* do not override ap */ } } object parallel { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* override ap */ } } } } import fewchaz.Implicits.sequential._

38

ReaderWriterStateT


type R = SmartProps type W = IO[Unit] //or any monoid type S = Unit type F[+A] = Furnace[A] type E = ErrorMsg type M[+A] = ReaderWriterStateT[F, R, W, S, A]

type Program[+A]= EitherT[M, E, A]

object Program { def furnace[A](f: R => Furnace[E \/ A]): Program[A] = EitherT[M, E, A](ReaderWriterStateT[F, R, W, S, E \/ A] { case (r, s) => f(r).map(or => (IO(()), or, s)) } )

def apply[A](f: R => E \/ A): Program[A] = furnace(f andThen (_.point[F]))

def either[A](value: E \/ A): Program[A] = apply(_ => value)

def fromTryCatch[A](value: => A): Program[A] = either(try \/-(value) catch { case NonFatal(e) => -\/(ErrorMsg(e)) })

def report(msg: String): Program[Unit] = EitherT.right[M, E, Unit](ReaderWriterStateT[F, R, W, S, Unit] { case (r, s) => (IO(println(msg)), (), s) }) }

39

...ReaderWriterStateT


lazy val program = for { _ <- fromTryCatch(ProductUniverse.setUniverseProvider(new CoreUniverseProvider)) _ <- report(s"About to run the program with: [${args mkString " "}]") dd <- date _ <- report(s"Running for carry date [$dd]") ps <- positions(dd) _ <- report(f"Found ${ps.size}%,d positions in DI1 futures") cc <- //Only try and get marks and D1 if there *are* positions ps.nonEmpty ?? (for { ms <- marks(dd) _ <- report(f"Found ${ms.size}%,d marks on DI1 futures") d1 <- d1Rate(dd) _ <- report(s"D1 rate is $d1") xx <- either(costs(dd, ps.values.toStream, ms)(d1)) } yield xx) _ <- report(s"Costs for positions are: $cc") fs <- //Must run the furnace section regardless of whether we now have costs furnaceInsertOrUpdate(cc, dd) rs <- { import scala.concurrent.duration._ fromTryCatch(Await.result(fs, atMost = 5.minutes)) } _ <- report(f"Created ${rs.size} furnace events") } yield ()

40

Nothing but a Gnab Gib


case class Result(log: W, v: E \/ Unit)

val result = (for { c <- config //E \/ SmartProps l <- c.maybeLogin(prefix = Some("furnace")).missingConfig //E \/ Login r <- \/-(Furnace.unsafePerformIO(program.run.run(c, ()))(l)) } yield Result(r._1, r._2)) valueOr { s => Result(IO(println(“Unable to invoke program”)), -\/(s)) }

//The end of the world (result.log |+| IO(println(result.v))).unsafePerformIO

41

This presentation

• http://www.slideshare.net/oxbow_lakes/fpx-talk-2014

• Fewchas: https://gist.github.com/oxbowlakes/8666295

• Furnace Scala API: https://gist.github.com/oxbowlakes/8686567

References

• Runar Bjarnason talk: http://skillsmatter.com/podcast/scala/stackless-scala-free-monads

• Scalaz: https://github.com/scalaz/scalaz

Extrasfor the masochists


http://www.slideshare.net/oxbow_lakes/fpx-talk-2014

https://gist.github.com/oxbowlakes/8666295

https://gist.github.com/oxbowlakes/8686567

http://skillsmatter.com/podcast/scala/stackless-scala-free-monads

https://github.com/scalaz/scalaz

42

Contact us


GSA Capital Partners LLP

[email protected] +44 (0)20 7959 8850

London Office

Stratton House5 Stratton StreetLondon W1J 8LA

T +44 (0)20 7959 8800F +44 (0)20 7959 8845

New York Office

1140 Ave of the Americas9th FloorNew York NY 10036

Economy & Finance

Fpx talk 2014