Upload
lukasz-balamut
View
434
Download
1
Tags:
Embed Size (px)
DESCRIPTION
talk to the Scala User group in Berlin
Citation preview
Scala in Nokia Places API
Lukasz Balamut - Places API Team
17 April 2013
Purpose of this talk
I Why are we spending time migrating to a new language?I How are we doing that?I What have we learned?
and of course: we are hiring.
Our scala story.
Our migration in lines of code
We’ve discovered more and more things on the way.
The Concordion tests problemIn June 2011 we had problems with Concordion (HTML based acceptance testframework),
I hard to integrateI HTML tests were not readable
Solution:
I Multi-line Strings in ScalaI Continue using JUnit
val expectedBody = """
{"id":"250u09wh-83d4c5479b7c4db9836602936dbc8cb8",
"title" : "Marfil (Le)",
"alternativeNames" : [
{"name" : "Le Marfil",
"language" : "fr"
}]
}"""
The test names problem
I JUnit + camel case - not that readable for long names
@Test
def preservesSearchResponseItemOrder() { /* ... */ }
I JUnit + underscores - violates naming convention, but better to read
@Test
def preserves_search_response_item_order() { /* ... */ }
I ScalaTest
test("preserves search response’s item order") { /* ... */ }
It is supported by tools we use and produces nice output when run (as a bonus)
[info] DiscoverAroundAcceptanceTest:
[info] - calls nsp with 15Km radius
[info] - preserves search response’s item order
[info] - when requesting zero results, no request to search is made
Lift-Json
I JSON (de)serialisation can be done in really nice way
e.g. getting place id:
{"place": {
"a_id": "276u33db-6f084959f97c45bc9db26348bafd4563"
}}
Also there is a generic way of parsing json, that gives you XPath like access:
val id = (json \ "place" \ "a_id").extract[String]
I Lift-json provides also easy and efficient case class JSON (de)serialisation,hance case classes
Scala case classes in the modelWe model our upstream and downstream datamodels as case classes.
case class Place (
name: String,
placeId: PlaceID
//(...)
)
I Each case class is a proper immutable data type - the new Java Bean ;),I A lot of boilerplate code is generated by the compiler e.g.:
I accessors,I equals/hashCode,I copy methods,I toStringI apply/unapply (for pattern maching),
After decompiling the scala code above we will end up with 191 lines of Java:
package pbapi.model.api;
import scala.*;
import scala.collection.Iterator;
import scala.runtime.BoxesRunTime;
import scala.runtime.ScalaRunTime$;
public class Place
implements Product, Serializable
{
public static final Function1 tupled() {return Place$.MODULE$.tupled();
}
public static final Function1 curry() {return Place$.MODULE$.curry();
}
public static final Function1 curried() {return Place$.MODULE$.curried();
}
public Iterator productIterator() {return scala.Product.class.productIterator(this);
}
public Iterator productElements() {return scala.Product.class.productElements(this);
}
public String name() {return name;
}
public PlaceID placeId() {return placeId;
}
public Place copy(String name, PlaceID placeId) {return new Place(name, placeId);
}
public PlaceID copy$default$2() {return placeId();
}
public String copy$default$1() {return name();
}
public int hashCode() {return ScalaRunTime$.MODULE$._hashCode(this);
}
public String toString() {return ScalaRunTime$.MODULE$._toString(this);
}
public boolean equals(Object obj) {if(this == obj) goto _L2; else goto _L1
_L1:
Object obj1 = obj;
if(!(obj1 instanceof Place)) goto _L4; else goto _L3
_L3:
String name$6;
PlaceID placeId$4;
Place place1 = (Place)obj1;
String s = place1.name();
PlaceID placeid = place1.placeId();
name$6 = s;
placeId$4 = placeid;
if(gd22$1(name$6, placeId$4) ? ((Place)obj).canEqual(this) : false) goto _L2; else goto _L5
_L4:
if(false) goto _L2; else goto _L5
_L2:
true;
goto _L6
_L5:
false;
_L6:
return;
}
public String productPrefix() {return "Place";
}
public int productArity() {return 2;
}
public Object productElement(int i) {int j = i;
j;
JVM INSTR tableswitch 0 1: default 24
// 0 39
// 1 46;
goto _L1 _L2 _L3
_L1:
throw new IndexOutOfBoundsException(BoxesRunTime.boxToInteger(i).toString());
_L2:
name();
goto _L4
_L3:
placeId();
_L4:
return;
}
public boolean canEqual(Object obj) {return obj instanceof Place;
}
private final boolean gd22$1(String s, PlaceID placeid) {s;
String s1 = name();
if(s != null) goto _L2; else goto _L1
_L1:
JVM INSTR pop ;
if(s1 == null) goto _L4; else goto _L3
_L2:
s1;
equals();
JVM INSTR ifeq 57;
goto _L4 _L3
_L4:
placeid;
PlaceID placeid1 = placeId();
if(placeid != null) goto _L6; else goto _L5
_L5:
JVM INSTR pop ;
if(placeid1 == null) goto _L7; else goto _L3
_L6:
placeid1;
equals();
JVM INSTR ifeq 57;
goto _L7 _L3
_L7:
true;
goto _L8
_L3:
false;
_L8:
return;
}
public Place(String name, PlaceID placeId) {this.name = name;
this.placeId = placeId;
super();
scala.Product.class.$init$(this);
}
private final String name;
private final PlaceID placeId;
}
package pbapi.model.api;
import scala.*;
import scala.runtime.AbstractFunction2;
public final class Place$ extends AbstractFunction2
implements ScalaObject, Serializable
{
public final String toString() {return "Place";
}
public Option unapply(Place x$0) {return ((Option) (x$0 != null ? new Some(new Tuple2(x$0.name(), x$0.placeId())) : None$.MODULE$));
}
public Place apply(String name, PlaceID placeId) {return new Place(name, placeId);
}
public Object readResolve() {return MODULE$;
}
private Place$() {}
public static final Place$ MODULE$ = this;
static {new Place$();
}}
Scala case classes - Configuration
I Easy to update configurationI Easy to compare environments’ configuration
case class HttpClientConfig(
serviceName: String,
serviceBaseUrl: URL,
readTimeout: Int = 0,
connectionTimeout: Int,
proxy: Option[ProxyConfig] = None,
oauthKeys: Option[OAuthConfig] = None,
displayUrl: Option[String] = None
)
"httpConfig":{"serviceName":"recommendations",
"serviceBaseUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/nearby/",
"readTimeout":4000,
"connectionTimeout":500,
"displayUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/nearby/"
}
The Scala Type System
I A great way to express assumptionsI Have them checked by the compiler through the whole codebaseI At every point in the code know what to expect or you will be reminded by
compiler.
The Scala Type System - Options 1
I Avoid null references (see The Billion Dollar Mistake )
This will not compile!
case class UpstreamResponse ( unreliableField: Option[String] )
case class MyResponse ( reliableField: String )
def transform(upstream: UpstreamResponse) =
MyResponse(
reliableField = upstream.unreliableField //<-- incompatible type
)
The Scala Type System - Options 2
But this will:
case class UpstreamResponse ( unreliableField: Option[String] )
case class MyResponse ( reliableField: String )
def transform(upstream: UpstreamResponse) =
MyResponse(
reliableField = upstream.unreliableField.getOrElse("n/a") // enforced by compiler
)
The Scala Type System - Options 3
how we were learning e.g.: to transform string when it is defined
def transform(str: String): String
reliableField =
if (upstream.unreliableField.isDefined)
transform(upstream.unreliableField.get)
else "n/a"
reliableField =
upstream.unreliableField match {case Some(f) => transform(f)
case None => "n/a"
}
reliableField =
upstream.unreliableField.map(transform) | "n/a"
The Scala Type System - Standard Types
I Immutable Map, List, Sets
val map: Map[String, String] = Map("a" -> "a", "b" -> "b")
val list: List[String] = "a" :: "b" :: Nil
I Tuples to express Pairs, Triplets etc.
val pair: Pair[String, Int] = ("a", 1)
val (a, b) = pair // defines a:String and b:Int
The Scala Type System - error handlingI Types can help with exceptional cases
val parsed: Validation[NumberFormatException, Int] = someString.parseInt
val optional: Option[Int] = parsed.toOption
val withDefault: Int = optional.getOrElse(0) //if there was parsing problem 0
I Exception to Option
val sub: Option[String] =
allCatch.opt(line.substring(line.indexOf("{\"")))
I or Either
val parsed: Either[Throwable, Incident] =
allCatch.either(new Incident(parse(jsonString))
and handle it later
parsed match {case Right(j) => Some(j)
case Left(t) => {println("Parse error %s".format(t.getMessage))
None
}}
The Scala Type System - functions
I we defer translation (Translated) and text rendering (FormattedText) toserialisation time using function composition so we don’t need to pass usercontext through the whole stack e.g.:
case class Place (
//(...)
attribution: Option[Translated[FormattedText]],
//(...)
)
case class Translated[A](translator: TransEnv => A) {def apply(env: TransEnv): A = translator(env)
def map[B](f: A => B): Translated[B] =
new Translated[B](env => f(apply(env)))
//(...)
}
XML literals
XML can be part of the code and compiler is checking syntax of it:
def toKmlCoordinates(style: String): String =
(<Placemark>
<name>{"bbox " + this}</name><styleUrl>{style}</styleUrl><Polygon>
<outerBoundaryIs>
<LinearRing>
<coordinates>
{west + "," + north + ",0"}{east + "," + north + ",0"}{east + "," + south + ",0"}{west + "," + south + ",0"}{west + "," + north + ",0"}
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>).toString
ScalacheckAutomatic testing of assumptions, using set of generated values.
e.g. the code below tests if
BBox.parse(a.toString) == a
is true for any a
val generator: Gen[BBox] =
for {s <- choose(-90.0, 90.0)
n <- choose(-90.0, 90.0)
if (n > s)
w <- choose(-180.0, 180.0)
e <- choose(-180.0, 180.0)
} yield new BBox(w, s, e, n)
property("parse . toString is identity") {check {
(a: BBox) =>
BBox.parse(a.toString) == a
}}
may yield this failing test message after several evaluations.
GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
(BBoxTest.scala:32)
Falsified after 2 successful property evaluations.
Location: (BBoxTest.scala:32)
Occurred when passed generated values (
arg0 = 33.50207944036654,-63.96980425691198,59.04945970748918,8.222590173180834
)
Scala (Scalding) in our analytics jobse.g. popular places job
class PopularPlacesJob(args: Args) extends AccessLogJob(args) {
implicit val mapMonoid = new MapMonoid[String, Long]()
val ord = implicitly[Ordering[(Long, String)]].reverse
readParseFilter()
.flatMap(’entry -> ’ppid) {entry:AccessLogEntry => entry.request.ppid
}.mapTo((’entry, ’ppid) -> (’ppid, ’time, ’app)) {
arg: (AccessLogEntry, String) => (arg._2, arg._1.dateTime, arg._1.appId.getOrElse("?"))
}.groupBy((’ppid, ’app)) {
_.min(’time -> ’minTime)
.max(’time -> ’maxTime)
.size(’num)
}.groupBy((’ppid)) {
_.min(’minTime)
.max(’maxTime)
.sum(’num)
.mapPlusMap(((’app, ’num) -> ’apps))
{ Map(_:(String, Long)) }{ _.toList
.map(a => (a._2, KnownClients.appName(a._1, "N/A")))
.sorted(ord)
.mkString(";")
}}.reverseSortAndLimit(’num)
.writeWithHeader(args("output"))
}
Not so good in Scala
I Almost whole team needed to learn the new language - luckily we haveToralf ;)
I Tools are not as mature as those in Java (but they are getting better veryfast)
I Longer compilation, compiler has a lot more to do (e.g. type inference)
I Multiple inheritance (traits)
I Operators
I changes in every language release
What we have learned, discovered?
I When done right - it’s possible to painlessly migrate code to newlanguage, developing new feautres in the same time
I How to use good typesystem and enjoy it
I Take advantage form whole great immutable world of painlessprogramming
I Use functions as first class citizens of language
Plans
I Scala 2.10I Run Transformations in Futures, Validations etc.I Akka actors, Futures compositionI Kill last Java files?I Stop using Spring
Thank you!
I project site:places.nlp.nokia.com
I My contact:[email protected]@lbalamut
I open position in our team: