Scala-ActiveRecordType-safe Active Record model for Scala

@teppei_tosa_en

Who am I

鉄平 土佐TEPPEI TOSA

iron peace place name

The first official conference in Japan.http://scalaconf.jp/en/

Typesafe members came to Japan and gave speeches.

Talked about the case example of building our original BRMS “BIWARD” in Scala.

Japanese engineers talked aboutScala tips or their libraries.

• “Stackable-controller” by @gakuzzzzhttps://github.com/t2v/stackable-controller

• “How we write and use Scala libraries not to cry” by @tototoshihttp://tototoshi.github.io/slides/how-we-write-and-use-scala-libraries-scalaconfjp2013/#1

For example,

http://scalaconf.jp/en/

Scala-ActiveRecordType-safe Active Record model for Scala

• https://github.com/aselab/scala-activerecord

• Latest version : 0.2.2

• Licence : MIT

Features

• Squeryl wrapper

• type-safe (most part)

• Rails ActiveRecord-like operability

• Auto transaction control

• validations

• Associations

• Testing support

Most of the other ORM Libraries

Wrap SQL

val selectCountries = SQL(“Select * from Countries”)

Have to define mappings from the results of SQL to the models.

val countries = selectCountries().map(row => row[String](“code”) -> row[String](“name”)).toList

The motivation of Scala-ActiveRecord

• Want to define the model mapping more easily.

• Want not to define the find methods in each model classes.

• Want not to write SQLs.

Other libraries example

1. Anorm

2. Slick ( ScalaQuery )

3. Squeryl

1. Anorm

• Anorm doesn’t have the Model layer.

• Have to define similar methods in each Class.case class Person(id:Pk[Long], name:String)object Person {! def create(person:Person):Unit = {! ! DB.withConnection { implicit connection =>! ! ! SQL("insert int person(name) values ({name}")! ! ! ! .on('name -> person.name)! ! ! ! .executeUpdate()! ! }! }! ...}

2. Slick ( ScalaQuery )

• Query Interface is good.

• Definding tables syntax is redundant.

• Have to define the mapping between table and model in each table.

case class Member(id:Int, name:String, email:Option[String])object Members extends Table[Member]("MEMBERS") {! def id = column[Int]("ID", O.PrimaryKey, O.AutoInc)! def name = column[String]("Name")! def email = column[Option[String]]("EMAIL")! def * = id.? ~ name ~ email <> (Member, Member.unapply _)}

3. Squeryl

• The best one in these libraries.

• Scala-ActiveRecord wraps this with some improvement.

1. Optimize the generated SQLs

2. Automate the transaction control

3. Use CoC approach to build relationship

When queries are combinedSqueryl generates sub-query SQL.

Squeryl

val query = from(table)(t => where(t.id.~ > 20) select(t))

from(query)(t => where(t.name like "%test%) select(t))

Scala

select * from ! (Select * from table where table.id > 20) q1where q1.name like "test"

SQL

It negatively affect performance.

When queries are combinedSqueryl generates sub-query SQL.

Scala-ActiveRecord

val query = Table.where(_.id.~ > 20)

query.where(_.name like "%test%").toList

Scala

select * from tablewhere table.id > 20 and table.name like "test"

SQL

It can generate more simple SQL statement.

Automate the transaction control

Squeryl

Scala-ActiveRecord

• Call “inTransaction” automatically at accessing Iterable#iterator.

• When the save or delete method is called, “inTransaction” is executed by default.

• Off course, you can call “inTransaction” expressly.

inTransaction { books.insert(new Author(1, "Michel","Folco"))! ! val a = from(authors) (a=> where(a.lastName === "Folco") select(a))}

Use CoC approach tobuild relationship.

Squerylobject Schema extends Schema{! val foo = table[Foo]! val bar = table[Bar]! val fooToBar = oneToManyRelation(Foo, Bar).via(! ! (f,b) => f.barId === b.id! )}

class Foo(var barId:Long) extends SomeEntity {! lazy val bar:ManyToOne[Bar] = schema.fooToBar.right(this)}

class Bar(var bar:String) extends SomeEntity {! lazy val foos:OneToMany[Foo] = schema.fooToBar.left(this)}

Use CoC approach tobuild relationship.

Scala-ActiveRecordobject Table extends ActiveRecordTabels {! val foo = table[Foo]! val bar = table[Bar]}

class Foo(var barId:Long) extends ActiveRecord {! lazy val bar = belongsTo[Bar]}

class Bar(var bar:String) extends ActiveRecord {! lazy val foos = hasMany[Foo]}

Getting Started

Define the dependency in SBT project definition

Add the following settings in build.sbt or project/Build.scala.

libraryDependencies ++= Seq( "com.github.aselab" %% "scala-activerecord" % "0.2.2", "org.slf4j" % "slf4j-nop" % "1.7.2", // optional "com.h2database" % "h2" % "1.3.170" // optional)

resolvers += Resolver.sonatypeRepo("releases")

Using Scala ActiveRecord Play2.1 Plugin

Add the following settings in project/Build.scalaval appDependencies = Seq( "com.github.aselab" %% "scala-activerecord" % "0.2.2", "com.github.aselab" %% "scala-activerecord-play2" % "0.2.2", jdbc, "com.h2database" % "h2" % "1.3.170")

val main = play.Project(appName, appVersion, appDependencies).settings(

resolvers ++= Seq( Resolver.sonatypeRepo("releases") ))

Add the following settings in conf/play.plugins

9999:com.github.aselab.activerecord.ActiveRecordPlugin

Database Support

H2 database

MySQL

PostgrSQL

Derby

Oracle

Defining Schema

Model implementation

case class Person(var name:String, var age:Int)! extends ActiveRecord

object Person! extends ActiveRecordCompanion[Person]

Schema definition

object Tables extends ActiveRecordTable {! val people = table[Person]}

CRUD

Create

val person = Person("person1", 25)

person.save // return true

val person = Preson("person1", 25).create

// return Person("person1", 25)

ReadPerson.find(1)

// Some(Person("person1"))

Person.toList

// List(person("person1”), ...)

Person.findBy("name", "john")

// Some(Person("John"))

Person.where(_.name === "john").headOption

// Some(Person("john"))

UpdatePerson.find(1).foreach { p =>

! p.name = "Ichiro"

! p.age = 37

! p.save

}

Person.forceUpdate( _.id === 1)(

! _.name := "ichiro", _.age := 37

)

DeletePerson.where(_.name === "john").foreach(_.delete)

Person.find(1) match {

! case Some(person) => person.delete

! case _ =>

}

Person.delete(1)

Query Interface

Find single objectval client = Client.find(10)

// Some(Client) or None

val John = Client.findBy("name", "john")

// Some(Client("john")) or None

val john = Client.findBy(("name", "john"), ("age",25))

// Some(Client("john",25)) or None

Get the search result as List

Scala

Clients.where(c =>! c.name === "john" and c.age.~ > 25).toList

Clients! .where(_.name == "john")! .where(_.age.~ > 25)! .toList

generated SQL

select clients.name, clients.age, clients.idfrom clientswhere clients.name = "john" and clients.age > 25

Using iterable methodsval client = Client.head

// First Client or RecordNotFoundException

val client = Client.lastOption

// Some(Last Client) or None

val (adults, children) = Client.partition(_.age >= 20)

// Parts of clients

Ordering

Client.orderBy(_.name)

Client.orderBy(_.name asc)

Client.orderBy(_.name asc, _.age desc)

LimitClient.limit(10)

OffsetClient.page(2, 5)

ExistenceClient.exists(_.name like “john%”)// true or false

Specify selected fields

Client.select(_.name).toList

// List[String]

Client.select(c => (c.name, c.age)).toList

// List[(String, Int)]

Combine QueriesScala

Clients.where(_.name like "john%")! .orderBy(_.age desc)! .where(_.age.~ < 25)! .page(2, 5)! .toList

generated SQLselect clients.name, clients.age, clients.idfrom clientswhere ((clients.name like "john%") and (clients.age < 25))order by clients.age desclimit 5 offset 2

Cache Control

val orders = Order.where(_.age.~ > 20)

//execute this SQL query and cheche the query

orders.toList

//don't execute the SQL query

orders.toList

When the query is implicitly converted, the query is cached.

Validations

Annotation-based Validation

case class User(

! @Required name:String,

! @Length(max=20) profile:String,

! @Range(min=0, max=150) age:Int

) extends ActiveRecord

Object User extends ActiveRecordCompanion[User]

Exampleval user = user("", "Profile", 25).create

user.isValid // false

user.hasErrors // true

user.errors.messages // Seq("Name is required")

user.hasError("name") // true

User("", "profile", 15).saveEither match { case Right(user) => println(user.name) case Left(errors) => println(errors.message)}// "Name is required"

Callbacks

Available hooks• beforeValidation

• beforeCreate

• afterCreate

• beforeUpdate

• afterUpdate

• beforeSave

• afterSave

• beforeDelete

• afterDelete

Example

case class User(login:String) extends ActiveRecord {! @Transient! var password:String = _! var hashedPassword:String = _! override def beforeSave() {! ! hashedPassword = SomeLibrary.encrypt(password)! }}

val user = User("john")user.password = "raw_password"user.save// stored encrypted password

Relationship

One-to-Many

case class User(name:String) extends ActiveRecord {! val groupId:Option[Long] = None! lazy val group = belongsTo[Group]}

case class Group(name:String) extends ActiveRecord {! lazy val users = hasMany[User]}

groups

id

name

users

id

group_id

name

One-to-Many

val user1 = User("user1").createval group1 = Group("group1").create

group1.users << user1

group1.users.toList// List(User("user1"))

user1.group.getOrElse(Group("group2"))// Group("group1")

Generated SQL sampleScala

group1.users.where(_.name like "user%")! .orderBy(_.id desc)! .limit(5)! .toList

generated SQLSelect users.name, users.idFrom usersWhere ((users.group_id = 1) And (users.name like "user%"))Order by users.id Desclimit 5 offset 0

Many-to-Many (HABTM)

case class User(name:String) extends ActiveRecord {! lazy val groups = hasAndBelongsToMany[Group]}

case class Group(name:String) extends ActiveRecord {! lazy val users = hasAndBelongsToMany[user]}

groups

id

name

groups_users

left_id

right_id

users

id

name

val user1 = User("user1").createval group1 = Group("group1").createval group2 = Group("group2").create

user1.groups := List(group1, group2)

user1.groups.toList// List(Group("group1"), Group("group2"))

group1.users.toList// List(User("user1"))

Many-to-Many (HABTM)

Many-to-Many(hasManyThrough)

groups

id

name

memberships

id

user_id

group_id

isAdmin

users

id

name

Many-to-Many(hasManyThrough)

case class Membership(! userId:Long, projectid:Long, isAdmin:Boolean = false) extends ActiveRecord {! lazy val user = belongsTo[User]! lazy val group = belongsTo[Group]}

case class User(name:String) extends ActiveRecord {! lazy val memberships = hasMany[Membership]! lazy val groups = hasManyThrough[Group, Membership](memberships)}

case class Group(name:String) extends ActiveRecord {! lazy val memberships = hasmany[Membership]! lazy val users = hasManyThrough[User, Membership](memberships)}

Conditions Options

case class Group(name:String) extends ActiveRecord {! lazy val adminUsers =! ! hasMany[User](conditions = Map("isAdmin" -> true))}

group.adminUsers << user// user.isAdmin == true

ForeignKey option

case class Comment(name:String) extends ActiveRecord {! val authorId:Long! lazy val author = belongsTo[User](foreignKey = "authorId")}

Join TablesScala

Client.joins[Order](! (client, order) => client.id === order.clientId).where(! (client, order) => client.age.~ < 20 and order.price.~ > 1000).select(! (client, order) => (client.name, client.age, order.price)).toList

generated SQL

Select clients.name, clients.age, order.priceFrom clients inner join orders on (clients.id = orders.client_id)Where ((clients.age < 20) and (groups.price > 1000))

Eager loading associationsThe solution for N+1 problem.

Scala

Order.includes(_.client).limit(10).map {! order => order.client.name}.mkString("\n")

generated SQLSelect orders.price, orders.id From orders limit 10 offset 0

Select clients.name, clients.age, clients.idFrom clients inner join orders on (clients.id = orders.client_id)Where (orders.id in (1,2,3,4,5,6,7,8,9,10))

Logging and Debugging

See the generated SQLs

Use the toSql method

println(User.where(_.name like "john%").orderBy(_.age desc).toSql)

Set logging level with “debug” in logback.xml

<configuration> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder> <pattern>%highlight(%-5level) %cyan(%logger{15}) - %msg %n</pattern> </encoder> </appender> <root level="DEBUG"> <appender-ref ref="STDOUT" /> </root></configuration>

In SBT console

build.sbt or project/Build.scalainitialCommands in console := """import com.github.aselab.activerecord._import com.github.aselab.activerecord.dsl._import models._SomeTables.initialize(Map("schema" -> "models.SomeTables"))"""

In the console> consolescala> User.forceInsertAll{ (1 to 10000).map{i => User("name" + i)} }scala> User.where(_.name === "name10").toList

Testing

Setting for test

build.sbt or project/Build.scalalibraryDependencies ++= Seq( "com.github.aselab" %% "scala-activerecord" % "0.2.2", "com.github.aselab" %% "scala-activerecord-specs" % "0.2.2" % "test", "org.specs2" %% "specs2" % "1.12.3" % "test")

resolvers += Resolver.sonatypeRepo("releases")

application.conftest { schema = "models.Tables" driver = "org.h2.Driver" jdbcurl = "jdbc:h2:mem:test"}

Test Exampleimport com.github.aselab.activerecord._

object SomeModelSpecs extends ActiveRecordSpecification {

override val config = Map("schema" -> "com.example.models.Tables")

override def beforeAll = { super.beforeAll SomeModel("test1").create }

override def afterAll = { super.afterAll }

"sample" should { // Some specifications code }}

Performance

ActiveRecord Overhead

• How Sql statements are generated.

• The time to create active-record object from the results of query.

ORM Race

•Anorm

•Slick

•Squerl

•Scala-ActiveRecord

Race Condition• Create the “User” table which has only 3 columns

• Insert 1000 records into the “User” table

• Select all from the table with same statements

• Time their trip to the end of creation objects

• Exclude the time to creating DB connection

• Use Play framework 2.1.1

• Use H2-database

• Run in Global.onStart

• Run 5 times

• Compare with the average times

The Racers

Anorm Squeryl

Slick Scala-ActiveRecord

SQL("SELECT * FROM USER") .as(User.simple *)

Query(Users).list

from(AppDB.user) (s => select(s)) .toList

User.all.toList

The Race Results

39.8ms

116.8ms

177.2ms

258.8ms

Squeryl

Anorm

Scala-ActiveRecord

Slick

Future

Validation at compiling(with “Macro”)

• The findBy method and conditions of association will be type-safe.

• Validate whether foreign-key is specified with existing key or not.

Support Serialization

Form Model XML

JSON

MessagePack

Validation

View

Bind

ViewHelper

Support Web framework

• CRUD controller

• Form Helper for Play and Scalatra

• Code generator as SBT plugin

Secret

DEMO with YATTER( Yet Another twiTTER )

YATTER’s tables

Follows

id

userid

follows_users

id

user_id

follow_id

Users

id

name

Tweets

id

userId

textManyToMany

OneToMany

OneToOne(But not supported yet)

https://github.com/ironpeace/yatter

#scalajp

Mt.FUJI

Tokyo Station

Japanese Castle

Sushi

Okonomiyaki

@teppei_tosa_enhttps://github.com/aselab/scala-activerecordhttps://github.com/ironpeace/yatter

Thank you