Embed Size (px)
Citation preview
1
Scala technology supportingthe Dwango account system
Seitaro Yuki / @pab_tech / DWANGO Co., Ltd.
ドワンゴアカウントシステムを支えるScala技術結城清太郎 / @pab_tech / 株式会社ドワンゴ
2
OverviewDwango is often named as a leading company to adoptScala among the enterprises in Japan.The account system is known to be largest Scala projectin our company, responsible for registering niconico usersand authenticating them.In this session we will introduce a variety of Scalaprogramming techniques that are used in the accountsystem
ドワンゴのアカウントシステムで使われているScalaプログラミング技術について解説します
3
ContentsWeb applications by continuation monadsDependency injection by Minimal Cake PatternNew transaction monad using subtyping
継続モナド、Minimal Cake Pattern、トランザクションモナドについて解説します
4
Web applications by continuation monads
継続モナドによるWebアプリケーション
5
Technical problems:How to construct web components
Web application needs to be decomposed to requestprocessing, response processing, filtering processing, etc.Each component needs to be easy to understand and testWe want a simple and powerful component technology
課題: Webアプリケーションの部品化
6
Simple and powerful component technology:Monads
We will explain how to build a web application usingcontinuation monads
継続モナドを使ったWebアプリケーションの構成方法を説明します
7
Similarity of a typical web application andcontinuation monads
まずは一般的なWebアプリケーションの構造と継続モナドの類似性から見ていきましょう
8
Typical web application structureTypical web applications have nested structure likeonions
e.g. Java Servlet, Python WSGI, Ruby RackMiddleware performs the following operations
It processes a request, and passes the result to the nextmiddlewareIt modifies a response that came back, and returns theresult to the before middlewareOr, when an error occurs it returns the result withoutcalling the next middleware
典型的なWebアプリケーションの構造
9
Typical web application structureWeb application
Web Browser Web Server Filter A Filter B Main Processing
request
request
request
request
response
response
response
response
典型的なWebアプリケーションの構造
10
Authentication middlewareLet's consider authentication middlewareWe assume that users logined already somewhereAuthentication middleware retrieves a session ID fromcookies, confirms using a storage such as Redis, passesthe result to the next middlewareOr, if there is no session ID or the session is invalid,authentication middleware redirects to a login form
認証処理をおこなうミドルウェアについて考えてみる
11
Authentication middlewareWeb Browser Web Server Authentication Filter Main Processing
request
request
alt ["Authentication is successful"]
request
request
response
response
response
["Authentication is failed"]redirect
response
認証処理のシークエンス図
12
Code this authenticationmiddleware using continuation
monad
では、この認証処理を継続モナドを使って書いてみましょう
13
What is continuation?Continuation represents the rest of thecomputation at a given point in theprocess's execution
Continuation from A
Continuation from B
B
R
Caller
A
Caller
プログラムの実行のある時点において評価されていない残りのプログラムを継続と呼ぶ
14
Continuation-passing style (CPS)CPS is a style of programming in which control is passed acontinuation explicitly
Caller
Caller
A
A
B
B
R
R
pass a continuation after A and execute
pass a continuation after B and execute
pass a continuation after R and execute
execute the continuation
execute the continuation
execute the continuation
継続を明示的に受け渡しするプログラミングスタイルを継続渡しスタイルと呼ぶ
15
CPS and web applications are similarWeb application seems to be able to be written in CPSFurther if continuation is monad, it becomes a simple andpowerful component
CPSとWebアプリケーションは似ている継続がモナドならさらによい
16
Continuation monad
A is a type of value at a given point, R is a final result typeA => R represents a continuation(A => R) => R represents a function that take acontinuationCont[R, A] is a constructor that creates a monad from afunction that take a continuation継続モナドは、継続を受け取る関数を受け取って、モナドを作る
case class Cont[R, A](run: (A => R) => R) { def map[B](f: A => B): Cont[R, B] = Cont(k => run(a => k(f(a)))) def flatMap[B](f: A => Cont[R, B]): Cont[R, B] = Cont(k => run(a => f(a).run(k))) }
17
Let's make web application usingcontinuation monads
継続モナドを使ってWebアプリケーションを作ってみよう
18
Authentication of continuation monad
継続モナドによる認証処理
def readSessionIdFromCookie(request: Request): Option[SessionId] = ???
def readSessionFromRedis(sessionId: SessionId): Option[Session] = ???
def redirectLoginForm: Response = ???
def auth(request: Request): Cont[Response, Session] = Cont((k: Session => Response) => readSessionIdFromCookie(request) .flatMap(readSessionFromRedis) .map(k) .getOrElse(redirectLoginForm)
19
Web application using for-expression
for式を使ってWebアプリケーションを組み立てる
def compress(request: Request): Cont[Response, Unit] = ???
def mainProcessing(request: Request, session: Session): Cont[Response,
def webApplication(request: Request): Cont[Response, Response] = for { session <- auth(request) _ <- compress(request) response <- mainProcessing(request, session) } yield response
// execution webApplication(request).run(identity) // => Response
20
Let's look at Play Action compositionmethods for comparison
比較のためにPlay標準のAction合成方法を見てみよう
21
Play Action composition methodsActionFunction
def invokeBlock
ActionBuilder ActionRefiner
ActionFilter ActionTransformerAction
The methods of Play Action composision is performed byfunctional composition
PlayのAction合成方法ActionFunctionの合成によっておこなわれる
22
Actually continuation also appears in Play
block: P[A] => Future[Result] is continuationBut it was not a monad
実はPlayでも継続の概念が使われているしかしモナドではなかった
trait ActionFunction[-R[_], +P[_]] { def invokeBlock[A](request: R[A], block: P[A] => Future[Result]): Future}
23
Problems of Play Action compositionFunctional composition of ActionFunction is less flexiblethan monadic compositionFor example, AuthenticatedBuilder returnsAuthenticatedRequestTherefore, AuthenticatedBuilder can be composed to onlyActionFunction receives AuthenticatedRequest
PlayのAction合成の問題点ActionFunctionの合成は柔軟性に欠ける
24
Advantages of using a continuation monadContinuation monad is simple
It can be represented by 6 linesMonadic composition is more flexible rather thanfunctional compositionIt is possible to combine with other kinds of monad byusing monad transformers
The account system uses ContT[Future, Result, A]for error handling
継続モナドをWebアプリケーションの記述に使う利点
25
Dependency Injection by MinimalCake Pattern
Minimal Cake PatternによるDependencyInjection
26
Technical problems:Large-scale system development
The account system has 230,000 lines in only code ofScala
For comparison, Play has 90,000 lines, Slick has 30,000lines
How to modularizeHow to test
課題: 大規模システム開発23万行のScalaコードがある
27
Let's solve these problems by thedependency injection
Dependency Injectionでこれらの課題を解決しよう
28
DI technology that we have adoptedDI technology that we have adopted is defined as"Minimal Cake Pattern"We will explain the following contents
Comparison of various other DI technologies of ScalaWhat is difference from the existing Cake PatternAdvantages for large-scale system development
ここでは我々が採用している"Minimal CakePattern"というDI技術について解説をします
29
Various methods of DI in ScalaDI Container such as GuiceCake patternReader monad
様々なScalaのDependency Injection方法
30
DI Container such as GuiceAdvantages
Experience in JavaAlso adopted in Play
Dependencies in source code are eliminatedcompletely
Shortening of compile timeDisadvantages
DI container often causes a run-time error of objectcreationWe should learn about the DI container
GuiceなどのDIコンテナを使う方法の利点と欠点
31
Cake patternAdvantages
Checking at compile timeFrameworks such as DI container are not required
DisadvantagesCake patterns would complicate a system as acomponent technologyIncrease boilerplates
Cakeパターンを使う方法の利点と欠点
32
Reader monadAdvantages
Same as Cake PatternDisadvantages
It will be full of reader monads in everywhereIt is difficult to manage a large number ofdependenciesIt is sensitive to change such as IO monads in HaskellWe conclude that reader monads should not be usedfor DI
Readerモナドを使う方法の利点と欠点
33
Which is good for large-scale systemdevelopment?
大規模システム開発ならどの方法がいいのか？
34
Rethink of Cake PatternCake Pattern is a proposed component technique in"Scalable Component Abstractions" by Odersky's et al.But, there are some problems as a component technique
By using an abstract type, types are likely to becometoo complicatedCakes are likely to be tightly coupled to each other
We will consider Cake Pattern not as a componenttechnique
コンポーネント技術としてのCake Patternの問題点
35
Mimimal Cake PatternDo not use advanced features such as abstract typesProvide only one field variableDo not nest classDefine the Cake Pattern with such features as "MimimalCake Pattern"
Mimimal Cake Patternとはコンポーネント技術ではないCake Patternである
36
Example of not use DI
まずはDIを使わない例
// Interface trait Repository
// Implementation object RepositoryImpl extends Repository
// Client object Service { // ↓Here's the problem referring to the Implementation val repository: Repository = RepositoryImpl }
37
Example of Mimimal Cake Pattern (Interface)
Add Uses trait to declare interfaceUses trait just provides one field variableDo not use abstract types and nested classes
Interfaceの例Usesトレイトを追加する
trait UsesRepository { val repository: Repository }
trait Repository
38
Example of Mimimal Cake Pattern(Implementation)
Add MixIn trait to provide the implementation
実装の例MixInトレイトを追加する
trait MixInRepository extends UsesRepository { val repository = repositoryImpl }
object RepositoryImpl extends Repository
39
Example of Mimimal Cake Pattern(Client)
Extend UsesRepository for using RepositoryUse MixInRepository in MixInService for providingService implementationWe can replace the implementation here
UsesとMixInを使ったServiceの例ServiceもMixInトレイトを提供する
trait Service extends UsesRepository
trait MixInService extends UsesService { val service = new Service with MixInRepository }
40
Development style using Minimal CakePattern
Our development style is based on Minimal Cake PatternInterface, Uses trait, implementation, MixIn traits andtest form a development unitWe should write a comment for all methods in theinterface, including all error valuesTests should include all error cases in the interfaceOnly when there are all present, the code is ready to bereviewed
Mimimal Cake Patternによる開発スタイル
41
Advantage of Minimal Cake PatternThe account system consists of 400 or more modules
We need to simplify cake pattern by discarding extrafeatures not related to DI
Static checking is desirable in large-scale systemIf we use DI container, it should be verified by startingthe system each time
DI has established our development styleWe want to re-evaluate Cake Pattern by limiting only to DI
Mimimal Cake Patternの利点の説明DIだけのCake Patternを再評価したい
42
New transaction monad using subtyping
サブタイピングを使った新たなトランザクションモナド
43
Technical problems:Abstraction of storage access
We got the power to abstract components by DIWe want to do even abstraction of storage access, such asMySQL and RedisHow to abstract transaction objects of the infrastructurelayer, such as DB sessionHow to compose transactionsWe want to facilitate a programming of Master/Slavestructure
技術的な課題: ストレージアクセスの抽象化
44
We will introduce a new transaction monadto solve these problems
これらの問題を解決する新しいトランザクションモナドを紹介します
45
Transaction monad: FujitaskFujitask enables abstraction of storage access andcomposition of transactionsFujitask treats transaction objects in the same way as areader monadFujitask determines whether the composed monad willquery Master or Slave by using subtyping
トランザクションモナド: Fujitask
46
Common problems of the Master / Slavestructure
In programming for Master/Slave structure, often weshould explicitly specify whether each code will queryMaster or SlaveA run-time errors when send a query of update to SlaveIt is hard to notice that we send a query that should besent to Slave to Master because no error occursFujitask will solve the these problems
Master/Slave構成のよくある問題Fujitaskはこのような問題を解決します
47
Basic parts of Fujitask
Fujitaskの基本部分
trait Task[-R, +A] { lhs => // To be implemented def execute(r: R)(implicit ec: ExecutionContext): Future[A]
def flatMap[S <: R, B](f: A => Task[S, B]): Task[S, B] = ???
def map[B](f: A => B): Task[R, B] = ???
def run[S <: R]()(implicit runner: TaskRunner[S]): Future[A] = ??? }
trait TaskRunner[R] { // To be implemented def run[A](task: Task[R, A]): Future[A] }
48
How to use FujitaskWe'll explain how to use the Fujitask through theimplementation of ScalikeJDBC
ScalikeJDBC版のFujitaskの実装を通じてFujitaskの使い方を説明します
49
Define types of transaction objects
Define kinds of transactionThese are assigned to a left type variable of Task[R, A]The important point is that ReadWriteTransaction inheritsReadTransaction
トランザクションオブジェクトを定義するReadWriteがReadを継承するのがポイント
trait Transaction
trait ReadTransaction extends Transaction
trait ReadWriteTransaction extends ReadTransaction
50
Implement transaction objects ofScalikeJDBC
These classes inherit Transaction and wrap DBSession ofScalikeJDBC
ScalikeJDBCのトランザクションオブジェクトの型を定義する
abstract class ScalikeJDBCTransaction(val session: DBSession)
class ScalikeJDBCReadTransaction(session: DBSession) extends ScalikeJDBCTransaction(session) with ReadTransaction
class ScalikeJDBCReadWriteTransaction(session: DBSession) extends ScalikeJDBCTransaction(session) with ReadWriteTransaction
51
Implement TaskRunner ofScalikeJDBC (1)
Define ask method to get the transaction objectDerived from the ask function of Reader monad
ScalikeJDBC版のTaskRunnerを実装するaskメソッドを定義する
package object scalikejdbc { def ask: Task[Transaction, DBSession] = new Task[Transaction, DBSession] { def execute(transaction: Transaction) (implicit ec: ExecutionContext): Future[DBSession] = Future.successful(transaction.asInstanceOf[ScalikeJDBCTransaction } }
52
Implement TaskRunner ofScalikeJDBC (2)
Define TaskRunner for ReadTransaction
ReadTransaction用のTaskRunnerを定義する
package object scalikejdbc { implicit def readRunner[R >: ReadTransaction] : TaskRunner[R] = new TaskRunner[R] { def run[A](task: Task[R, A]): Future[A] = { val session = DB.readOnlySession() val future = task.execute(new ScalikeJDBCReadTransaction(session)) future.onComplete(_ => session.close()) future } } }
53
Implement TaskRunner ofScalikeJDBC (3)
Define TaskRunner for ReadWriteTransaction
ReadWriteTransaction用のTaskRunnerを定義する
package object scalikejdbc { implicit def readWriteRunner[R >: ReadWriteTransaction]: TaskRunner[ new TaskRunner[R] { def run[A](task: Task[R, A]): Future[A] = { DB.futureLocalTx(session => task.execute(new ScalikeJDBCReadWriteTransaction(session))) } } }
54
Now, we can use Fujitask ofScalikeJDBC
TaskRunner can be implemented for another DB libraryand Redis client in the same wayNext, we create domain code using this Fujitask ofScalikeJDBC
これでScalikeJDBCのFujitaskを使うことができるようになりました
55
Define an interface of Repository
Repositoryのインターフェースを定義する
trait UserRepository {
def create(name: String): Task[ReadWriteTransaction, User]
def read(id: Long): Task[ReadTransaction, Option[User]]
}
56
Implement Repository using ScalikeJDBCFujitask
ScalikeJDBC版Fujitaskを使ったRepositoryの実装
object UserRepositoryImpl extends UserRepository {
def create(name: String): Task[ReadWriteTransaction, User] = ask.map { implicit session => val sql = sql"""insert into users (name) values ($name)""" val id = sql.updateAndReturnGeneratedKey.apply() User(id, name) }
def read(id: Long): Task[ReadTransaction, Option[User]] = ask.map { implicit session => val sql = sql"""select * from users where id = $id""" sql.map(rs => User(rs.long("id"), rs.string("name"))).single.apply() }
}
57
Implement Service using Repository
Fujitaskで作ったRepositoryを使ってみる
trait MessageService extends UsesUserRepository with UsesMessageRepository {
def createByUserId(message: String, userId: Long): Future[Message] = { val task = for { userOpt <- userRepository.read(userId) user = userOpt.getOrElse( throw new IllegalArgumentException("User Not Found")) message <- messageRepository.create(message, user.name) } yield message
task.run() } }
58
How Fujitask worksTransactions are combined by for-expressionuserRepository.read has ReadTransaction andmessageRepository.create has ReadWritetransaction,then the combined monad has ReadWritetransaction bysubtypingAnd, when task.run is invoked, an appropriateTaskRunner is selected by implicit parameter mechanism
Fujitaskはどのように動作するか
59
The advantage of FujitaskStorage access can be abstracted by FujitaskWe could compose transactions by FujitaskMaster/Slave structure Programming is made easier
Fujitaskの利点
60
Sharding by Fujitask
Use ShardedTask having a shard key instead of the Task ifwe want shardingType of shard key is checked statically by ShardedTaskAlso, the value of the shard key is checked at run timewhether the same in flatMap
Fujitaskによるシャーディング
abstract class ShardedTask[Key, -R, +A](val shardKey: Key)
61
Conclusion
総括
62
Let's consider the commonalitiesamong these techniques
最後に今回説明した技術の共通点を考えたい
63
SimpleContinuation monad can be represented by 6 linesMinimal Cake Pattern is the minimized Cake Pattern for DIEven though Fujitask has a variety of storage accessfunctions, it can be written in about 30 lines
今回我々が紹介した技術はどれもシンプルである
64
Use new features introduced in ScalaContinuation monad can be briefly composed by Scalamonads syntax
Also, It can be combined with other monads in monadtransformers by higher kinded types
Minimal Cake Pattern uses a mixin of traitsFujitask uses variances and ad hoc polymorphism byimplicit parameter
どれもScalaで新しく導入された機能を使っている
65
Without frameworksWeb components are implemented by web framework inother languages
We have proposed a method to use continuationmonad
Dependency Injection has been implemented by DIcontainer in Java
We can use cake pattern instead of DI containerWe would have to use AOP If we try to implement thefunctions of the Fujitask in other languages
我々の手法ではフレームワークは必要ではない
66
The Dwango account system is supported bythe advanced features of Scala
Thanks to the advanced features of Scala, our systembecame more simple, more understandable and moreconvenient
ドワンゴアカウントシステムはScalaの高度な言語機能に支えられている
67
So, we want to conclude this session that ...
よって、このセッションをこう締め括りたい
68
We will continue to use the advancedlanguages like Scala
我々はScalaのような進歩した言語を使っていきたい
69
And, we hope that programming languagescontinue to make progress
そして、これからもプログラミング言語は進歩しつづけてほしい
70
Thank you for listening
ご清聴ありがとうございました
