Dexador Rises

Dexador Rises

Lisp Meet Up #31 August 26, 2015

Eitaro Fukamachi Somewrite Co., Ltd.

I’m Eitaro Fukamachi @nitro_idiot fukamachi

My Products

• Clack

• Caveman2

• Woo

• quickdocs.org

• Qlot

Contributing

• Roswell

We Work Remotely

We Work RemotelyWe’re hiring!

Today, let me talk about HTTP client.

HTTP client

Client Server

HTTP client

Client Server

HTTP Request

GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*

HTTP client

Client Server

HTTP Request

HTTP Response

GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*

HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive

HTTP client

Client Server

HTTP Request

HTTP Response

GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*

HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive

ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo

HTTP client libraries

• Drakma

• trivial-http

• Carrier

HTTP client libraries

• Drakma

• De facto. Full-featured.

• trivial-http

• Simple. A few features. No SSL.

• Carrier

• Asynchronous.

HTTP client libraries

• Drakma (usocket)

• De facto. Full-featured.

• trivial-http (usocket)

• Simple. A few features. No SSL.

• Carrier (cl-async/libuv)

• Asynchronous.

HTTP client libraries

• Drakma (usocket)

• De facto. Full-featured.

• trivial-http (usocket)

• Simple. A few features. No SSL.

• Carrier (cl-async/libuv)

• Asynchronous.

75 LIBRARIES

Required by

3 LIBRARIES

None

Drakma

• Ediware

• Since 2006

• Still maintained at GitHub

Mature??? Easy to use??? Fast???

No.

Pitfall 1: Force URL encoding

• Force URL encoding with PURI

Pitfall (1/3) of Drakma

(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” tag))

tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => PURI:URI-PARSE-ERROR

• Force URL encoding with PURI

Pitfall (1/3) of Drakma

(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)))

tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”

• Force URL encoding with PURI

Pitfall (1/3) of Drakma

(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)))

tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”

tag = “AKB48” => OK tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"

• Force URL encoding with PURI

Pitfall (1/3) of Drakma

(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)) :preserve-uri t)

tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => OK

tag = “AKB48” => OK tag = “乃木坂46” => OK

Pitfall 2: Poor language support

• Poor language support with flexi-streams

Pitfall (2/3) of Drakma

(drakma:http-request “http://www.hatena.ne.jp/”)

(drakma:http-request “http://www.google.co.jp/”)

• Poor language support with flexi-streams

Pitfall (2/3) of Drakma

(drakma:http-request “http://www.hatena.ne.jp/”) ;; => body as UTF-8 string

(drakma:http-request “http://www.google.co.jp/”) ;; => body as byte vector

WARNING: Problems determining charset (falling back to binary): :SHIFT_JIS is not known to be a name for an external format.

Pitfall 3: Error handling

• Tend to forget error handling

Pitfall (3/3) of Drakma

(let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)

• Tend to forget error handling

Pitfall (3/3) of Drakma

(let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)

It fails if the HTTP response code is 4xx or 5xx.

• Tend to forget error handling

Pitfall (3/3) of Drakma

(multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …))

Raise an error unless the status is not 200

• Tend to forget error handling

Pitfall (3/3) of Drakma

(multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …))

Raise an error unless the status is not 200

Want to retry???

• Tend to forget error handling

Pitfall (3/3) of Drakma

(block nil (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (go retry)) (return body))))

With Auto-Retrying

• Tend to forget error handling

Pitfall (3/3) of Drakma

(block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (go retry)) (return body)))))

Retry only 5 times

Many pitfalls.

Ridiculous.

We just wanted to send an HTTP request.

Dexador changes it.

• Full-featured. usocket based.

• Use fast-http, QURI, Babel, cl-cookie

Dexador: Another choice

Dexador: APIs

(dex:get “http://lisp.org/“) (dex:post “http://lisp.org/“) (dex:head “http://lisp.org/“) (dex:put “http://lisp.org/“) (dex:delete “http://lisp.org/“)

Dexador: Language support

;; Shift_JIS (dex:get “http://www.google.co.jp/“)

;; EUC-JP (dex:get “https://mixi.jp/“)

Dexador: Error handling

(handler-case (dex:get “http://cliki.net/“) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))

Dexador: Error handling

(handler-case (dex:get “http://cliki.net/“) (dex:http-request-forbidden () ;; for 403 forbidden ) (dex:http-request-service-unavailable () ;; for 503 service unavailable ) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))

Dexador: Error handling

;; Ignore errors and continue (handler-bind ((dex:http-request-failed #'dex:ignore-and-continue)) (dex:get "http://lisp.org"))

Dexador: Auto-Retrying

;; Auto-retry on 503 error (handler-bind ((dex:http-request-service-unavailable #’dex:retry-request)) (dex:get "http://lisp.org"))

Dexador: Auto-Retrying

;; Retry only 5 times (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5))) (dex:get "http://lisp.org"))

Dexador: Auto-Retrying

;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))

Dexador: Auto-Retrying

;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))

(block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (sleep 3) (go retry)) (return body)))))

Dexador

Drakma

You may ask…

You may ask…

…is it fast?

BenchmarkSending GET request 30 times to Local Server

(lower is better)

0

0.009

0.018

0.026

0.035

Drakma Dexadorw/out conneciton-pool

0.024s

BenchmarkSending GET request 30 times to Local Server

(lower is better)

0

0.009

0.018

0.026

0.035

Drakma Dexadorw/out conneciton-pool

0.013s

0.024s

Sending GET request 30 times to Local Server(lower is better)

0

0.009

0.018

0.026

0.035

Drakma Dexadorw/out conneciton-pool

0.013s

0.024s

Benchmark

x1.8 faster!

Sending GET request 30 times to Local Server(lower is better)

0

0.009

0.018

0.026

0.035

Drakma Dexadorw/out conneciton-pool

0.013s

0.024s

Benchmark

x1.8 faster!????

How about requesting over network?

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.505s

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s x1.2 faster

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s 0.0036 sec/req

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s 0.0036 sec/req

Too trivial improvement…

How come?

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s

Network Latency +

Connection establishment

Network Latency +

Connection establishment

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

0.396s

0.505s

Network Latency +

Connection establishment

Network Latency +

Connection establishment

The largest bottleneck

Differences of Servers/ClientsHTTP Server HTTP Client

C

S

C C

C

S

Can’t help Network Latency and bandwidth.

Can skip connection establishment…?

• Reuse connections once established

(Implicit) Connection-pooling

• Reuse connections once established

(Implicit) Connection-pooling

;; Establish a new connection (dex:get “http://lisp.org/index.html“)

;; Reuse the above connection (dex:get “http://lisp.org/index.html“)

• Reuse connections once established

(Implicit) Connection-pooling

;; Establish a new connection (dex:get “http://lisp.org/index.html“)

;; Reuse the above connection (dex:get “http://lisp.org/index.html“)

0.727 sec

0.380 sec

Benchmark (again)

BenchmarkSending GET request 30 times to Local Server

(lower is better)

0

0.006

0.012

0.018

0.024

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.013s

0.024s

BenchmarkSending GET request 30 times to Local Server

(lower is better)

0

0.006

0.012

0.018

0.024

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.005s

0.013s

0.024s

BenchmarkSending GET request 30 times to Local Server

(lower is better)

0

0.006

0.012

0.018

0.024

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.005s

0.013s

0.024s

x4.8 faster!

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.396s

0.505s

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.219s

0.396s

0.505s

BenchmarkSending GET request 30 times to Remote Server

(lower is better)

0

0.15

0.3

0.45

0.6

Drakma Dexadorw/out conneciton-pool

Dexadorw/ connection-pool

0.219s

0.396s

0.505s Still x2.3 faster!

• It’s pretty common to request to the same host multiple times

• Can expect some performance improvements in real applications

In real applications

Status

• Still BETA (v0.9.7)

• Stabilizing. More performance improvements are secondary importance.

• Bug reports are welcome

• Tested with SBCL, CCL, ABCL and ECL on Travis CI

Status

Try the new player and send me a feedback.

Thanks.

EITARO FUKAMACHI 8arrow.org @nitro_idiot fukamachi

See Also

• Dexador: github.com/fukamachi/dexador