Building Hermetic Systems (without Docker)

Building Hermetic Systems (without Docker)

Will Farrell @wkf

Who am I?

I’m the Lead Software Architect at MojoTech in Providence, RI, that’s who.

We’re a software consultancy, and we’ve built many web and mobile apps.

What can you expect to learn?

• How hermeticity is an example of functional design thinking

• How to identify “leaks” in not-so-hermetic systems

• How to leverage Clojure to build and manage hermetic systems

What are hermetic systems?

Airtight

Pure

Why is hermeticity desirable?;; ...

(ns determinism (:require [hermeticity]))

(defn system [well-defined-input] well-defined-output)

;; ...

–Every Developer Ever

“It works on my machine.”

What about Docker?

–Every Developer Ever

“Let’s use Docker.”

–Every Developer Ever, Eventually

“Let’s never use Docker again.”

xkcd

Why are there leaks?

• The JVM

• The operating system

• The electricity

• …you get it

Which leaks can we fix?

External Libraries

What’s wrong with libraries?(defproject motrics "0.1.0-SNAPSHOT" :description "Mojotech Metrics" :url "https://github.com/mojotech/motrics" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :repositories {"elasticsearch-releases" "https://maven.elasticsearch.org/releases"} :dependencies [[org.clojure/clojure "1.7.0"] [com.stuartsierra/component "0.3.1"] [com.taoensso/timbre "4.2.1"] [com.taoensso/carmine "2.12.2"] [com.github.kstyrc/embedded-redis "0.6"] [org.elasticsearch/elasticsearch "2.2.0"] [org.elasticsearch.plugin/shield "2.2.0"] [environ "1.0.2"] [tentacles "0.5.1"] [suspendable "0.1.0"]] :main ^:skip-aot motrics.core :target-path "target/%s" :test-paths ["src/test"] :source-paths ["src/main"] :min-lein-version "2.0.0" :profiles {:uberjar {:aot :all} :dev {:jvm-opts ["-Xmx300m"] :source-paths ["src/dev"] :repl-options {:init-ns motrics.repl} :dependencies [[reloaded.repl "0.2.1"]]}})

Remember these?

External Services

What’s wrong with services? $ ping api.github.com ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host ping: cannot resolve api.github.com: Unknown host

Should we rewrite everything?(No)

What are components?(defrecord ExampleComponent [options cache database scheduler] component/Lifecycle

(start [this] (println ";; Starting ExampleComponent") ;; In the 'start' method, a component may assume that its ;; dependencies are available and have already been started. (assoc this :admin (get-user database "admin")))

(stop [this] (println ";; Stopping ExampleComponent") ;; Likewise, in the 'stop' method, a component may assume that its ;; dependencies will not be stopped until AFTER it is stopped. this))

What are systems?(defn example-system [config-options] (let [{:keys [host port]} config-options] (component/system-map :db (new-database host port) :scheduler (new-scheduler) :app (component/using (example-component config-options) {:database :db :scheduler :scheduler}))))

What about embedding?

Embed Elasticsearch(defrecord EmbeddedElasticsearchServer [config] component/Lifecycle

(component/start [this] (update this :node #(or % (-> (:elasticsearch-server config) (update :settings (partial merge default-node-settings)) (elasticsearch/new-node) (elasticsearch/start-node)))))

(component/stop [this] (some-> this :node elasticsearch/stop-node) (dissoc this :node))

suspendable/Suspendable

(suspendable/suspend [this] this)

(suspendable/resume [this that] (if (= (-> this :config :elasticsearch-server) (-> that :config :elasticsearch-server)) (assoc this :node (:node that)) (do (component/stop that) (component/start this)))))

REPL namespace

(ns my-app.repl (:require [reloaded.repl :as repl :refer [system start stop go reset reset-all]] [com.stuartsierra.component :as component] [my-app.elasticsearch :refer [new-embedded-elasticsearch-server]] [my-app.core :as my-app] [my-app.config :as config]))

Manage services(def config (merge config/config {:elasticsearch-server {:settings {"cluster.name" "elasticsearch" "http.enabled" "true" "http.host" "127.0.0.1" "http.port" "9200" "transport.host" "127.0.0.1" "transport.tcp.port" "9300"}}}))

(repl/set-init! #(-> (component/system-map :worker-service (my-app/new-worker-service config) :embedded-elasticsearch-server (new-embedded-elasticsearch-server config)) (component/system-using {:worker-service [:embedded-elasticsearch-server]})))

Embeddable Services

• datomic

• zookeeper

• h2

• kafka

• elasticsearch

• rabbitmq

• postgresql

• redis

• neo4j

• and more!




Are we done?

Entropy

What’s wrong with entropy?

(str/reverse “entropy”) ;; That was easy!

Where do we get entropy?

xkcd

How do we reproduce random?(defn random-seq "Generate a lazy sequence of random numbers based on an intial seed." [seed n] (let [r (java.util.Random. seed)] (repeatedly #(.nextInt r n))))

(take 5 (random-seq 1 10)) ;;-> (2 5 7 1 8)

(take 5 (random-seq 1 10)) ;;-> (2 5 7 1 8)

(take 5 (random-seq 2 10)) ;;-> (3 6 5 5 9)

Can randomness be a service?

So, can randomness be a service?

Time

What’s wrong with time?

What’s wrong with time?

Can’t we just use Datomic?

How can we model time?(defn application [from-tick to-tick] “Run our important application.” (let [clock (new-clock from-tick to-tick)] ;; ... ))

;; Run from tick 0 to 100 (application 0 100)



But how do we design a system?

–Will Wright (he made SimCity)

“Reticulating splines.”

So now our system is perfect?

What did we learn?

• How to evaluate systems and find leaks

• How to plug leaks when you find them

• How to make good trade-offs when building hermetic systems

Questions?

Thanks!Will Farrell

@wkf