Halloway EvidentCodeAtScale

7/31/2019 Halloway EvidentCodeAtScale

1/98

evident

late 14c., from O.Fr. evident and directly from L.evidentem (nom. evidens) "perceptible, clear,

obvious, apparent" from ex- "fully, out of" (see ex-)

+ videntem (nom. videns), prp. of videre "to

see" (see vision).

http://www.etymonline.com/index.php?term=evident

1
http://www.etymonline.com/index.php?term=evidenthttp://www.etymonline.com/index.php?term=evidenthttp://www.etymonline.com/index.php?term=vision&allowed_in_frame=0http://www.etymonline.com/index.php?term=vision&allowed_in_frame=0http://www.etymonline.com/index.php?term=ex-&allowed_in_frame=0http://www.etymonline.com/index.php?term=ex-&allowed_in_frame=0http://www.etymonline.com/index.php?term=evident&allowed_in_frame=0http://www.etymonline.com/index.php?term=evident&allowed_in_frame=0


2/98


3/98

3


4/98

declarativeprogramming

4


5/98

In computer science, declarative programming is aprogramming paradigm that expresses the logic of a

computation without describing its control flow.[1] Many

languages applying this style attempt to minimize or

eliminate side effects by describing whatthe program should

accomplish, rather than describing how to go aboutaccomplishing it.

http://en.wikipedia.org/wiki/Declarative_programming

5
http://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Side_effect_%28computer_science%29http://en.wikipedia.org/wiki/Side_effect_%28computer_science%29http://en.wikipedia.org/wiki/Declarative_programming#cite_note-0http://en.wikipedia.org/wiki/Declarative_programming#cite_note-0http://en.wikipedia.org/wiki/Control_flowhttp://en.wikipedia.org/wiki/Control_flowhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Programming_paradigmhttp://en.wikipedia.org/wiki/Programming_paradigmhttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Computer_science


6/98

rock/paper/scissors/

lizard/Spock

http://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spock

6
http://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spockhttp://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spock


7/98

(defdominates{:paper:rock:rock:scissors

:scissors:paper})

declarative

7


8/98

(defdominates{:paper:rock:rock:scissors

:scissors:paper})

declarative, functional

(defnwinner [play-1 play-2] (cond (= play-1 play-2) nil (= (dominates play-1) play-2) play-1 :else play-2))

8


9/98

so everything should bedeclarative, right?

9


10/98

some reasons not tobe declarative

functional requirement

non-functional requirement

constraint

10


11/98

abstraction

11


12/98

ideal numberof methods

0

12


13/98

publicinterfaceMap { intsize(); booleanisEmpty(); booleancontainsKey(java.lang.Objecto); booleancontainsValue(java.lang.Objecto);

Vget(java.lang.Objecto); Vput(Kk, Vv); Vremove(java.lang.Objecto); voidputAll(java.util.Map


14/98

publicclassListTablesRequest { public ListTablesRequest(); publicjava.lang.StringgetExclusiveStartTableName(); publicvoidsetExclusiveStartTableName

(java.lang.StringexclusiveStartTableName); publicListTablesRequestwithExclusiveStartTableName (StringexclusiveStartTableName); publicjava.lang.IntegergetLimit(); publicvoidsetLimit(java.lang.Integerlimit); publicListTablesRequestwithLimit(Integerlimit); publicjava.lang.StringtoString();

publicinthashCode(); publicbooleanequals(Objectobj);}

concretion

14


15/98

Abstractions may be formed by reducingthe information content of a concept or

an observable phenomenon...

http://en.wikipedia.org/wiki/Abstraction

15
http://en.wikipedia.org/wiki/Abstractionhttp://en.wikipedia.org/wiki/Abstractionhttp://en.wikipedia.org/wiki/Phenomenonhttp://en.wikipedia.org/wiki/Phenomenonhttp://en.wikipedia.org/wiki/Concepthttp://en.wikipedia.org/wiki/Concepthttp://en.wikipedia.org/wiki/Informationhttp://en.wikipedia.org/wiki/Information


16/98

v1

F

v2

F

v3

F

v4

epochal time model

16


17/98

v1

F

v2

F

v3

F

v4

abstraction here reduces

the information I must provide

abstraction here reduces

the information I can get17


18/98

abstractions in Datomic

about 50 protocols / interfaces

(inter)action

between services

between components

perception some new (generic!) data structures

no concretions

18


19/98

publicinterfaceEntity {

Objectget(Objectkey);Set keySet();

}

perception example

19


20/98


21/98

number of db

update methods inDatomic

2

21


22/98

abstraction guidelines

fine grained

separate perception and action

perception is generic

22


23/98

values

23


24/98


25/98

[?customer :email ?email]

Constrains the results returned,binds variables

data pattern

25


26/98



data pattern

entity attribute value

26


27/98



data pattern

constant

27


28/98



data pattern

variablevariable

28


29/98

[42:email ?email]

Find a particular customers email

constants anywhere

29


30/98

[42 ?attribute]

What other attributes does

customer 42 have?

variables anywhere

30


31/98


32/98

[:find?customer:where [?customer :email]]

find clause

variable to

return

32


33/98

[:find ?customer:where [?customer:email]

[?customer:orders]]

Find all the customers who

have placed orders.

implicit join

33


34/98

declarative < evident

declarative evident

tables, documents datoms

ORM maps, entities

ambient db explicit db

34


35/98

(use '[datomic.api :only (q) :as d])

(q '[:find ?customer :where [?customer :id]

[?customer :orders]]

db)

API

35


36/98

(use '[datomic.api :only (q) :as d])

(q '[:find ?customer :where [?customer :id]

[?customer :orders]]

db)

q

36


37/98


38/98

(q '[:find ?customer

:where [?customer :id][?customer :orders]]

db)

input(s)

38


39/98

:in $database ?email

in clause

Names inputs so you can refer to themelsewhere in the query

39


40/98

Find a customer by email.

parameterized query



:where [$database ?customer :email ?email]]

db

"[email protected]")

40
mailto:[email protected]:[email protected]


41/98


first input


:in$database ?email


db


41


42/98


second input



:where [$database ?customer :email?email]]

db


42


43/98


verbose?


:in$database ?email


db


43


44/98


shortest name possible


:in$ ?email

:where [$ ?customer :email ?email]]

db


44


45/98


elide $ in where


:in$ ?email

:where [ ?customer :email ?email]]

db


no need to

specify $

45
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


46/98

[(< 50 ?price)]

predicates

Functional constraints that canappear in a :where clause

46


47/98

[:find ?item:where [?item:item/price ?price]

[(< 50 ?price)]]

Find the expensive items

adding a predicate

47


48/98

[(shipping ?zip ?weight) ?cost]

Take bound variables as inputsand bind variables with output

functions

48


49/98

[(shipping ?zip ?weight) ?cost]

function args

function arguments

49


50/98

[(shipping ?zip ?weight)?cost]

function returns

bind return

values

50


51/98

[:find ?customer ?product:where [?customer :shipAddress ?addr]

[?addr :zip ?zip][?product :product/weight ?weight]

[?product :product/price ?price][(Shipping/estimate ?zip ?weight) ?shipCost][(


52/98

publicclassShipping { publicstaticBigDecimalestimate(Stringzip1, intpounds);

}

Functions can be plainJVM code.

arbitrary functions

52


53/98

(q '[:find ?op ?e ?a ?v

:in $ ?renamings :where[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)

[[?op ?e ?a ?v]]]](db customer-conn)renamings)

find people with interests

53


54/98

(q '[:find ?op ?e ?a ?v



as entities

54


55/98

(q '[:find?op ?e ?a ?v



and make ... datoms?

55


56/98

(->>(q '[:find ?op ?e ?a ?v :in $ ?renamings :where

[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)[[?op ?e ?a ?v]]]]

(db customer-conn)renamings)

seq (d/transact hobbies-conn))

ETL job

56


57/98

->>

the ETL macro

57


58/98

schema is data

58


59/98

{:db/id #db/id[:db.part/db]

:db/ident:part/name:db/valueType:db.type/string:db/cardinality:db.cardinality/one:db/fulltext true:db/doc"The name of the part"

:db.install/_attribute:db.part/db}

attribute schema

59


60/98

(create-eav :part/business-key :db.type/string :db.cardinality/one "The name of the part" :db.part/db)

why not this?

60


61/98

value propositions

powerful extant API

make from any language

read without evaluation

names, not positions

extend easily

build from programs

61


62/98

navigation is data

62


63/98

publicinterfaceEntity { Objectget(Objectkey);

Set keySet();// evil enhancementObject reverseGet(Object key);

}

Entity revisited

63


64/98

(get-in jack [:town])

Jacks town

64


65/98

(get-in jack [:town :_town])

who else is in Jacks town?

65


66/98

number of methods inDatomics graph

navigation API

066


67/98

evident code is not enough

specifications

doc strings

drawings (graffle)

outlines (org, Confluence)

tests (???)

67


68/98

tests are executabledocumentation (?)

68


69/98

what is a test?

create input

driver

capture output

validate

communicate

record

69


70/98

(are [x y] (= x y) (+) 0 (+ 1) 1 (+ 1 2) 3 (+ 1 2 3) 6

(+ -1) -1 (+ -1 -2) -3 (+ -1 +2 -3) -2

(+ 1 -1) 0 (+ -1 1) 0

(+ 2/3) 2/3 (+ 2/3 1) 5/3 (+ 2/3 1/3) 1 )

test, complected

validation

hand-curated

inputs

driver

outputs

70


71/98

generative testing

71


72/98

http://en.wikipedia.org/wiki/Mastermind_(board_game)

72
http://en.wikipedia.org/wiki/Mastermind_(board_gamehttp://en.wikipedia.org/wiki/Mastermind_(board_game


73/98

(defnrandom-secret[]

(gen/vec #(gen/one-of :r:g:b:y) 4))

input generation

73


74/98

(defnscore[c1 c2]

(let [exact (exact-matches c1 c2) ums (unordered-matches c1 c2)unordered (apply + (vals ums))]

{:exact exact:unordered(- unordered exact)}))

fn under test

74


75/98

(defnscoring-is-symmetric[secret guess score]

(= score (game/score guess secret)))

(defnscoring-is-bounded-by-number-of-pegs[secret guess score]

(< 0 (matches score)(count secret)))

validation

75


76/98

(defspec score-invariants

game/score[^{:tag `random-secret} secret^{:tag `random-secret} guess]

(assert (scoring-is-symmetric secret guess %)) (assert (scoring-is-bounded-by-number-of-pegs

secret guess %)))

driver

inputs

validations

fn under test

76


77/98

========================================================:secret | :guess | :score========================================================

(:r :r :r :r) | (:r :r :r :r) | {:exact 4, :unordered 0}(:r :r :r :r) | (:r :r :r :g) | {:exact 3, :unordered 0}(:r :r :r :r) | (:r :r :r :b) | {:exact 3, :unordered 0}(:r :r :r :r) | (:r :r :r :y) | {:exact 3, :unordered 0}

...

...or just enumerate all cases

77


78/98

test inputs and outputsare data

78


79/98

:dying "... .O.

..."

:off "O.O...O.O"

:on "|||

O.O|||"

visual inputs

http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5

79
http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5


80/98

==============================================================================:input | :bind-form | :bind-type | :count | ?a |==============================================================================42 | ?a | scalar | 1 | 42 |

[2 3 5 7] | ?a | scalar | 1 | [2 3 5 7] |42 | [?a ?b] | tuple | | |[2 3 5 7] | [?a ?b] | tuple | 1 | 2 |42 | [?a ...] | collection | | |[2 3 5 7] | [?a ...] | collection | 4 | 3 / 2 / 7 / 5 |42 | [[?a ?b ?c]] | relation | | |[2 3 5 7] | [[?a ?b ?c]] | relation | | |==============================================================================

tests write docs

80


81/98

once you add time,stateful interactions are

81


82/98

once you add time,

stateful interactions are

data!

82


84/98

---------------------------------------------------------------Legend: incrementing value on reconnect, or [T]ime[O]utreconnects: [A]ttempt [S]ucceed [F]ail [B]ackoff [Interrupt]

[T]throw [Z]sleep [C]leanup c[L]eanupFailed

---------------------------------------------------------------

0 A L Z B A I S1 C A S

interleaving

84


85/98

...........................................

...........................................

..............

{:retries 9,:blocks-planned 100,:blocks-completed #,:storage-fail-rate 0.05}

simulating failure

85


86/98

...........................................

...........................................

..............

{:retries 9,:blocks-planned 100,:blocks-completed #,:storage-fail-rate 0.05}

simulating failure

possible sources:

timing

logs

metrics86


87/98

tests are programs

87


88/98

(defnrun-with-transcript[forms]

(binding [*ns* *ns*] (let [temp (gensym)]

(in-ns temp) (clojure.core/use 'clojure.core) (doseq [f forms] (pprint f) (print "=> ") (pprint (eval f))

(println)) (remove-ns temp) :done)))

from REPL to regression

88


89/98

the sim

89

Setup


90/98

Execution

p

Sim DB

SimData

sim-schema.dtm

orders-schema.dtm

SImGenerator

OrdersGenerator

Runner

Feeder

Time

Logs

Metrics

Validation

Validator

Orders DB

OrdersData

90

Setup


91/98

Execution

p

Sim DB

SimData

sim-schema.dtm

orders-schema.dtm

SImGenerator

OrdersGenerator

Runner

Feeder

Time

Logs

Metrics

Validation

Validator

Orders DB

OrdersData

91

Setup


92/98

Execution

p

Sim DB

SimData

sim-schema.dtm

orders-schema.dtm

SImGenerator

OrdersGenerator

Runner

Feeder

Time

Logs

Metrics

Validation

Validator

Orders DB

OrdersData

92

Setup


93/98

Execution

p

Sim DB

SimData

sim-schema.dtm

orders-schema.dtm

SImGenerator

OrdersGenerator

Runner

Feeder

Time

Logs

Metrics

Validation

Validator

Orders DB

OrdersData

93

l


94/98

simulation

not a precious curated path

uses real input and output points

models time

separates testing activities in time and space

validates via logic programs

some effort to set up

94


95/98

how Clojureenabled Datomic

95

Cl j f


96/98

Clojure features

data

functions

references

protocols

deftype

96

Cl j hil h


97/98

Clojure philosophy

architecture

simplicity

power

focus

97


98/98

Documents

Halloway EvidentCodeAtScale