Embed Size (px)
Citation preview
LISP DEPLOYMENTKyle KerslakeRavenPack
Our Problems
■ Large applications■ Lots of required data■Subtle variations of these applications■ Long startup times (data loading)
Build + Run from Source
■Syncing repositories is easy■Git pull and reload to hotpatch (easy)
■Full process takes over 20 mins
Dump an image!
■Startup down to 2 minutes■One file to rule them all
■File is 7gb large (hard to move)■Requires deployment targets to be set up■Care needed for stored pathnames
Lisp images in Docker images
■ Infrastructure can handle big images■Deployment is almost as simple as source■Base Lisp container means no Lisp setup
■Hotpatching is more awkward■Additional tool to learn
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM;
JB+?�2H _�bFBM- `�bFBM!K++K2X`m
.2TiX Q7 *a- hl JmMB+?
�T`BH k- kyRN
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN R f 8
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
AM i?2 T`2pBQmb BMbi�HHK2Mib
*QMi`QHHBM; 2p2`vi?BM; rBi? GBbT Bb ;`2�i
GBbT K�+?BM2b r2`2 iQQHb Q7 b�M2` iBK2b
q2# �M/ ?�`/r�`2 �`2 ?Q``B#H2S`2pBQmb v2�`b, ?Qr iQ +QMi`QH ?�`/r�`2 /`Bp2`b
h?Bb iBK2, r2# BM T`�+iB+2
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN k f 8
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
h?Qm;?i7mH i?2`B/BQM
qQ`Fb QM r2#U�b 2p2`v i?2`B/BQM ě +Q#r2# bTB/2` ě Bb 2tT2+i2/ iQV
q`�Tb iQQHb HBF2 *G@>hJG8@S�`b2` �M/ .`�FK� �M/ *aa@a2H2+iQ`b
h?BMFb #27Q`2 �+iBM;,_2/2}M�#H2 TQHB+B2b 7Q` 2p2`vi?BM; pB� *GPa
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN j f 8
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
ai�i2 Q7 r2#
h?2v iQQF � r2# Q7 /Q+mK2Mib �M/ FBHH2/ Bi5q2# Q7 �TTHB+�iBQMb ě #mi bM�Tb?Qib biBHH rQ`Fa�p2 2p2`vi?BM; #27Q`2 `2�/BM;5
C�p�b+`BTi �M/ bivH2b �`2 MQi `2�HHv M22/2/ KQbi Q7 i?2 iBK2.2bB;M2`b �bF 7Q` b2K�MiB+ K�`FmT\6�+2#QQF `2[mB`2b K2i�/�i� 7Q` ;QQ/ T`2pB2r\G2iǶb mb2 i?2 `2bmHib
aiQT HQQFBM; 7Q` mT/�i2b #v 2v2b ě T`Q+2bb �`iB+H2 ?BbiQ`v �miQK�iB+�HHvmbBM; QMHv ?mK�M@Q`B2Mi2/ T�;2bU�M/ /QrMHQ�/ �`+?Bp2b B7 /2bB`2/V
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN 9 f 8
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
*`�rHBM; .aG
UrBi?@T�;2 Ut ]?iiTb,ffTH�M2iXHBbTXQ`;f]V]/BpO+QMi2Mi = T = �]U]?`27] tVUrBi?@T�;2 Ut t ,`2+m`b2 `2+m`b2V
UH2i T`2p ]i`,7B`bi@+?BH/ = i/ = �]U]?`27] tVV
UT�;2@r�HF2`@#`�M+?2`@T`Q;MUrBi?@T�;2 Ut t ,72i+? MBH ,F22T@#`�M+?2` iV
]HB = �]U]?`27] tVUB7 U+H@TT+`2,b+�M ]fkyR(y@3)f] tV
T�;2@r�HF2`@i2`KBM�iQ` tVVU`2+m`b2 T`2pVVVV
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN 8 f 8
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
X
*`�rHBM; .aG
UrBi?@T�;2 Ut ]?iiTb,ffTH�M2iXHBbTXQ`;f]V]/BpO+QMi2Mi = T = �]U]?`27] tVUrBi?@T�;2 Ut t ,`2+m`b2 `2+m`b2V
UH2i T`2p ]i`,7B`bi@+?BH/ = i/ = �]U]?`27] tVV
UT�;2@r�HF2`@#`�M+?2`@T`Q;MUrBi?@T�;2 Ut t ,72i+? MBH ,F22T@#`�M+?2` iV
]HB = �]U]?`27] tVUB7 U+H@TT+`2,b+�M ]fkyR(y@3)f] tV
T�;2@r�HF2`@i2`KBM�iQ` tVVU`2+m`b2 T`2pVVVV
?iiTb,ff;BiH�#X+QKKQM@HBbTXM2ifK`�bFBMfi?Qm;?i7mH@i?2`B/BQM
JX _�bFBM UhlJV GBbT BM i?2 KB//H2 Q7 r2# #`QrbBM; �T`BH k- kyRN 8 f 8
What’s new with Clasp & Cando?
Martin Cracauer <[email protected]>European Lisp Symposium 2019
Genoa, Italy
Clasp/Cando support organizationChristian Schafmeister and Martin Cracauer founded a company:
- ThirdLaw Technologies www.thirdlaw.tech- With Temple University and DOE- Paid support for open source Clasp and Cando- Commercial modules (e.g. Free Energy Perturbation)- Distributed computing framework to execute Cando output directories for
molecular dynamics package (e.g. Amber) on distributed computing fleets- Emphasis on security- Research to optimize cost and/or latency for those computing jobs
New youtube demos!
www.thirdlaw.tech
Clasp infrastructure and accessibility progress- Progress in Clasp startup time- Lots of general Lisp correctness patches (thanks Karsten Poeck)- Progress in compile time (Clasp 30 min, Cando+quicklisp 40 min)- Progress in debug info- Buildbot ensures branches stay buildable. -ish- Officially supported: Linux w/LLVM6 (e.g. Debian Stretch w/backports, Ubuntu
18.04), OSX/brew, OSX/macports, FreeBSD port- Contiguous update of Docker image: `docker run … cracauer/cando`
Linux users can also use the Docker image to extract the binary /opt/clasp or use it as a chroot instead of Docker.
Clasp, Cando and cl-Jupyter- Jupyter notebooks are the official GUI for Cando- Our hacked up cl-jupyter is on github/clasp-developers- Our buildbot builds the whole stack of Clasp, Cando, cl-jupyter and all
quicklisp modules required by that, and puts it into Docker- Can be used by anybody who just wants to use cl-jupyter, not specific to
Cando- Anyone wants to backport it to general Cl/quicklisp? Right now a bit
Clasp-specific
Docker image can be used to unpack /opt/clasp, which has it all.
Open source used in Clasp/CandoClasp doesn’t go on crusades to reinvent wheels. Partial list:
- SICL- LLVM (all code generation goes through LLVM optimization)- Boehm GC and MPS (open-source-ish)- CL-jupyter and various graphics packages for web from quicklisp- Alexandria- Closer-mop- Concrete syntax tree
Distributed computing for molecular dynamics- Not specific to molecular dynamic, can run anything that is data-in-disk-files
oriented- Focus on security (single user keys, proper keychain, proper isolation of
individuals, no network based security part of scheme)- Optimizations will be in Lisp- Many subjobs are Lisp jobs (most are GPU computing)
Distributed jobdir execution- Hundreds of subjobs- Subjobs can depend on previous jobs output- Incremental jobdirs, update while it is executing- No fixed machine assigning, new jobs picked up from graph as available
Heterogeneous fleet:
- Fast CPU (dumb jobs), many cores (multithreaded jobs), different GPUs- Jobdir has ~150 GPU subjobs in 4-8 “classes”- Different classes have different relative speed on different GPU types- Matching between subjob class and GPU type known-ish
Dependency graph above turns into job execution like this.
20 AWS ec2 GPU instances, Chris’ daughters gaming computer, GPU and CPU from Martin’s datacenter
www.cons.org/graph.dot.pdf
Lisp advocacyDrive home the message “we don’t just need readable software, we need changeable software”.
That requires compile-time computing. Keep every assumption you make during implementation in a single place. Generate all the other places where that assumption is needed from that one location, with compile-time code. You cannot hunt down human-driven spread of that assumption.
https://medium.com/@MartinCracauer
#BT @ #mbBM2bb BM T`Q;`2bb
1X EHQ+FK�MM>X S�mHb2M
hX SQ?H.X CmMF2`
ahE *QMbmHiBM; � .2p2HQTK2Mi :K#>
ykX �T` kyRN
#BT
1Mi2`T`Bb2 _2bQm`+2 SH�MMBM; U1_SV 7Q`6BM�M+2 � �++QmMiBM;>mK�M `2bQm`+2bP`/2` S`Q+2bbBM;amTTHv +?�BM K�M�;2K2MiS`QD2+i K�M�;2K2Mi*mbiQK2` `2H�iBQMb?BT K�M�;2K2MiXXX
#BT
#BT F2`M2H Y KQ/mH2b#BT BM/mbi`v#BT HQ;BbiB+b#BT BMbm`�M+2#BT XXX
BM/BpB/m�H K2i?Q/bf�//@QMbf+mbiQKBx�iBQMb
;2M2`B+
."@�++2bb>hhSUaV@a2`p2`1@J�BHBM;*ao BKTQ`if2tTQ`is"_GQ/ifT/7 T`BMi}H2bXXX
T`QD2+i /2p2HQTK2Mi
Qp2`pB2r
+H�bb2b
+H�bb2b
r?v *QKKQM GAaS\
J�+`Qb+H�bb /2b+`BTiBQMK2i?Q/ /2}MBiBQMT`BMi b2iiBM;b
2``Q` ?�M/HBM;r`BiBM; HQ; }H2bK2bb�;2 iQ #`Qrb2``QHH@#�+F QM 2``Q`
T`Q+2bb +H2�M mT`2bi�`i i�bF r?BH2 /2#m;;BM;
`2+QKTBHBM;XXX
#BT @ #mbBM2bb BM T`Q;`2bb
1X EHQ+FK�MM>X S�mHb2M
hX SQ?H.X CmMF2`
ahE *QMbmHiBM; � .2p2HQTK2Mi :K#>
ykX �T` kyRN
Exploring syntax
Modules and predicates
(Module cont.)
(Module cont.)
Object oriented
(Object oriented cont.)
