View
1.004
Download
3
Category
Preview:
DESCRIPTION
Talk given at EuroPython 2013
Citation preview
BATOUmulti-(component|host|environment|.*)
deployment
Wednesday, 3.July 13
@theuni
Wednesday, 3.July 13
Wednesday, 3.July 13
Wednesday, 3.July 13
Wednesday, 3.July 13
AUTOMATING DEPLOYMENTS IS HARD
Wednesday, 3.July 13
HOW DOES
CONVERGENCE HELP?
Wednesday, 3.July 13
HOW DOES THIS WORK WITH BATOU?
Wednesday, 3.July 13
SOME PERSPECTIVE
Wednesday, 3.July 13
Wednesday, 3.July 13
IT'S NOT THAT BAD.
Wednesday, 3.July 13
Wednesday, 3.July 13
service deployment
Fabric, Capistrano, ...
system configuration Puppet, Chef, ...
provisioning kickstart, Razor, imaging ...
Wednesday, 3.July 13
FTP
bashmkzopeinstance
zc.buildoutfabric
Wednesday, 3.July 13
CONVERGENCE
Wednesday, 3.July 13
"Everything that follows is a result of what you see here."(Dr. Alfred Lanning; I, Robot)
Wednesday, 3.July 13
SIMPLEos.mkdir('foo')with open('foo/bar', 'w') as myfile: myfile.write('asdf')os.chmod('foo/bar', 0755)
Wednesday, 3.July 13
•unexpected system state
•can't resume
•unnecessary updates
os.mkdir('foo')with open('foo/bar', 'w') as myfile: myfile.write('asdf')os.chmod('foo/bar', 0755)
SIMPLISTIC
Wednesday, 3.July 13
CORRECT(?)if not os.path.isdir('foo'): os.unlink('foo')if not os.path.exists('foo'): os.mkdir('foo')try: os.lstat('foo/bar')except OSError: passelse: if os.path.isdir('foo/bar'): shutil.rmtree('foo/bar') else: os.unlink('foo/bar')if (os.path.exists('foo/bar') and open('foo/bar', 'r').read() != 'asdf'): open('foo/bar', 'w').write('asdf'):current = os.stat('foo/bar').st_modeif stat.S_IMODE(current) != 0755: os.chmod('foo', 0755)
Wednesday, 3.July 13
SIMPLE
File('foo/bar', content='asdf', mode=0755, leading=True)
Wednesday, 3.July 13
class File(Component):
namevar = 'path'
def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)
File('foo/bar', content='asdf', mode=0755, leading=True)
Wednesday, 3.July 13
class File(Component):
namevar = 'path'
def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)
File('foo/bar', content='asdf', mode=0755, leading=True)
compute target state (no touching!)
Wednesday, 3.July 13
class File(Component):
namevar = 'path'
def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)
File('foo/bar', content='asdf', mode=0755, leading=True)
compute target state (no touching!)
composition operator
Wednesday, 3.July 13
class File(Component):
namevar = 'path'
def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)
File('foo/bar', content='asdf', mode=0755, leading=True)
compute target state (no touching!)
composition operator
order matters
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
run "anywhere"
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
run on target
run "anywhere"
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
run on target
run "anywhere"
after all sub-components
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
run on target
only if needed
run "anywhere"
after all sub-components
Wednesday, 3.July 13
class Presence(Component):
namevar = 'path' leading = False
def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)
def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass
run on target
only if needed
run "anywhere"
after all sub-components
keep delegating!
Wednesday, 3.July 13
class Directory(Component):
namevar = 'path' leading = False
def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)
Wednesday, 3.July 13
class Directory(Component):
namevar = 'path' leading = False
def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)
could be done with recursive composition
Wednesday, 3.July 13
class Directory(Component):
namevar = 'path' leading = False
def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)
could be done with recursive composition
refactor with sub-components if too
complex
Wednesday, 3.July 13
class Directory(Component):
namevar = 'path' leading = False
def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)
all methods optional: no configure()
could be done with recursive composition
refactor with sub-components if too
complex
Wednesday, 3.July 13
class Directory(Component):
namevar = 'path' leading = False
def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()
def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)
all methods optional: no configure()
could be done with recursive composition
pattern: just wipe out what's wrong
refactor with sub-components if too
complex
Wednesday, 3.July 13
CONVERGENCE
resume where needed
handle many system states transparently avoid
unnecessary updates
Wednesday, 3.July 13
COMPONENTS
composition of simple components
no magic bullet, just a lot easier to factor
your code
configure - verify - update
Wednesday, 3.July 13
Wednesday, 3.July 13
SINGLE-COMMAND
Wednesday, 3.July 13
REPEATABLERELIABLE
Wednesday, 3.July 13
SIMPLE
Wednesday, 3.July 13
ENTROPY
Wednesday, 3.July 13
EXPRESSIVENESSREADABILITY
Wednesday, 3.July 13
REUSABLE
Wednesday, 3.July 13
PLATFORM INDEPENDENCE
Wednesday, 3.July 13
DOMAIN AGNOSTIC
Wednesday, 3.July 13
NO ADDITIONAL RUNTIME
DEPENDENCIES
Wednesday, 3.July 13
CONTINUITY
Wednesday, 3.July 13
MINIMAL DOWNTIMES
Wednesday, 3.July 13
Wednesday, 3.July 13
PRACTICAL USAGE
Wednesday, 3.July 13
REQUIREMENTS
Python 2.7
SSH
virtualenvMercurial
Wednesday, 3.July 13
ENVIRONMENTS
[environment]service_user = myservicehost_domain = flyingcircus.iobranch = production
[hosts]multikarl00 = nginx, haproxymultikarl01 = postgres, redis, memcached, crontabmultikarl12 = supervisor, logrotate, doctotext, myappmultikarl13 = supervisor, logrotate, doctotext, myapp
Wednesday, 3.July 13
LOCAL
$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)$ bin/batou-local dev localhost$
Wednesday, 3.July 13
REMOTE$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OKDeploying test02.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OK
Wednesday, 3.July 13
OVERRIDES
class Hello(Component):
hostname = "foo"
[environment]...
[component:hello]hostname = bar
Wednesday, 3.July 13
SECRETS
class Hello(Component):
db_password = none
secrets/production.cfg
[hello]db_password = reallysecretstuff
Wednesday, 3.July 13
SECRETS
class Hello(Component):
db_password = none
secrets/production.cfg
[hello]db_password = reallysecretstuff
SciFibut close
Wednesday, 3.July 13
PROVIDE/REQUIREclass MyApp(Component):
def configure(self): self.provide('appserver', self.host.fqdn)
class HAProxy(Component):
def configure(self): self.backends = \ self.require('appserver')
Wednesday, 3.July 13
PLATFORMSclass HAProxy(Component): ...
@platform('flyingcircus.io', HAProxy)class SystemWideHAProxy(Component):
def configure(self): self += File('/etc/haproxy', ensure='symlink', link_to=self.parent.haproxy_cfg.path)
Wednesday, 3.July 13
VFS MAPPING
./
...
./work
./work/_/etc/haproxy.cfg
class HAProxy(Component):
def configure(self): self += File('/etc/haproxy')
[environment]...[vfs]sandbox = Developer
Wednesday, 3.July 13
FEATURESclass MyApp(Component):
features = ['instance', 'jobrunner']
def configure(self): if 'instance' in self.features: ...
[hosts]hosta = myapp:instancehostb = myapp:jobrunnerhostc = myapp:instance, myapp:jobrunnerhostd = myapp
Wednesday, 3.July 13
Wednesday, 3.July 13
CONVERGENCE
COMPOSITION
DETAILSWednesday, 3.July 13
QUESTIONS?
Wednesday, 3.July 13
batou.readthedocs.org
pypi.python.org/pypi/batou
bitbucket.org/gocept/batou
Wednesday, 3.July 13
Recommended