Buildout: creating and deploying repeatable applications in python

Buildout: creating and deploying repeatable applications in python

Mikel Larreategi - @erralin

PySS Donostia - 2014-09-27

Who are we?

We are a team of 15 people working on internet since 2001

We are not all developers:Management: 1

Content authoring, client support and management help: 4

Designers: 2

Developers: 9

What we do?

Websites!
(mostly)

In simple language: we do websites

What we do?

Collaborative news websites

Community sites

Corporate portals and intranets

Multilingual sites / Localization

R&D

But what kind of websites?We build sites for our customers so that they can manage the site through the web, without HTML, FTP or any other prior knowledge.

They manage the sites using a WYSIWYG editor

We also do much R&D, and many of our projects come after much time invested on R&D: we now work with real-time-twitter-stream parsing and content curation for instance.

How we do it?

PYTHON!

And we do everything in python, and we are proud of it.

We love python

Both for our internal scripts and also for all web development we do

OK, I admit we do some WordPress...

Our background

Back in 2001.....

We come from Zope world

In 2001 one of the founders discovered Zope and Python

Zope has been recognized as a python killer app

People say that Zope has used several concepts that later have been widely used.

There are two quotes in one Zope book by Phillip J. Eby (creator of setuptools) that say:Where Zope leads, Python follows

Those who do not studay Zope, are condemned to reinvent it

Zope

Official definition: Free, open-source, object-oriented web application server written in Python.

A way to publish python objects in a web environment.

Originally developed to replace/improve CGI-based development

The story says that Zope was invented in a plane, by Jim Fulton who was going to teach CGI development, and thought that object publishing was a saner way to do web-development

Zope

Introduced object oriented databases (ZODB) in 2001

Through-the-web development (don't tell anyone)

Restricted python (stripped-down python for security reasons)

DTML or ZPT as template language

Zope encouraged web-based-development to develop quick and powerful dynamic sites through the web

Yes python scripts and templates were written in a form in a management area

But, for security reasons, a stripped-down version of python was used: only some of the standard library packages could be used

They invented 2 templating languages:DTML: something like PHP, mixing DTML tags and HTML tags

ZPT: using XML namespaces to create XML-valid documents

If you needed to do more powerful things, you could write normal python packages on the filesystem

But we also do Django

Official definition: high-level python web framework that encourages rapid development and clean, pragmatic design.

A framework to develop sites easily and fastly usually with SQL based data-schema and storage

But not everything is Zope in CodeSyntax.

Although we developed mysql based applications on top of Zope writing raw SQL queries, we finally adopted Django to do SQL-schema-driven applications

Django is also great!

Zope until .../2.5/2.6

Monolithic tarball

Even included python (2.1) in earlier versions

Add-ons developed as Products: folders with __init__.py and some initialisation code

Drop add-ons on a folder and restart

First versions of Zope were monolithic tarballs, that you needed to extract and run using a custom script.

The filesystem products were just dropped in a given folder on the filesystem, and everything worked quite nicely.

Zope 2.7/2.8/2.9

From monolithic -> installable thing

./configure && make && make install

System python or custom compiled

Zope instance: install once, create N instances

ZEO for database decoupling

With Zope 2.7, Zope brought the concept of Zope instance: you install Zope once, and then you can create several instances:You just have to install Zope ones

You can even install products that can be used in all of your instances

But you can install custom products/packages only in your instance

Zope need to be compiled with a given python version and run cmmi

You can also create an independent ZEO instance, with that you can install the database in a separate instance on the same server or in another one.

But.... we needed more

But for large websites, we thought that we needed an integrated powerfull environment, and found Plone, which has many things already done (we can understand Zope as a set of tools, and Plone a product built on top of those tools that provides more features).

Plone is a CMS

Simple and powerful

Adaptable workflow

Security (users, groups, roles, permissions)

Extensibility

Usability

But the installation....

Plone 1.x, 2.0, 2.1, 2.5, 3.0, 3.1:Tarball Zope 2.6, 2.7, 2.8, 2.9 and 2.10

Plone 3 (2007) and 3.1 (2008):Zope Zope 3: small reusable components

Plone 3.2 (2009):Tarball eggs

Everyone started developing python eggs for Plone

As a complex product, Plone installation was each time more comples:On first versions, Plone was released as a tarball: download, untar it inside Zope and worked

But people started using and developing products for Plone (extending core functionality), and at the same time, Zope started to be rewritten into Zope 3, to provide small reusable components, providing a way to upgrade independent components of Zope, and encourage reusability outside Zope.

That's why with Plone 3.2 everyone started writing standard python eggs and something needed to be done with the installation, because doing python setup.py install to all products was a no-go

Buildout to the rescue

Jim Fulton (creator of Zope), created zc.buildout

To solve 2 problems:Application-centric assembly and deployment

Repeatable assembly of programs from python eggs

So, the creator of Zope, Jim Fulton, wrote zc.buildout to install Zope 3 applications, and Plone got benefit from that to install Plone.

Plone Architecture

Apache/Nginx:80Zope (w Plone):8080ZEO (w ZODB):8081

This is a typical Plone setup, with a web server, the application server, in which Plone is installed, and the database server.

Everything can be installed on the same server or split across several servers

Complex installation

With python eggs, the installation is not to tar xzf Plone-3.2.0.tgz

But, python setup.py install a lot of eggs:Plone 3.x: +150

Plone 4.x: +250

Complex setups possible

As you see, Plone's installation needed to install a lot of python eggs, and needed to create several scripts to run zope instances, configure database connection, .

Plone Architecture

Apache/Nginx:80Zope (w Plone):8082ZEO (w ZODB):8084Zope (w Plone):8081Zope (w Plone):8083

And the installation can be splitted across several servers, so the installation needs to be easy and repeatable.

You don't want to run the same application with different egg versions on the same server (all instances need to have the same version setup).

Plone Architecture

Apache/Nginx:80Zope (w Plone):8082ZEO (w ZODB):8084Zope (w Plone):8081Zope (w Plone):8083

SERVER 1

SERVER 2

SERVER 3

And if you split everything on 3 (or more) servers, you need to have everywhere the same versions

Needs

Easy deployment (2-3 commands)

Repeatable:We need to run several times in development (each time we add new eggs)

We need to run it in several servers (own, development, production, ...)

So, we (and Plone developers, and sysadmins), needed something that could be run easily and fastly (when in development several times a day)

It needed to be repeatable: we want the same layout, and same set of versions on development or production servers

And that's why buildout was created

How solves it buildout?

ini-format config file (buildout.cfg)

Ordered execution of parts

Each part runs a recipe:python egg that given an input, does something

Nothing else

Buildout gets that defining an ordered execution of parts.

Each part is run by a recipe

And a recipe is a do something script.

Each recipe knows what to do.

Extending buildout

Buildout parts can be extended from other files and override options (OOP concept)

Extension can happen over-the-net (extends= http://myserver/buildout.cfg)

Known Good Set of eggs for proper installation

Create your own recipe

You can even create your own recipes, or extend one buildout file from another.

For instance:In development servers, you need to start Zope in debug mode, on production not;

In development servers you install just one instance; on production 2;

In development servers the port numbers are ones; on production others

This is achieved creating your own file, saying there that you extend another one, and overriding the values. It's just and OOP thing.

Why repeatable?

You can set exact versions of eggs

You say which files you download from where

The generated filesystem layout is always the same

If all versions are set correctly you will always get the same installation

Why we say this is repeatable?Because the installation is done on all servers in the same way

You say in the buildout configuration file which files or which packages and which versions to download.

The generated filesystem layout is always the same

If all versions are set, the installation result will be always the same.

How you develop new eggs?

They have no exact version:We can switch from given versions to source checkouts (svn, git, hg, ) easily using an extension: mr.developer

Extensions are similar to recipes

Enable and deactivate develop-eggs

Is this just for deployment or also for development?

You need to say buildout the versions of product you install (feedparser = 5.1.2), but what if you are developing a new product?

We use an extension, called mr.developer, that easily plugs into buildout the so-called develop eggs, eggs without a fixed version that are being developed.

Moreover this extension eases the way to checkout and clone packages from git/svn/mercurial/etc.

Buildout example (minimal)

[buildout]parts = installation[installation]recipe = zc.recipe.eggeggs = feedparserinterpreter = custompy

Creates a python interpreter (custompy) with feedparser in the path

Buildout example: Plone

[buildout]extensions = mr.developershow-picked-versions = trueparts = instance zeo omelette zopepyextends = http://dist.plone.org/release/4.3.3/versions.cfg
(show plonebuildout.cfg)

Show plonebuildout.cfg

Q: You use this for django?

A: Yes!

Q: Why?

A: because we came to django after Plone, and we knew how it worked and we were used to it. All our deployments are done in the same way

And yes, we use this for django too.

Perhaps django installation is easier without buildout (virtualenv & pip), but we came into django after learning buildout for Plone, and buildout worked so well that we adopted buildout also for

Q: but...

A: yes, we could install django with pip in a virtualenv... but we were used to use a more complex solution that allowed us to create bigger things

Yes, OK, we could do it with pip and virtualenv, but this way, we use everywhere the same deploying mechanism and everyone know how to run things.

Buildout example: django

[buildout]parts = django staticfiles cron(show djangobuildout.cfg)

Show djangobuildout.cfg

What recipes do?

Plone world:Create a Zope instance and put all eggs in the path

Configure port numbers, database location, blob file location, object cache sizes,

Create a ZEO instance to decouple the database (configure ports, paths, etc)

What can buildout recipe do?

In plone environments the basic recipes:Create zope instances from zope installation

Create ZOE instances to decouple data-base installation

Configure paths, environment vars, database locations, paths, por numbers, object cache sizes....

What recipes do?

Django world:Create an interpreter with django installed without polluting the system installation

Run static file collector

Create WSGI scripts

Install gunicorn and configure to run with nginx

In django worlds:Install django in an isolated environment

Run static file collectors

Create WSGI scripts

What recipes do?

Any world:Create supervisor/circusd scripts to run services

Download packages from svn/git/hg

Download tarballs

Create cronjobs

But you have many other recipes to do almost everything:Create supervisor or circus scripts and configurations to run the services

Download packages

Create cron jobs

What recipes do?

Any world:Install Apache/nginx

Configure load balancers (pound/haproxy)

Update po files and upload to Google Docs for easy translation

Create egg omelette to easily navigate on installed eggs

...

Install apache

Configure load balancers depending on the server instance setup

Upload PO files to a Google Doc to easily translate interface messages

Create egg omelette to navigate in eggs...

Search pypi for recipes...

Real life examples

In all of our servers we have +120 buildouts

Everyone knows (Plone or django developer) how to run them:./bin/buildout -vv

It's predictable

But this is used in real-life?All Plone installations since 2009 are installed using buildout.

Buildout is the responsible of the most web-traffic on pypi

In all of our servers we have more than 120 buildouts

Example architecture with buildout: www.bertsozale.com

NGINX :80
(system package)VARNISH :8081HAPROXY :8082ZOPE :8083ZOPE :8083ZOPE :8083

ZEO :8084

All in one server

This is an example architecture we installed with buildout in one server (it was quite big server) for a customer.

It installed varnish (caching server), haproxy (load balancer), several Zope instances and a database server.

We also tested to put a database slave in another machine, and the master database was replicated life there to have a live backup-copy

Example architecture: EEA

APACHE:80POUND
:8080

ZEO :8080

ZOPE
:8081ZOPE
:8082ZOPE
:8083ZOPE
:8084ZOPE
:8081ZOPE
:8082ZOPE
:8083ZOPE
:8084ZOPE
:8081ZOPE
:8082ZOPE
:8083ZOPE
:8084SERVER 1

SERVER 2

SERVER 3

SERVER 4

SERVER 5

In another project for the European Environment Agency, we have this setup in 5 servers, everything installed using buildoutOne server with Apache and Pound:

Three servers, each with 4 zope instances

Another server with the databse.

Example architecture: EEA

4 servers, everything configured using one buildout file.

https://github.com/eea/esdrt.buildout

EEA Sysadmins encourage buildout setupRepeatability

No need of root user to install things

All this complex setup (remember, you need to deploy the exact same versions of everything on all servers), is handled by one or two buildout files, and everything is public.

You can check the configuration files on github.

The EEA sysadmins even encourage this buildout setup (they are not Plone experts), because all setups across their premises are handled in the same way, the users have the same permission-set and errors can be found easily.

Buildout vs fabric?

Use of buildout does not exclude use of fabric.

Fabric also used for automation and repeatable environments

All is about what is best for you:Simple fab file? do it

A lot of deploy code lines? try buildout

And what about fabric? Cannot you use fabric?Yes, the use of buildout doesn't exclude fabric (or any other tool).Fabric is a tool for automation and running remote task.Everything is about what is best for you. We feel confortable with buildout, we also think that it's easier to reuse all those existing recipes than building tasks with fabric.Anyway, we don't exclude the use of fabric, if that suits you, use it!

Buildout & fabric

Use of buildout doesn't exclude fabric (did I say that?)

Use fabric to run update and run buildout:Update git checkout

Run buildout

Restart services

Buildout & fabric

Some EEA projects use it

We don't, we ssh & run buildout

But there is literature about this:http://tinyurl.com/fab-buildout

http://tinyurl.com/fab-buildout-so

Fabric is a way to run commands on remote servers

You can even use it with buildout.There's literature about using fabric with buildout: not to handle the installation of all those packages, but to run buildout on the server and to update the svn/git checkouts.

Buildout: Summary

Assemble and deploy simple and complex setups

Repeatable installed ecosystem is always the same

Specially suited for python based applications (Plone, Zope, Django, Pyramid), but also for non-python things: Apache, Pound, HAProxy, Memcached, ...

So, as a summary:Buildout helps you assembling and deploying both simple and complex applications.

Those setups are repeatable: everywhere you run them, you will get the same setup

It's specially suited for python applications (Zope, Plone, Django, pyramid), . but you can also install Apache, Pound, HAProxy, Memcached, ...)

Buildout (the bad part)

If your deployments are not similar, finding and searching recipes can be hard.

Error messages, sometimes are not helpful.

setuptools & distribute fight in the last 6 months

Q & A ?

Internet Solutions www.codesyntax.com

Azitain industrialdea, 3-KE-20600 EibarTel.: +34 943821780