The Problem

Our intrepid systems administrator installs the operating system from disk

From other disks, the anti-virus, office suite,and other vendor-provided applications the system

needs are loaded.

Patches for the operating system and applications are downloaded, either directly or indirectly, and

are applied to the system.

What's wrong whit that?

● The CDROMs may be different versions on different installs

● The patches from external sources will amost certainly be different on different installs

● The systems administrator may not put systems together in an identical manner

● Basically no two systems will be alike● This will result in two “identical” systems with

different behavior

The system is released to developers, who write and compile code, and copy it to the system.

I said “developers.”(you could add DBAs and application owners to

this list, but you see where I'm going)

What's wrong with that?

● It almost guarantees that different code will be deployed to different “identical” systems

● If a system “blows up” and needs to be rebuilt, you have to throw operations resources and development resources at it to get it running again.

● If you need to re-create a system “as-built” you have to find all the sources again (this just won't happen).

For the next system, wash, rinse repeat.

And so on, and so on, until the work not only fills all the time of all the bodies, but additional bodies

are needed to keep up with the work.

Welcome to “meat cloud” computing. (also, you'll need managers, “senior” managers to manage them, and “Directors” to manage them.)

All the while, no one will really have any idea what's going on, (beyond a holistic level)

But what can you do?

Identify your “authoritative” sources of information.

Found 'em! Now what?

Umm, I think I may see the problem...

Let's start over...

What if, instead of installing operating systems and applications striaght from media, the sysadmin stored them in a “repository”?

And then, any time a system was built, a script would grab those packages, and install the

operating system, and the applications?

One small script (or recipe) for each application, describing how it should be installed.

Now the application (package) repository will always be changing, as will the recipies that install

the applications (packages)...

A “version control system” (vcs) shall be maintained for both the package repositories and

the installation scripts.

A host “meta-data” repository will instruct from which repository to pull packages, and which scripts to use to install and configure them.

Patches for the Operating Systems and applications could be staged in the appropriate repositories, and scripts written to install them.

Developers would need to package their applications for installation, and provide

automated scripts for installation.

Now when boxes need built or rebuilt, no human resources are needed other than to initiate the

process.

Now identify your “authoritative” sources of information...

Found them!

Now that's more like it.

There are 3 major components

● The package repository● The recipe store● The meta-data repository

The Package Repository● Nothing gets installed to a system, by hand or

in an auotmated fashion without the installer being copied to the package repository first.

● The repository will be version controlled by cloning with hard-links, and then and only then adding new content.

● The repositories versions will be given unique IDs that can be linked to hosts in the meta-data repository

● A host shall only pull from one repository, even if a new one needs to be cloned to satisfy this.

The Recipe Store

● The recipe store will contain recipes for installing any and everything we install.

● Every component will have a unique ID that can be tied to a host using the meta-data repository

● All recipies will be version controlled.● No items unique to a host will go in the recipe

store. ● Items unique to a host will be pulled from the

meta-data repository and “merged” with the recipe to complete the configuration

The meta-data repository

● The meta-data repository will describe the infrastructure at the highest level

● The meta-data repository must be able to be queried by any script or application, or by hand

● It must conform to an open standard● It must be network facing (no application tiers)● It must be secure (SSL/TLS)● It must handle authentication and authorization

internally

Additional components

● Monitoring rules may be added to the recipe store such that hosts tied to the monitoring rules via the meta-data repository will be monitored in the manner described in the store.

● Back-up rules may be added to the recipie store such that hosts tied to the back-up rules via the meta-data repository will back up accordingly

● Any other rules may be set up in the same manner...

Process flow

The players

On the “Install Inititiator”, the command is given to deploy the “Target Host.” The initiator queries the meta-data repository for the

specifics of the install

The meta-data repository replies with, what to install, where, and the personalization information specific to the host.

The install initiator (using the meta-data) instructs the target host which recipies to use to install which packages from a specific

repository.

The target host pulls the appropriate packages from the package repository, and the installation recipies and does the work needed

to install itself.

Once installation is complete the target host routinely queries the meta-data-repository and the recipie store for changes, and does any work defined therein. It is now in “configuration management”

mode.

If the recipe store or the meta-data repository indicate that the target host should point to a new repository, then the target host

will update it's packages accordingly. This is how patches are applied.

Design Terminology

● The “package repository” may be an apt repository, yum repository, a CIFS share, or a combination of them, depending on the operating system(s) you need to install

● The “recipe store” may be version-controlled shell scripts, batch files, cfengine policies, puppet manifests, or whatever you need to get the job done.

● The “meta-data repository” can be an LDAP server (or a set of them) or a bunch of YAML files in git/svn, again, whatever you need...

Questions?