#LinuxPiter

How devopsexhausts itself,and what will happen next

Kirill Vecherajetware.org

The next generation of models of software systems management

68

Kirill Vechera

CTO jetware.orgSoftware system management automationBuilding, configuring, testing, deployment, updating

BioSystem programmingSolaris, FreeBSD, Linux

System and network administrationUnix, OSPF, BGP

Internet services developmentC, Perl, Java, Python, Ruby

Annotation

The evolution of information systems management

Present instruments and emergingHow do we contribute to itWhy devops becomes unnecessary

Big100 — 100 000 servers

Complex10 — 100 apps10 — 1000 connections100 — 10 000 libraries and programs

Growing0.1 — 100 updates/day

Modern information system

ProblemsControlUpdateDiagnoseOptimizeRepair

Structure of systems

Monolithic

Distributed,tight coupled

Distributed,loosely coupled

Monolithic system

Fixed configurationEverything as a single copy

ServicesConnections

Loss of service and communication disrupts the whole systemCapacity is limited

Distributed system

Connected componentsComponentSingle service or multiple different services together

ServicesSingle service or multiple instances of a service

MacrolevelMicrolevel

Macrolevel

Componentsservices

Topologyconnections between the components

Microlevel

Internal arrangement of the componentPrograms, libraries, settings, dataConnections between internal services

Inside the component — monolithic subsystem

Distributed tightly coupled system

Everything is predeterminedConnections to the servicesBinding to the equipmentAdding/removing thecomponents or connectionsThe reaction to the failures

ManagementCentralizedConnections and bindings are organized by the administrator

Distributed loosely coupled system

Everything is changeableConnections to the servicesBinding to the equipmentAdding/removing thecomponents or connections

ManagementСontinuousAutomatedCentralized

Systems management evolution

1. Hands

2. Scripts

3. Tools

Equally — both monolithic system, and tightly coupled distributed systems

What follows IaC tools?

Switching to loosely coupled systemsSystems management is stratifiedWe need macro-level menegementWe need micro-level menegement

Loosely coupled system

Minimal requirementsAutomated managementConnections to the servicesBinding to the equipmentAdding/removing thecomponents or connections

Minimal requirements to the component

Portability

Replicability

Interface

Isolation from neighbours

Equipment independent

Separate date storage

Connections for services

AWS helped to get accustomed

Services oriented architechtureImmutable images of virtual machinesSeparate data storageMultiple service instances

Docker

PaaS basement

Development tool

Service execution engine

Docker-container as a component of a systemLightweight replacement of the virtual machineApplication isolation and dependencyInterface - the network and dataStimulates the use of immutable image

DockerfileFast building and replicability

All requirements for the component of loosely coupled system are complete

The evolution of distributed systems management – macrolevel

1. Hands

2. Scripts

3. Tools

4. Orchestration

5. Self-organization

Distributed systems management

4. Orchestration AlternativesKubernetesMesos MarathonDocker Swarm mode

The evolution – microlevel, monolithic system

1. Hands

2. Scripts

3. Tools

4. ???

5. Self-organization

The evolution of distributed systems managementMacro-levelOrchestration

Micro-levelDocker ???

A service in a Docker-container

You'd imagine inside ...

But in fact

The evolution of systems management

Distributed system, macro-levelService application, micro-level

Docker

Kubernetes

Docker for micro-level management

Simple provisioningDockerfile: shell, deb/rpm, chef/puppet/salt

Quick update: layersInterface to macro-level: manualComponents integrationsMicroserices: separate containers, manual

Micro-level, service application

Frequent updatesMany dependencies (components)Programs, libraries, date

Dependency versionsComponents integrationSettingsInterface to the macro-level

Actual problems

Versions incompatibilitybugs or API changes in the new versionsbugs or vulnerabilities in the old versions

Human error when updatinginternal settingsorchestration meta-data

Easy rider

Software development, corporate customersCustomized software solutionsBusiness Intelligence

Continuous integrationSeveral hundred projectsCommon componentsCustomized compoments

Some merges & acquires before

Easy rider

Primary softwareOracle, SAP Hana, PostgreSQL, MySQLJava, C++, Python, JavaScript, PHP

AndR, Excel, ImageMagick, other stuff

ОСRHEL, Oracle Linux, SLESown patches for JRE, libc, OMP etc

Windows

Easy rider

Delivery to the clientSaaSVirtual machines.exe + jarRPM packages

Easy rider

BeforePlatformX build serversDifferent OSes

ManagementScripts — make, shell, pythonJenkinsPuppet

Easy rider

ProblemsPoor testingLong wait for the result rarely tested→

Nonconformant test and the work environmentErrors raised on the customer side

Build process often breaksA lot of manual labor

Easy rider

What is neededWell controlled environmentSaaS, Virtual machines

Quality building and testingReproducible buildsFast buildsMinimum manual interventionSimilar to Maven or Gradle

Easy rider

ConsideringJuju CharmsNix

Nix package manager

Built package — pure function ofSource codeConfiguration valuesDependencies (another built packages)Building rules

Nix

Everything consist of a packagesEvery packageSeparate directoryRead-onlyIs built when required as dependencyor installed pre-built from repository

Nix – user environment

Union point for several nix-packagesEvery user environment — in a separate read-only directoryEvery packages combination — independent environmentAdding or removing a packageCopy current environment to the new oneAdd or remove package to the new environmentSwitch current environment to the new (symlink replaced)

Nix – user environment

Directories structure complies FHSPresent bin, etc, lib, include, share, libexecAbsent var, log, tmp — read-only

Symlinks from user environment to the corresponding files in a packages

Nix — user environment

~/.nix-profile

/nix/store/0123-user-env

bin

share man man8

mysqld

mysqld.8.gz

php

/nix/store/4567-mysql-5.5.50 bin mysqld

share man man8 mysqld.8.gz

share man man8 mysqld.8.gz

/nix/store/89ab-php-5.6.1 bin php

Easy rider

NowHardware (SaaS & CI)~ 180 own machines (SaaS — 90 & CI — 90)< 200 AWS (mostly CI)500 — 2000 containers

ManagementNix + Docker + KubernetesJenkinsPuppet

Easy rider

Problems with NixNo version dependenciesAll dependency consumers rebuildingLong wait for the result - up to 2 hours

Nix — rebuild on argument changes

mysql: libc openssl

0123-libc 4567-openssl → 89ab-mysql

patched libc → cdef-libc

cdef-libc 4567-openssl → 3210-mysql

Easy rider

SearchingHas Nix's advantagesAutomated buildsDependenciesIsolationRepeatability

PlusNo unnecessary buildsSuitable for multiple versions

Easy rider

ConsideringHabitatSnappyGentooJetware

Jetware vs Nix, in common

Consists of a packagesRead-onlyWith versionsWith build variations (flavours)

Packages with dependenciesAutomated installationAutomated building

Configuration variables of a package

Jetware — packages

Not a pure functionDependencieson package, on version conditions

Incapsulated packagesPackages self-integration and self-configuration

Jetware — runtime environment

Union point for packagesKeeping packages data and settingsSeparated from OS environmentworks directly on any OSlibc — just a package

Jetware — runtime environment

Package — outsidePackage — namespacePackage keeps all (read-only)Data and settings — inside bin

lib

var log

etc

tmp runlibc

Runtimeenvironmentconfiguration

Jetware — export into environment

Immutable dataOnly minimumbin, libsymlinks

bin

lib

var log

etc

tmp run

libc

Jetware — export into environment

Changeable dataCopiedSeparate sectionsOwn FHS

bin

lib

var log

etc

tmp run

libc

Packages self-integration

Data export to another packagesRelations, files or APIExamplepackage mysql exports own C-headers into cc package

Settings value for another packagesExamplemongodb_tuning assigns mongodb.port=22333

Easy rider

Demo implementation with JetwareBuilds by dependencies, versionsOwn subprojects and third-party components

Repositories with testing staging flow→Docker-container creation40 — 500 packages1.5 — 20 seconds

Expected~ 180 120 own machines (CI — 30, 90 — SaaS)< 200 0 machines from AWS

The evolution of distributed systems managementMacro-levelOrchestration

Micro-levelSelf-constuction

What’s next?

Systems management fusion both on macro-level and micro-level~ 1 — 3 years

Life cycle of a service

Developer1 Source code + libraries

2 Compiling, building3 Dependencies integration

4 Testing5 Export to image

6 Deployment7 Put into operation

???

Orchestration

Does the program need OS?

Program needsKernel for syscallsFile systemRuntime environmentLibraries and executablesAnother data — resources

OS environment — not needed

Must the program be implanted to OS?

To access the program we needTo know where it is placedTo feed the data inputTo accept the data output

The program can be anywhere — OS is not needed

Tight binding of OS and application

Historical heritageInfluence of Linux distributions

Self-constructing autonomous program

Is not bound to OSDescribesRuntime dependencies (packages, services)InterfaceConsuming and suppling resources (port, stdin/stdout etc)

Build rules (optional)SourcesBuild dependencies

Management on macro-level

Scheduling objectApplication: package and version

Starting an applicationPackage building or retriving from a repositoryRuntime environment constructionService deployment and executionPublishing interface data for service discovery

Service package description

postgresql-cluster-user-geotags-5.1 requires: postgresql-9.1 postgresql-cluster-config-2.1 storage-ssd-1: env: volume: user_geotags size: 1gb

Versions and updates

Developer releases the versioncommit in repository tagged with version

Another service has reference on package-version service is building and executing

No reference on package-versiongarbage collector stops the service

The evolution of distributed systems management

Macro-level and micro-level fuse

Robots replacing devops

Developers and sysadmins returning to the more intellectual work

And then?

Some years laterThe fifth stage of evolution - self-organization

How devops exhausts itself,and what will happen next

Kirill Vechera

http://jetware.orgSkype: kirill.vecheracv-c@jetware.org