Dr. Ralf S. Engelschall€¦ · Manually deploy all applications into a single, shared, and unmanaged !lesystem location. Dependencies are resolved manually. Examples: Windows Fonts,

Dr. Ralf S. Engelschall

Content

know

understand

apply

ConceptsConcepts

Focus

Methods Technologies

scope ofthis training subsequent

task of trainee in practice

Statements

Examples

Rationales

Diagrams(written)

Trainer(verbal)

Trainer(verbal)

Scope

Computer Science

SoftwareEngineering

Software &SystemsArchitecture

Goal

Step 1: Your Insight (Believe)Concepts have a larger life-time than particular technologies and products.

Step 2: Our PreparationConcepts have to be assembled in a concise form to be handy in practice.

Step 3: Your ApplicationConcepts can be applied in practice both proactive/constructive and reactive/analytical.

2. B

EIN

G G

OO

D A

TCO

NCE

PTU

ALI

ZATI

ON

1. T

HIN

KIN

G L

IKE

AN

ARC

HIT

ECT

Type

Theory

(Conceptual)Model

Practice

AbstractionGeneralization

InstantiationSpecialization

Focus

Literatureknows aboutmore things

Theory

Industryknows aboutmore things

Practice

the mostrelevant concepts

ArchitectureFundamentals

Licensed to Technische Universität M

ünchen (TUM

) for reproduction in Computer Science lecture contexts only.

Intellectual Content: Version 1.0.2 (2019-06-28), Authored 2010-2019 by Dr. Ralf S. Engelschall

Graphical Illustration: Version 1.0.2 (2019-06-28), Copyright ©

2018-2019 Dr. Ralf S. Engelschall <http://engelschall.com

>, All Rights Reserved.U

nauthorized Reproduction Prohibited.A

F00.0

Architecture Fundamentals

RequirementsREQSystematically and structurally elicit,organize, document and agree onscope, functionality and constraints

EnvironmentENVDesign all external applicationarchitecture aspects to allow seamlessintegration into its environment

ResourcesRESProvision all human resources,environments and toolingsto drive the project

AnalysisANASystematically and structurally analyzeand document the functionality into abinding and complete speci�cation

DesignDESDesign all internal applicationarchitecture aspects to allow an e�cientand e�ective implementation

ProjectPRJPlan all time, costs and scope aspects to allow an e�ective ande�cient execution of project

DebuggingDBGLocate the origin of a functionalor non-functional problem withinthe application code or data

ImplementationIMPCraft all application code and dataaccording to the requirements, analysisand design speci�cations

ExecutionEXEContinuously ensure that at any time allproject members know about and canperform their assigned work packages

TestTSTTest application at all its structurallayers against de�ned functionaland non-functional test cases

AssemblyASMAutomate hierarchical building of allcode and data artifacts and automatedeployment & operations of application

ControllingCTLContinuously compare planned andactual status of time, costs and scopeand adjust project accordingly

MeasurementMSMMeasure static and dynamic aspectsof application and project and provideresulting key performance indicators

DocumentationDOCCraft documentation for the targetaudiences manager, developer, userand administrator

DeliveryDLVPlan and control the continuouschange and delivery process of theapplication across releases

InspectionINSReview and audit all aspects of thesoftware engineering process of theproject and judge current quality

TrainingTRNCreate and perform presentationsand workshops based on theapplication and its documentation

Con�gurationCFGConsistently identify and version allrevisions of all project artifacts acrossall storage types and their branches

ANALYTICAL CONSTRUCTIVE STEERINGPL

ANN

ING

EXEC

UTI

ON

AL

What?

Project =Disciplines xProcess

Why?

How?Who?When?

How? Who?Where?

secondary / support disciplines primary / focus / ”king” disciplines domain Analysis domain Developmentdomain Design domain Analyticsdomain Documentation domain Managementdomain Con�guration

What?

Licensed to Technische Universität München (TUM


Intellectual Content: Version 1.0.7 (2010-07-21), Authored 2006-2010 by Dr. Ralf S. Engelschall, inspired by Rational Uni�ed Process (RUP)

Graphical Illustration: Version 1.0.9 (2019-06-28), Copyright © 2007-2019 D

r. Ralf S. Engelschall <http://engelschall.com>, All Rights Reserved.

Unauthorized Reproduction Prohibited.01.1

AF

Software Engineering Disciplines

Manually deploy all applications into a single, shared, and unmanaged filesystem location. Dependencies are resolved manually. Examples: Windows Fonts, Unix 1990th /usr/local.

Pro: simple deploymentCon: incompatibilities, hard uninstallation

Bare AmalgamationAMA

Amalgamation

Application

Establish an application out of multiple Managed Packages. Examples: OpenPKG Stack, Docker Compose, Kubernetes/Kompose, Kubernetes/Helm.

Pro: independent, flexibleCon: overhead

Package/Container StackSTKApplication

Stack

Package/Container

Manually deploy all applications into multiple, distinct, and unmanaged filesystem locations. Dependencies are resolved manually. Examples: macOS *.app, OpenPKG LSYNC.

Pro: simple deployment, easy uninstallationCon: no repair mechanism

Unmanaged HeapHEA

Application

Heap

Bundle an application with its stripped-down OS dependencies and run-time environment into a container image. Examples: Docker/ContainerD, Kubernetes/CRI-O, Windows Portable Apps.

Pro: independent, simple deploymentCon: less variant possibilities, no depend.

Container ImageCON

Let individual installers deploy applications into multiple, distinct, and managed filesystem locations. Dependencies are manually resolved or bundled. Examples: macOS *.pkg, Window MSI, InnoSetup.

Pro: easy uninstallation, repairableCon: requires installer, diversity, no depend.

Managed HeapINSBundle an application with its full OS dependencies and run-time environment into a virtual machine image and deploy and execute this on a hypervisor. Examples: VirtualBox, VMWare, HyperV, Parallels, QEMU.

Pro: all-in-one, independentCon: overhead, sealed, unflexible

Virtual Machine ImageIMG

Let a central package manager deploy all applications into a single, shared, and managed filesystem location. Dependencies are automatically resolved. Examples: DPKG/APT, RPM, FreeBSD pkg, MacPorts, NPM.

Pro: easy uninstall., repairable, dependenciesCon: P.M. pre-installation, P.M. single instance

Managed PackagePKGBundle an application with its full OS dependencies, run-time environment and underlying hardware. Examples: AVM Fritz!Box, SAP HANA.

Pro: all-in-one, independentCon: expensive, sealed, unflexible

Solution ApplianceAPP

Software

Solution Appliance

Hardware

OS

Application

Container Image

OS (guest, user-land)

Application

Container Runtime

Virtual Machine Image

OS (guest)

Application

Virtual Machine Hypervisor

Package/Container Manager

Package/Container

Application

Application

Application

Application

Application

Heap

Application

Heap

Application

Heap

Package Manager

Application

Package

Application

Package

Application

Package

Installer

Application

Heap

Installer

Application

Heap

Installer


ünchen (TUM







F13.1

Software Deployment

ON-DEMAND

PAY-PER-USE

ELASTICITY

SELF-SERVICE

MULTI-TENANCY

MEASURED SERVICE

NETWORK ACCESS

Cloud Characteristics

logi

cally

/ ch

rono

logi

cally

rela

ted

Public Cloud

Private Cloud

Community Cloud On-Premises

Provider-Hosted

Cloud Scope Cloud Location

outs

ide-

insid

e re

latio

nshi

p

outs

ide-

insid

e re

latio

nshi

p

SaaSSoftware

as a Service

Cloud Approaches

FaaSFunction

as a Service

PaaSPlatform

as a Service

IaaSInfrastructure

as a Service

tech

nica

l lay

ers

CaaSContaineras a ServiceNetwork

Tools & Libraries

Operating System

Application RuntimeService EventsService Mesh

Computer

Container Runtime

ApplicationService

aka“Serverless

Computing”

User Session Function Call Continuously

AUTOMATABLE

logi

cally

/ ch

rono

logi

cally

rela

ted

Who? Where?How?

What?

Duration:

Proprietary StandardizedInterfaces:

Cost shifting: CAPEX to OPEX!Goal shifting: cheap to flexible!


ünchen (TUM







F13.2

Cloud Computing Resources

APIGateway

(Traefik)

Microservice(Frontend)

MessageQueue

(NATS + NATS Streaming Server)

SessionStore

(Corvus + Redis)

CachingProxy

(Squid)

ArtifactRepository

(Nexus)

ServiceRegistry(Kubernetes)

ProcessManager

(Kubernetes)

CredentialStore

(Vault + Notary)

MetricStore

(InfluxDB-Relay + InfluxDB)

EventStore(Jaeger)

EnterpriseIntegration

(WSO2 ESB)

EntityStore

(CockroachDB)

BLOBStore(Minio)

Authentication& Authorization(Dex + SPIRE + Opa)

ContinuousDelivery

(Drone)

VersionControl(Gitea + Git)

Full-TextStore

(ElasticSearch)

NameResolution

(CoreDNS)

Communication [C]

Persistence [P]

Tracing [T]

Gateway [G]

ProcessManager

(Kubernetes + CRI-O)

ServiceRegistry(Kubernetes)

Microservice(Core)

Microservice(Core)

Microservice(Frontend)

Common Group

Microservice Application (HA)

Foreign Application (non-HA)

Platform Application (HA)

Microservice Group

Microservice(Cross)

Microservice(Backend)

Major Design Criterias:1. Targets DevOps approach.2. Targets Continuous Delivery process.3. Targets Microservice Architecture.4. Targets Container Image deployment.5. Targets Service Mesh communication.6. Targets Server Cluster setup.7. Provides High-Availability of Service Platform8. Provides High-Availability of Application Microservices.9. Provides Scalability of Application Microservices.

M G I DT P C

2

M G I T P C

M G I T P C

1 3 N

Partitioned Cluster Setup (5x2+1+N Machines):

D

21 3 N

(Virtual) Machine

…

…

M G I T P C

M G I T P C

M G I T P C

M G I T P C

M G I T P C

M G I T P C

M G I T P C

Systems of Record DevelopersUsersSystems of Engagement

Management [M] Delivery [D]

GraphStore(Neo4J)

BlockchainStore

(Tendermint)

File-TreeStore

(GlusterFS + XFS)

Microservice(Backend)

Microservice(Cross)

Microservice(Cross)

Authentication& Authorization

(SPIRE Agent)

Standard Cluster Setup (5+1+N Machines):

Integration [I]

Major Approach Idea:With Cloud-Native Architecture one maximizes theleverage of PaaS-like, high-available, and scalable Cloudservices at the level of Software- and Systems-Architecturefor a whole set of applications.

CNCF Cloud-Native Definition 1.0:Cloud-native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.


ünchen (TUM







F13.3

Cloud-Native Architecture

M.N branch

trunk(M branch)

feature development

mergingM.N.0

time

bugfixmaintenance

mergesM.NbK-S-

YYYYMMDD

M.NbK-D

M.N.K.1

upstream vendortracking branch

alphaphase

betaphase

releasecandidate

phase

releasephase

developmentphase

maintenancephase

product diversityaccess unlimited

product focusaccess unlimited

product stabilityaccess controlled

product freezeaccess restricted

VCS commitactivity

M.N.1

M.N.K[.0]

M.N.0.1

M.N.0.K

M.N.K.K

M.N.K branch

M.Na1 M.NaK M.Nb1 M.NbKM.Nrc1

M.NrcK


ünchen (TUM


Intellectual Content: Version 1.3.0 (2012-09-19), Authored 2011-2012 by Ralf S. EngelschallG

raphical Illustration: Version 1.2.1 (2012-09-19), Copyright © 2008-2012 Ralf S. Engelschall <http://engelschall.com



F14.1

Version Control Architecture

source:checkoutcheckout

source:checkincheckin

F

stage0[:build]bootstrapgenerate

source:unpackunpack

stage0:cleanbootstrap:clean

realclean

stage1[:build]setup

configure

stage1:cleansetup:cleandistclean

S

stage2[:build]compile

stage2:cleancompile:clean

clean

C

stage3[:build]link

build

stage3:cleanlink:clean

L

stage4[:build]bundle

all

stage4:cleanbundle:clean

B

artifact:uploadartifact:download

source:packpack

source:downloadsource:upload

stage5[:build]package:pack

package

stage5:cleanpackage:clean

P package:upload

package:downloadI

DS

VCS

AR

install:uninstalluninstall

install:installinstall

package:unpack PB

install:updateupdate

install:upgradeupgrade

install:migratemigrate

install:repairrepair

runtime:stopstop

install:validatevalidate

runtime:startstart

runtime:reloadreload

runtime:scrubscrub

R

X

XXXXExternal State/Resource

Process State

State Transition Process Activity(semi-automated or automated)

runtime:statusstatus

sourceediting

runtime:testtest

stage2:testcompile:test

stage3:testlink:test

test

stage4:testbundle:test

stage5:testpackage:test

SC

3: D

evel

oper

Sho

rt-C

ircui

t Tra

nsiti

on

SC

4: D

evel

oper

Sho

rt-C

ircui

t Tra

nsiti

on

SC

1: D

evel

oper

Sho

rt-C

ircui

t Tra

nsiti

on

State Transition (short-circuit)State Transition (external)

Development:BuildStandard Process

Operations:DeploymentStandard Process

generation

oper

atio

n

mode

lossless lossy

fragmentation

transformation

duplication

integration

elimination

Examples:editing from scratchmaking random data

Examples:downloadinguploading

Examples:unpacking archive

Examples:object extraction

Examples:format changingrecoding

Examples:source compilationparsing

Examples:linking objects

Examples:packing archive

Examples:cleaning up

BUILDPATTERNS

source:tracktrack

Subversion, Git, Mercurial, Maven, Gradle, NPM, YARN, UPD, Autoconf, Automake, CMake, OpenPKG RPM

APT, dpkg, RPM,OpenPKG Build, Docker

APT, dpkg, RPM,OpenPKG RPM, Docker

OpenRC, SystemD, Init, SupervisorD,OpenPKG RC, Docker

1

1 2 3

1 3

4 5

6

Vendor

Packager (Source)

Packager (Binary)

Deployer

Operator

2

Archived

InstalledRunningBundled Packaged

C / L / B

Compiled/Linked

SC

2: D

evel

oper

Sho

rt-C

ircui

t Tra

nsiti

on

ACTIVITY NAMING: each activity has at least an official name and zero or more calling aliases

DEVELOPMENT: intentionally, there are short-circuit transitions between states

CONSISTENCY: directly triggering an activity causes intermediate activities (between old and new state) to be implicitly triggered first

ROUND TRIP: every forward-only activity should have a corresponding backward activity

GLO

BAL

RULE

S

package:meta-write

package:meta-read

1. edit2. compile3. start4. stop

DEV

ELO

PER

LOO

P

PREPARATION1 2 BUILDING DISTRIBUTION3

5 INSTALLATIONRUNTIME6 4 RESOLUTION

NSIS, InnoSetup, Tar, InfoZip,NPM, Maven, Gradle, Docker, OpenPKG Build

stage2:lintcompile:lint

lint

stage1:lintsetup:lint

stage3:lintlink:lint

stage4:lintbundle:lint

Setup Compiled Linked

stage5:lintpackage:lint

Bundled

stage0:lintbootstrap:lintsource:lint

ArtifiactRepository

(Binary)Distribution

Site

(Source)Distribution

Site

VersionControlSystem

Fresh Bootstrapped

Make, Maven, Gradle, Ant/Ivy,NPM, YARN, Docker Build, OpenPKG RPM

runtime:reconfigurereconfigure

install:configureconfigure

runtime:backupbackup

runtime:restorerestore

XXXProcess Activity(manually or semi-automated)

A BA A B

A

B

DS

BPackaged

1

4

31

3

Software Developer

Continuous Integration System

Continuous Deployment System

2

Release Engineer 1 2

2

65

AUTOMATION: each activity has to be automated inside a command-line driven tool (and is just triggered from a control head tool)


ünchen (TUM







F14.2

Assembly Process Architecture

Development Environments Testing Approaches

Operations Environments

(Potentially destructive) Environment for the (separated) development of all artifacts of the system, usually located directly at each developers.

DEV Development

(Potentially destructive) Environment for integrating all artifacts of the system, usually located centrally to the developers and driven automatically.

INT Integration

Environment for testing the system as a whole in a production-resembling context.

TST Test

Environment for staging the system as a whole in a production-equal context in order to deploy it to the Production environment subsequently.

STG Staging

Destructive environment mirroring the Staging environment for experimentation purposes, including operating system and major dependency upgrades.

DRY Dry-Run

Environment for running the system as a whole in production.

PRD Production

Destructive environment mirroring the Production environment for failover situation, including disaster recovery situations.

FOV Failover

Whitebox testing the inner details of individual units (components) against their functional specification.Specialization: Regression Testing

UT UnitTesting

Whitebox testing the outer interplay of individual units (components) against the technical design of the solution.Specialization: Smoke Testing

IT IntegrationTesting

Blackbox testing the system as a whole against the functional/domain and non-functional/quality requirements of the solution.

ST SystemTesting

Blackbox testing and formal approval of the system as a whole against the end-to-end functionality and user experience requirements.

UAT (User) AcceptanceTesting

Destructive environment mirroring the Test environment for demonstration and experimentation purposes, including trainings and showcases.

SBX Sandbox

Blackbox testing the system as a whole against the availability and proper operation of the intended services.Specialization: Service Monitoring

OT OperationTesting

DEV INT TST SBX

Version Control System

(Development)

ArtifactRepository

(Development)

ArtifactRepository

(Operations)

DRYSTG PRD FOV

Testing Targets:

EnvironmentMirroring

Artif

act F

low

Original EnvironmentsReplicated Environments

Quality Promise,Bugfix Effort


ünchen (TUM







F14.3

Environments & Quality Assurance

Development Environment Production Environment

Operations

Staging EnvironmentIntegration Environment

Development

Test Environment

ED

VC

AR

RT

CI

AR

BX

CD

DX

RT RT RT RT

BX

CD

DX

IA

DX

WC

SC

DS

BS

TC

DS

SCBS

DS

DX

TC

TC

DC

CD

DX

DSTC

SC

BS

DS

SCBS

DS

UsersTesters

Developers

Dev

elop

er

Administrator

TC

ICIC

DSTC

DS

RTVC AR

CI CD

OperationsDevelopment

DevOps Pipeline Pattern

Acto

rsSt

oresS S

A

Actor-Store Pattern

Acto

rsSt

ores

TC

DSTC

TC

Rule 1: Actors are linked by either Store-connected artifact flows or plain triggersRule 2: The configuration of all Actors and Stores have to be kept as small as possible by just performing orchestration and retrieving the actual commands via external scripts.Rule 3: On each Version Control (VC) commit, the application should have to be automatically redeployed as an updated version on the Run-Times (RT).Rule 4: Development and Operations can be split via two synchronised Artifact Repositories (AR), because of network topology constraints, or act on a single common one.Rule 5: Deployment on Staging can be both automatically and manually triggered.

Rules

ED: Editor / IDECI: Continuous IntegrationBX: Build ExecutionCD: Continuous DeploymentDX: Deployment ExecutionIA: Infrastructure Automat.

ActorsWC: Working CopyVC: Version ControlDC: Distribution CopyAR: Artifact RepositoryRT: Run-Time

StoresSC: Source ComponentIC: Interm. ComponentTC: Target ComponentBS: Build ScriptDS: Deployment ScriptTS: Test Script

ArtifactsFlowsArtifact FlowEvent FlowControl Flow

TC

TC

IC IC

TS TS

TS

TS

TS

TS

TSTS

DSTC

TS

DSTC

TS


ünchen (TUM


Intellectual Content: Version 1.0.1 (2019-10-31), Authored 2019 by Dr. Ralf S. Engelschall w

ith input from Christian Reiber and D

r. Thomas Schöpf


2019 Dr. Ralf S. Engelschall <http://engelschall.com



F14.4

DevOps Toolchain

Distraction-free low-fidelity illustration of the solution and its base features, displaying its pure structure and core functionality only.

WF Wireframe

High-fidelity mostly interactive sample, mockup, model or simulation of the solution and its base features, show-casing its structure and functionality.

PT Prototype

Pure realization of most-risky aspects of the solution, proofing their feasibilities. Might still be based on a different technology than WS, MVP and FP.

PoC Proof of Concept

Realization of all technical, fundamental aspects of the solution, ensuring the domain specific aspects can be realized later on top of it without problems.

WS Walking Skeleton

Early version of solution with just enough functionality to enable full turn of Build-Measure-Learn loop with minimal amount of effort and time.

MVP Minimum-ViableProduct

Final version of the solution with all intended functionality and targeting the mainstream market.

FP Full Product

Early version of the solution with incomplete and unstable functionalities to get feedback on product. Usually tagged as “M.NaK” (K > 0).

A Alpha

Early version of the solution with complete but still unstable functionalities to stabilise product. Usually tagged as “M.NbK” (K > 0).

B Beta

Mature version of solution with complete and stable functionalities to catch last-minute problems. Usually tagged as “M.NrcK” (K > 0) around RTM.

RC Release Candidate

Final version of the solution with complete and stable functionalities, available for production use. Usually tagged as “M.N.0”.

R Release

Updated version of the solution with complete and stable functionalities, available for production use. Usually tagged as “M.N.K” (K > 0).

RU Release Update

Arbitrary permanent points-in-time during development. This is the default tag for the source code. Intended for no availability releases.

DEV Development

Distinct temporary point-in-time for a release of the current version without a version increase. Intended for limited availability releases.

SNP Snapshot

Distinct temporary point-in-time for a release of the current version with a version increase. Intended for early and general availability releases.

REL Release

Edition of the solution for the Open Source Community. Contains just the base functionality and has limited volunteering support.

CE CommunityEdition

Edition of the solution for the Enterprise market. Contains the base and additional functionality and has full commercial support.

EE EnterpriseEdition

No public availability of solution at all. The scope for all Development and sometimes Snapshot point-in-times.

XA NoAvailability

Limited public availability of solution. Usually for releases after the End-of-Life-Announcement (EOLA) or for releases with specific customer features.

LA LimitedAvailability

Early public availability of solution for early market. Usually for Beta or Release Candidate levels or for Release and initial Release Update levels.

EA EarlyAvailability

Late public availability of solution for mainstream market. Usually for Release and sometimes just for Release Update levels.

GA GeneralAvailability

Distribution channel for all quarterly releases (“YYYY.QN”) with experimental features turned off. Intended for fast mainstream market and production use.

STABLE StableChannel

Distribution channel for all monthly releases (“YYYY.MM”) with experimental features turned on. Intended for early market or testing purposes.

EDGE EdgeChannel

Distribution channel for all daily snapshots (“YYYY.MM.DD”) with experimental features turned on. Intended for testing purposes only.

BLEED BleedChannel

Evolution Stages (ES) Maturity Levels (ML) Points-In-Time (PiT)

Availability Scopes (AS)Product Editions (PE) Distribution Channels (DC)

Version Control (VC) Version Scheme (VS) Product Life-Cycle (PLC)

<Name>[/<ES>]<M>.<N><ML><R>[-<PiT>[-<PE>[-<AS>[-<Date>]]]]

Foo/PT 1.2b3-SNP-CE-LA-2018.01.01Foo 1.2.3-REL-EE-GAFoo 1.2a3-DEV

Version Number (VN)

Major Version of solution. Usually bumped on major technical or domain-specific changes only. A bump resets the Minor Version and the Revision, too.

M Major Version

Minor Version of the solution within the Major Version. Usually bumped on new features. A bump resets the Revision, too.

N Minor Version

The Revision of the Maturity Level within Major and Minor Version. Bumped for every A/B/RC/R/RU Maturity Level.

K Revision

Edition of the solution with just the standard functionality and regular support.

STD StandardEdition

Edition of the solution with both the standard and extra functionalities and extended support.

PRO ProfessionalEdition

Release toManufactoring

(RTM)

End-of-LifeAnnouncement

(EOLA)

Last-Order-Date(LOD)

End-of-Life

(EOL)

XALA

GAEA

t

BLEED

EDGE

STABLE

3 months4 weeks1 day

Distribution channel for all (half-)year releases (“YYYY[.N]”) with experimental features turned off. Intended for slow mainstream market and production use.

SOLID SolidChannel

SOLID

Which? Who? Where?

When?What? When?When?

arbitrary technology target technology


ünchen (TUM







F15.2

Software Release Management

Documents

Dr. Ralf S. Engelschall€¦ · Manually deploy all applications into a single, shared, and unmanaged !lesystem location. Dependencies are resolved manually. Examples: Windows Fonts,