Enabling Continuous Delivery with Database...

Enabling Continuous Delivery with

Database Practices

Pramod Sadalage @pramodsadalage ThoughtWorks Inc.

Agenda

Why Continuous Delivery

Collaboration with the data team

Automation for data

Refactoring databases

Deploying database changes

Why Continuous Delivery?

To get fast feedback from users, release frequently

Reduce risk of releasing

Learning and responding to customer needs is critical

Achieve Continuous Delivery

Close collaboration between everyone involved in delivery

Extensive automation of all parts involved in delivery

process

Reduce the gap between development and operations

CD should involve the database team

How does continuous delivery apply to

databases?

Collaboration techniques

Pair the DBA’s and Developers

Version control all database scripts and artifacts

Automation techniques

Let developers provision application database

√× JaiAndy

Laptop/Desktop

Application

Database

Continuously Integrate database changes

Integration environmentLocal dev environment

ANT Maven Gradle Rake

Continuous Integration Engine

Source Control

Dev Database

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Check in application code and database migration scripts

Update and build

Database changes applied by DBA Team

Source Control

Dev Database

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Migration scripts (DBA)

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Migration scripts

Integration Database

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Migration scripts

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Apply migration scripts

Migration scripts

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Migration scripts

Artifacts

Package Migration scripts

War Jar

Source Control

PROD UAT QA

Dev DB

PROD UAT QA

Environment

Update and build

Migration scripts

Artifacts

Package Migration scripts

War Jar

Apply Migration scripts

War Jar

Benefits of CI with databases

• Test application code and database at one place

• Generate code and database artifacts

• Integrate application and database changes in an independent environment

• Show current state of application and database to all

DB artifacts in CI

application artifact for build <<n>>

database artifact for build <<n>>

Deployment

• Database migration/upgrade should be a development time task not deployment time task

Deployment

• Package all the migration scripts, during Continuous Integration cycle

Deployment

• Apply the migration scripts

Deployment

• Apply the migration scripts

• Deploy frequently to reduce risk

Database refactoring

http://databaserefactoring.com/

A database refactoring is a small change to your database schema (the DDL, data, and DB code) which improves its design without changing its semantics.

A database refactoring improves its design while retaining both its behavioral and informational semantics.

A database refactoring is a small change to your database schema (the DDL, data, and DB code) which improves its design without changing its semantics.

Timeline of Change

Implement the refactoring

{Deploy new changes, migrate data, put in scaffolding code

Expand

Timeline of Change

Transition

Transition Period (old and new)

{Start

Expand

Timeline of Change

Transition

Transition Period (old and new)

{Start

Expand

Refactoring completed

Remove old schema, scaffolding code

Contract

Keeping Old and New alive

• DB Should be able to handle multiple versions of the application

• Create Interfaces in the database

• Wrap tables with views

• Create calculated columns

• Create triggers to sync data

Expand Contract an example

Start name = “Pramod Sadalage”

Expand name = “Pramod Sadalage” firstname = “Pramod” lastname = “Sadalage”

Contract firstname = “Pramod” lastname = “Sadalage”

More on expand contract

Expand

Without data migration name = “Pramod Sadalage” firstname = null lastname = null

With data migration name = “Pramod Sadalage” firstname = “Pramod” lastname = “Sadalage”

Expand

Contract firstname = “Pramod” lastname = “Sadalage”

Simple scenario - DBDeploy

ALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60);

--//@UNDO

ALTER TABLE Customer DROP COLUMN firstname VARCHAR2(60); ALTER TABLE Customer DROP COLUMN lastname VARCHAR2(60);

With synchronized dataALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60); CREATE OR REPLACE TRIGGER SynchronizeName BEFORE INSERT OR UPDATE ON Customer REFERENCING OLD AS OLD NEW AS NEW FOR EACH ROW BEGIN IF :NEW.Name IS NULL THEN :NEW.Name := :NEW.firstname||' '||:NEW.lastname; END IF; IF :NEW.name IS NOT NULL THEN :NEW.firstname := extractfirstname(:NEW.name); :NEW.lastname := extractlastname(:NEW.name); END IF; END; /

—//@UNDO ……

Migrate and Synchronize data

ALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60); UPDATE Customer set firstname = extractfirstname (name); UPDATE Customer set lastname = extractlastname (name);

CREATE OR REPLACE TRIGGER SynchronizeName BEFORE INSERT OR UPDATE …. —//@UNDO …… UPDATE Customer set name = firstname ||’ ‘||lastname WHERE name IS NULL; ALTER TABLE Customer DROP COLUMN firstname; ALTER TABLE Customer DROP COLUMN lastname;

Contract

ALTER TABLE Customer DROP COLUMN name;

—//@UNDO

ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname ||’ ‘||lastname WHERE name IS NULL;

Contract

ALTER TABLE Customer DROP COLUMN name;

—//@UNDO

ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname ||’ ‘||lastname WHERE name IS NULL;

SET UNUSED name; When drop takes

forever

Keep legacy apps happyALTER TABLE Customer DROP COLUMN name; ALTER TABLE CUSTOMER ADD (name AS (generatename (firstname,lastname)) );

—//@UNDO

ALTER TABEL Customer DROP COLUMN name; ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname ||’ ‘||lastname WHERE name IS NULL;

Virtual column in Oracle,

Generated Column in

MySQL.

Another example

Migration script

ALTER TABLE custordr rename to customerorder;

CREATE OR REPLACE VIEW custordr AS SELECT custordrid, ponumber, ordrdt, shipdate, sptoadid FROM customerorder ; --//@UNDO

DROP VIEW custordr; ALTER TABLE customerorder RENAME TO custordr;

data in migrations

data in migrationsINSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 22, 'John Doe Services', 1 ); INSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 23, 'Bob Products', 1 ); INSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 24, 'Carr Machinery', 1 );

data in migrations

INSERT INTO currency ( currencyid, name, code ) VALUES ( 2, 'Canadian Dollar', 'CAD' ); INSERT INTO currency ( currencyid, name, code ) VALUES ( 3, 'Australian Dollar', 'AUD' ); INSERT INTO currency ( currencyid, name, code ) VALUES ( 4, 'EMU Euro', 'EUR' );

data in migrations

DELETE FROM contact ct WHERE NOT EXISTS (SELECT 1 FROM customer p WHERE ct.contactid=p.contactid) and EXISTS (SELECT 1 FROM (SELECT customerid, COUNT(*) FROM contact WHERE customerid IS NOT NULL GROUP BY customerid HAVING COUNT(*) >1) ct2 WHERE ct.customerid=ct2.customerid);

• Large refactorings are risky

• Sequence of many small refactorings can create the desired change

• Migration scripts should be checked in and run on local dev/ci/qa/uat/prod etc.

• Changes to data are also migrations

Tracking Changes

• Each change is a delta/migration script

Tracking Changes

• Each change is a delta/

• Migration scripts are development time activity not deployment time tasks

Tracking Changes

• Package migration scripts for automated deployment

Tracking Changes

• Package migration scripts for automated deployment

• Same scripts for: developers, QA, UAT and Production

Continuous Delivery

ant -propertyfile qa.properties upgrade ant -propertyfile live.properties upgrade

Deployment should be easy

Thanks @pramodsadalage

sadalage.com databaserefactoring.com

Enabling Continuous Delivery with Database...

Documents

WIP and Kaizen-Key Tools for Enabling Continuous Improvement · WIP and Kaizen: Key Tools for Enabling Continuous Improvement Michael DePaoli Agile/Lean Coach, VersionOne [ August

Appian Continuous Delivery Framework - BPM Consulting · Appian Continuous Delivery Framework iACD Incessant’s iACD solution provides a Continuous Delivery Framework enabling organizations

Enabling and accelerating C-H functionalization through ... · Enabling and Accelerating C−H Functionalization Through Continuous‐Flow Chemistry PROEFSCHRIFT ter verkrijging van

Enabling Technologies for the Continuous …pqri.org/wp-content/uploads/2015/11/Trout.pdfEnabling Technologies for the Continuous Manufacturing of APIs: ... Aspirin: Crystal orientation

White Paper: Compartmentalized Continuous Integration: Enabling Rapid, Flexible Collaboration for Complex Software Projects

Enabling Continuous Delivery For Major Retailer using open source Jenkins

Enabling Continuous Delivery with Oracle Cloud Services...Continuous Delivery Is Reshaping The Future Of ALM Source: Application Development & Delivery Professionals on July 22, 2013

1 Blackboard Outcomes System: Enabling Continuous Improvement

Shattering Wi-Fi Records at Levi’s Stadium and Enabling ... · Shattering Wi-Fi Records at Levi’s Stadium and Enabling Mobile Engagement 3GBPS: CONTINUOUS WI-FI BANDWIDTH FOUR

Continuous Shape Shifting: Enabling Loop Co-optimization ...web.eecs.umich.edu/~anijain/ShapeShifterppt.pdf · Continuous Shape Shifting: Enabling Loop Co-optimization via Near-Free

Kuksa: A Cloud-Native Architecture for Enabling Continuous ... · Kuksa: A Cloud-Native Architecture for Enabling Continuous Delivery in the Automotive Domain Ahmad Banijamali1[0000

Compartmentalized Continuous Integration: Enabling rapid ...info.perforce.com/rs/perforce/images/AlteraWhitePaper...Compartmentalized Continuous Integration: Enabling rapid ﬂexible

Enabling Continuous Delivery and Staged Delivery within a ... · Enabling Continuous Delivery and Staged Delivery within a Common Process Jeff Westenhaver ... DevOps Major Shifts

DockerCon SF 2015: Docker, Enabling Continuous (Food) Delivery

Enabling Agile Testing Through Continuous Integration Agile2009

Gids enabling continuous delivery for your web app

Enabling Continuous Delivery (CD) in Enterprises with Testing

4dev prezentacja cd 2014 · Enabling Continuous Delivery ... Survey. Our highest priority is $ to satisfy the customer$ through early and $ continuous delivery! of valuable software

Using Capacity Building to Drive Continuous Improvement in … · 2015-03-24 · a mechanism for continuous feedback, enabling managers and employees to promptly address issues as

Compartmentalized Continuous Integration: Enabling Rapid, Flexible Collaboration for Complex Software Projects