Child Welfare Digital Services

Sprint Review Presentation

Sprint Dates: 08/17/17 to 08/30/17

Sprint Review of PI 3.6

DevOps Engineering

Agenda

Review the Sprint Goal

Demo #1: Perry APM - New Relic (Jim)

Demo #2: OpenScap (Steve)

Demo #3: Log Access (ELK) in pre-int (Alex)

Demo #4: Jira Request

Team Metrics

Impediments

Risks

What’s Next

Review the Sprint Backlog

2

Sprint Goal

Keep the lights on (maintain existing environments)

Deliver & Maintain new and existing environment in the CD

pipeline:Preview/ Performance

Improved Measurement & Logging

Access to logs via Web Browser for Integration Env

More NewRelic Monitoring of Infrastructure (L1/L2) and App

Performance (L3/L4)

Team workflow by work type

Complete DOE Team Jira Pilot

Support Dev Teams

3

1st Demo of Functionality – Perry APM - New Relic

4

NewRelic JVM Agent Installed on Perry in Pre-Int Environment

• Demo on transaction filtering

• Web transaction response time breakdown capability

• HTML error transaction tracing

2nd Demo of Functionality – openSCAP

5

OpenSCAP is an open source implementation of the NIST SCAP protocol

(Security Content Automation protocol)

Works on:

• RHEL 5+

• CentOS 5+

• Other xUx distributions are not supported.

• Will not work on AWS RHEL AMI.

OpenSCAP automates the compliance and assurance scans for (Among many

others) NIST compliance.

OpenSCAP can do automated remediation.

When used in conjunction with an open stack tool such as Ansible, openSCAP

can provide secure configuration and compliance, in support of configuration

management and compliance management .

3rd Demo of Functionality – Log Access (ELK) in pre-int

6

4th Demo of Functionality –Jira Request

7

SharePoint: https://osicagov.sharepoint.com/sites/projects/CWS-NS/de/

DevOps Engineering – Sprint Metrics: PI 3.6

8

BUILD METRICS: PI 3.6 Only

PI 3.4, 3.5, and 3.6 Combined

Change from PI3.4 to 3.5

Builds Δ 105

Percentage 60%

Change from PI3.5 to 3.6

Builds Δ -102

Percentage 61%

Row Labels

Sum of #

Successful

Sum of #

Failed

Sum of # Total

Builds

preint 53 6 60

Preview 24 19 43

Integration Environment 16 1 17

Devops-Builds 2 8 10

demo-integration 7 3 10

app01 0 9 10

preview1 3 2 5

performance

environment 4 0 4

Common 1 1 2

Grand Total 110 49 161

Row Labels

Sum of #

Successful

Sum of #

Failed

Sum of # Total

Builds

preint 207 32 240

Integration Environment 75 14 90

Preview 44 37 82

app01 2 43 47

performance

environment 22 12 34

demo-integration 20 11 31

integration02 7 11 18

Devops-Builds 2 8 10

Deploy environments 3 6 10

preview1 4 2 6

trn02-staging 2 2 4

Production 1 2 3

sandbox 2 0 3

Common 1 1 2

preint02 1 0 1

Grand Total 393 181 5810

50

100

150

200

250

300

PI3.1 PI3.2 PI3.3 PI3.4 PI3.5 PI3.6

DEVOPS BUILD METRICS# Success # Failed # Total Builds

DevOps Engineering – Sprint Metrics: PI 3.5

9

Sprint Metrics

Planned (Points) 53

Unplanned (Points) 11

Total Story Points 64

Completed Backlog Items (Planned) 40

Completed Backlog Items (Unplanned) 7

Total Completed - Stories/Chores/Bugs 47

Sprint Velocity 37 (-9)

Total % Complete 58%

Average Velocity over 3 sprints 36.7

Velocity History over 3 sprints100%

Sprint Backlog (PI 3.5): 64 Points

10

Key:

Unplanned and/or Operational Support

Chores

PlannedId Title Labels Type Estimate Current State Created at Accepted at Requested By Owned By

148527001DevOps - build continuous integration for Geo Service API cals, dependency<cals_dev, devops feature 2accepted 8-Jul-17 4-Aug-17Leonid Marushevskiy Alex Kuznetsov

149963787 IT Service Continuity Report - Due 08/05/17 devops contractual deliverables chore accepted 4-Aug-17 4-Aug-17Alaine Karoleff Sreekar Peddi

149827761**Integration** Deploy CALS API to Int Environment pi-3 integration, planned pi-3 feature 2accepted 31-Jul-17 5-Aug-17Rasim Bokhari Alex Kuznetsov

149978159System Availability Report - Due 08/04/17 devops contractual deliverables chore accepted 4-Aug-17 5-Aug-17Alaine Karoleff Ashish Kumar

149870629Capacity Management Report - Due 08/05/2017 devops contractual deliverables chore accepted 2-Aug-17 5-Aug-17Alaine Karoleff Jim Bader

149870726Service Asset and Configuration Report - Due 08/04/17 devops contractual deliverables chore accepted 2-Aug-17 5-Aug-17Alaine Karoleff Arpit Patil

150030257Integration deployment (ferb-api) pi-3 operational-support chore accepted 7-Aug-17 7-Aug-17John Simko Ashish Kumar

150023873Integration02 deployment (ferb-api) pi-3 operational-support chore accepted 7-Aug-17 7-Aug-17John Simko Alex Kuznetsov

149977735Deploy CALS API updates to Integration Environment cals, on-call, pi-3 unplanned, sre chore accepted 4-Aug-17 7-Aug-17John Simko Ashish Kumar

150023757Integration deployment (ferb-api) pi-3 operational-support, sre chore accepted 7-Aug-17 7-Aug-17John Simko Alex Kuznetsov

150068796Count user Access issue to Backend chore accepted 8-Aug-17 8-Aug-17Sreekar Peddi Sreekar Peddi

150027394Integration deployment (perry) ci/cd, pi-3 operational-support feature 2accepted 7-Aug-17 8-Aug-17John Simko Ashish Kumar

149971961Deploy Rundeck Updates to Integration on-call, pi-3 unplanned, sre feature 2accepted 4-Aug-17 10-Aug-17John Simko Ashish Kumar

150057325Chad - Tool Access configuration-management, tools feature 2accepted 8-Aug-17 10-Aug-17Chad Bratton Ashish Kumar

150094759**Integration** Deployment of cals-api ci/cd, pi-3 integration, pi-3 operational-support feature 1accepted 9-Aug-17 10-Aug-17Leonid Marushevskiy Ashish Kumar

150114883**GitHub** -- Add DevOps Engineers to GitHub configuration-management, github, identity-administration feature 1accepted 9-Aug-17 10-Aug-17David Sanabria Chad Bratton

150026113Deploy Rundeck Updates to Integration02 on-call, pi-3 unplanned, sre feature 2accepted 7-Aug-17 10-Aug-17Leonid Marushevskiy Ashish Kumar

146105593On Call Support 3.5 - Alex and Ashish pi-3 operational-support, pi-3 unplanned feature 2accepted 25-May-17 11-Aug-17Chad Bratton Alex Kuznetsov

150107626Integration deployment (Deploy Rundeck updates) pi-3 operational-support, sre feature 1accepted 9-Aug-17 11-Aug-17John Simko Ashish Kumar

150104055**Integration** Deployment of cals ci/cd, pi-3 integration, pi-3 operational-support feature 1accepted 9-Aug-17 11-Aug-17John Simko Arpit Patil

149819489**JIRA** Migration Research feature 1accepted 31-Jul-17 11-Aug-17Chad Bratton Chad Bratton

150053196**DB2** Integration Unauthorized access pi-3 integration bug accepted 8-Aug-17 11-Aug-17David Sanabria Vinod Reddy

148393121DevOps - Deploy Perry to CALS DEV environment cals, dependency<cals_dev, devops feature 1accepted 6-Jul-17 11-Aug-17Leonid Marushevskiy Alex Kuznetsov

150150417**Operational Task** Grant VPN and other access to Srikanth pi-3 operational-support feature 1accepted 10-Aug-17 11-Aug-17Arpit Patil Arpit Patil

150029993Configuration Management - Process Improvement configuration-management, improvements feature 1accepted 7-Aug-17 11-Aug-17Chad Bratton Chad Bratton

145652637DEV - API - provide access to application logs on CALS API Development Environment back-end, devops feature 1accepted 17-May-17 11-Aug-17Alex Kuznetsov Alex Kuznetsov

150116360**preint** DiskSpaceUtilization Warning bug accepted 9-Aug-17 14-Aug-17David Sanabria Ashish Kumar

150185103**Performance Analyst Role** Define and Document Role & Responsibilities --Draft administration, performance chore accepted 11-Aug-17 14-Aug-17David Sanabria Jim Bader

150190457**INT01/02** - Retire Integration v1 VPCs pi-3 integration, sre chore accepted 11-Aug-17 14-Aug-17David Sanabria Vinod Reddy

149968457**Demo** Tear Down "Old" Demo pi-3 demo environment, sre chore accepted 4-Aug-17 14-Aug-17John Simko Vinod Reddy

150233592Integration deployment (ferb-api) pi-3 operational-support chore accepted 14-Aug-17 14-Aug-17John Simko Arpit Patil

148634483 **PERF** Implement infrastructure monitoring using New Relic monitoring, pi-3 new relic implementation, pi-3 perf, sre feature 1accepted 11-Jul-17 14-Aug-17Rasim Bokhari Jim Bader

11

Sprint Backlog (PI 3.5): 64 Points

12

150242570Integration deployment (intake-app) pi-3 operational-support chore accepted 14-Aug-17 14-Aug-17John Simko Srikanth Chowdhary

149968450**Perf v1** Retire "Old" Perf pi-3 perf, sre feature 2accepted 4-Aug-17 15-Aug-17John Simko Vinod Reddy

149947741**S3 Bucket SSL verification Error** bug - moderate, dependency<web, sre bug accepted 4-Aug-17 15-Aug-17Arpit Patil Arpit Patil

148014339**NewRelic** -- Implement APM for Intake digital service "Pre-Int" pi-3 monitoring, pi-3 new relic implementation, sre, system performance feature 2accepted 28-Jun-17 15-Aug-17David Sanabria Ashish Kumar

150280108Demo-Integration deployment pi-3 operational-support chore accepted 15-Aug-17 16-Aug-17John Simko Srikanth Chowdhary

150066958**App Cocina** -- Testing Elasticsearch 5.5 with applications app-cocina, ci/cd, elastic-search, pi-3 engineering feature 2accepted 8-Aug-17 16-Aug-17Sreekar Peddi Arpit Patil

150109507Deploy Rundeck Updates to Demo-Integration on-call, pi-3 demo-integration, pi-3 operational-support, pi-3 unplanned, sre feature 1accepted 9-Aug-17 16-Aug-17Rasim Bokhari Ashish Kumar

150176602**Performance ** Deploy Sanpshot to Performance ci/cd, pi-3 perf, planned pi-3 feature 2accepted 11-Aug-17 16-Aug-17John Simko Vinod Reddy

150235917**User Management** - Eugene Drozddevelopment, identity-administration, pi-3 operational-support, pi-3 unplanned feature 1accepted 14-Aug-17 16-Aug-17David Sanabria Srikanth Chowdhary

150244745Demo-Integration People search Failed pi-3 operational-support bug accepted 14-Aug-17 16-Aug-17David Sanabria Vinod Reddy

150230542**Integration** database connectivity issue bug - critical, pi-3 integration, pi-3 operational-support bug accepted 14-Aug-17 16-Aug-17Arpit Patil Michael Sampath

150232349**Database** Developer Access to DB2 database, development, pi-3 operational-support, pi-3 unplanned, security feature 1accepted 14-Aug-17 16-Aug-17David Sanabria Arpit Patil

150245136Demo-Integration deployment ci/cd, pi-3 demo-integration, pi-3 operational-support feature 2accepted 14-Aug-17 16-Aug-17John Simko Vinod Reddy

150274576**DEMO_INT** Rundeck- Connectivity Issue pi-3 demo-integration, pi-3 integration, pi-3 operational-support bug accepted 15-Aug-17 16-Aug-17David Sanabria Vinod Reddy

150311770re-deploy NewRelic Infra-agents in *new* Preview environment. infra, r1 - application performance monitoring chore accepted 16-Aug-17 16-Aug-17Jim Bader Vinod Reddy

148383309DevOps - Run integration tests on Pre-Int after deploy of cals-api automated-testing, cals, dependency<cals_dev, devops feature 2delivered 5-Jul-17 Leonid Marushevskiy Alex Kuznetsov

148379281DevOps: create continuous integration (CI) for Perry in State Jenkins ci/cd, development, devops, perry, pi-3 development feature 2delivered 5-Jul-17 Leonid Marushevskiy Alex Kuznetsov

150246697Config Mgmt. Logging configuration-management, operations bug delivered 15-Aug-17 Chad Bratton Chad Bratton

149333659**Proxy** Non-prod proxy service architecture, pi-3 non-prod proxy, pi-3 proxy implementation, technical-debt feature 3finished 21-Jul-17 Rasim Bokhari Sreekar Peddi

143616317CALS Core County Connectivity to Sandbox cals, dependency<cals_dev, pi-2: public-sandbox, pi-2: sandbox.cwds.ca.gov feature 3finished 12-Apr-17 John Simko Michael Sampath

150146498On Call Support 3.6 WK 1- Arpit and Srikanth on-cal, pi-3 operational-support, pi-3 unplanned feature 2started 10-Aug-17 Arpit Patil Arpit Patil

150304836Add NewRelic Infra agents in *new*Performance environment. infra, r1 - application performance monitoring feature 1finished 16-Aug-17 Jim Bader Alex Kuznetsov

148387491**Prod** Add reminders for Production Password renewal on Calendar/Add Chore in every other Sprint technical-debt chore started 6-Jul-17 Rasim Bokhari Sreekar Peddi

150116334**Monitoring** Daily Monitoring of Environments/CI improvements, pi-3 monitoring chore started 9-Aug-17 Rasim Bokhari Ashish Kumar

150152580ElasticSearch 5.5.1 In Preint preint feature 2started 10-Aug-17 David Sanabria Arpit Patil

150185722**Preview ** Deploy Intake to Preview ci/cd, planned pi-3 feature 2started 11-Aug-17 John Simko Sreekar Peddi

150246933**IAM** -- Template for Team Roles & Access configuration-management, identity-administration chore started 15-Aug-17 David Sanabria Chad Bratton

150028691Turn on Sealed / Sensitive test data in **preint** and **int** environments feature 1started 7-Aug-17 Gregg Hill Ashish Kumar

149036595New Relic APM Configuration on Perry *preint* monitoring feature 2started 17-Jul-17 Rich Bach Rich Bach

150280322**Case Management Environment** Create a new Environment and CI process for CM case-management, ci/cd, development, pi-3 development feature 2started 15-Aug-17 Rasim Bokhari Arpit Patil

150304244Config Management - Add Jenkins Build Number configuration-management, operations feature 1started 16-Aug-17 Chad Bratton Ashish Kumar

149828071**MGMT** - ELK Stack - create new ELK infrastructurepi-2: implement logstash, pi-2: management environment, pkg-elk-stack, security feature 3started 31-Jul-17 Alex Kuznetsov Alex Kuznetsov

150309306Performance deployment ci/cd, pi-3 operational-support chore started 16-Aug-17 John Simko

150310287Integration deployment ci/cd, pi-3 integration, pi-3 operational-support feature 1started 16-Aug-17 John Simko Vinod Reddy

Team Influence Team Control Organizational / External

High Impact Medium Impact Low Impact

Impediments

13

Unplanned In-flow work stream

Start Finishing Stop Starting

User Story Ready State

Risks & Issues

ITSM/Change Management Process/Tool

Configuration Management - Low

Off hour Production Support

Identity & Access Management Cost for DevOps

14

What’s Next?

POC for automated smoke test for deployments

DevOps Metrics & Dashboard

Improved Measurement & Logging

Access to logs via Web Browser for other Environments

More NewRelic Monitoring of Infrastructure (L1/L2) and

App Performance (L3/L4)

Operational Plan & Buildout of Sandbox/Production

Operations Support (Deployments, IDs, Bugs)

15