Secure + DynamicApp Config at GapTech

Nivesh GopathiDirector Platform ServicesGapTech

Spencer GibbSoftware EngineerPivotal

Introduction• What is GapTech

• In Development / Pilot

• Disclaimer: Not an endorsement

Overview• Business Problem:

– App Secrets– Dynamic Config– SCCS

• Challenges• Solve:

– App Secrets– Dynamic Config– Gotchas

• Next Up• Q & A

App Secrets• Users, passwords, keys, tokens• Requirements:

– Distributed Platform capability– Secure (at rest & in motion)– Consistent– Deep Access Control– Audit Trail– Spring boot/cloud (primary)– Non JVM (secondary)

Dynamic Config• Feature toggles/weights

– Branch by abstraction– Kill switches

• Requirements:– Distributed Platform capability– Timely (in seconds)– Consistent (across all live and future instances)– Deep Access Control– Audit trail– Spring boot/cloud (primary)– Non JVM (secondary)

Spring Cloud Config Server• Server and client-side support for managing externalized config• Scalable• Maps identically to Spring Environment and PropertySource

abstractions• Leverage @ConfigurationProperties or @RefreshScope• Default back-end: Git• Details at http://projects.spring.io/spring-cloud/spring-cloud.html#

_spring_cloud_config• See Clint Checketts’ Implementing Config Server And Extending It talk

Challenges• Git is a shared log (mutable), it is not (and does not pretend to be) an audit

trail• Shows timestamps for local commits• Not built for secrets management• Hot, cross-DC Git replication strategy (tier 1 Prod system)• Repo lockdown policies and processes• Spring cloud sidecars to support non JVM apps• See the

Moving From Monolithic Architecture To Spring Cloud And Microservices talk

Solve• Let’s build a proprietary solution with a short lived team to fill in the gaps

• Let’s collaborate with the broader community on a turnkey solution, for the community

Vault• Hashicorp Vault

– Securely store, access secrets (with leasing)– Detailed audit log

• Sanitized, simple logs– Token based fine grained access control– LDAP integration, dynamic secrets for some systems– HA Cluster (with Consul back-end)– Can be sealed on system compromise– https://github.com/hashicorp/vault– https://www.vaultproject.io/– See Mark Paluch’s Managing Secrets At Scale talk

App Secrets - SCCS Integration• Spring Cloud Config Vault integration

– Vault as an SCCS Environment repository– Supports properties style and nested keys for Vault– https://github.com/spring-cloud/spring-cloud-config

• App passes in Vault access token• Can use multiple back-end repos (Git and Vault)• Can use with @ConfigurationProperties• Make sure you mask secrets on env end-point

Deploy Time

Startup Time

Consul• Hashicorp Consul

– Hierarchical key/value store (among other things)– Distributed HA, DC aware– Highly scalable (health checks with gossip protocol)– https://github.com/hashicorp/consul– https://www.consul.io/– See Spencer Gibb’s Spring Cloud with Consul and Vault talk

Dynamic Config - SCCS Integration• In progress• Using @RefreshScope

– /refresh end point will blow away proxy, lazy load config on next call– Constructor may get called multiple times– Need to manage lifecycle

• Spring Cloud Config Consul watcher– Server side watch for Consul paths/keys– Can define watch polling times– Triggers /refresh end-point on event– https://github.com/spring-cloud/spring-cloud-config

Update time

Gotchas• Consul DC awareness and quorum (use replicate – one way)

• Vault caching (across clusters)

• Multi-tenancy – Consul over-notification

• Config versioning strategy

• Token rotation may need a deployment

Next up• Consul watcher extension• Vault HA improvements and UI• Smart spring cloud config clients for non-JVM apps

– Python– Node.js– .NET

• PCF Tile

Q & A

Biography of the PresenterNivesh Gopathi is the Director of Platform Services at GapTech. Prior to joining Gap, he led automation and delivery teams at enterprises and upstarts.

Nivesh has a passion for enabling CD and DevOps. He has architected and delivered large scale platform, cloud and container transformations on critical revenue generation digital assets. He holds an MS in CS from the University of Missouri.

He can be reached via https://www.linkedin.com/in/niveshgopathi

Check out GapTech careers!

References• Spring Cloud Config

– https://github.com/spring-cloud/spring-cloud-config– https://docs.pivotal.io/spring-cloud-services/config-server/

• Vault– https://github.com/hashicorp/vault– https://www.vaultproject.io/

• Consul– https://github.com/hashicorp/consul– https://www.consul.io/