Haufe #msaday: "Building a Microservice Ecosystem"

Building a microservice ecosystem

Daniel Bryant@danielbryantuk

OpencRedo

Today• What is a ‘microservice ecosystem’?

• Local dev of services becomes challenging

• Think about your build pipeline(s)

• Testing requires a paradigm shift

• Push to production as early as possible

24/11/2016 @danielbryantuk

@danielbryantuk• Chief Scientist at OpenCredo, CTO at SpectoLabs

ü Transforming organisations through technology and teams

ü Agile, Lean, Architecture, CI/CD, DevOps

ü Microservices, cloud, Containers, Java, Go, Docker, Kubernetes

• London Java Community Associate

• Adopt OpenJDK and JSR

• InfoQ Editor, DZone MVB, VOXXED, O'Reilly


My Biggest Claim to Fame in Microservices?


Seriously Though...…


skillsmatter.com/skillscasts/7004-our-journey-to-world-gifting-domination-how-notonthehighstreet-com-embraced-docker

So, What is a microservice?

“Loosely coupled service oriented architecture with bounded contexts”

Adrian Cockcroft

“Applications that fit in your head”James Lewis


Why? The move towards Cloud-native

• Hypothesis-driven business (and development)

• Microservice architecture (and 'Micro' teams)

• Devops mindset/culture (shared, learning/feedback & Mechanical sympathy)

• Continuous delivery

• Automated, self-service platform

Hat tip @caseywest


Exploring the Ecosystem


Adrian Cockcroft’s Thoughts


www.slideshare.net/adriancockcroft/microxchg-microservices

Alexis Richardson’s Thoughts


gotocon.com/goto-london-2015/#!#schedulePopupExtras-7011

wikibon.com/wp-content/uploads/container_implementations.png

Technology Choices


24/11/2016

What do I mean by ‘ecosystem’?• Build– Local development, pipelines and integration

• Test– From local integration to End-to-end

• Deploy

• Operate

• Observe– Monitoring/logging/alerting


Don’t forget about (RE)ArchitecturE Complexity



www.infoq.com/news/2015/04/raffi-krikorian-rearchitecting

What do I mean by ‘ecosystem’?• Build– Local development, pipelines and integration

• Test– From local integration to End-to-end

• Deploy

• Operate

• Observe– Monitoring/logging/alerting


Build


Developing Locally: The Basics

• GitHub’s Boxen (Puppet)

• Pivotal’s Sprout (Chef)

• Mac-dev-playbook (Ansible)

• Hashicorp Vagrant

• Docker Compose (Docker machine/native)


Developing Locally

• The naive approach

– Replication of env per service

– …and dependencies and data stores and...

– Soon gets crazy

• Local profiles + mocking/stubbing

– Spring profiles + Mockito etc


Developing Locally

• Service virtualisation– Hoverfly, Mountebank, Wiremock (Saboteur)

• ‘Production-in-a-box’ (IFTTT)– Docker Compose, Vagrant, cf_nise_installer

• Environment leasing– Create your own env (e.g. Hailo)


Developing Locally

www.opencredo.com/2015/09/20/working-locally-with-microservices/24/11/2016 @danielbryantuk

Create a Pipeline


Key lesson learned (the Hard Way)

• Push through to production as early as possible!

• Detect and fix issues

– Technology

– Process

– People


Test


Always Remember...

24/11/2016 @danielbryantukblog.bbv.ch/2012/06/13/acceptance-test-driven-development/

Microservice Testing Basics

• Toby Clemson’s article

martinfowler.com/articles/microservice-testing


Microservice Testing Basics

• My article

https://www.specto.io/blog/recipe-for-designing-building-

testing-microservices.html


(multiple) Pipelines• Big-bang release

– Beware of rubber stamping (distributed monolith)

– ‘semver’ if you must (semver.org) e.g. 1.2.1

• Gated release– Critical path testing in Stage

• Single service Continuous delivery

– Consumer-based contracts

– Backwards compatibility


Service-level Integration Testing

• Contracts

– Pact-JVM github.com/DiUS/pact-jvm

– PACT broker

github.com/bethesque/pact_broker

– Examples:

github.com/mstine/microservices-pact


My Opinions• BDD critical paths throughout application

– Including API journey

• Contract tests (failure is a conversation)

• BDD services API (e.g. Serenity BDD)

• Component test and unit test (as normal)– Maven surefire/failsafe


Final Words on Testing• Don’t forget the ‘ilities”

• Security / reliability– ZAP (from the OWASP team)

– github.com/continuumsecurity/bdd-security

• Performance / scalability

– Jmeter (Jenkins Performance plugin)

– Gatling

– flood.io


Deploy


Separate Deploy and Release

• Feature flags– Difficult at scale (and distribution)

– Enable at ingress

• Incremental (phased) rollout

• Canary vs blue/green

• Avoid datastore migrations (if possible)


Centralise Configuration

• Consul & consul-template

• Etcd/ZK & confd

• Netflix Archaius

• Spring Cloud config

• Watch for application rollback!


Operate


Building Blocks

• HashiCorp Terraform

– VMWare? cloud-init and vcloud-tools

– “Boot my secure government cloud”

• “CAPS”– Chef, Ansible, Puppet, SaltStack

– Automated sysadmin

• Docker / Ami / RPM / JAR

– Machine vs Os/lang artifact

• Standardise on an OS

– Amazon Linux vs mainstream distros


DevOps and Programmable Infrastructure

• Devops involves programming…

• Introduce SOLID principles

• Good CI/CD principles

– Gitflow etc

– Testing


Service Discovery

• External

– HAProxy / nginx / ELB etc

• Client-side

– Netflix Ribbon (with Prana)

– ‘Baker Street’ (extending SmartStack)

– srv-router

• Kubernetes and CF are good to go


External


Client-side


Observe


When bad things happen, people are always involved

24/11/2016 @danielbryantuk|@oakinger

Monitoring and People

24/11/2016 @danielbryantukwww.infoq.com/news/2015/06/too-big-to-fail

Start with the Basics

• Health checks– Coda Hale (DropWizard) Metrics

– Spring Boot actuator

• KPIs for apps (and business)– Assertions / invariants

– Throughput

– Queue length


Logging

• What every engineer should know

• 10 Tips for Proper Application Logging

• ElasticSearch-Logstash-Kibana (ELK)

– Buffer/proxy log sending or…

– Mount directory into container


Monitoring• Push

– Spring Boot actuator e.g. InfluxDbExporter

• Pull – E.g. Telegraf (shout to Tareq Abedrabbo)

• InfluxDB vs prometheus vs graphite vsopentsdb

• Information radiators– Aggregate vs individual


Aggregation: Sick Cattle, Not Sick Pets

Opentracing & OpenZipkin


Problems?

• Rob Ewaschuk’s “Philosophy on Alerting”

• Brendan Gregg’s USE method

– “check utilization, saturation, and errors.”

• “DevOps Troubleshooting”– Kyle Rankin


Let’s wrap this up...


Summary• development of 3+ services is challenging

• Think about your build pipeline(s)

• Testing requires a paradigm shift

• Operations requires much more automation

• Push through to prod as early as possible


Bedtime reading


THANKS...

@danielbryantuk

[email protected]

http://muservicesweekly.com/

(Credit to Tareq Abedrabbo for inspiration/guidance)


Logging vs monitoring

• Logging

– Primarily used post-incident

– Machine readable

– Hard to do right

– Noise/signal

• Monitoring

– Useful in Real time during incident

– Trend analysis

– Easier to do
