Kubernetes Ingress for AWS Cost Saving

Kubernetes Ingress for AWS Cost SavingTrendMicro Consumer WSE AWSEEric C Huang2017/10/25

goo.gl/VrjuSp

AWS Elastic Load Balancer Is Too

EXPEN$IVE

70% cheaper Kubernetes cluster on AWS

Service Types

● ClusterIP (Default Type)● NodePort● LoadBalancer

ClusterIP

Define a ClusterIP Service

ClusterIP

● port● targetPort

Create a ClusterIP Service via kubectl

● kubectl run echo-server-dev \--image=gcr.io/google_containers/echoserver:1.4 \--port=8080 \--replicas=2 \--namespace=awse

● kubectl expose deployment echo-server-dev-service \--port=80 \--target-port=8080 \--namespace=awse

Verify a Service

● kubectl get services echo-server-dev --namespace=awse -o yaml

● kubectl get deployments echo-server-dev --namespace=awse -o yaml

● kubectl get endpoints echo-server-dev --namespace=awse -o yaml

How to Connect a ClusterIP Service?

● kubectl run nettools \--image=jonlangemak/net_tools \--namespace=default

● kubectl exec nettools-xxx -it bash● curl http://echo-server-dev.awse

○ [service-name].[namespace]

NodePort

NodePort

● port● nodePort● targetPort

How to Connect a NodePort Service?

● curl http://[node ip]:[node port]/

LoadBalancer

LoadBalancer

● port● nodePort● targetPort● CLB (provider: aws)

LoadBalancer

How to Connect a LoadBalancer Service?

● curl http://[CLB]/

LoadBalancer with TLS

● CLB○ HTTPS / TCP + SSL○ Certificate (from ACM)○ TLS Protocol + Cipher

● Route 53 A Alias -> CLB

How to Connect a LoadBalancer Service with TLS?

● curl https://[Route 53 A Alias]/

Ingress

Ingress

● An Ingress is a collection of rules that allow inbound connections to reach the cluster services.

● Ingress Types○ Simple Fanout○ Name Based Virtual Hosting○ TLS

Simple Fanout

Name Based Virtual Hosting

Different Ingress Controllers

● Ingress Controller○ Nginx: https://github.com/kubernetes/ingress-nginx○ Voyager (HAProxy): https://github.com/appscode/voyager/tree/3.2.2○ Træfik: https://docs.traefik.io/user-guide/kubernetes/○ ...etc

Nginx Ingress Controller

● Handle 404:○ nginx-default-backend deployment○ nginx-default-backend ClusterIP service

● Reverse Proxy:○ ingress-nginx deployment○ ingress-nginx LoadBalancer service

Ingress

Verify Ingress

● Simple Fanout○ curl -H “Host:foo.bar.com” http://ingress-nginx.kube-system/foo○ curl -H “Host:foo.bar.com” http://[node ip]:[nodeport]/bar○ curl -H “Host:foo.bar.com” http://[CLB]/foo

● Name Based Virtual Hosting○ curl -H “Host:foo.bar.com” http://ingress-nginx.kube-system○ curl -H “Host:bar.foo.com” http://[node ip]:[nodeport]○ curl -H “Host:foo.bar.com” http://[CLB]

TLS

● AWS CLB Annotations:○ service.beta.kubernetes.io/aws-load-balancer-backend-protocol: “http”○ service.beta.kubernetes.io/aws-load-balancer-ssl-cert:

"arn:aws:acm:ap-northeast-1:xxx:certificate/xxxx"○ service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"○ service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"○ ( service.beta.kubernetes.io/aws-load-balancer-internal: “false” )○ ( service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: “sg-xxx” )

● Route 53 A Alias -> AWS CLB

Ingress + CLB + TLS

Verify Name Based Virtual Hosting with TLS

● curl https://foo.bar.com● curl https://bar.foo.com

Q & A