Upload
others
View
35
Download
0
Embed Size (px)
Citation preview
© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
[email protected] Conf, April 2018
Speaker: Goutam Tadi
Greenplum for ContainersGreenplum on Pivotal Container Service
1
Cover w/ Image
Agenda
■ Greenplum Architecture
■ Greenplum Data Platform
■ Kubernetes on PCF
■ Greenplum on Kubernetes
■ Demo
■ Q+A
2
MASSIVELY PARALLEL DATA WAREHOUSE
WHAT IS GREENPLUM?
3
Greenplum = Massively Parallel Postgres for Analytics
StandbyMaster
…
MasterHost
SQL
Interconnect
Segment Host
Node1
Segment Host
Node2
Segment Host
Node3
Segment Host
NodeN
LocalStorage
OtherRDBMSes
SparkGemFireCloudObject
StorageHDFS KafkaETL
SpringCloud
Data Flow
Master ServersQuery planning and dispatch
Segment ServersQuery processing and data storage
Interconnect
External Sources & PipelinesParallel loading and streaming
Greenplum Data Platform
ANALYTICALAPPLICATIONS
NATIVE INTERFACES
PIVOTALGREENPLUMPLATFORM
MULTI-STRUCTURED DATA
SOURCES &PIPELINES
Structured Data
JDBC, ODBC
SQL
ANSI SQL
FLEXIBLE DEPLOYMENT
LocalStorage
OtherRDBMSes
SparkGemFireCloudObject
StorageHDFS
JSON, Apache AVRO, Apache Parquet and XML
Teradata SQL
Other DB SQL
Apache MADlib
ML/Statistics/Graph
Python. R,Java, Perl, C
Programmatic
Apache SOLR
Text
PostGIS
GeoSpatial
Custom Apps BI / Reporting Machine Learning AI
On-Premises
NEXT GENERATION
DATA PLATFORM
KafkaETLSpringCloud
Data Flow
MassivelyParallel(MPP)
PostgreSQLKernel
PetabyteScale
Loading
QueryOptimizer(GPORCA)
WorkloadManager
PolymorphicStorage
Command Center
SQL Compatibility
(Hyper-Q)
DS AnalystsIT Dev
PublicClouds
PrivateClouds
FullyManaged
Clouds
5
Faster Deployments… How?
6
Have youHave you...● Ran out of disk space ?
● Been able to provision more than 100 postgres instances in few minutes?
● Faced issues in recovering failures ?
● Faced issues in expanding the database?
7
experienced these before with any database?
RELIABLY DEPLOY AND RUN
CONTAINERIZED WORKLOADS.
WHAT IS PKS?
8
Kubernetes on Pivotal Cloud Foundry
Continuously deliver any app to every major private and public cloud with a single platform.
9
10
Faster Deployments… How?
Greenplum Data Platform + PKS
11
Kubernetes 101
Kubernetes Master
12
Kubernetes Master
kubelet kube-proxy docker
Node
kubelet kube-proxy docker
Node
13
Kubernetes 101
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
14
Kubernetes 101
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
15
Kubernetes 101
Kubectl
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
Storage volumes
16
Kubernetes 101
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
Storage volumes
17
Kubernetes 101
Kubernetes MasterLoad Balancer Service
Node Node
Storage volumes
18
Pod
kubelet kube-proxy docker
Node
Pod
Node
Kubernetes 101
Greenplum on Kubernetes
Node
Pod
mirror
kubelet kube-proxy docker
19
Greenplum Service
Pod
kubelet kube-proxy docker
standby
Pod
kubelet kube-proxy docker
Node
Storage volumes
primary
Pod
kubelet kube-proxy docker
master
Benefits
Greenplum on PKS
20
1. On Demand Cluster Provisioning
PKS
Alana
21
1. On Demand Cluster Provisioning
PKS
Alana
Give me a Greenplum
Cluster
22
1. On Demand Cluster Provisioning
PKS
Alana
Give me a Greenplum
Cluster
Cluster Alana
23
1. On Demand Cluster Provisioning
PKS
Alana
Give me a Greenplum
Cluster
Cluster Alana
gpdb-alana:5432
24
1. On Demand Cluster Provisioning
PKS
Alana
Cluster Alanapsql gpdb-alana:5432
25
1. On Demand Cluster Provisioning
PKS
Alana
Cluster Alanapsql gpdb-alana:5432
Dev Team26
1. On Demand Cluster Provisioning
PKS
Alana
Cluster Alanapsql gpdb-alana:5432
Dev Team27
1. On Demand Cluster Provisioning
PKS
Alana
Cluster Alanapsql gpdb-alana:5432
Dev Team
Cluster Dev
28
1. On Demand Cluster Provisioning
PKS
Alana
Cluster Alanapsql gpdb-alana:5432
Dev Team
Cluster Dev
29
1. On Demand Cluster Provisioning
PKS
Alana
Dev Team
Cluster Alana
Cluster Dev
psql gpdb-alana:5432
psql gpdb-dev:5432
30
2. Service Discovery
master
ContainerPod
We can always discover a container by DNS.
For example, DNS address for different roles:master.greenplum.svc.cluster.localstandby.greenplum.svc.cluster.localsegment-0a.greenplum.svc.cluster.localsegment-0b.greenplum.svc.cluster.local
standby
ContainerPod
Mirrorsegment-0b
ContainerPod
Primarysegment-0a
ContainerPod
31
3. HA without Rebalancing
seg-0a
ContainerPod
seg-0b
ContainerPod
seg-1a
ContainerPod
seg-1b
ContainerPod
Primary
Mirror
32
3. HA without Rebalancing
seg-0a
ContainerPod
seg-0b
ContainerPod
seg-1a
ContainerPod
seg-1b
ContainerPod
Mirror becomes primaryAnd no need for rebalancing.
Primary
Mirror
33
4. Kubernetes Plugins Support : Container Storage Interface
34
seg-1b
ContainerPod
Cloud Storage
4. Kubernetes Plugins Support : Logging
35
seg-0b
ContainerPod
seg-1b
ContainerPod
syslog / stderr syslog / stderr
LoggingAgent Log Store
HEY PKS! GIVE ME A GREENPLUM CLUSTER OF “N”
SEGMENTS
GREENPLUM ON PKS DEMO
36
Deploy Greenplum on PKS
Demo
37
Expand Greenplum on PKS
Demo
39
Greenplum Segment Failover
Demo
41
More Components,StatefulSets,
Custom Resource Definitions
Future Work
43
Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.45