Upload
paypal
View
1.183
Download
1
Embed Size (px)
Citation preview
©2015 PayPal Inc. Confidential and proprietary.
PayPal’s Private Cloud @ Scale
Jigar Desai• @jigardesai • December 09 2015
©2015 PayPal Inc. Confidential and proprietary. 2
Table of contents Introduction
OpenStack@PayPal
Our Journey
Lessons Learned
Future
Q&A
3©2015 PayPal Inc. Confidential and proprietary.
Introduction to PayPalA Technology Perspective
©2015 PayPal Inc. Confidential and proprietary. 4
About PayPalPayPal is a leading technology platform company that enables digital and mobile payments on behalf of consumers and merchants worldwide. We put our customers at the center of everything we do. We strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on any platform and through any device.
©2015 PayPal Inc. Confidential and proprietary. 5
Architecture @ PayPalRobust Infrastructure, Reusable Platforms, Payments Operating System, Delightful Experiences
Cust
omer
Facin
gIn
frast
ruct
ure
Analytics Platform
HadoopTeradata FPTIBI
Technology Platform
Framework
Data Access
Tools MediaMessagingSOA CrawlerCache
Infrastructure & Operations Layer
Power Data Center Hardware Network Database OperationsToolsCloudStorage Monitoring
Experience
Payments Operating System
LoginIdentity EmailCart CouponsPayment Ship SupportRisk InvoiceWallet POS Notify Merchant
6©2015 PayPal Inc. Confidential and proprietary.
OpenStack@PayPal
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
x 100K Cores
x 10 PB Storage
OpenStack Cloud
> 10K Physical Servers
x 10K VMs
x 1000engineers
x 10M Lines
x 1000 Releases/year
> 1000 Services
x 1000 Builds/day
x 1000Deploy/day
Key StatisticsDeveloper Statistics
9 Availability Zones
3 Regions
©2015 PayPal Inc. Confidential and proprietary. 8
Our OpenStack Journey
2012 2013
2014
2015
• OpenStack POC with 16 servers
• Built 2 AZs with OpenStack Essex release (96 hypervisors)
• 5 apps in production
• Migrated from Essex to Folsom
• Built 3 more Azs • 350 apps in
production• Support for dev
and QA environments
• Upgrade to Havana
• Migration of Nova to Neutron
• 1000+ apps in production
• 100% of web and mid tier apps on OpenStack
• Deployed our largest AZ with 2500 nodes using Nova cells
• Kilo upgrade in progress
©2015 PayPal Inc. Confidential and proprietary. 9
The Current Stack
Datacenters (Geographic regions, Availability Zones)
OpenStack Cloud (Infrastructure as a Service)
Platform as a Service
Web/Mid Tier Applications
Common Infrastructure
Common Platforms and Services
• 100% of web and mid tier applications on OpenStack
• Proprietary PaaS on top of OpenStack to automate E2E application development life cycle
• Support for polyglot applications
• Test and production environments available on-demand
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
Steps to Production 25 Manual Steps Automated
Time to Market
QA Deployment
Build Time
Prod Deployment
Infrastructure
2 years back
Now
8 Weeks
5 Hours
2.5 Hours
4 Hours
1 Month
1 Day
30 Minutes
5 Minutes
< 30 Minutes
30 Minutes
Impact On Life Of A Developer
11©2015 PayPal Inc. Confidential and proprietary.
Challenges And Lessons Learned
©2015 PayPal Inc. Confidential and proprietary. 12
Challenges And Lessons Learned• Operating OpenStack at scale requires building an entire ecosystem of tools and back office
functionality like monitoring, configuration management, log aggregation, hardware remediation etc.
• We faced several control plane reliability issues as our deployment grew from several hundreds of compute to several thousands specially around RabbitMQ and services interaction
• Scaling OpenStack to 10s of thousands of nodes is quite challenging specially around Neutron and RabbitMQ.
• Security is evolving but not mature enough yet • Upgrading to a newer version of OpenStack is not straightforward specially with internal
patches and takes months for larger deployments with multiple regions and availability zones
• Distributed nature of OpenStack results into challenges with debug-ability of system and identifying the root cause
• OpenStack is a journey and commitment. It’s success depends on building a talented team of engineers that takes pride in leveraging and contributing to OpenStack community
13©2015 PayPal Inc. Confidential and proprietary.
The Future
©2015 PayPal Inc. Confidential and proprietary. 14
The Future Stack
Datacenters (Geographic regions, Availability Zones)
OpenStack Cloud (Infrastructure as a Service)
Web and Mid Tier Applications
Common Infrastructure
Common Platforms and Services
• New PaaS is being built using Mesos and Docker on top of OpenStack
• Goal is to achieve industry leading datacenter efficiency and utilization
• High application resiliency against infrastructure failures
• Hybrid cloud POC • PaaS support for
stateful applications
Public Clouds
Questions?Jdesai at paypal dot com