View
1
Download
0
Category
Preview:
Citation preview
Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant
Darin Lively, Brian Ameling, & John Henley 2018-05-08
© Wal-Mart Stores, Inc. 2018
© Wal-Mart Stores, Inc. 2018
This presentation contains specific configuration and parameters optimized to Walmart's environment. These are likely not the exact configurations or parameters that you will use in your environment. But if they can scale, we know you can, too.
Any reference in this presentation
to any specific commercial
product, process, or service, or the
use of any trade, firm or
corporation name is for
information and convenience
purposes only, and does not
constitute an endorsement or
recommendation by Walmart Inc.
PLEASE NOTE
PHOTO DOWNLOADED FROM WALMART CORPORATE
https://corporate.walmart.com/photos/customers-checkout-after-shopping-walmarts-black-friday-event-on-thursday-nov-26-in-rogers-ark
SCALE W A L M A R T
© Wal-Mart Stores, Inc. 2018
Walmart’s Satellite 6 Journey
PART ONE
PART TWO
PART THREE
PART FOUR
PART FIVE
Walmart & Red Hat Joint Engineering Large-Scale Satellite Operations Satellite Infrastructure Considerations Managing Satellite Clients at Scale Operational Improvements
© Wal-Mart Stores, Inc. 2018
Partnership
● Two companies working together on a common goal to meet and exceed their business demands.
Collaboration ● collaborative relationship with Red hat that can be mirrored by any size customer
○ RHT listens, helps pull together sat releases that benefit the larger customer base (whatever scale you're at)
■ BZs & RFEs are here and the process is open to anyone.
○ Walmart + Red Hat had a positive Red Hat field Engineering experience
■ this is over, but we’re creating some programs to continue the benefits
JOINT ENGINEERING
EFFORT
COLLABORATIVE PARTNERSHIP
REPEATABLE PROCESS, OPEN
TO ANYONE
IMPROVING SATELLITE FOR
ALL
© Wal-Mart Stores, Inc. 2018
Large-Scale Satellite Operations
© Wal-Mart Stores, Inc. 2018
11,700 STORES
150+ DISTRIBUTION CENTERS
28 COUNTRIES
2.3M ASSOCIATES
WHAT WALMART BUSINESS SCALE LOOKS LIKE
© Wal-Mart Stores, Inc. 2018
270M CUSTOMERS/WEEK
234K PACKAGES
182K PACKAGE
DOWNLOADS/ HOUR PEAK
46 ACTIVATION
KEYS
22 CAPSULES
60K HOSTS
64 CONTENT VIEWS
9 LIFECYCLE
ENVIRONMENTS
3.9K PULP REPOS
WHAT WALMART SATELLITE SCALE LOOKS LIKE
1 SATELLITE
© Wal-Mart Stores, Inc. 2018
© Wal-Mart Stores, Inc. 2018
Satellite Infrastructure Considerations
© Wal-Mart Stores, Inc. 2018
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
DC1
DC2
DC3
...
DC1
DC2
DC3
...
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
SECURE
DC1
DC2
DC3
...
DC1
DC2
DC3
...
DC1 FOREMAN-
PROXY SERVICES
& DMZ HOST CAPSULE
Secure
SECURE
DC1
FOREMAN- PROXY SERVICES
& DMZ HOST CAPSULE
Secure
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
DC1
DC2
DC3
...
5 6
Hiera customizations
Apache overrides
Limits files
Kernel tunables
Postgres
Logrotate
Pulp concurrency
CUSTOMIZATIONS
© Wal-Mart Stores, Inc. 2018
HIERA CUSTOMIZATIONS
/etc/foreman-installer/custom-hiera.yaml
© Wal-Mart Stores, Inc. 2018
APACHE OVERRIDES
/etc/httpd/conf.d/zzz-custom-overrides.conf
© Wal-Mart Stores, Inc. 2018
LIMITS FILES
/etc/systemd/system/*
© Wal-Mart Stores, Inc. 2018
KERNEL TUNABLES
/etc/sysctl.conf
© Wal-Mart Stores, Inc. 2018
POSTGRES
/var/lib/pgsql/data/postgresql.conf
© Wal-Mart Stores, Inc. 2018
LOGROTATE
© Wal-Mart Stores, Inc. 2018
PULP CONCURRENCY (Reduced Published Times)
© Wal-Mart Stores, Inc. 2018
Managing Satellite Clients at Scale
© Wal-Mart Stores, Inc. 2018
USING PUPPET TO AUTOMATE CLIENT SIDE OPERATIONS
PACKAGING FILES SERVICES
PUPPET MODULES
© Wal-Mart Stores, Inc. 2018
USING PUPPET ENTERPRISE TO ROLLOUT IN SCALE
ENTERPRISE CONSOLE
CLASSIFICATION
GROUPS
MIGRATION LOGIC
6 5
PUPPET ENTERPRISE
© Wal-Mart Stores, Inc. 2018
MODIFIED PUPPET MODULE TO RUN MULTIPLE LINUX CLIENTS
UPSTREAM SOURCE
SUSE ENTERPRISE LINUX
© Wal-Mart Stores, Inc. 2018
HOW SCALE IMPACTED ROLLOUT
WHERE: WHAT: WHEN: HOW:
Started in one location Touched 60,000 boxes Delivered on an aggressive timeline Rolled out successfully
© Wal-Mart Stores, Inc. 2018
Operational Improvements
© Wal-Mart Stores, Inc. 2018
Walmart had challenges, but they turned them into efficiency gains with customization
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
Added automation to roll out to a huge number of clients, multiple locations, small team
1/5 1/6 1/8 1/9 1/10 1/11 1/12 1/13 1/16 1/17 1/18 1/19 2/14 3/1 3/3
BUSINESS CHALLENGE
Need to get clients migrated in a reasonable amount of time. This required automation.
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
BUSINESS CHALLENGE
During client registration, Satellite defaults out of the box are not optimal for large scale, and they get maxed out.
Added randomization to even the distribution
BEFORE
AFTER
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
BUSINESS CHALLENGE
Users need new content available faster.
Performed customizations to dramatically reduce publish times
Initially impossible to publish all in one day (manually through the UI)
Scripted – still more than 24 hours because of capsule syncs
Added shared filesystem = 12 hour publish
Multi-threaded script = 4 hour publish
6.3 = 1.5 hour publish
OVER TIME
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
PUBLISH TIME (HOURS)
© Wal-Mart Stores, Inc. 2018
BUSINESS CHALLENGE
Walmart needed version control. Template changes from one user changed the template for everyone.
Moved to fully version-controlled template process to improve workflow
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
BUSINESS CHALLENGE
Need visibility into the operational health of the infrastructure
© Wal-Mart Stores, Inc. 2018
collectd > graphite > grafana
Ansible playbook to redeploy grafana dashboards
IMPORTANT METRICS TO COLLECT
METRICS
standard cpu/memory/disk/network
foreman tasks active (per task type)
foreman tasks per minute (per task type)
katello event queue
qpid queue depths
dynflow plans
dynflow orphaned execution plans
pulp tasks running
pulp tasks state
pulp repo count
postgres connections
postgres query length
apache processes
passenger processes
apache scoreboards
capsule apache busy servers
Scalability
Build capacity for IT of any size
Collaboration
Share your ideas and optimizations
Infrastructure
Create a reference architecture then customize
SCALING SUCCESS
Migration
Know your environment and track key metrics
© Wal-Mart Stores, Inc. 2018
THANK YOU plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
© Wal-Mart Stores, Inc. 2018
Recommended