View
398
Download
0
Category
Preview:
Citation preview
Jez Halford
Jez Halford
Freelance web development consultant
jezhalford.com
@jezhalford
● World’s largest designer of semiconductors
● Over 60 billion ARM chips shipped to date
● Began moving into server CPUs a few years ago
Shoot for the moonHewlett Packard EnterpriseMoonshot
● Very high server density
● More efficient cooling
● Built in fault-tolerance
Moonshot
● 45 cartridges, supporting up to4 servers per cartridge
● 4 redundant power supplies
● 2 ethernet switches
● 5 cooling fans
● iLo Management interface
Shoot for the moon
CPU Eight 64 bit cores @ 2.4GHz
Memory 64GB
Storage 120GB / 240GB / 480GB SSD
Network 2x 10 Gigabit NICs
ProLiant m400
“Drinking our own Champagne”
Goals
● Robust, highly available website
● Scalable to handle very high load
● Use standard, generally available technologies where possible
Milestones
● Modify the PHP app from Windows-specific to platform agnostic
● Design and prove the server architecture
● Go live
Windows to anything
● Remove hard-coded paths
● Improve autoloading
Windows to anything
● Config injection over environment detection
if (gethostname() == ‘qa-box’) {
// set environment for QA
}
if (getenv(‘APP_ENV’) == ‘qa’) {
// set environment for QA
}
Server Architecture
PHP on ARM
sudo apt-get install php7
High Availability
● Two NICs per cartridge
● Bonded to work as one
● Allows one chassis switch OR one NIC to fail and operation to continue
http://www.unixmen.com/linux-basics-create-network-bonding-on-ubuntu-14-10/
Runningon multiple servers● Load balancing● Filesystem● Application cache● Session storage● Log aggregation● Data persistence
Runningon multiple servers● Load balancing● Filesystem● Application cache● Session storage● Log aggregation● Data persistence
Load Balancing
● Paid-for version of nginx● Support & extra features
http { upstream myapp1 { server srv1.example.com; server srv2.example.com; server srv3.example.com; }
server { listen 80;
location / { proxy_pass http://myapp1; } }}
Load Balancing
ARM Nginx Node
ARM Web Node
ARM Web Node
ARM Web Node
Web
Bonded IP address
Load Balancing
F5
ARM Nginx
Node A
ARM Nginx
Node B
ARM Web Node A1
ARM Web Node A2
ARM Web Node A3
Web
ARM Web Node B1
ARM Web Node B2
ARM Web Node B3
Moonshot Chassis A
Moonshot Chassis B
Caching statics at the load balancer● Nginx respects Cache-Control headers● In our setup these come from Apachehttp {
upstream myapp1 { server srv1.example.com; server srv2.example.com; server srv3.example.com; }
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;
server {listen 80;
location / { proxy_pass http://myapp1; proxy_set_header Host $host; proxy_cache STATIC; proxy_cache_valid 200 1d; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; }
}
Runningon multiple servers● Load balancing ✔● Filesystem● Application cache● Session storage● Log aggregation● Data persistence
Filesystem
● Both are free, open source distributedfilesystem solutions
● Multiple nodes work together to form a single mountable filesystem
● Generally slow at writing, but great forstoring PHP source and static assets
Filesystem
● Copy files to each node at regularintervals
● Nominated “master” node is the source
● Has a replication lag
Filesystem
We used existing, external and mountable storage ARM already had
Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache● Session storage● Log aggregation● Data persistence
Application Cache
● 64GB on each node!
● Potentially 2.8 Terabytes per 45 node moonshot chassis
Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage● Log aggregation● Data persistence
Session Storage
Options!
Session Storage
● Zero configuration
● Slow access
● Persistent
● Needs either a shared filesystem or sticky sessions
Session Storage
● Simple
● Very easy configuration
● Non-persistent
● It’s for caching, not sessions
Session Storage
● Fast
● Persistent
● Generally fault-tolerant with Redis Cluster
● Cannot always guarantee consistency
Session Storage
Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation● Data persistence
Log Aggregation
● Lots of apps on lots of servers
● Lots of logs generated
● Debugging is hardif we have to look inlots of places
Log Aggregation
rsyslog● Aggregate logs from MySQL, Apache, Nginx across
several servers...
● ...into one file per app on a “management” server
Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation ✔● Data persistence
Data Persistence
Data Persistence
● Active / Active, not Master / Slave
● True horizontal scalability
● High availability
A PXC Cluster
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
A PXC Cluster
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
ARM Web Node
Split Brain
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
ARM DB Node 6
Split Brain
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
ARM DB Node 6
Split Brain
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
ARM DB Node 6
Consistency over availability
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Networking failure
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Networking failure
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
More than half the nodes fail
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
More than half the nodes fail
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Single node failures
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Rejoining
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Mixed failures
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
Multiple node failures
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
PXC Startup
ARM DB Node 1
sudo service mysql bootstrap-pxc
Cluster size: 1
PXC Startup
ARM DB Node 1
ARM DB Node 2
sudo service mysql start
Cluster size: 2
PXC Startup
ARM DB Node 1
ARM DB Node 2
ARM DB Node 3
sudo service mysql start
Cluster size: 3
PXC Startup
ARM DB Node 1
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
sudo service mysql start
Cluster size: 4
PXC Startup
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
sudo service mysql start
Cluster size: 5
PXC
● Very good horizontal scaling
● An absolute nightmare in cascading failures
● Those failures are very rare
Load balancing the database
● How does the application know which DB node to talk to?
● HAProxy! A TCP Load Balancer
Load balancing the database
ARM Web Node A1
PHP
HAProxy
127.0.0.1:3306Individualdatabasenodes...
An individual web node:
ARM Web Node A1
Load balancing the database
ARM Web Node A2
ARM Web Node A3
HAProxy
HAProxy
HAProxy
ARM DB Node 3
ARM DB Node 5
ARM DB Node 1
ARM DB Node 2
ARM DB Node 4
Load balancing the database
● Each HAProxy instance carries out regular health checks on each node
● When a node fails a check it is removed from the pool until it passes again
● The health checks need to verify PXC cluster state, as well as responsiveness to incoming connections
https://github.com/olafz/percona-clustercheck
Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation ✔● Data persistence ✔
#
F5
Bonded IP address
ARM Nginx
Node A
ARM Nginx
Node B
ARM Web Node A1
ARM Web Node A2
ARM Web Node A3
Web
ARM Web Node B1
ARM Web Node B2
ARM Web Node B3
ARM DB Node 1
ARM DB Node 5
ARM DB Node 2
ARM DB Node 3
ARM DB Node 4
PXC ClusterMoonshot Chassis A
Moonshot Chassis BMoonshot Chassis C
F5 ARM Web
Node A1
Web
#
ARM Web
Node A2ARM Web
Node A3
ARM Web
Node B1
ARM Web
Node B2
ARM Web
Node B3
ARM DB
Node 1
ARM DB
Node 5
ARM DB
Node 2
ARM DB
Node 3
ARM DB
Node 4
PXC Cluster
Bonded IP address
ARM Nginx
Node A
ARM Nginx
Node B
ARM Web
Node A1
#
ARM Web
Node A2ARM Web
Node A3
ARM Web
Node B1
ARM Web
Node B2
ARM Web
Node B3
ARM DB
Node 1
ARM DB
Node 5
ARM DB
Node 2
ARM DB
Node 3
ARM DB
Node 4
PXC Cluster
Bonded IP address
ARM Nginx
Node A
ARM Nginx
Node B
Moonshot Chassis A
Moonshot Chassis B
Moonshot Chassis C
Moonshot Chassis D
● Lower total cost of ownership
● Highly fault tolerant
● Uses standard technologies
What next?
● Several providers of hosted ARM servers
AaaS{ ARM-as-a-Service }
scaleway.com
armasaservice.com
mininodes.com
unixy.net
nx-box.net
Hewlett Packard Enterprise Moonshot
https://www.hpe.com/uk/en/servers/moonshot.html
Thanks for listening
joind.in/talk/01006
@jezhalford jezhalford.com
Recommended