Embed Size (px)
Citation preview
OpenStack-‐Ansible and AutomataPublic vs Private Cloud Performance
Hugh Ma | Michael O’Rourke
2
Flex | Ciii A Global Server and Rack Integration BusinessDeploy
Node, rack & multi-‐rack level solutions with native hardware only, native & third party hardware for hybrid & JDM solutions
OTD to Commit, BTO/ CTO Systems Integration
Consistently at 99%
Universal Test System (UTS)
Automated logging, analysis & reporting
Lead Time to Customer To request: 1 – 4 Days Capability
Order Acknowledgement Time
4 Hours
SMI with Suppliers Consistently at >65% of spend
Cork, Ireland
Brazil
Guadalajara, MX
Milpitas, CA
Austin, TX
Columbia, SC
Penang, Malaysia
Zhuhai, China
Singapore
Hungary, Campus
India
3
Flex | Ciii Types of Product Engagements
• Rack level Interoperability• Cable management and
Diagrams• Functional cabinet to fully
tested cluster assembly & test
• Customer/Manufacture/provision of all fibre optic, data & power cables
• System/cluster level stress testing, including R/W activity, latency measurements, thermal performance, etc.
• Component level design• Leverage existing PCBA• Design new mid-‐plane• Design new risers• Custom Bezel design• System level validation• Thermal / EMC / Safety
• Leverage Industry Standard offerings
• Solutions Design & Engineering
• Define Best of Breed, Price for Performance & Life cycle management
• Leverage existing system level validation*
• OEM Branding and Packaging
COTS Integration
Cloud and Data Center
“Light” Customization
“Heavy” Customization
• COT’s Integration +• System Modifications to
OEM needs, cooling, application drive, carriers
• Non AVL component selections
• BIOS defaults change• BMC defaults change
Complexity
Time to Market
4
Rack engineering workflow
Architect
CloudLabs Engineering flex1
Integrate Test Manufacture32 5
Optimize4
Services
Support6
Automata and FlexConfig
5
Why use Ansible as the framework foundation?
Ansible• Agent-‐less• OpenStack Support• Network Switch Support• Real-‐time Job Events• Community
Ansible Tower• Role-‐based Access• Job Scheduling• Credentials Management• SCM Support• Inventory Management• Surveys
6
Server Provisioning with Stacki
ü Open Sourceü Simplified Baremetal Provisioningü Mac address targeted PXE Bootingü Customizable Local CentOS and Ubuntu Repositoriesü Hardware & Software RAID Supportü NIC Bondingü Post-‐Install Configurationü http://www.stacki.com/
7
OpenStack -‐ Ansible
• Using Ansible to deploy OpenStack• First released in April 2015 (Kilo)• Currently on Mitaka (Stable) and Newton (Master branch)• OpenStack-‐Ansible deployment steps:
8
OpenStack Cluster Configuration
Deployment Host
9
OpenStack-‐Ansible Deployment Process
• Deployment Host• Executes the deployment playbooks via Ansible• Holds all the necessary deployment configuration files• Can also be a Target Host!
• Target Hosts• All the OpenStack cluster nodes (Controllers, Compute, and Storage)• Needs proper network configuration and dependency packages
• Deployment Configuration• Define the proper network bridges and container networking variables. (VLAN, VXLAN, FLAT, etc…)• Define node roles.
• Deployment• Run the playbooks: setup-‐hosts, setup-‐infrastructure, and setup-‐openstack.
10
OpenStack-‐Ansible – Example Network Configuration – Single Interface Setup
Bridge Interface Network Bridge-‐port (VLAN-‐id)
Br-‐mgmt 172.29.236.0/22 Eth2.41
Br-‐vxlan 172.29.240.0/22 Eth2.42
Br-‐storage 172.29.244.0/22 Eth2.43
Br-‐ex 10.118.112.0/20 Eth2.44
11
OpenStack-‐Ansible Experience
• Pros:• Well-‐Documented• Installation and reconfiguration is modular• Very close to upstream OpenStack• Responsive community | #openstack-‐ansible @ irc.freenode.net• ”Mistake friendly” thanks to containers• Supports popular 3rd party network and storage vendors (PLUMgrid, CEPH, etc…)
• Cons:• Requires manual host imaging and initial configuration• Requires manual network configuration
12
Automata Workflow
12
13
Benchmark Top-‐Level Play
!!!"!"name:"Running"Common"Ops""""hosts:"all"""serial:"10"""become:"yes"""pre_tasks:"""""!"include:"roles/common/tasks/clean!up.yml"""""""""ignore_errors:"yes"""roles:"""""!"common""!"name:"Benchmark"Suite""""hosts:"client"""pre_tasks:"""""!"local_action:"command"uuidgen""""""""register:"job_id""""""""run_once:"true"""""!"set_fact:"""""""""result_path:"'/tmp/result!{{"job_id.stdout"}}/'""""roles:"""""!"passmark"""""!"stress"""""!"mprime"""""!"reboot"""post_tasks:"""""!"local_action:"""""""""module:"slack"""""""""token:"xxx"""""""""msg:""Server"Validation"Suite"by"@{{"operator"}}"on"{{"ansible_date_time.iso8601"}}"completed!"Job"ID:"{{"job_id.stdout"}}""""vars:"""""num_hosts:""{{"groups['client']"|"reject('equalto',"'Ansible!Control')"|"list"|"length"}}""""""operator:""blackbox""
!
Common Tasks
Generate Job ID
Benchmark Roles
14
Benchmark Tasks
These tasks show:• Execution of a Benchmark • Completion with proper results
15
Parsing Tasks
These tasks contain the logic for parsing result files of a benchmark.!!!"!"name:"Cat"STDOUT"of"Execution"""command:"cat"/tmp/test/{{"ansible_hostname"}}_{{"benchmark_name"}}.out""!"name:"Run"Parser"on"Result"File"and"Generate"CSV"""script:"parser.py""{{"benchmark_name"}}""/tmp/test/{{"ansible_hostname"}}_{{"benchmark_name"}}.out""!"find:""""""paths:""/tmp/test""""""patterns:""*.csv""""register:"found_csv""!"name:"Fetching"Result"CSV"File"to"Control"Host"""fetch:"src={{"item['path']"}}"dest={{"result_path"}}"flat=yes"""with_items:""{{"found_csv.files"}}"""!"name:"Fetching"Raw"Output"to"Control"Host"""fetch:"src=/tmp/test/{{"ansible_hostname"}}_{{"benchmark_name"}}.out"dest={{"result_path"}}"flat=yes""!"include:"system!identity.yml"""become:"yes""!"name:"Write"out"Config"to"File"""local_action:"copy"content={{"time"}}"dest={{"result_path"}}config.ran"""run_once:"true"
!
Run Parser.py
Fetch Results
Fetch System Logs
16
Demo Video
17
The Ansible Experience
• The “Good”• Easy to train other members of the team on usage• Intuitive playbook structure• Flexibility in custom modules and plugins• Repeatable Benchmarking• Cuts down manual work!
• The “Kinda Bad”… but not really• Error messages are sometimes obscure• To OpenSSH or to Paramiko?
• The “Could Be Better” • Active/Active HA for Tower (On the Roadmap!)• Canceled Jobs != Failed Jobs
18
Looking Ahead
• Expand Benchmark Coverage• Application Benchmarks• Database Benchmarks• Hadoop Deployments• Container Deployments
• Full Stack Validation Suite with Stacki and Ansible Openstack• One playbook to validate them all
• Community Contribution• Support OpenStack Benchmarking Projects (i.e Browbeat)
19
Cloud Benchmarking with Unixbench
20
Unixbench – System Benchmark
• Released in 1995 to measure performance of an entire system• Sensitive to HW as well as SW configuration and OS version• Sub-‐test scores compared against Sun SparcStation 2061 (1994) to create sub-‐test index• Sub-‐test indices combined (geo-‐mean) to form System Index
Sub-‐test Name Description System Component
Dhrystone Synthetic integer test (1984) CPU
Whetstone Synthetic floating-‐point test (1972) CPU
execl() Count of system exec calls in 1 second OS
File Copy Writes file to disk then makes a copy Disk I/O
Pipe Throughput Measuring 512B transfers/second Memory, OS/Shell
Pipe-‐based Context Switching “2-‐way conversation” with increasing integer Memory, OS/Shell
Shell Scripts Shell script execution time OS/Shell
Sys call Latency induced by entering OS OS
21
Victoria, bare-‐metal v. single virtualized instance
20% perf drop
22
Medium vm performance as active vm’s increase in Rack (23 servers)
40% perfdrop as cluster fills
23
Unity Large vm performance as active vm’s increase in Rack (23 servers)
37% drop as cluster fills
24
Unity X-‐Large vm performance as active vm’s increase in Rack (23 servers)
31% drop as cluster fills
25
Unity, medium VMs provide optimal cumulative Unixbench performance
26
VMs on Unity Private Cloud match Public Cloud Perf at 50% Utilization
27
Private Cloud (27 Victoria Cluster) Cost Calculations
HardwareCPU: $1,340 (2x E5-‐2630 @ $670)Memory: $1,600 (8x 32GB DDR4 DIMMs, SK Hynix @ $200)Storage: $212 (1x 480GB SSD)-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐Server: $5,500
For 27 Victoria in a rack: $148,500.00Rack + 3 Switches + 10G Cabling: $9,785
Total cost to buy: $158,285
Power / Cooling Average summer power cost: $0.243/kWhAverage winter power cost:. $0.188/kWh
Idle power usage/server: 120WUnder-‐load power usage/server: 240W27 Victoria under load: 6480WAverage daily operating cost: $36.81Amortizing cost of rack etc.: $158,285/(365.25 * 5)
Daily cost: $86.67 / dayCooling needed: 1.69 tons
Including power/cooling: $131.79 / dayHourly expense: $131.79 / 24hr = $ 5.49 / hour
28
Public v. Private Cloud; Private has better value as low as 25% utilization
