Interop - Crash Course In Open Source Cloud Computing

  • Published on
    11-May-2015

  • View
    855

  • Download
    2

Embed Size (px)

DESCRIPTION

This will be an overview of the open source software that can be used to deploy and manage a cloud computing environment. The session will include information on storage, networking(e.g. OpenDaylight) and compute virtualization (Xen, KVM, LXC) and the orchestration(Apache CloudStack, OpenStack) of the three to build their own cloud services.

Transcript

  • 1.Mark Hinkle Senior Director, Open Source Solutions Citrix Inc. mark.hinkle@citrix.com mrhinkle@gmail.com @mrhinkle Crash Course Open Source Cloud Computing

2. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing ABOUT ME I Help Build Open Source Ecosystems Open Source Experience Manage Citrix Open Source Business Office Apache CloudStack Committer Advisory boards Gluster and Xen Project Joined Citrix via Cloud.com acquisition July 2011 Zenoss Core open source project to 100,000 users, 1.5 million downloads Former LinuxWorld Magazine Editor-in-Chief Open Management Consortium organizer Author - Windows to Linux Business Desktop Migration Thomson NetDirector Project - Open Source Configuration Management 3. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing Slides Available on Slideshare: http://www.slideshare.net/socializedsoftwar e Creative Commons Attributions-ShareAlike 4.0 International Share copy and redistribute the material in any medium or format Adapt remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. ShareAlike If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. 4. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing VETTING OPEN SOURCE PROJECTS How can you tell if theyre legit Code Velocity Committers Committer Reputation User-driven or Vendor-Driven Innovation User Activity Corporate Support* Reputation of Foundation* 5. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPEN SOURCE ANALYSIS Visualizing Community Activity http://www.ohloh.net http://activity.openstack.org 6. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing 60 SECOND CLOUD DEFINITION 5 CHARACTERISTICS OF CLOUD 1. On-Demand Self-Service 2. Broad Network Access 3. Resource Pooling 4. Rapid Elasticity 5. Measured Service User Cloud a.k.a. SOFTWARE-AS-A-SERVICE Developer Cloud a.k.a. PLATFORM-AS-A-SERVICE Systems Cloud a.k.a. INFRASTRUCTURE-AS-A- SERVICE 7. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing Vertical Scaling (Scale-Up) Allocate additional resources to VMs, requires a reboot, no need for distributed app logic, single-point of OS failure Horizontal Scaling (Scale-Out) Application needs logic to work in distributed fashion (e.g. HA-Proxy and Apache Hadoop) SCALE-UP SCALE OUT Elasticity and the cloud 8. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing VIRTUALIZATION Carving up compute resources OPEN SOURCE Xen Project Citrix XenServer KVM VirtualBox OpenVZ LXC PROPRIETARY VMware Microsoft Hyper-V OracleVM (Based on Xen Project) 9. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPEN VIRTUALIZATION FORMATS Virtualization Payloads Open Virtualization Format (OVF) is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines. Formats for hypervisors/cloud technologies: Amazon - AMI KVM QCOW2 VMware VMDK Xen Project IMG Hyper-V - VHD Virtual Hard Disk LXC local file system/mount point - Docker* 10. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing SOURCING CLOUD APPLIANCES Packaging Engines for VMs Tool/Project What you can do with them Bitnami BitNami provides free, ready to run environments for your favorite open source web applications and frameworks, including Drupal, Joomla!, Wordpress, PHP, Rails, Django and many more. Boxgrinder BoxGrinder is a set of projects that help you grind out appliances for multiple virtualization and Cloud providers Oz Command-line tool that has the ability to create images for common Linux distributions to run on KVM SUSE Studio SUSE Studio supports building and deploying directly to cloud services such as Amazon EC2. 11. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing LINUX CONTAINERS (LXC) Lightweight Linux Virtualization Lets your run a Linux system within another Linux system A container is a group of processes on a Linux box, put together the provide an isolated environment From the inside, it looks like a VM Externally it looks like normal processes chroot on steroids 12. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing LXC VS. VMs Containers compared to Hardware Virtualization Source: http://www.slideshare.net/jpetazzo/presentations 13. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing DOCKER CONTAINER PACKAGING Open source LXC packaging engine Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, public clouds and more. To learn more please visit our website: www.docker.io 14. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing MultiplicityofGoodsMultiplicityof methodsfor transporting/storing DoIworryabouthow goodsinteract(e.g. coffeebeansnextto spices) CanItransport quicklyand smoothly (e.g.fromboatto traintotruck) CARGO TRANSPORT PRE-1960 15. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing MultiplicityofGoods Multiplicityof methodsfor transporting/storing DoIworryabouthow goodsinteract(e.g. coffeebeansnextto spices) CanItransportquicklyand smoothly (e.g.fromboattotrainto truck) in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery. SOLUTION INTERMODAL SHIPPING CONTAINER 16. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing Static website Web frontendUser DB Queue Analytics DB Developm ent VM QA server Public Cloud Contributor s laptop MultiplicityofStacks Multiplicityof hardware environments Production Cluster Customer Data Center Doservicesandapps interactappropriately? CanImigratesmoothly andquickly An engine that enables any payload to be encapsulated as a lightweight, portable, self- sufficient container that can be manipulated using standard operations and run consistently on virtually any hardware platform DOCKER IS A SHIPPING CONTAINER FOR CODE 17. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing INFRASTRUCTURE AS-A-SERVICE Compute Orchestration Year Started License Virtualization Technologies Apache CloudStack 2008 Apache Xenserver, Xen Cloud Platform, KVM, Vmware Hyper-V Eucalyptus 2006 GPL Xen, KVM, VMware (commercial version) OpenNebula 2005 Apache Xen, KVM, VMware OpenStack 2010 (Developed by NASA by Anso Labs previously) Apache VMware ESX and ESXi, , Xen, XenServer, KVM, LXC, QEMU and Virtual Box 18. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPENSTACK The Boy Band of the Open Source Cloud 19. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPENSTACK SHARED SERVICES Span Compute, Storage and Networking IDENTITY SERVICE IMAGE SERVICE TELEMETRY SERVICE ORCHESTRATI ON SERVICE 20. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing EVEN MORE OPENSTACK PROJECTS Span Compute, Storage and Networking Cinder (Block Storage Service) Metering/Monitoring(Ce ilometer) Orchestration (Heat) Trove(Database Service) Bare Metal (Ironic) Queue Service (Marconi) 21. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPENSTACK SOLUTION PROVIDERS If you cant do it yourself 22. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing CLOUD APIS Everything (should) have an API in the Cloud deltacloud daisein jclouds libcloud fog 23. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing CLOUD STORAGE Virtualized, Distributed usually on Commodity Hardware Project Description Ceph Distributed file storage system developed by DreamHost GlusterFS Scale Out NAS system aggregating storage over Ethernet or Infiniband OpenStack Storage Long-term object storage system Riak CS Riak CS is open source software designed to provide simple, available, distributed cloud storage at any scale. Riak CS is S3-API compatible and supports per-tenant reporting for billing and metering use cases. Sheepdog Distributed storage for KVM hypervisors 24. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing PLATFORM-AS-A-SERVICE Abstracted Cloud-Scale Run-Time Environments Project Sponsors Languages/Frameworks CloudFoundry Vmware -> Pivotal -> CloudFoundry Foundation Spring for Java, Ruby for Rails and Sinatra, node.js, Grails, Scala on Lift and more via partners (e.g. Python, PHP) Cloudify Gigaspaces [Groovy for deployment recipes] OpenShift Origin Red Hat Java, Ruby, PHP, Perl and Python Apache Stratos WSO2 - >Apache Stratus PHP, Tomcat, MySQL cartridges 25. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing SOFTWARE DEFINED NETWORKING(SDN) Virtualization meets the network Decoupling of the control and data planes of the network to improve efficiency. Communication from a SDN controller via a protocol to network devices both physical and virtual. Automation Dynamic Networks Security Heterogeneous Management Abstractions allow for programmable networks. Network can be changed quickly via a controller Network offerings can match virtualization offerings for finer grained security in a highly volatile compute landscape. Single control point for various devices. 26. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing Business Applications Network Services SDN Control Software API API Network DevicesNetwork DevicesNetwork Devices Network DevicesNetwork DevicesNetwork Devices Application Layer Control Layer Infrastructure Layer Control Data Plane Interface (e.g. OpenFlow) SDN OVERVIEW 27. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPENFLOW Virtualization meets the network OpenFlow enables networks to evolve, by giving a remote controller the power to modify the behavior of network devices, through a well-defined "forwarding instruction set". The growing OpenFlow ecosystem now includes routers, switches, virtual switches, and access points from a range of vendors. 28. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing OPEN VSWITCH Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). To learn more please visit our website: http://openvswitch.org/ 29. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing Project Description Floodlight The Floodlight controller is an enterprise-class, Apache-licensed, Java-based OpenFlow Controller. Indigo Indigo is an open source project to support OpenFlow on a range of physical switches. By leveraging hardware features of Ethernet switch ASICs, Indigo supports high rates for high port counts, up to 48 10-gigabit ports. Multiple gigabit platforms with 10-gigabit uplinks are also supported. Open Daylight Linux Foundation Collaborative Project based on Cisco One Controller and plugins from numerous vendors in development. E.g IBM DOVE OpenStack Network Pluggable, scalable, API-driven network and IP management Open vSwitch Open vSwitch is a open source (ASL 2.0), multilayer virtual switch designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). Hitchhikers Guide to the Open Cloud by @mrhinkle 29 OPEN SOURCE SDN 30. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing BIG DATA 31. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing NOSQL DATABASES Horizontally scalable unstructured data retrieval Name Type Description Apache Cassandra Wide Column Store/Families API: many Query Method: MapReduce, Replicaton: , Written in: Java, Concurrency: eventually consistent , Misc: like "Big-Table on Amazon Dynamo alike", initiated by Facebook CouchDB Document Store API: Memcached API+protocol (binary and ASCII) , most languages, Protocol: Memcached REST interface for cluster conf + management, Written in: C/C++ + Erlang (clustering), Replication: Peer to Peer, fully consistent, Misc: Transparent topology changes during operation, provides memcached-compatible caching buckets HBase Wide Column Store/Families API: Java / any writer, Protocol: any write call, Query Method: MapReduce Java / any exec, Replication: HDFS Replication, Written in: Java Hypertable Wide Column Store/Families PI: Thrift (Java, PHP, Perl, Python, Ruby, etc.), Protocol: Thrift, Query Method: HQL, native Thrift API, Replication: HDFS Replication, Concurrency: MVCC, Consistency Model: Fully consistent Misc: High performance C++ implementation of Google's Bigtable. MongoDB Document Store API: BSON, Protocol: C, Query Method: dynamic object-based language & MapReduce, Replication: Master Slave & Auto-Sharding, Written in: C++,Concurrency Redis Key Value/ Tuple Store API: Tons of languages, Written in: C, Concurrency: in memory and saves asynchronous disk after a defined time. Append only mode available. Different kinds of fsync policies. Replication: Master / Slave, Misc: also lists, sets, sorted sets, hashes, queues. Riak Key Value / Tuple Store API: JSON, Protocol: REST, Query Method: MapReduce term matching , Scaling: Multiple Masters; Written in: Erlang, Concurrency: eventually consistent (stronger then MVCC via Vector Clocks) 32. By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com Crash Course in Open Source Cloud Computing MAP REDUCE Algorithm for Parallelized Data Set Processing Problem Data Master Node Wo...