OSCON 2014 - Crash Course in Open Source Cloud Computing

  • Published on
    08-Sep-2014

  • View
    1.224

  • Download
    8

Embed Size (px)

DESCRIPTION

This crash course is designed to give an overview of cloud computing architecture and the open source software that can be used to deploy and manage a cloud computing environment. Topics to be discussed in this session will include virtualization (KVM, LXC, and Xen Project), orchestration (Apache CloudStack, Eucalyptus, Open Nebula, and OpenStack), and storage (GlusterFS, Ceph, and others). The talk will also provide insight into how to deliver Platform-as-a-Service (PaaS) and what technologies can be used to compliment this evolving cloud computing paradigm. Systems administrators and IT generalists will leave the discussion with a general overview of the options at their disposal to effectively build and manage their own cloud computing environments using free and open source software and understand the capabilities and benefits of a host of technologies.

Transcript

  • Mark Hinkle Senior Director, Open Source Solutions Citrix Inc. mark.hinkle@citrix.com mrhinkle@gmail.com @mrhinkle Last updated: 7/20/2014 Crash Course In Open Source Cloud Computing
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing ABOUT ME I Help Build Open Source Ecosystems Open Source Experience Manage Citrix Open Source Business Office Apache CloudStack Committer and PMC Member Advisory boards Gluster and Xen Project Joined Citrix via Cloud.com acquisition July 2011 Zenoss Core open source project to 100,000 users, 1.5 million downloads Former LinuxWorld Magazine Editor-in-Chief Open Management Consortium organizer Author - Windows to Linux Business Desktop Migration Thomson NetDirector Project - Open Source Configuration Management
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing Slides Available on Slideshare: http://www.slideshare.net/socializedsoftwar e Creative Commons Attributions-ShareAlike 4.0 International Share copy and redistribute the material in any medium or format Adapt remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. ShareAlike If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing AGENDA Vetting Open Source Cloud Projects What is Cloud in 60 Seconds Virtualization Infrastructure-as-a-Service SDN Open Source for the Amazon Web Services
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing VETTING OPEN SOURCE PROJECTSHow can you tell if theyre Legit Code Velocity Committers Committer Reputation User-driven or Vendor-Driven Innovation User Activity Corporate Support* Reputation of Foundation*
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing OPEN SOURCE ANALYSIS Visualizing Community Activity http://www.ohloh.net http://activity.openstack.org
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing 60 SECOND CLOUD DEFINITION 5 CHARACTERISTICS OF CLOUD 1. On-Demand Self-Service 2. Broad Network Access 3. Resource Pooling 4. Rapid Elasticity 5. Measured Service User Cloud a.k.a. SOFTWARE-AS-A-SERVICE Developer Cloud a.k.a. PLATFORM-AS-A-SERVICE Systems Cloud a.k.a. INFRASTRUCTURE-AS-A- SERVICE Just because Software Marketing GuysThink its the Internet
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing Vertical Scaling (Scale-Up) Allocate additional resources to VMs, requires a reboot, no need for distributed app logic, single-point of OS failure Horizontal Scaling (Scale-Out) Application needs logic to work in distributed fashion (e.g. HA-Proxy and Apache Hadoop) SCALE-UP SCALE OUT Elasticity and the cloud
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing HYPERVISORS AND CONTAINERS Differences in virtualization Type 1 Hypervisors VMware, Xen Project, Hyper-V Type 2 Hypervisors KVM, VirtualBox Containers LXC
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing VIRTUALIZATION Carving up compute resources OPEN SOURCE Xen Project Citrix XenServer KVM VirtualBox OpenVZ LXC PROPRIETARY VMware Microsoft Hyper-V OracleVM (Based on Xen Project)
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing OPEN VIRTUALIZATION FORMATS Virtualization Payloads Open Virtualization Format (OVF) is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines. Formats for hypervisors/cloud technologies: Amazon - AMI KVM QCOW2 VMware VMDK Xen Project IMG Hyper-V - VHD Virtual Hard Disk LXC local file system/mount point - Docker*
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing LINUX CONTAINERS (LXC) Lightweight LinuxVirtualization Lets your run a Linux system within another Linux system A container is a group of processes on a Linux box, put together the provide an isolated environment From the inside, it looks like a VM Externally it looks like normal processes chroot on steroids
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing THE PORTABILITY PROBLEM Containers compared to Hardware Virtualization Different file formats for virtual machines VMware uses vmdk file format, Xen and Hyper- V use VHD, KVM uses Raw or QCOW2 Guest images may be processor architecture bound VMware and Xen can manage SCSI devices, but KVM cannot KVM and Xen can use virtio drivers but not VMware VMware uses a proprietary agent inside the guest OS (VMware tools) which does not work with Xen or KVM Xen uses VirtIo and ParaVirtualized drivers, Xen uses
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing CONTINUOUS INTEGRATION Rebuild Applications on any Cloud and/or Virtualized Infrastructure Code Application is stored in a repository (Subversion,Git) Build Code is built (Jenkins) Test Unit tests are automated (Jenkins) Deploy Deploy code to server various ways Code Build Test Deploy Thoughtworks Go Open Source Continuous Deliver System
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing DOCKER CONTAINER PACKAGING Open source LXC Packaging Engine Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, public clouds and more. To learn more please visit: www.docker.io
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing WHAT IS DOCKER System for Managing and Deploying LXC Containers Compliment to LXC not a replacement Managed daemonized processes on Linux using LXC Create ability to re-use and manage similar applications Content agnostic Hardware agnostic Easy to automate Integrated with other tools: Chef, OpenShift, Puppet, VMware, etc.
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing KUBERNETES Container Cluster Management Scheduler Kubernetes builds on top of Docker to construct a clustered container scheduling service. Kubernetes enables users to ask a cluster to run a set of containers. The system will automatically pick worker nodes to run those containers on, which we think of more as "scheduling" than "orchestration To learn more please visit: https://github.com/GoogleCloudPlatform/kubernetesGreek for Shipmaster
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing APACHE MESOS One to many tools for managing large numbers of devices Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers. Largely supported by Twitter, used by LinkedIn, AirBNB too. Features Fault-tolerant replicated master using ZooKeeper Scalability to 10,000s of nodes Isolation between tasks with Linux Containers Multi-resource scheduling (memory and CPU aware) Java, Python and C++ APIs for developing new parallel applications Web UI for viewing cluster state To learn more please visit: http://mesos.apache.org/
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing 19 APACHE ZOOKEEPER Centralized Server to Service Distributed Apps ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications To learn more please visit: http://zookeeper.apache.org/
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing INFRASTRUCTURE-AS-A-SERVICE Compute Orchestration Project Year Started License Virtualization Technologies Apache CloudStack 2008 Apache (Bare Metal), Xenserver, KVM, LXC VMware Hyper- V Eucalyptus 2006 GPL Xen, KVM, VMware (commercial version) OpenNebula 2005 Apache Xen, KVM, VMware OpenStack 2010 (Developed by NASA by Anso Labs previously) Apache VMware ESX and ESXi, , Xen, XenServer, KVM, LXC, QEMU and Virtual Box
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing OPENSTACK The Boy Band of the Open Source Cloud
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing OPENSTACK SHARED SERVICESSpan Compute, Storage and Networking IDENTITY SERVICE IMAGE SERVICE TELEMETRY SERVICE ORCHESTRATION SERVICE
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing EVEN MORE OPENSTACK PROJECTS Span Compute, Storage and Networking Cinder Block Storage Service Ceilometer Metering/Monitoring Heat Orchestration Trove Database Service Ironic Bare Metal (Ironic) Marconi Queue Service
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing OPENSTACK SOLUTION PROVIDERS If you cant do it yourself OpenStack is not a product. If you are building a large infrastructure, its more like a tool kit. It gives you a lot of technologies that do take a lot of effort to integrate. Chris Kemp, OpenStack Board Member and Co-Founder CEO of Piston Computing
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing CLOUD APIS Everything (should) have an API in the Cloud Deltacloud(ruby) Daisein(java) Jclouds(java) Libcloud(python) Fog(ruby)
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing CLOUD STORAGE Virtualized, Distributed usually on Commodity Hardware Project Description Ceph Distributed file storage system developed by DreamHost -> InkTank -> Red Hat (block, object, file) GlusterFS Scale Out NAS system aggregating storage over Ethernet or Infiniband (file) OpenStack Storage Long-term object storage system (object) Riak CS Riak CS is open source software designed to provide simple, available, distributed cloud storage at any scale. Riak CS is S3- API compatible and supports per-tenant reporting for billing and metering use cases. (object) Sheepdog Distributed storage for KVM hypervisors, distributed iSCSI
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing PLATFORM-AS-A-SERVICE Abstracted Cloud-Scale Run-Time Environments Project Sponsors Languages/Frameworks CloudFoundry VMware -> Pivotal -> CloudFoundry Foundation Spring for Java, Ruby for Rails and Sinatra, node.js, Grails, Scala on Lift and more via partners (e.g. Python, PHP) Cloudify Gigaspaces [Groovy for deployment recipes] OpenShift Origin Red Hat Java, Ruby, PHP, Perl and Python Apache Stratos WSO2 - >Apache Stratus PHP, Tomcat, MySQL cartridges
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing SOFTWARE DEFINED NETWORKING(SDN)Virtualization meets the network Decoupling of the control and data planes of the network to improve efficiency. Communication from a SDN controller via a protocol to network devices both physical and virtual. Automation Dynamic Networks Security Heterogeneous Management Abstractions allow for programmable networks. Network can be changed quickly via a controller Network offerings can match virtualization offerings for finer grained security in a highly volatile compute landscape. Single control point for various devices.
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing Business Applications Network Services SDN Control Software API API Network DevicesNetwork DevicesNetwork Devices Network DevicesNetwork DevicesNetwork Devices Application Layer Control Layer Infrastructure Layer Control Data Plane Interface (e.g. OpenFlow) SDN OVERVIEW
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail.com OSCON 2014 - Crash Course in Open Source Cloud Computing BENEFITS OF SDN Network Virtualization is the final frontier of Software Defined Datacenter Dynamically update networks Automate network functionality Program security into the network Centrally apply policies to network and services Optimize networks
  • By Mark R. Hinkle @mrhinkle mrhinkle@gmail...