© 2014 Sungard AS. All rights reserved.

Cloud Usage with CloudStackAug 2014

Chandra Prakash Mishra

2

In next 45 minutes …

Agenda

• CloudStack Usages

• Cloudstack configuration to enable Usage Capture

• Usage Data Generation Process

• Usage Type

• Usage Event

• Usage Records

• How can Access Usage Data

• Use Cases

3Footer – Apply across document

Usage Requirements

• Meter their cloud's consumption and monetize the cloud offerings by charging for usage

• Report on the cloud’s resource usage for monitoring/capacity planning

• Existing billing system, just need access to the usage data

• Organizations tend to have different billing criteria/charging models and require access to usage data for their specific needs

4Footer – Apply across document

Usage in CloudStack today

• CloudStack does not offer any native usage monitoring or billing capability • Root Admin can view current System Capacity • Domain Admin and Users can view VM statistics

5Footer – Apply across document

Usage Server

• CloudStack includes a Usage Server that create summary usage records for the various resources consumes in cloudstack.

• The usage server is an optional, separately installed component called “cloudstack-usage”.

• The usage records are stored in a separate database called “cloud_usage”.• Its Runs once a day by default .It can be configured to run multiple times per

day and it’s behavior is controlled by the following global configuration settings.

enable.usage.server : Whether the usage is active ?

usage.stats.job.aggregation.range : Usage is aggregated for specified number of minutes. e.g default:1440

usage.stats.job.exec.time : Time when the usage server will start processing. Default: 00.15

usage.execution.timezone : The timezone to use for usage.stats.job.exec.time. Default: timezone of the management server

usage.aggregation.timezone : The timezone to use for Cloudstack usage stats aggregation (default GMT)

6Footer – Apply across document

CloudStack Event

• Usage events are stored in usage_event table. Any resource for which usage has to be tracked needs to log usage events when the the resource is created/allocated and when its deleted/released.

• Network usage is the only exception which doesn’t use usage events. Once these events are logged usage server will process them and generate Cloudstack usage records.

Description of various columns in usage_event table:

• type – Event type as described in cloudstack-oss/api/src/com/cloud/event/EventTypes.java• account_id – Id of the account which owns the resource• zone_id – Data center to which this resource belongs to• resource_id – Id of the resource. For VM usage this will be VmId, for template templateId etc• resource_name – Name of the resource.• offering_id – Id of Service Offering in case of Vm usage, Disk offering in case of volume usage etc..• template_id – Id of the template used for a volume / Vm. • size – Size of the resource. Volume Size or Template size etc.• resource_type - Additional type attribute for a resource. Could be used for hypervisor_type etc.• processed – Is set to 0 by default. This field is updated to 1 when the usage server completes processing of this event.

7Footer – Apply across document

Usage Record Generation

The usage server schedule a job at the time specified by usage.stats.job.exec.time parameter .This job query cloud databases cloud.usage_event table and generate usage records in the cloud_usage database table cloud_usage as follows

• CloudStack events have been logged into cloud.usage_event table .

• The cloudstack_usage job gets a list of the latest usage events.

• Inserts these events into cloud_usage.usage_events table.

• Parses cloud_usage.usage_event and populates Helper tables .

• Helper table data used to populate cloud_usage.cloud_usage table with aggregation range wise data .

8

Usage Type

Type id

Type Name Description

1 RUNNING_VMTracks the total running time of a VM per usage record period. If the VM is upgraded during the usage period, you will get a separate Usage Record for the new upgraded VM.

2 ALLOCATED_VMTracks the total time the VM has been created to the time when it has been destroyed. This usage type is also useful in determining usage for specific templates such as Windows-based templates.

3 IP_ADDRESS Tracks the public IP address owned by the account.

4 NETWORK_BYTES_SENTTracks the total number of bytes sent by all the VMs for an account. Cloud.com does not currently track network traffic per VM.

5NETWORK_BYTES_RECEIVED

Tracks the total number of bytes received by all the VMs for an account. Cloud.com does not currently track network traffic per VM.

6 VOLUME Tracks the total time a disk volume has been created to the time when it has been destroyed.

7 TEMPLATETracks the total time a template (either created from a snapshot or uploaded to the cloud) has been created to the time it has been destroyed. The size of the template is also returned.

8 ISOTracks the total time an ISO has been uploaded to the time it has been removed from the cloud. The size of the ISO is also returned.

9 SNAPSHOTTracks the total time from when a snapshot has been created to the time it have been destroyed.

11 LOAD_BALANCER_POLICYTracks the total time a load balancer policy has been created to the time it has been removed. Cloud.com does not track whether a VM has been assigned to a policy.

12 PORT_FORWARDING_RULE Tracks the time from when a port forwarding rule was created until the time it was removed.13 NETWORK_OFFERING The time from when a network offering was assigned to a VM until it is removed.14 VPN_USERS The time from when a VPN user is created until it is removed.

9Footer – Apply across document

Usage Records

Currently there is no UI which displays usage records. API call listUsageRecords fetches usage records. Usage Records show the amount of resources consumed by guest instances.

• Allocated & Running VMs • Network Usage • IP Address • Disk Volume • Template, ISO and Snapshot • Load Balancer or Port Forwarding • Network Offering • VPN User

10Footer – Apply across document

Vm Usage Record Format

For running and allocated virtual machine usage, the following fields exist in a usage record:

account – name of the accountaccountid – ID of the accountdomainid – ID of the domain in which this account resideszoneid – Zone where the usage occurreddescription – A string describing what the usage record is trackingusage – String representation of the usage, including the units of usage (e.g. 'Hrs' for VM running time)usagetype – A number representing the usage type (see Usage Types)rawusage – A number representing the actual usage in hoursvirtualMachineId – The ID of the virtual machinename – The name of the virtual machineofferingid – The ID of the service offeringtemplateid – The ID of the template or the ID of the parent template. The parent template value is present when the current template was created from a volume.usageid – Virtual machinetype – Hypervisorstartdate, enddate – The range of time for which the usage is aggregated; see Dates in the Usage Record

11Footer – Apply across document

Usage Record Example

Allocated VMEvery time a VM is created VM.CREATE event is logged. Similarly VM.DESTROY event is logged when VM is destroyed. Corresponding entries in cloud.usage_event table would like the following:+———————-——————-——-———–————-———–———–—-———-+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+———————-——————-——-———–————-———–———–—-———-+| 11 | VM.CREATE | 2 | 2011-02-09 11:15:27 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 || 12 | VM.DESTROY | 2 | 2011-03-09 10:16:18 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 |+———————-——————-——-———–————-———–———–—-———-+

Duration between VM.CREATE and VM.DESTROY is reported as Allocated VM usage.

Running VMUsage for Running VM is implemented similar to allocated VM usage. Events VM.start and VM.stop are tracked here instead.

+———–———-——————-——-———–————-———–———–—-———-+| id | type     | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+———–———-——————-——-———–————-———–———–—-———-+| 12 | VM.START |          2 | 2011-02-09 11:16:18 |       1 |           5 | i-2-5-KK      |           7 |           2 | NULL |         0 || 15 | VM.STOP  |          2 | 2011-02-09 12:00:24 |       1 |           5 | i-2-5-KK      |           7 |           2 | NULL |         0 |+———–———-——————-——-———–————-———–———–—-———-+

Duration between VM.START and VM.STOP is reported as Running VM usage. 

12Footer – Apply across document

How we can access usage data

13Footer – Apply across document

CloudStack API

• The usage records can be accessed through the CloudStack API .

• This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed

• CloudStack provides the following Root Admin API calls and responds with XML and JSON responses:

1. listUsageTypes – lists available Usage Types 2. listUsageRecords - provides Usage records for a date range 3. generateUsageRecords - asynchronous usage record generation

14Footer – Apply across document

ListUsageRecords API

The base API request is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature • Optional request parameters:

15Footer – Apply across document

<listusagerecordsresponse> <count>1816</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord></listusagerecordsresponse>

Example response from listUsageRecords

16Footer – Apply across document

CloudMonkey CLI

• CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API.

• It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing.

• Filtering can be used to limit the result set. • It supports argument passing and shell automation.

• The host, port, apikey and secretkey can be configured on setup.

17Footer – Apply across document

CloudMonkey list usagerecords

The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10 enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07

18Footer – Apply across document

SQL

• The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack

• Usage Data can be exported in a CSV format from a SELECT query

• The SQL query can become complex if the ID fields in the data need to be de-referenced

• Typically multiple passes of the query results may be required to extract the required information

19Footer – Apply across document

CloudStack Message Queue

• CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers .

• Use case: A third-party cloud usage solution can subscribe to CloudStack events and generate usage data which can be consumed by their billing software .

For more details Please refer this link• https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus

20Footer – Apply across document

Customer Use Cases

21Footer – Apply across document

MS ExcelMS Excel

·MS Excel can be used to analyse CloudStack Usage Records ·The usage records are imported from the cloud_usage table into MS Excel

via an MySQL ODBC connection ·Either PivotTables or reports can be used to present the data ·This is convenient and great for integration testing between CloudStack and

external systems

22Footer – Apply across document

Splunk

• Splunk is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs

• We then connected splunk to the CloudStack Usage database and automated the collection and indexing of Usage Data

• Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting

23Footer – Apply across document

Amysta

Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity Mgmt Amysta consolidates public cloud expenses e.g. AWS and private cloud consumption in one

dashboard view 

Fore more details please refer this link:https://www.youtube.com/watch?v=ZA-VPb7WAy8http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing

24Footer – Apply across document

SunGard Cloud Portal

25Footer – Apply across document

Ubersmith's

Ubersmith's CloudStack Billing Service Module allows you to have Ubersmith bill for CloudStack usage.

Fore more details please refer this link :http://www.ubersmith.com/kbase/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=161&nav=0

Thank You!