Upload
dorian-garrison
View
105
Download
4
Embed Size (px)
DESCRIPTION
Cloud Usage with CloudStack. Sept 2014 Chandra Prakash Mishra. In next 45 minutes …. Agenda CloudStack Usages Cloudstack configuration to enable Usage Capture Usage Data Generation Process Usage Type Usage Event Usage Records How can Access Usage Data Use Cases. - PowerPoint PPT Presentation
Citation preview
© 2014 Sungard AS. All rights reserved.
Cloud Usage with CloudStackSept 2014
Chandra Prakash Mishra
2
In next 45 minutes …
Agenda
• CloudStack Usages
• Cloudstack configuration to enable Usage Capture
• Usage Data Generation Process
• Usage Type
• Usage Event
• Usage Records
• How can Access Usage Data
• Use Cases
3Footer – Apply across document
Usage Requirements
• Meter their cloud's consumption and monetize the cloud offerings by charging for usage
• Report on the cloud’s resource usage for monitoring/capacity planning
• Existing billing system, just need access to the usage data
• Organizations tend to have different billing criteria/charging models and require access to usage data for their specific needs
4Footer – Apply across document
Usage in CloudStack today
• CloudStack does not offer any native usage monitoring or billing capability
• Root Admin can view current System Capacity
• Domain Admin and Users can view VM statistics
5
Usage Server
• Optional
• Separately installed
• Provide aggregated usage records
• Run at least once per day (Default):1440
• Take data from the events log in cloud DB “cloud.usage_event”
• Create summary usage records in cloud_usage DB “cloudusage.usage_event”
• Access usage records using listUsageRecords“http://localhost:8096/client/api?command=listUsageRecords&startdate=2013-05-25&enddate=2013-06-05”
6
Usage Server Config
Name Description value
enable.usage.server Flag for enabling usage true
usage.stats.job.exec.time Time when the usage server will start processing
00:15
usage.execution.timezone The timezone to use for usage job execution time
Usage server time zone
usage.stats.job.aggregation.range Usage is aggregated for specified number of minutes
1440
usage.aggregation.timezone The time zone to user for usage stats aggregation
GMT
7Footer – Apply across document
Usage Server Config
Usage Server
Name Value
usage.stats.job.exec.time 00:15
usage.execution.timezone PST
usage.stats.job.aggregation.range 60
usage.aggregation.timezone GMT
• Usage Job will run at PST 00:15
• Generate usage records for 1hrs from 07:15:00 GMT to 08:14:59 GMT
• Next exec.time is 01:15
Usage Type
8
ID Usage Type ID Usage Type
1 RUNNINGVM 8 ISO
2 ALLOCATEDVM 9 SNAPSHOT
3 IPADDRESS 11 LOADBALANCERPOLICY
4 NETWORKBYTESSENT 12 PORTFORWARDING
5 NETWORKBYTESRECEIVED 13 NETWORKOFFERING
6 VOLUME 14 VPNUSERS
7 TMEPLATE
9
CloudStack Event
• Usage events are stored in usage_event table. Any resource for which usage has to be tracked needs to log usage events when the the resource is created/allocated and when its deleted/released
• Network usage is the only exception which doesn’t use usage events• Once these events are logged usage server will process them and generate Cloudstack usage
records
Description of various columns in usage_event table:
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| 12 | VM.START | 2 | 2011-02-09 11:16:18 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 || 15 | VM.STOP | 2 | 2011-02-09 12:00:24 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 |+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +| 2 | NET.IPASSIGN | 2 | 2011-02-09 11:16:18 | 1 | 10 | 10.91.30.121 | NULL | NULL | 1 | 0 || 9 | NET.RELEASE | 2 | 2011-02-09 12:00:24 | 1 | 21 | 10.91.30.121 | NULL | NULL | 0 | 0 |+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Running VM
Public IP address
10Footer – Apply across document
Usage Record Generation
• CloudStack events have been logged into cloud.usage_event table
• The cloudstack_usage job gets a list of the latest usage events
• Inserts these events into cloud_usage.usage_events table
• Parses cloud_usage.usage_event and populates Helper tables
• Helper table data used to populate cloud_usage.cloud_usage table with aggregation range wise data
11
Network Usage Record Generation
• Network usage is processed differently and is not driven by usage events
• In Virtual Networking case Iptables rules are set in domR to track public traffic
• Management server queries for Network stats periodically and populates them in user_statistics table account-wise
• Cloud.user_statistics is queried and populated to cloud_usage.user_statistics with latest stats
• Network helper entry is created in usage_network table
• Network usage for current event is calculated as:
bytesSent = (total bytes sent) – (bytes sent prior to this event);bytesReceived = (total bytes received) – (bytes received prior to this event);
Usage Calculation
12
1-1-VM Event Timedeployed 2013-06-05 12:00
start 2013-06-05 12:00
stop 2013-06-05 18:00
start again 2013-06-05 23:00
startdate 2013-06-05 00:00:00
enddate 2013-06-05 23:59:59
RUNNINGVM(Usage Type 1)
6+1 = 7 hours
ALLOCATEDVM(Usage Type 2)
12 hours
The Usage Record
13Footer – Apply across document
Usage Record Format
Virtual Machine Usage Record Format: For running and allocated vm usage
account name of the account
accountid ID of the account
domainid ID of the domain in which this account resides
zoneid Zone where the usage occurred
description A string describing what the usage record is tracking
usage String representation of the usage, including the units of usage (e.g. 'Hrs' for VM running time)
usagetype A number representing the usage type (see Usage Types)
rawusage A number representing the actual usage in hours
virtualMachineId The ID of the virtual machine
name The name of the virtual machine
offeringid The ID of the service offering
templateid The ID of the template
usageid Virtual machine
type Hypervisor
startdate, enddate The range of time for which the usage is aggregated; see Dates in the Usage Record
14
Usage Record Format
account name of the account
accountid ID of the account
domainid ID of the domain in which this account resides
zoneid Zone where the usage occurred
description A string describing what the usage record is tracking
usagetype A number representing the usage type (see Usage Types)
rawusage A number representing the actual usage in hours
usageid Device ID (virtual router ID or external device ID)
type Device type (domain router, external load balancer, etc.)
startdate, enddate The range of time for which the usage is aggregated; see Dates in the Usage Record
Network Usage Record Format:
For more details Please refer this link:https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/API_Developers_Guide/usage-record-format.html
15Footer – Apply across document
How we can access usage data
16Footer – Apply across document
CloudStack API
• The usage records can be accessed through the CloudStack API
• This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed
• CloudStack provides the following Root Admin API calls and responds with XML and JSON responses
• listUsageTypes – lists available Usage Types
• listUsageRecords - provides Usage records for a date range• • generateUsageRecords - asynchronous usage record generation
17Footer – Apply across document
ListUsageRecords API
The base API request is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature • Optional request parameters:
18
<listusagerecordsresponse> <count>1816</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description> <usage>2.34 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.34</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord> ………</listusagerecordsresponse>
Example response from listUsageRecords
19Footer – Apply across document
CloudMonkey CLI
• CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API
• It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing
• Filtering can be used to limit the result set • It supports argument passing and shell automation
• The host, port, apikey and secretkey can be configured on setup
20Footer – Apply across document
CloudMonkey list usagerecords
The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10 enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07
21Footer – Apply across document
SQL
• The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack
• Usage Data can be exported in a CSV format from a SELECT query
• The SQL query can become complex if the ID fields in the data need to be de-referenced
• Typically multiple passes of the query results may be required to extract the required information
22Footer – Apply across document
CloudStack Message Queue
• CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers
• Use case: A third-party cloud usage solution can subscribe to CloudStack events and generate usage data which can be consumed by their billing software
For more details Please refer this link: https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus
23Footer – Apply across document
Customer Use Cases
24Footer – Apply across document
MS ExcelMS Excel
·MS Excel can be used to analyse CloudStack Usage Records ·The usage records are imported from the cloud_usage table into MS
Excel via an MySQL ODBC connection ·Either PivotTables or reports can be used to present the data ·This is convenient and great for integration testing between CloudStack
and external systems
25Footer – Apply across document
Splunk
• Splunk is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs
• We then connected splunk to the CloudStack Usage database and automated the collection and indexing of Usage Data
• Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting
26Footer – Apply across document
Amysta
Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity Mgmt Amysta consolidates public cloud expenses e.g. AWS and private cloud
consumption in one dashboard view
Fore more details please refer this link:https://www.youtube.com/watch?v=ZA-VPb7WAy8http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing
27Footer – Apply across document
SunGard Cloud Portal
Thank You!