Upload
hoangtuyen
View
217
Download
3
Embed Size (px)
Citation preview
Leveraging Ambari to Build Comprehensive Management UIs
For Your Hadoop Applicationsby Christian Tzolov
@christzolov
Whoami
Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member
[email protected] blog.tzolov.net @christzolov
How To Provision, Manage And Monitor Enterprise Applications At Scale?
Contents• Ambari Overview
• Stacks and Services (demo)
• Blueprints (demo)
• Views (demo)
• Metrics & Monitoring
• On the Cloud
Apache Ambari
A platform to provision, manage and monitor Hadoop and Enterprise Applications
Deployment Topology
Ambari Web
(REST API) Ambari Server
admin node
Ambari Agent
cluster node 1
Ambari Agent
cluster node 2
Ambari Agent
cluster node N
Special PowersServices & Stacks
Blueprints
Views
Monitoring
Services & Stacks
Service Concepts
• Stack - Set of (managed) Services and software packages for them
• Service - Set of (distributed) Components
• Component - Service building block of type Master, Slave, Client
Service Architecture
Service Layout|_ services |_ <service_name> metainfo.xml metrics.json |_ configuration {configuration files} |_ package |_ scripts {command scripts}
• metainfo.xml - defines service, components and the command management scripts
• configuration - service config files used in and deployed through the command scripts
• package/scripts - commandScripts and templates.
Metainfo.xml<metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/> <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo>
<components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>
Service / Command Scriptimport sys from resource_management import * class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute()
import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()
Service / Demo
Ambari Blueprint
Why?• Installation Automation
• Declarative definition of a cluster
• Specify a Stack, the Component layout and the Configurations to materialize a cluster
• REST API
Blueprint Flow
Ambari Server
node 1
Cluster 1
node 2 node N
Cluster 2
. . . .
Cluster Z
. . . .
Blueprint (json)
Host Mapping (json)
REST API
Blueprint Deploy{ "Blueprints" : { "stack_name" : "PHD", "stack_version" : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]}
POST /api/v1/blueprints/my-blueprint
{ "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] }
POST /api/v1/clusters/my-cluster
Blueprint REST API/api/v1/blueprint
Cluster Logical Structure and configuration
/api/v1/clusterPhysical cluster mapping. Materializes a blueprint with real infrastructure
/api/v1/clusters/<cluster name>?format=blueprint
Exports blueprint for an existing cluster
Blueprint Demo
Ambari Views
View Components
Ambari Server
Ambari Web
Server Side (java)
Client Side (js,html..)
REST API
VIEW
PAC
KAG
E (J
AR)
• Client Side - any client-side technologies
• Server Side - Java application, Servlet, a REST service or ResourceProvider.
• View Package - JAR bundles of the view definition and resources
• View Context - Instance attributes and configuration
context
View Packageview.xml
<server packages + classes>
index.html <supporting UI assets>
WEB-INF (lib/*.jar)
View Context• On the Server Side
• View-instance Config
• View-instance Attributes
• Execution context, authenticated user principal
ViewContext
getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String
View Dashboard
View REST APIGET /api/v1/views List deployed views
GET /api/v1/views/<View Name> List all View instances
GET /api/v1/views/<View Name>/versions/<View Version>
Show specific View instance
POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name> [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]
Create new View Instance
Google Polymerhttps://www.polymer-project.org/1.0/
View Demo
https://github.com/tzolov/ambari-webpage-embedder-view
https://youtu.be/qaR_6OBeKsQ
Ambari Monitoring
Ambari Metrics SystemCollect, Aggregate and Serve System Metrics
• Metric Collector - Collects and Aggregates
• Metric Monitor - System-level metrics
• Metric Hadoop Sinks - Service-level metrics
AMS Architecture
Ambari & Cloud
Virtualization & Cloud• Vagrant - PivotalHD
(https://github.com/tzolov/vagrant-pivotalhd)
• Cloudbreak (http://sequenceiq.com/cloudbreak/)
Thank You!