Upload
hortonworks
View
573
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Learn about the new features of Ambari 1.7.0 http://ambari.apache.org/
Citation preview
Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Apache Ambari 1.7.0
November 2014
What’s New
Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
What’s New in Ambari 1.7.0
Core ServicesResourceManager HA
Capacity Scheduler Refresh Queues
HDFS Rebalance
Manage -env.sh Files
Set <final> Config Properties
Download Client Configs
Config Versioning + History
Stacks
Stack Advisor
Ambari PlatformAmbari Administration
Ambari Views Framework
Ambari Blueprints Export w/configs
Ubuntu 12 Platform Support
UsabilityCheck THP enabled
Abort Background Operations
Delete ZooKeeper Servers
For a complete list of changeshttps://issues.apache.org/jira/browse/AMBARI/fixforversion/12326546
Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Core Services
Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Resource Manager HA
Adds a second ResourceManager into cluster
ResourceManagers run Active / Standby
Requires ZooKeeper (and at least 3 ZK Servers)
Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Resource Manager HA
Select Host for the second ResourceManager
Stops and starts components of Services dependent on YARN
• MapReduce, Hive, WebHCat
Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Capacity Scheduler Refresh Queues
Refresh Capacity Scheduler Queues to apply scheduler changes
Useful when performing non-destructive CS changes
• Adding new queues
• Adjusting existing queue capacity
Deleting queues is a destructive CS change
• Requires ResourceManager Restart
Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Capacity Scheduler Change Actions
Note: when performing “refresh” with ResourceManager HA configured, Ambari performs “refresh” against both the Active and Standby ResourceManager
Change Valid Actions Comments
Add New or Edit Existing Queue
Refresh Capacity Scheduleror
Restart ResourceManager
When using “refresh”, if the only YARN config change is to CS, the “restart
required” flag will clear automatically
Delete Queue Restart ResourceManagerResourceManager restarts are usually
fast (~ 1 minute) compared to a NameNode restart
Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Refresh Queues Failure on Destructive Change
Important: Command fails if you attempt refresh w/ destructive changes
Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
HDFS Rebalance
Set balance threshold
Check progress via Background Ops dialog
Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Configuration Enhancements
Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Summary of Config Enhancements
Download Client Configs
Manage -env.sh files
Set <final> Config Properties
Configuration Change Notes
Add Bulk Custom Properties
Config Versioning + History
Page 12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Download Client Configs
Download client configuration files for a Service
Useful to build “edge” nodes (or “gateway” nodes) outside of Ambari
Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Manage -env.sh Configs
Manage the templates for –env.sh config files
hadoop-env.sh, hive-env.sh, etc
• Modify template from UI• Avoid having to customize
in Stack definition
Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Set <final> Config Properties
Declare a property as <final>
• via Ambari Web and REST API
User applications may not alter the property
<property> <name>dfs.datanode.data.dir</name> <value>/hadoop/hdfs/data</value> <final>true</final></property>
Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Add Bulk Custom Properties
Add custom configuration properties in Single or Bulk mode
Single or
Bulk Mode
Single or
Bulk Mode
Bulk Mode
(key=value textarea)
Single Mode
(key/value input)
Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Configuration Change Notes
Store notes with each configuration change
Page 17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Config Versioning and History
Service Config Versions – saved per service
List of Config History
Compare Versions
Filter by “Changed Properties”
Revert Changes (i.e. “Make Current”)
Audit Log of Changes/var/log/ambari-server/ambari-
config-changes.log2014-09-12 15:14:19,011 INFO - Cluster 'MyCluster' changed by: 'admin'; service_name='YARN' config_group='default' config_group_id='-1' version='2'
Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Config History TabHistory of Changes
Filter, Sort,
Search
Page 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Configs Tab
Most Recent Versions (view, compare, revert)
Compare Versions
RevertVersion
Filter by “Changed”
Page 20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Administration
Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ambari Administration
Introducing the Ambari Admin
Introducing Groups
Introducing Permissions
LDAP Users and Groups
Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Introducing the Ambari Admin
Ambari Admins can manage Ambari, including:
- Managing users and groups
- Elevating other users to “Ambari Admins”
- Installing clusters and managing cluster permissions
- Creating view instances and managing view permissions
Page 23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Introducing Groups
Added Groups support to Ambari
Ability to manage group membership
Supports Local or LDAP groups
Page 24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Introducing Permissions
Permissions in Ambari define what can be done for a given Resource Type
Terminology: Resource Types in Ambari are CLUSTER and VIEW
Support for Operator and Read-Only CLUSTER permissions
Grant permissions to users or groups
Page 25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
LDAP Users + Groups
Ambari supports configuring for LDAP authentication
New support for importing / sync’ing LDAP users and groups with Ambari
Step 1:Setup Ambari for LDAP
Authentication
Step 2:Import/sync LDAP users and groups with Ambari
Page 26 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
LDAP Authentication
Setup Ambari for LDAP Authentication
ambari-server setup-ldap
Configure primary and secondary LDAP server URLs, SSL optional, anonymous binding, etc.
AmbariServer
ExternalLDAP
Local users & groups
DB LDAP users & groups
Authenticate
Page 27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
New LDAP Authentication Config Properties
Ambari 1.6.1 or Earlier Ambari 1.7.0
Primary LDAP URL Required Required
Secondary LDAP URL Optional Optional
SSL Optional Optional
Base DN Required Required
Bind Anonymously true/false true/false
Bind Manager Optional Optional
User Name Attribute Required Required
User Object Class Required
Group Name Attribute Required
Group Object Class Required
Group Member Attribute Required
Page 28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Import / Sync’ing LDAP Users + Groups
Import / synchronize LDAP users and groups with Ambari
ambari-server sync-ldap
Options:• Import a specific set of users and groups (--users and --groups), or
• Import all LDAP users and groups (--all), or
• Sync “existing Ambari LDAP” users and groups (--existing)
AmbariServer
ExternalLDAP
Local users & groups
DB LDAP users & groups
Authenticate
Import/Sync
Page 29 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Summary of Changes
Ambari 1.6.1 or Earlier Ambari 1.7.0
Principals Users Users and Groups
Principal Types Local or LDAP Local or LDAP
User Administration All cluster “admins” can administer users.
Only users designated as “Ambari Admins” can administer users and groups and “Manage Ambari”.
Cluster Permissions Admin or Read-Only Operator or Read-Only
Cluster Access All users have read-only cluster access implicitly.
Must explicitly grant cluster permissions.
Disable User Access to Ambari
Not available Ability to set a user as “inactive”.
LDAP Authentication Supported Supported with additional parameters.
LDAP User Default Ambari Access
Implicit access on login. Must import / sync LDAP users with Ambari to enable access.
Page 30 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
What Happens on Ambari Upgrade?
Existing Ambari 1.6.1 or Earlier After Upgrade to Ambari 1.7.0
Local Users Migrated
LDAP Users Migrated
LDAP Authentication Re-run setup-ldap to set new properties
Non-Admin Users Granted “Read-Only” permission on cluster
Admin UsersGranted “Operator” permission on cluster
Flagged as “Ambari Admin”
IMPORTANT: Review + narrow this list as appropriate
Page 31 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework
Page 32 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ambari Views Framework
Goal: enable the delivery of custom UI experiences in Ambari Web
Developers can extend the Ambari Web interface• Views expose custom UI features for Hadoop Services
Ambari Admins can entitle Views to Ambari Web users• Entitlements framework for controlling access to Views
Page 33 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Example Views
Capacity Scheduler“Queue Manager” View
Hive Tez Query“Jobs” View
Page 34 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework
Views Framework vs. Views
Views
Core to Ambari
Page 35 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Components
• Serve client-side assets (such as HTML + JavaScript)
• Expose server-side resources (such as REST endpoints)
VIEWClient-side
assets(.js, html)
AMBARI WEB
VIEWServer-side resources
(java)
AMBARI SERVER
{rest}Hadoop
and other systems
Page 36 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Delivery
1. Develop the View (just like you would for a Web App)
2. Package as a View (basically a WAR)
3. Deploy the View into Ambari
4. Ambari Admins create + configuration view instance(s) and give access to users + groups
Develop DeployPackage Create Instance(s)
Page 37 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Versions and Instances
• Deploy multiple versions and create multiple instances of a view
• Manage accessibility and usage
Page 38 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Choice of Deployment Model
• For Hadoop Operators:Deploy Views in an Ambari Server that is
managing a Hadoop cluster
• For Data Workers:Run Views in a “standalone” Ambari Server
AmbariServer
HADOOPStore & Process
AmbariServer
Operators manage the cluster, may have Views deployed
Data Workers use the cluster and use a “standalone” Ambari Server for Views
Page 39 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views REST API
Resource Description
GET /api/v1/views List of available views
GET /api/v1/views/{viewName}/versions List of available view versions
GET /api/v1/views/{viewName}/versions/{version}/instances
List of view instances in a given version
GET /api/v1/views/{viewName}/versions/{version}/instances/{instanceName}
Details about a specific view instance.
Page 40 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Deployment
Place view-package.jar on Ambari Server
/var/lib/ambari-server/resources/views
Restart Ambari Server to start deployment
ambari-server restart
Deployment registers the view with Ambari and extracts the view
/var/lib/ambari-server/resources/views/work
Once complete, the view is available to create instances
Dev Tip: UI assets in extracted view can be modified w/o restart
Page 41 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework Terminology
Term Description
View Definition Describes the view resources and core view properties such as name, version and any necessary configuration properties.
View Package Packages the view client and server assets (and dependencies)
View Deployment Deploying a view into Ambari
View Version A specific version of a view
View Instance An instantiation of a specific view version
Framework Services View context, instance data, configuration properties and events
Page 42 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Definition (view.xml)
name – the internal name of the view (must be unique)
label – the public display name of the view
version – the version of the view
parameter – (optional) metadata about view instance properties
resource – (optional) the names of the classes required to support a view sub-resource
instance – (optional) property sets which define static view instances<view> <name>MYVIEW</name> <label>My View</label> <version>1.0.0</version> <parameter> <name>scope</name> <description>The scope of the instance.</description> </parameter></view>
Dev Tip: Use Static instances for development
Page 43 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Package
View Definition: view.xml
Resource / Service classes : JAX-RS annotated
UI classes : html, Servlets deployed as web app (WEB-INF/web.xml).
Application logic : Supporting classes
Dependencies : 3rd party jars or classes├── WEB-INF│ └── web.xml├── org│ └── apache│ └── ambari│ └── view│ └── weather│ ├── CityResource.class│ ├── CityResourceProvider.class│ ├── CityService.class│ └── WeatherServlet.class└── view.xml
view-package.jar
Page 44 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Context
• Available to the view server-side through injection.• Provides access to instance configuration.• Provides access to the view and instance attributes.• Provides access to run time information about the current
execution context, including authenticated user principal.
public interface ViewContext { public String getUsername(); public String getViewName(); public String getInstanceName(); public Map<String, String> getProperties(); public void putInstanceData(...); public String getInstanceData(...);}
Page 45 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Configuration Parameters
Describe required and optional view instance configuration parameters
<parameter>
Supports variables
${viewName} ${instanceName} ${username}
Access parameters from view context
ViewContext.getProperties()
Page 46 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Instance Data
Save key/value data per instance per user
Useful for information such as “user prefs” or other lightweight data
Access instance data from view context
ViewContext.putInstanceData(...)
ViewContext.getInstanceData(...)
Page 47 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Context Provides Authenticated User
VIEW
AMBARI SERVER
Ambari DB
LDAP
{rest}
<html>
User AuthN Source
User-Permission Mapping
Authenticate
Provide user and impersonator via
ViewContext
Page 48 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Server-Side Resources
Servlets
Ambari ResourceProviders
Plain ol’ REST endpoints (using whatever you want)
<resource> <name>city</name> <plural-name>cities</plural-name> <id-property>id</id-property> <resource-class>org.apache.ambari.view.weather.CityResource</resource-class> <provider-class>org.apache.ambari.view.weather.CityResourceProvider</provider-class> <service-class>org.apache.ambari.view.weather.CityService</service-class> </resource></view>
Page 49 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Framework and View Events
<view-class>viewClazz</view-class>
org.apache.ambari.view.View
ViewContext.getController()
ViewController.registerListener(…)
ViewController.fireEvent(…)
Listener.notify(…)
Deploy Create instances
onDeploy() onCreate()onDestroy()
instance
instanceinstance
fireEvent()
notify()notify()
Page 50 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Where Can I Learn More?
https://cwiki.apache.org/confluence/display/AMBARI/Views
https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md
https://github.com/apache/ambari/tree/trunk/ambari-views/examples
https://github.com/apache/ambari/tree/trunk/contrib/views
Page 51 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Stack Advisor
Page 52 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Stack Advisor
Extends Ambari Stacks to include a “Stack Advisor”
Provides recommendations for and performs validation on component layout & configuration
Improves Stack pluggability
Exposes new REST endpoints:
/recommendations
/validations
REST endpoints used during Cluster Install Wizard and Configs UI
Page 53 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Usability
Page 54 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Host Check: Transparent Huge Pages
Page 55 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
ZooKeeper Add/Delete
Add ZooKeeper Servers via Service Actions menu
Delete ZooKeeper Servers from Hosts page
Page 56 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Summary Tab “Restart Required” Banner
Page 57 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
NameNode HA: ZKFC Status Shown in Summary
Page 58 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Maintenance Mode Prompt
On Service Stop or Restart, include option to Turn On Maintenance Mode to suppress alerts
Page 59 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Launch HA Wizards from Services page
HDFS: NameNode HA YARN: ResourceManager HA
Page 60 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Abort Background Operations
Ability to abort operations that are running• On abort:
• Tasks that are queued on Server are removed (so they do not queue on Agents)
• Tasks that are queued on Agents are removed (so they do not execute)
• Tasks executing on Agents are killed (SIGTERM, pause, then SIGKILL if needed)
Page 61 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Learn More
Page 61
Resource Location
Apache Ambari Project Page http://ambari.apache.org
Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI
Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI