Upload
vanduong
View
226
Download
5
Embed Size (px)
Citation preview
1
Proactive Monitoring for PowerCenter:
What’s Next in Governance and Operational Environmental Monitoring
Chris Carlson
Director, Product Evangelist
Informatica
#IW2012
2
Agenda
• Overview of Proactive Monitoring for
PowerCenter
• Background of the Solution
• Technical Overview
• What’s Next in Operations and Governance
• Product Demonstration
3
Overview
• Capabilities and features of Proactive Monitoring for
PowerCenter Operations and Governance
• What’s new: environmental variables (CPU, memory, etc.) and
governance rules (enforcing best practices)
• How to use Proactive Monitoring to achieve faster response
times to potential operational and governance issues in your
PowerCenter environment
The Proactive Monitoring for PowerCenter solutions have dozens of prebuilt
rules and pattern logic combined with the flexibility of self-service that allows
users to easily modify applications without going through long cycles.
These solutions address a diverse array of potential issues, including session
and workflow breakdowns, trending data issues, and the vast area of
development governance.
4
Why did we Build this? What Are Some Customer Pains?
“We invest a lot on resources (manpower) to monitor
PowerCenter – and we still miss many issues”
“We have tools to report on performance after an
issue occurs, but nothing that helps us prevent it”
“I want out of the custom scripting business for
monitoring. Not scalable and not effective anymore”
“We have no way at all to enforce best practices”
5
Informatica Built these Solutions to Address Three Common Customer Concerns…
1. Customers asked for a “complete, PowerCenter-centric” Monitoring Solution
2. Customers want to become “Proactive” in seeing issues before they turn into problems. “Smart push” the alerts instead of static dashboard displays.
3. Scalable way to identify anomalies such as Workflows running longer/shorter than normal or returning more/less rows than normal.
6
Time Spent on PowerCenter Deployments
Development
Testing
Monitoring of
PowerCenter Workflows
and
Environment
Best
Practices
Where is your time spent? ROI Associated with PMPC
• 70% Reduction on Time Spent Manually Monitoring Workflows
• 80% Reduction on Time Spent Manually Monitoring Best Practices
• Soft ROI • Reduced Help Desk Costs • Avoid Issues • Increased Reliability
Regain substantial time for new development, new projects, etc…
7
What is Proactive PowerCenter Monitoring?
Prebuilt Solution to Monitor PowerCenter
Delivers Alerts to people, dashboards and systems
Complete Operational and Best Practices
Extensible – build
unlimited number of new rules
8
Move from Reactive to Proactive
Time
Event(s)
• Workflow is started
• Changes are made
• Workflows are Named
• Paths are hardcoded
Issue(s)
• Workflows run long
• Developers make many
changes
• Source rows rejected
• Development Practices
slip to Production
Problems and Losses
• Schedules are missed
• No rows are loaded
• Zombie Workflows
• Workflows Fail
• Phone Calls At 3am
• Business doesn’t get
reports
Proactive Reactive
9
Customer Case Study
PowerCenter
Continuous growth in complexity
Case Study: A media enterprise
No of folders: 112 No of schedules: 6,336
No of mappings: 18,180 Work flow executions per day:
3,310
No of transforms: 93,940 Session executions per day:
10,402
No of sessions: 21,879
No of workflows: 13,109
One Shared Service Environment
Challenges:
• Existing monitoring tools did
not look at PowerCenter
internals
• No Possible Way to Monitor
over 10,000 Sessions Per Day
with existing tools or manually
Solution:
• Proactive PowerCenter for
Operations monitors 24X7
in a “Proactive Manner”
• Can now catch data issues
like “less data” than
expected because of
completeness of solution
10
Solution Overview
Proactive PowerCenter Monitoring
License-Limited + Rule Pack
Alert Enrichment
DBs, Web Services and more…
Alert Editing, Creation and Management
Custom: Environmental (CPU, Memory..)
DB (SQL)
JMS Web Services
• Operational
• Design Time
• Start/Stop Workflows
• Create Help Desk Ticket
• Update Control Tables
SQL Sources to Expose Repository Features and Allow Easy Rule and Report Building Environmental Monitoring Runtime Statistics Receiver Ping PowerCenter Services
Notification Framework for Alerts Prebuilt EMAIL Responses Prebuilt Dashboard Alerts PowerCenter Workflow Response to Execute Commands
11
Component Architecture
Monitor & Alert
Proactive Monitoring
for PowerCenter (Source Feeds, Rules/Templates,
Watchlists, Alerts)
Workflow
Controls
Alerts
(Dashboards, e-mail, DBs, other systems, etc)
Real time Workflow and
Session Data PowerCenter
Runtime
Workflow
Statistics PowerCenter
Repository
PowerCenter Server (CPU,
Memory)
PowerCenter Repository
(Table space, DB Ping)
Environment
Monitoring
12
Alert Scenario Categories
Category Scenarios
Stop Light - Are Domain services running?
Breakdowns - Did a Session/Workflow fail? When?
- Did something recently change?
- Are there design-time artifacts in
production (e.g. hardcoded paths)
Data Issues - Failed rows in Session/Workflow
- Succeeded Workflow with Zero
records loaded
Trending - Workflows deviating from norm
(defined or dynamic threshold)
Environmental - Failures correlated with CPU load,
memory usage, process
13
Proactive Monitoring for PowerCenter Operations Alerting Rules & Templates (ships with over 23 scenarios)
Operational Monitoring
1 Workflow/session was successful but zero
(0) records loaded
2 Rejected Records in Session
3 Process running on integration service node
with more than X% CPU usage
4 Repository service database table space
usage is more than X%
5 Session/Workflow Failures
6 SLA Violations: Workflow/Folder/Repository
Level
7 Significant increase in Workflow/Session
Elapsed Times
8 Significant decrease in applied row counts
9 Workflows missing schedules
10 Too many concurrent workflows
11 Automatic Restart of Zombie Workflows
12 Ping Domain/Integration Service/Repository
Service
Plus optional Environmental Variables:
Node Operating System CPU, Memory , and Processes
PowerCenter Repository Database – Heartbeat and
Tablespace checks
• Identify top memory and CPU consumers
• Correlate failures to environmental issues
• Faster identification of root causes
• Build new alerts for any number of correlations
14
Sample Alert Missed Schedule Enriched with Environmental Data
15
Ensuring the Right People get the Right Amount of Alerts
• Targeted Alerting
• Administrators
• Operations
• Developers
• Architects
• Set Alerts on…
• Folders
• Repository
• Domain
• Workflow
• Session
• Transforms
16
Integrate Alerts with your IT processes
• Multiple output channels (one, many, all):
• Send email to user / distribution group
• SMS TXT, pager
• Instant Message alerts (XMPP)
• Trouble ticketing systems
• Network management systems
• Dashboards and reports
• Alert management features:
• Alert aggregation, suppression, snooze
• “Do not disturb” / exception lists
• URL Link to knowledge bases and troubleshooting steps for 1st/2nd level support personnel
• Configure to support monitoring for alert acknowledgement
17
Proactive PowerCenter Governance: What is it?
It is a framework to implement alerting and reporting to monitor
best practices related to the use of
(Transform/Session/Workflow) attributes to control the
behavior of PowerCenter
• Who can use it?
• Managers of PowerCenter Development teams
• PowerCenter Administrators
• Which attributes are covered?
• Almost all the attributes that are configured through PowerCenter Designer and Workflow Manager (in the Properties and Session Config tabs) are covered
• How many attributes are there?
• Approximately 250 attributes
18
Proactive Monitoring for PowerCenter Governance Alerting Rules & Templates (future)
Governance & Best
Practices Naming Conventions:
Mappings/Sessions/Workflows
Invalidated Mappings/Sessions/Workflows
Transforms saved with Default Values
Saving objects without comments
Frequent Changes to Mappings/Sessions
Hardcoded Paths in Object definitions
Illegal Command tasks used in Workflows
Duplicate Mappings/Sessions/Workflows
Customizable Session Configuration alerts
License allows for limitless rules
Initially we created a set of “High Level” Best
Practices Rules
Templates with over 250 pre-mapped attributes makes
creating governance rules fast and effective.
19
Top Governance Attribute Alerts (that can impact production)
• Lookup SQL Override is set
• Reset option in Sequence Generator is checked
• Tracing is set to Verbose
• Commit Interval is set too high
20
Scheduled Governance Reporting
21
Demonstration
• Run Time Failure after Design Time Change
• Workflows Runs Longer than its past Average
and Misses SLA
• Design Time Best Practices Enforcement***
• Build a New Throughput Rule Using the
RulePoint Wizard
22
Informatica Marketplace Overview A Trusted, Open Ecosystem
• Virtual Marketplace for Data Integration Apps
• Solutions across all technology areas – DI, DQ, MDM, Cloud, etc.
• Open Ecosystem – Apps from Partners, ISVs, Consultants, and Developers
• Seal of Approval ensures App quality
• More than 600 Apps, over 200 Free!
• 15k visits per month, 2k downloads
http://marketplace.informatica.com
R
M
23
Thank You