Upload
leif-davidsen
View
498
Download
1
Embed Size (px)
Citation preview
© 2015 IBM Corporation
AME-2285: Using IBM MQ in Managed File Transfer EnvironmentsLeif Davidsen
Senior Product Manager
IBM Messaging
Files and file content are in every part of the enterprise
• Businesses and their applications use files across their
infrastructure to store valuable data
• Where the data is created is not always where it needs to be
consumed
• Files and file contents need to be moved to deliver value
1AME2285 - @LeifDavidsen
How do most organizations move files today?
• FTP• Typically File Transfer Protocol (FTP) is combined with writing
and maintaining homegrown code to address its limitations
• Why is FTP use so widespread?• FTP is widely available – Lowest common denominator
• Promises a quick fix – repent at leisure
• Simple concepts – low technical skills needed to get started
• FTP products seem “free”, simple, intuitive and ubiquitous
• Legacy File Transfer products• A combination of products often used to provide silo solutions
• Often based on proprietary versions of FTP protocol
• Can’t transport other forms of data besides files
• Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure – especially with SOA
• People• From IT Staff to Business staff and even Security Personnel
• Using a combination of email, fax, phone, mail, memory keys…
Most organizations rely on a mix of home-grown code,
several legacy products and different technologies … and
even people!
2AME2285 - @LeifDavidsen
Shortcomings of Basic FTP
Limited Flexibility
Limited Security
Limited visibility and traceability
Limited Reliability
Unreliable delivery – Lacking
checkpoint restart – Files can
be lost
Transfers can terminate
without notification or any
record – corrupt or partial
files can be accidentally used
File data can be unusable
after transfer – lack of
Character Set conversion
Often usernames and
passwords are sent with file –
as plain text!
Privacy, authentication and
encryption often not be
available
Non-repudiation often lacking
Transfers cannot be monitored and managed
centrally or remotely
Logging capabilities may be limited and may only
record transfers between directly connected systems
Cannot track the entire journey of files – not just from
one machine to the next but from the start of its
journey to its final destination
Changes to file transfers often require updates to many
ftp scripts that are typically scattered across machines
and require platform-specific skills to alter
All resources usually have to be available concurrently
Often only one ftp transfer can run at a time
Typically transfers cannot be prioritized
3AME2285 - @LeifDavidsen
FTP introduces risks in data movement
Process Risk
Delays in transferring files impacts collaboration with customers/partners
Integration files that are delayed impact SLAs
Failure of file delivery impacts the processes themselves
Security Risk
Data encryption and governance of sensitive information transmitted in files
Inability to apply corporate security policies to person-initiated file transfers
No visibility over the type and sensitivity of the data being transferred
No ability to support audit requirements
IBM’s Managed File Transfer solution addresses all of these risks
4AME2285 - @LeifDavidsen
Who benefits from Managed File Transfer
• Application Owners• Producers and consumers of data held in files
• Application developers likely to own files, own code and own existing file transfer mechanisms
• Increase reliability, improved timely access to data
• Infrastructure owners• Currently provide connectivity between systems, through middleware solutions
• Provide technical expertise to assist application owners
• Reduce recurring error handling and recovery, leverage middleware tools
• CxOs• Need to improve productivity of staff and IT
• Speed up business performance without impacting security
• Customers• Less risk of loss or exposure of data
• Improved business effectiveness and better, more accurate fulfilment
5AME2285 - @LeifDavidsen
IBM MFT Vision
Accelerate and simplify governance of the growing volume of business-
critical data movement within and beyond the enterprise with Smarter MFT
Maximize the agility and performance of dynamic business networks by reducing
the complexity, risk, and cost of file transfer
Visibility• Single view of transfer activity
• Transaction and business monitoring• Dashboards, analytics, and scorecards
Usability & Management• Persona-based, easy-to-use interfaces
• Unified control and configuration of infrastructure
• Community on-boarding and coordination
Universal• Any transport, any protocol, and any
partner• Global and cloud-enabled deployment• Broad platform coverage and industry
standards
Security & Performance• Assured delivery and high availability
• Protection of file data in transit and at rest• Accelerated transport and low latency
Connectivity• Integration with BPM to drive business processes
• Leveraging ESBs to enable service orientation• SOA Registry/Repository for lifecycle governance
6AME2285 - @LeifDavidsen
Comprehensive IBM Managed File Transfer
Monitor file transfer activity & SLAsIBM® Sterling Control Center
Peer-to-peer file transfers via file
systemIBM® Sterling Connect:Direct®
Message Centric
TransfersMQ MFT
Scope of this presentation
• A single packaged offering for IBM Managed File Transfer
• Sterling Connect:Direct and Sterling Control Center are packaged with
IBM MQ Managed File Transfer as WebSphere Managed File Transfer
• IBM MQ Advanced also includes these additional capabilities
B2B Onboarding and file transfer via
mailboxingIBM® Sterling File Gateway
Cloud service for B2B file exchangeIBM® Sterling File Transfer Service
7AME2285 - @LeifDavidsen
IBM offers comprehensive MFT Capabilities
IBM MQ Managed File Transfer provides file
transfer optimized for data delivery across IBM MQ
networks
Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and
between enterprises across Connect:Direct
protocol
Sterling Control Center Manage file transfer
activity across your file transfer servers including
Connect:Direct, FTP, MQ MFT and Sterling File
Gateway
Addressing multiple use cases and scenarios for both internal and multi-
enterprise file transfer
8AME2285 - @LeifDavidsen
MQ V8 Advanced – Universal Messaging including Managed File Transfer
WebSphere Managed
File Transfer
All functions available for one single price
Sterling C:D
Sterling C:C
MQ V8
MQ Telemetry
MQ AMS
MQ MFT
Sterling C:D
Sterling C:C
MQ MFT
MQ Advanced V8 Enterprise Messaging with publish/subscribe and
queuing for any application, on any OS, using
any language
Connects applications reliably and securely to
send data anywhere to any solution
Extends application messaging to Mobile,
Internet of Things and Managed File Transfer
Single, simplified licensing to provide all the
capabilities to connect the business, and
maximize data value
Do more with your MQ investment, lowering
operational overheads, and raising efficiencies
Ensures all possible functions and capabilities
are available instantly to allow the business to
respond faster to changing needs
Moves data from anywhere to anywhere with
included end to end monitoring and management
MQ Advanced V8
Available as WebSphere Managed File
Transfer Service, or WebSphere Managed
File Transfer Managed Endpoints
9AME2285 - @LeifDavidsen
A consolidated transport for both files and messages
• Traditional approaches to file transfer result in parallel infrastructures
• One for files – typically built on FTP
• One for application messaging – based on IBM MQ, or similar
• High degree of duplication in creating and maintaining the two infrastructures
• MQ MFT reuses the MQ network for managed file transfer and yields
• Operational savings and simplification
• Reduced administration effort
• Reduced skills requirements and maintenance
File TransfersApplication
Messaging
Consolidated Transport
for Messages & Files
10AME2285 - @LeifDavidsen
Applications exchanging file data
Components of a typical MQ MFT network
Agents
– The endpoints for
managed file transfer
operations
Commands
– Send instructions to agents
Log database or file
– A historical record of file
transfers
Coordination queue manager
– Gathers together file
transfer events
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands Log databaseor file
11AME2285 - @LeifDavidsen
Log databaseor file
Agents
Act as the end points for file transfers
Long running MQ applications that transfer files by splitting them into MQ messages
– Efficient transfer protocol avoids excessive use of MQ log space or messages building up on queues
Multi-threaded file transfers
– Can both send and receive multiple files at the same time
Generate a log of file transfer activities which is sent to the “coordination queue manager”
– This can be used for audit purposes
Associated with one particular queue manager (or MQ Appliance)
Agent state on queues
Applications exchanging file data
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands
12AME2285 - @LeifDavidsen
Log databaseor file
Commands
Send instructions to agents
and display information about
agent configuration
– Via MQ messages
Many implementations of
commands:
– MQ Explorer plug-in
– Command line programs
– Open scripting language
– JCL
– Documented interface to
program to
Applications exchanging file data
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands
13AME2285 - @LeifDavidsen
Log Database & File
Keeps a historical account of
transfers that have taken place
– Who, where, when… etc.
Implemented by the ‘logger’
component which connects to the
coordination queue manager
– Stand alone application
- Can log to database or file
– Or JEE application
- Can log to database only
Queryable via Web Gateway
– Also a documented interface
Applications exchanging file data
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands Log databaseor file
14AME2285 - @LeifDavidsen
Log databaseor file
Coordination Queue Manager
Gathers together information
about events in the file transfer
network
Not a single point of failure
– Can be made highly available
– Messages stored + forwarded
MQ publish / subscribe
– Allows multiple log databases,
command installs
– Documented interface
Applications exchanging file data
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands
15AME2285 - @LeifDavidsen
Log databaseor file
MQ MFT and the MQ Appliance
The MQ Appliance can be the Queue
Manager to provide both regular MQ
Queue Manager capabilities
No other MQ deployment needed
Also Coordination Queue Manager
capabilities
No files are stored on the appliance
No MQ MFT Agent needed on
appliance to support this
Highly available and robust
Secure with MQ AMS entitlement
built in
– Content encrypted based on policies
Applications exchanging file data
IBM MQ
Agent Agent Agent
“Coordination”
Queue Manager
Commands
16AME2285 - @LeifDavidsen
Example usage of monitoring + program execution
1. Application writes
file to file
system
Existing
Application
MQ
MFT
Agent
MQ
MFT
Agent
Existing
Application
2. Agent monitors file
system, spots arrival
of file and based on
rules, transfers the file
3. MFT transports file
to destination
4. At destination MQ MFT
writes file to file system
5. MFT can also start another
application to process the file
17AME2285 - @LeifDavidsen
Securing file data with SSL and IBM MQ AMS
MQ MFT supports transport
level encryption using SSL
Data is encrypted before it is
sent over a channel and
decrypted when it is received
IBMMQ
QueueManager
IBMMQ
QueueManager
svrconn channel
sndr/rcvrchannels
IBMMQ
QueueManager
IBMMQ
QueueManager
svrconn channel
sndr/rcvrchannels
When combined with MQ
Advanced Message Security
– Allows file data to be encrypted
at the source system and only
decrypted when it reaches the
destination system
– Data is secure even when at
rest on a queue
Agent Agent
AgentAgent
AME2285 - @LeifDavidsen
Handling both files and messages across MQ MFT
One file to one message
MQMFT
One file to a group of messages
One message to one file
A group of messages (or all messages on the queue) to one file
• File can be split based on:
• Size
• Binary delimiter
• Regular expression
• One message becomes one file
• Optionally, a delimiter can be
inserted between each message
used to compose the file
• One file becomes one message
MQMFT
MQMFT
MQMFT
19AME2285 - @LeifDavidsen
Universalmessaging backbone
IBM MQ Advanced V8 Architecture Summary
WebSphere Managed File Transfer (Includes MQ MFT, Sterling C:D, Sterling C:C)
• End-to-end audit trail across file transfers
• Reliable and efficient transfer of file data regardless of size
• Centralized monitoring of transfer events and management of transfers
MQ Advanced Message Security
Signs and encrypts data sent over MQ
to provide privacy and integrity
IBM MQ
Scalable, reliable, and efficient transport
for data
File-to-File
Message-to-File
File-to-Message
Message LevelSecurity
Mobile & M2M Messaging
IBM MQ Telemetry
Lightweight, low power, low bandwidth
messaging to mobiles and sensors
20AME2285 - @LeifDavidsen
A grocery retailerIBM messaging backbone delivers transaction data to central hub in near-real time
The need:
At this company, transaction data from more than 2,400 grocery stores
reached a central data warehouse via daily batch transfers, delaying
analysis and hampering theft prevention efforts.
The solution:
The company implemented an integrated message queuing
infrastructure that can transfer data from any store endpoint to the
central hub in near-real time
IBM MQ MFT software transfers files between the central hub and the
stores’ ESBs and in turn to other endpoint applications in the store.
The benefit:
Dramatically reduces time between customer transaction in
store and data reaching data warehouse
Provides a rock-solid messaging integration engine in a small footprint
Delivers a reliable infrastructure for transaction data and
file-based information
“WebSphere Message
Broker is rock-solid. It has
a relatively small footprint,
and it is a world-class
integration engine.”
—A software engineer,
grocery retailer
Solution components:
Software
IBM® Integration Bus
(formerly known as
IBM WebSphere®
Message Broker)
IBM MQ
IBM MQ Managed File
Transfer
21AME2285 - @LeifDavidsen
IBM offers comprehensive MFT Capabilities
IBM MQ Managed File Transfer provides file
transfer optimized for data delivery across IBM MQ
networks
Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and
between enterprises across Connect:Direct
protocol
Sterling Control Center Manage file transfer
activity across your file transfer servers including
Connect:Direct, FTP, MQ MFT and Sterling File
Gateway
Addressing multiple use cases and scenarios for both internal and multi-enterprise file transfer
22AME2285 - @LeifDavidsen
IBM® Sterling Connect:Direct
• What it does– High volume, large size, mission-critical file transfers
– Automation, retry-restart, for unattended operation, with never breached security
• What it allows clients to do– Move files with confidence and upgrade unreliable, unsecured FTP
– Handle growth in data volume and size without increase in staff
• How do I know I need it?– Regulatory, industry, or company security/compliance requirements continue to grow
– Business operations require assured delivery of critical data
Reliable and high performance solution for secure, point-to-point, bulk data transfers
Peer-to-peer file transfersIBM® Sterling Connect:Direct®
23AME2285 - @LeifDavidsen
Detail: IBM® Sterling Connect:Direct
• MFT solution using peer-to-peer file transfer to move data across the
enterprise and between trading partners
• Remediates FTP and failed audits by adding security, monitoring, and
reliability without replacing FTP scripts
• High-volume, bulk data transfers for corporate-to-bank and financial
data connectivity
• Data movement across a broad range of platforms, including
mainframe, midrange, and distributed
• Small footprint, robust integration with 3rd party schedulers, and rapid
deployment
24AME2285 - @LeifDavidsen
Trading Partner
Integrating IBM Sterling Connect:Direct and IBM MQ MFT
MFT
Audit
Agent
IBM MQ
Agent Agent
C:D
Bridge
Agent
C:D
Node
C:D
Node
C:D
Node
C:D
Node
C:D
Audit
Reference
Inside the MFT audit trail…
The audit information for each MFT transfer
references related C:D audit information
The Connect:Direct Bridge
capability supports managed file
transfers that span MFT and
C:D with a joined up audit trail
27AME2285 - @LeifDavidsen
Bridging MQ and C:D networks at a large publisher
Solution architecture for a large publisher using the C:D bridge as a way to consolidate its IT infrastructure while still retaining the communication protocols used to communicate with
partners
MQ
Pain Points
Existing solution is a hodgepodge of
different data transports from different
vendors
Difficult to integrate existing applications,
built on MQ messaging, with file data
arriving from business partners
IBM’s MFT suite helps
Bridge the MQ and C:D environments
Leverage existing MQ skills and investment
Message-to-file and file-to-message
End-to-end visibility of files and messages
Modernize batch applications
Modernize MFT with RESTful Web 2.0
interfaces
Internal connectivity standardized on an MQ-backbone
Existing MQ applicationsCustomer #1
Customer #2
Customer #n
…
C:D bridge
Customers request that publications
can be submitted to the publisher via the C:D
protocol
Publisher
28
IBM offers comprehensive MFT Capabilities
IBM MQ Managed File Transfer provides file
transfer optimized for data delivery across IBM MQ
networks
Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and
between enterprises across Connect:Direct
protocol
Sterling Control Center Manage file transfer
activity across your file transfer servers including
Connect:Direct, FTP, MQ MFT and Sterling File
Gateway
Addressing multiple use cases and scenarios for both internal and multi-enterprise file transfer
29AME2285 - @LeifDavidsen
IBM® Sterling Control Center
Delivering higher levels of service for key integration and transfer services
• What it does
• Consolidated logging, reporting, and analytics for all transfers and integration
processes
• Rule-driven service level management
• Alerts and notifications
• Configuration management
• How it is delivered
• Java-based solution for management and visibility of the file transfer and B2B
integration infrastructure
• Engine, agents, and user consoles
• Interfaces to existing system management tools
30AME2285 - @LeifDavidsen
IBM® Sterling Control Center
• What it allows customers to do
• Have visibility into all file transfer activity including Sterling Connect:Direct and
IBM MQ MFT from a central console
• React to transfer events before they impact SLAs
• Easily provide compliance reporting and mobile monitoring
• Central configuration management for Sterling Connect:Direct® servers
• How do I know I need it?
• Increasing numbers of transfers or B2B interactions are governed by SLAs
• Increasing compliance, governance and audit requirements
• Pressure to decrease IT operational cost for B2B or file transfer operations
• What differentiates IBM Sterling?
• Product specific knowledge of file transfer events, business processes and SLAs
to build rules around
• Access from mobile devices
• Scale and performance – test environment with moderately-sized hardware
performed well with 5K servers producing 540K processes and 4.3M events
per day
31AME2285 - @LeifDavidsen
IBM offers comprehensive MFT Capabilities
IBM MQ Managed File Transfer provides file
transfer optimized for data delivery across IBM MQ
networks.
Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and
between enterprises across Connect:Direct
protocol
Sterling Control Center Manage file transfer
activity across your file transfer servers including
Connect:Direct, FTP, MQ MFT and Sterling File
Gateway
Addressing multiple use cases and scenarios for both internal and multi-enterprise file transfer
Available together as WebSphere Managed File Transfer, or as Sterling
Connect:Direct Advanced. These entitlements are also combined inside the
IBM MQ Advanced offering
34AME2285 - @LeifDavidsen
Adding MQ Managed File Transfer to your infrastructure
• Key benefits and highlights
• Add reliability, manageability, security when moving your file data
• Infrastructure team can manage a wider set of data movement through a single middleware layer
• Speeds delivery of value to businesses from file-based assets
– File to message; message to file
– File to file transfer also available
– Also bridge to FTP
• Deploy flexibly throughout your infrastructure to match your MQ environment
• Deploy MQ MFT agents where your files are
• Deploy MQ Advanced or MQ MFT Service on your MQ Queue Managers
• Leverage the MQ Appliance to provide MQ services in a MFT solution
35AME2285 - @LeifDavidsen
1990s 2000s 2010s
V2 V5 V5.1 V5.3V5.2 V6 V7 V7.0.1V1.1
Multi-platform Managed File Transfer
V7.1 V7.5
System PatternApplication Pattern
MQIAssured delivery
Integrated Messaging
Hypervisors
JMS 1.1RCMS
Common criteriaEclipse
HTTPAJAXREST
Web 2.0
.NET
SSLSOAP
JMS
XML
Mobile
V2.1
Pub/sub
Multiple clusterXMIT queue
Multi-instanceQMGR
20th Anniversary of MQ at IMPACT
Timeline for MQ and Messaging product releases
V8
IBM MQ
Advanced for
Developers
IBM MQ
Advanced
IBM
MessageSight
IBM MQ
Advanced
Message
Security
IBM MQ
Managed File
Transfer
IBM MQ Low
Latency
IBM
MQSeries IBM MQTTIBM MQ
Everyplace IBM MQ
Appliance
37AME2285 - @LeifDavidsen
Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the
agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.
Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank YouYour Feedback is
Important!
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.