Click here to load reader

Informatica PowerExchange for MQ Series

  • View
    219

  • Download
    3

Embed Size (px)

DESCRIPTION

Informatica PowerExchange for MQ Series. HOW-TO | CIF CONSULT | Redouane BELBAHRI. Agenda. Messaging Fundamentals Message Recovery Testing MQ. Agenda. Messaging Fundamentals. 1. Messaging Fundamentals. - PowerPoint PPT Presentation

Text of Informatica PowerExchange for MQ Series

  • AgendaMessaging FundamentalsMessage Recovery Testing MQ

  • Agenda1Messaging Fundamentals

  • Messaging FundamentalsWebSphere MQSeries enables application programs to communicate with each other using messages and queues.This form of communication is referred to as commercial messaging. There are two methods for applications to communicate:Fire and ForgetRequest/Response

    Currently PowerExchange for Websphere MQ follows send-and-forget model, also known as Datagram, where once the message is put in the target queue, no reply is sent to the source queue. ReplytoQ and ReplyToQmgr ports are populated, but PowerCenter does not use these to send a reply to follow request-and-response model.

  • Pre-requisites for working with MQInstall the WebSphere MQ clientBefore you upgrade or install PowerExchange for WebSphere MQ, install the WebSphere MQ client or WebSphere MQ server on the machine that runs the PowerCenter Integration Service.Configure Channels for Queue ManagersIf the WebSphere MQ system does not have a client-connection channel, configure a channel for each queue manager you want to connect to. WebSphere MQ does not allow you to connect to multiple queue managers in a session.The Queue should be on the Distributed environment not on Mainframe

  • Application CommunicationsProgramAProgramBPut RecordGet RecordRecord QueueFire and ForgetReply toQueueProgramAProgramBRequest/ResponsePut RecordGet RecordRecord QueuePut Record Processed MessageGet Record Processed Message

  • What is a message?Message = Header(s) + Application Data

  • What is a Queue?A queue is simply a place to put data.This figure shows how messaging works in the simple case where the program putting the message and program getting the message are both on the same computer and connected to the same queue manager.

  • What is Queue Manager?A queue manager is the subsystem software which controls access to the individual queues assigned to it. The queue manager logs all activity with each individual queue thus creating an audit trail. Multiple queue managers can coexist with each other. The limiting factor is the availability of system resources.LOG/AUDIT TRAILPUTDATAGETDATAQUEUE MANAGERCOMMUNICATIONSAPPLICATIONS ANYWHEREACF2/SECURITYQUEUESTCP/IP

  • Publish/SubscribePub/SubBrokerPublisherPublisherSubscriberSubscriberWebSphere MQSeriesMessage/QueuingPublish DataPub/SubBroker(List Server)PublisherPublisherSubscriberSubscriberPublish DataSubscribe to Published Data

  • WebSphere MQ SourcesPowerCenter treats the message data component of a message as a field containing message data. The PowerCenter Integration Service can read message data in the following formats: Flat file (fixed-width or delimited) XML COBOL Binary

  • WebSphere MQ TargetsThe PowerCenter Integration Service can write data to the message header and the message data fields of a WebSphere MQ message. It can write message data to a WebSphere MQ message in the following formats: Flat file (fixed-width or delimited) XML COBOL Binary

  • What are the differences between the server and client connectivity to MQIf MQ Server is on a different machine than PowerCenter server, then MQ client should be installed on the machine hosting PowerCenter. If MQ server is on the same machine as PowerCenter, MQ client is optional.

  • Can you send a reply to the MQ sender using PowerCenter?Currently PowerExchange for Websphere MQ follows send-and-forget model, also known as Datagram, where once the message is put in the target queue, no reply is sent to the source queue. ReplytoQ and ReplyToQmgr ports are populated, but PowerCenter does not use these to send a reply to follow request-and-response model. If the requirement is to send a reply back to the sender then this has to be achieved on the MQ server with the data populated in the ReplytoQ and ReplyToQmgr ports.

  • High Volume System ConsiderationsPersistent: The Queue Manager first writes the message out to a log before it allows it to be visible in the queue. In a very high volume flow, if this is not handled correctly, it can lead to performance degradation and cause the logging to potentially fill up the file system. Non-persistent messages: are immediately visible in the queue for processing, but unlike persistent messages, if the Queue Manager or server crashes they cannot be recovered. To handle this type of flow volume. PowerCenter workflows can be configured to run in a Grid environment.

  • MQ Message FormatWhen reading an MQ Series message insure that the posted message is of format: MQSTRJMS client can send a message in MQHRF2 format which is not readable by PowerCenter.

    Note: to change in JMS Destination destination =jmsTemplate102.getDestinationResolver().resolveDestinationName(session, queueName, isPubSubDomain);((MQQueue)destination).setTargetClient(JMSC.MQJMS_CLIENT_NONJMS_MQ);

  • Does PowerExchange for MQSeries support messages with MQ RFH and RFH2 header formats?PowerExchange for MQSeries does not support MQ RFH and RFH2 header formats.A feature request (CR 188295) has been submitted to address this in a future release of PowerCenter.

  • What are "Rows per Message" and "Message Format" options in a static MQ target definition?When you configure a flat file or XML target with a static MQ queue connection the following options are available:Rows per message Message FormatRows per messagethe Rows per message' indicates the number of rows that would constitute a single message.Message FormatThe Message Format is the PowerCenter server to load data in the specific MQ format (such as MQSTR).

  • Associated Source QualifierYou create an associated source qualifier in a mapping depending on the WebSphere MQ message data format.You can create the following types of associated source qualifier transformations for WebSphere MQ message data:Source Qualifier transformation for flat file message dataNormalizer transformation for COBOL message data

  • Static WebSphere MQ Target DefinitionsUse a static WebSphere MQ target definition in a mapping when you only want to write message data to the WebSphere MQ target. When you use a static WebSphere MQ target definition, you can pass data in flat file format to the WebSphere MQ message data field. The message header fields in the target remain static. No data passes to the message header fields from the pipeline.When you use a static WebSphere MQ target, use a flat file target definition for the message data and configure the session to write messages to a message queue. The PowerCenter Integration Service does not maintain transactional consistency.

  • Agenda22Message Recovery

  • Does PowerExchange for MQSeries allow graph recovery and roll forward?PowerCenter supports recovery and roll back for MQSeries mappings. Guaranteed Message Delivery (GMD) which can be enabled for mappings that are configured with destructive read mode.

  • LoadTransformationExtractionMSQ Application 2 Application 1MSQWebSphere MQ Series ProcessFlat FilesLoadLoadLoadPowerCenter Integration Service can recover unprocessed messages from a failed session. When you enable message recovery for a real-time session, the PowerCenter Integration Service stores source messages or message IDs in a recovery file, recovery table, or recovery queue.

  • How is real-time session recovery information handled by PowerCenter? PowerCenter 8.1.x (or below) uses file-based recovery. The Integration Service writes real-time session recovery information to a local cache. When you enable the recovery option, theIntegration Service stores all the messages it reads from the messaging source in a local cache before processing the messages for the target. In recovery run, messages in cache are processed first and then normal processing resumes from the real-time source. At times messages are re-processed from recovery cache, resulting in duplicates in the target.Effective in PowerCenter 8.5.1, the Integration Service writes real-time session recovery information to a database table. The table helps maintain data integrity during recovery so no data is lost or duplicated. Messages are processed and committed to target first and then acknowledged back to the source system.

  • Rules and Guidelines for Message RecoveryThe Integration Service fails sessions that have message recovery enabled and contain any of the following conditions:The source definition is the master source for a Joiner transformation.You configure multiple source definitions to run concurrently for the same target load order group.The mapping contains an XML target definition.You edit the recovery file or the mapping before you restart the session and you run a session with a recoverystrategy of Restart or Resume.The Integration Service cannot connect to the recovery queue or recovery topic.The Integration Service does not write the recovery message to the recovery queue or recovery topic.If the number of messages that the Integration Service reads or writes from the message queue exceeds the message size limit, increase the message size limit or decrease the flush latency.

  • Cleanup Using Session RecoveryIf your mapping contains an MQSeries source, you can enable Destructive Read and session recovery in the session properties to ensure queue cleanup. During the session, the PowerCenter Server removes messages from the source queue after writing messages to the recovery cache folder. TheDestructive Read option does not delete rows that do not meet the Source Filter condition

  • Recovery cache file namesRecovery cache file names use the following format:

    pmgmd_[sourcestate][sessionstate][targetstate]_________.datExamplepmgmd_sourcestate_50fc0313_9762_4670_98e9_7e2e5b7a93b8_268_5_191_61_0_0_1_1.dat

    In this example:50fc0313_9762_4670_98e9_7e2e5b7a93b8 is the Repository GUID268 is the workflow ID 5 is the folder ID 191 is the session ID 61 is the transformation ID

  • Recovery cache file namesTo determine the name of the workflow use the workflow ID in the following query:

    SELECT WORKFLOW_NAME FROM REP_WORKFLOWS WHERE WORKFLOW_ID=268 AND SUBJECT_ID=5To determine the name of the session and folder use the session ID in the following query:

    SELECT SESSION_NAME, SUBJECT_AREA FROM REP_LOAD_SESSIONS WHERE SESSION_ID=191 AND SUBJECT_ID=5

  • Recovery TypesInternally, recovery can be divided into two types: incremental recovery and full load recovery.In incremental load recovery, the Integration Service uses the ROW_COUNT information in the PM_RECOVERY table to determine the number of rows committed to the target. When the session is recovered, previously loaded rows will be skipped. Thus, incremental recovery ensures that there are no duplicates loaded to the target. The Integration Service performs incremental recovery for relational, normal load targets.In full load recovery, data is loaded from the beginning during the recovery run. The Integration Service performs full load recovery for relational bulk load, flat file (including regular, MQSeries, and external loader), XML, and SAP BW targets. Recovery for non-relational targets is done using the pmgmd file created at the $PMCacheDir cache directory.

  • Recovery and TruncationDuring the recovery session run, the Integration Service ignores the Truncate Target Table option for relational normal load targets, since the load will continue from the last commit point. For relational bulk load targets the Integration Service allows the Truncate Target Table option in both normal and recovery run. Please see Table 1 for truncation and data load behavior in case of bulk and normal relational targets.

    At the end of a successful session run the Integration Servicedeletes all recovery information in the target database and in $PMCacheDir. TheIntegration Service always resets the recovery information when it runs the session normally, regardless of whether the previous session run succeeded or failed.

  • Recovery Scenarios (1/2)Recovery can be performed under the following scenarios: The workflow suspends. If the workflow suspends due to session failure, you can recover the failed session and resume the workflow. The workflow fails. If the workflow fails as a result of session failure, you can recover the session and run the rest of the workflow. The workflow completes, but a session fails. If the workflow completes, but a session fails, you can recover the session alone without running the rest of the workflow.In order to run a recovery session, the following options are available in the Workflow Manager: Recover Task. Use this option when you want to recover a single session. Recover Workflow From Task. Use this option when you want to recover the session and continue the workflow.

  • Recovery Scenarios (2/2)

    The following options are available in the Workflow Monitor: Resume/Recover. Use this option to resume a suspended workflow in recovery mode. Recover Workflow From Task. Use this option when you want to recover the session and continue the workflow. Recover Task. Use this option when you want to recover a single session.

  • Unrecoverable SessionsPowerCenter does not support session recovery in the following cases: The session uses non-pass-through partitioning. All partition points must be pass-through partitioning unless the partition point passes data to an Aggregator or Rank transformation. If Enable Recovery is enabled in the session properties, you can only select the default partition types for each partition point. The session uses database partitioning. PowerCenter does not support recovery for a session that uses database partitioning. You want to debug the mapping. You cannot perform recovery from the Debugger. You want to perform a test load. You cannot perform a test load if the session is enabled for recovery. If you enable recovery, the Workflow Manager disables the Test Load option.

  • What are the failure scenarios with JMS or MQ source with session recovery? The session reads the message from the, writes to the targets. The session then updates the recovery queue with the message ID of the just written message and issues a commit to both the message written and the recovery queue update. Once the recovery queue is updated and target committed, the message is removed from the source JMS or MQ queue.Failure scenarios during the above process:

  • Failure scenarios during the above processThe target goes down before commit:When the session restarts, it does not find the message ID committed in the target recovery queue. So the message is re-read from the queue and process goes on as usual.The target goes down after issue of commit:The session would have removed the message from the source as commit is done. Upon restart, the session would find the committed message in the target and process continues reading next available messages.The source goes down after read and before target commit:Upon restart, the session would re-read the message afresh as the message is not present in the target recovery queue.

  • Failure scenarios during the above processThe source goes down after read, after commit to target and before delete of source message:The session would have failed after commit but before delete message from the source. Upon restart, the session would read the target recovery queue, find the message still exists in the source but already committed to target. The session would just delete the message from the source.The source or target JMS is already down before the session starts:The session connects to source and target upon init. Recovery queue is out of the picture as session would fail during init of connection itself.

  • Controlling Queue Clean-up with Destructive ReadIf you select Destructive Read, and the mapping contains an associated source qualifier, the PowerCenter Integration Service ignores the Destructive Read option. To remove messages from the source queue for mappings that contain an associated source qualifier, use RemoveMsg(TRUE).Use RemoveMsg(TRUE) when a mapping contains an associated source qualifier.RemoveMsg(TRUE) removes messages from the queue after committing all data to the target. For example, enter the following filter condition:RemoveMsg(TRUE)

  • Agenda3Testing MQ

  • Test Using SOAPUI

    hello]]>

    Use this template:

  • Testing using RFHUTILIn DEV http://www-01.ibm.com/support/docview.wss?uid=swg24000637RFHUTIL allow you to put msg in a professional manner but it does also some msg conversion and high level message browsingExcellent tool for any EAI/Middleware developer on Top of MQ

  • Test connection between MQ client and MQ server outside InformaticaUse the amqsputc sample program to put a message on the queue at the server workstation. Go to the /usr/mqm/samp/bin directory, which contains these IBM sample programs.Put a message on the queue at the server using the ./amqsputc myQueue myQManagerName command. The Sample amqsput0 start target queue is myQueue message is displayed.Type some message text on one or more lines, followed by a blank line. The Sample amqsput0 end message is displayed. Your message is now on the queue and the command prompt is displayed again. Use the amqsgetc sample program to get the message from the queue back to the clientTo get the message from the queue located on the server, enter the /amqsgetc myQueue myQManagerName command.The sample program starts and your message is displayed. After a pause, the sample ends and the command prompt is displayed again.

  • Overriding Connection Attributes in the Parameter FileIf you use a session parameter to define a connection for a source or target, you can override the connection attributes in the parameter file. Use the $QueueConnectionName session parameter.When you define a connection in the parameter file, the Integration Service searches for specific, user-defined session parameters that define the connection attributes. For example, you create a Message Queue connection parameter called $QueueConnectionMyMQ and define it in the [s_MySession] section in the parameter file. The Integration Service searches this section of the parameter file for the rows per message parameter, $Param_QueueConnectionMyMQ_Rows_Per_Message.

  • Is it possible to use pmrep createConnection to create QUEUE connections in the repository? No, currently it is not supported to create QUEUE connections using the pmrep createConnection command. The QUEUE connections will have to be created from the Workflow Manager tool.

  • Many Thanks for you time today!

    **When you read data from a WebSphere MQ source, you read data from the message header and the message data components of a WebSphere MQ message. The PowerCenter Integration Service treats the message header component as message header fields. The message header fields contain information about the message on the queue. Using the message header data, you can filter messages you want to read. For example, you can read messages based on the message format, or you can filter messages that originated from a particular application. When you read data from a WebSphere MQ source, you read data from the message header and the message data components of a WebSphere MQ message. The PowerCenter Integration Service treats the message header component as message header fields. The message header fields contain information about the message on the queue. Using the message header data, you can filter messages you want to read. For example, you can read messages based on the message format, or you can filter messages that originated from a particular application. *Working with WebSphere MQ Targets OverviewThe PowerCenter Integration Service writes target data to a WebSphere MQ message queue. It can write data toone message queue or multiple message queues. The PowerCenter Integration Service writes data to themessage header fields and the message data field in the message. Depending on the type of data that you want towrite to the message queue, choose a dynamic or static target definition.Message RecoveryWhen you configure message recovery for a real-time session, the PowerCenter Integration Service can recover unprocessed messages from a failed session. When you enable message recovery for a real-time session, the PowerCenter Integration Service stores source messages or message IDs in a recovery file, recovery table, or recovery queue. If the session fails, run the session in recovery mode to recover the messages that the PowerCenter Integration Service did not process. LINK: https://communities.informatica.com/infakb/whitepapers/1/Pages/15211.aspx?docid=15211&type=external&index=124Use the REP_LOAD_SESSIONS view on the repository database to determine the name of the session associated with the session instance ID.

    The Recovery Cache Directory is not configurableProblem Description When specifying a value for Recovery Cache Directory in the session properties, the session automatically uses the INFA_HOME/server/infa_shared/Storage Directory. Cause This is a known issue and a change request (CR 215937) has been created to address this in a future release. Solution Workaround

    To ensure the pmgmd* files are written to a specific directory set the $PMStorageDir on the Integration Service.Use the REP_LOAD_SESSIONS view on the repository database to determine the name of the session associated with the session instance ID.

    The Recovery Cache Directory is not configurableProblem Description When specifying a value for Recovery Cache Directory in the session properties, the session automatically uses the INFA_HOME/server/infa_shared/Storage Directory. Cause This is a known issue and a change request (CR 215937) has been created to address this in a future release. Solution Workaround

    To ensure the pmgmd* files are written to a specific directory set the $PMStorageDir on the Integration Service.**In general, it is safe to assume a changed(edited and saved) mapping or session will result in an unrecoverable session or an indeterministic data load, depending on the extent of the changes. To perform consistent data recovery, the session properties for the recovery session must be the same as the session properties for the normal failed session. You might get inconsistent data if you perform recovery under the following circumstances:

    The Integration Service runs in Unicode mode and you change the session sort order. The Integration Service code page or source and target code pages change after the initial session failure. Data movement mode changes after the initial session failure. The session performs incremental aggregation and the Integration Service stops unexpectedly. The mapping uses a Sequence Generator transformation. The mapping uses a Normalizer transformation. The sources or targets change after the initial session failure. The mapping uses a lookup transformation that performs a lookup on the target table, or the data in the lookup table changes between session runs. An XML/VSAM source generates keys that the Integration Service uses to populate the target. You edited the mapping in a way that causes the Integration Service to distribute data differently.Note: The above design and behavior holds good for any type of JMS source/target. DX Active MQ, Tibco, IBM MQ etc.

    *Note: The above design and behavior holds good for any type of JMS source/target. DX Active MQ, Tibco, IBM MQ etc.

    *You first have to define your connection string , in this case set MQSERVER=QM12.BSL02.HUB/tcp/lhwwmqa6(1415)C:\Program Files (x86)\IBM\WebSphere MQ\bin>amqsputc.exe QM12D001Sample AMQSPUT0 starttarget queue is QM12D001

    *Message_Format*The following are the connection types supported with PowerCenter 9.1 infacmd createconnection command:SEQ, IMS, SAP, ODBC, VSAM, DB2Z, DB2, ORACLE, ADABAS, WEB SERVICES, SQL SERVER, and DB2I.

    *