Wso2 esb-maintenance-guide

Chanaka FernandoSenior Technical Lead

WSO2 Inc.

WSO2 ESB maintenance guide

Agenda•Monitoring/Tracing messages

• Logging with ESB

• Troubleshooting

•Debugging

•Best practices

Source : http://bonfirehealth.com/week-13-insights-spark-integration/

Monitoring ESBMessage Tracing is used to trace, track and visualize the message body in each intermediate stage of its transmission. This is useful for a number of reasons, including audit trails and debugging.

l Management console

l BAM mediator

l Mediation statistics data agent

l Service statistics data agent

l BAM message tracer

l JMX monitoring

Monitoring ESB (Management Console)Management console

l Mediation statistics graphical view

l System statistics

l SOAP tracer

l Mediation tracer

l System logs

l Application logs

Monitoring ESB(BAM mediator)BAM Mediator can be used track the messages at different points of mediation flow

Create a BAM server profile and configure the data streams for the BAM mediator

Publish data using the BAM mediator inside the mediation flow

Unique ActivityID will be created for each message. This will be used for correlating the messages at different points

Activity monitoring dashboard can be used to view and correlate the messages published to BAM

Monitoring ESB(Mediation Stat agent)Mediation Statistics data agent in the ESB can be used to monitor the statistics like average response times, number of requests related to mediation components like proxy services, sequences, etc .. from the BAM

BAM tool box available OOTB

Provides request count/ response time/ average/ min/ max and total for proxy/sequence/endpoint level

Monitoring ESB(Service Stat agent)Service statistics agent can be used to monitor the statistics for deployed services like proxy services

Monitoring ESB(BAM message tracer)Message Tracing is used to trace, track and visualize a message's body in each intermediate stage of its transmission.

This is useful for a number of reasons, including auditing and debugging.

You can use the Activity Dashboard of WSO2 BAM to trace messages going through WSO2 ESB.

It allows you to track and trace messages in real-time within Cassandra, without having to write Hive scripts

Monitoring ESB(jmx monitor)

ESB exposes a number of management resources as JMX Mbeans

These MBeans can be accessed remotely using a JMX client such as Jconsole.

JMX Service URL at server start up

INFO - JMXServerManager JMX Service URL :service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

Can be used to monitor memory/cpu/threads/classes related information

Latency related information also available

Can be used with any tool which is capable of monitoring jmx port

Logging with WSO2 ESBWSO2 ESB uses log4j for logging messages

Logs are saved in ESB_HOME\repository\logs directory

Types of log files

l HTTP Access logs – records all http requestsl WSO2 carbon logs – provides application and runtime

specific informationl Service logs – records mediation time audit messagesl Trace logs – records tracing information about the

services which has tracing enabledl Errors logs – records errors occurred in ESB

Logging with WSO2 ESB

Log mediator can be used to log specific messages during mediation flow.

<log [level="string"] [separator="string"]><property name="string" (value="literal" | expression="xpath")/>*</log>

4 levels of logging

l Simple - To, From, WSAction, SOAPAction, ReplyTo, MessageID and any properties.l Headers - All SOAP header blocks and any property.l Full - All attributes included in log level "simple", the SOAP envelope and any

property.l Custom - The only property specified in the Log mediator


Wire level messages can be viewed by enabling wire logging

For NHTTP transportlog4j.logger.org.apache.synapse.transport.nhttp.headers=DEBUGlog4j.logger.org.apache.synapse.transport.nhttp.wire=DEBUG

For passthrough transportlog4j.logger.org.apache.synapse.transport.passthru.wire=DEBUGlog4j.logger.httpclient.wire.content=DEBUG

You can configure log4j per service using log4j.properties file. An example is given below.

log4j.category.SERVICE_LOGGER.StockQuoteProxy=DEBUG, SQ_PROXY_APPENDERlog4j.additivity.SERVICE_LOGGER.StockQuoteProxy=falselog4j.appender.SQ_PROXY_APPENDER=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.SQ_PROXY_APPENDER.File=logs/stock-quote-proxy-service.loglog4j.appender.SQ_PROXY_APPENDER.datePattern='.'yyyy-MM-dd-HH-mmlog4j.appender.SQ_PROXY_APPENDER.layout=org.apache.log4j.PatternLayoutlog4j.appender.SQ_PROXY_APPENDER.layout.ConversionPattern=%d{ISO8601}[%X{ip}-%X{host}] [%t] %5p %c{1} %m%n


Hierarchy of Levels:

TRACE - Designates finer-grained informational events than the DEBUG. DEBUG - Designates fine-grained informational events that are most useful to debug an application. INFO - Designates informational messages that highlight the progress of the application at coarse-grained level. WARN - Designates potentially harmful situations. ERROR - Designates error events that might still allow the application to continue running. FATAL - Designates very severe error events that will presumably lead the application to abort.


Performance Considerations of LoggingThis section describes several performance considerations of logging.1. Using logger.isDebugEnabled() method saves string concatenation. For example,if(logger.isDebugEnabled()){logger.debug("java logging level is DEBUG Enabled");}2. Saving sensitive information like passwords, social security/credit card numbers etc. as plain text in log files is notrecommended. Either do not store such information in application or remove it as soon as you are done.3. Use proper log levels from the options given below:INFO > DEBUG: Changing the level from INFO to DEBUG increases resource usage, because all the stacktraces and information useful to debug the application will appear.WARN > INFO: Changing WARN to INFO increase the log file size several times than before, because theWARN level is only displayed when a warning should be given.ERROR > WARN: Changing ERROR to WARN can slight increase the file size.FATAL > ERROR: Changing FATAL to ERROR increases size only slightly. The difference of these twolevels is non-fatal errors.

Troubleshooting WSO2 ESBl Enable debug logs when something goes wrong in WSO2 ESBl It can be done for all the components by enabling DEBUG level for the root logger.

Better to enable debug logs for the relevant componentsl Enabling DEBUG level logs for synapse core (Which is the core of the WSO2

ESB)Shutdown the ESB server- open log4j.properties file from a text editor. The log4j.properties file is located at <ESB_HOME>/repository/confdirectory.- Set log level to debug for log4j.category.org.apache.synapse as followslog4j.category.org.apache.synapse=DEBUG- Start ESB server

This will print useful information related to the mediation flow inside ESB. We can use this log for troubleshooting

DEBUG - IterateMediator Start : Iterate mediatorDEBUG - IterateMediator Splitting with XPath : //m0:getQuote/m0:request resulted in 4 elementsDEBUG - IterateMediator Submitting 1 of 0 messages for processing in parallel

Troubleshooting WSO2 ESBl Trace logs trace the entire path when a message travels along a mediation

sequence. Tracing can be enabled for a proxy or for a sequence. This can be done by adding the following attribute for the proxy/sequence configuration;

l trace="enable"l When tracing is enabled, trace logs can be seen from the mediation tracer

management console UI or from the wso2-esb-trace.log file. This file is located at the <ESB_HOME>/repository/logs directory.

l This will provide more fine grained information than the debug logs

Here is a sample trace log

22:15:43,020 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER Proxy Service SplitAggregateProxy received a new message from :127.0.0.122:15:43,020 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER Message To: /services/SplitAggregateProxy22:15:43,020 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER SOAPAction: urn:getQuote22:15:43,020 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER WSA-Action: urn:getQuote22:15:43,021 [-] [PassThroughMessageProcessor-1] TRACE TRACE_LOGGER Envelope : <?xml version="1.0"encoding="utf-8"?><soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body></soapenv:Body></soapenv:Envelope>22:15:43,023 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER Using the anonymous in-sequence of the proxy service formediation22:15:43,023 [-] [PassThroughMessageProcessor-1] INFO TRACE_LOGGER Start : Sequence <anonymous>22:15:43,023 [-] [PassThroughMessageProcessor-1] TRACE TRACE_LOGGER Message : <?xml version="1.0"encoding="utf-8"?><soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body></soapenv:Body></soapenv:Envelope>

Troubleshooting WSO2 ESBWhile developing integration scenarios with the ESB, sometimes we might get into situations where the expected results are not received for the service invocations. - The message payload sent by the ESB is not in the format that is expected by the backend service- The content type of the sent message is not supported by the backend service- Some of the required headers like ‘Authorization Header’ is missing in the request sent from the ESB.To debug these type of issues, we could inspect the messages that are transferred over the connection and verify whether they are in the format that is expected by the backend service. After inspecting the transferred messages, we can amendthe messages to make it compatible with the backend service.

To inspect messages passed in the connections, we can use two mechanisms.

l TCPMon

l WireLogs

Troubleshooting WSO2 ESB (TCPMon)We can use TCPMon tool to monitor messages going through http transport towards ESB and from ESB. This is really helpful in troubleshooting scenarios

There are two connections in this scenario.1. Client-to-Proxy Service connection2. Proxy-to-Backend Server connection

Troubleshooting WSO2 ESB (TCPMon)We can monitor messages flowing between the Proxy service and the Backend by placing TCPMon as follows. In this occasion we need to temporarily change the port of the endpoint to point to the TCPMon which can be done withoutrestarting the ESB.

Troubleshooting WSO2 ESB (TCPMon)We can monitor messages flowing between client and the Proxy service by placing TCPMon as follows

Troubleshooting WSO2 ESB (WireLogs)WireLogs functionality is also used to monitor the messages flowing through WSO2 ESB over http transport. It has the following advantages over TCPMon tool

l Can monitor messages going through https transportl Do not need to change the synapse configuration

You can enable wire logs by following these steps

- Shutdown the ESB server- Open log4j.properties file from a text editor. This file is located at the <ESB_HOME>/repository/conf directory.- Uncomment the following entry;log4j.logger.org.apache.synapse.transport.http.wire=DEBUG- Start the ESB server

The above described procedure applies to WSO2 ESB 4.7.0 or higher versions. The entry to uncomment in log4j.properties is slightly different in previous versions.For ESB 4.6.0 :log4j.logger.org.apache.synapse.transport.passthru.wire=DEBUG (add this line to log4j.properties)For ESB 4.5.1 or previous versions (PTT is not available in these versions):log4j.logger.org.apache.synapse.transport.nhttp.wire=DEBUG

Troubleshooting WSO2 ESB (WireLogs)

[2015-01-11 11:08:20,076] DEBUG - wire >> "POST /services/RespondProxy.RespondProxyHttpSoap12Endpoint HTTP/1.1[\r][\n]"[2015-01-11 11:08:20,076] DEBUG - wire >> "Content-Type: application/soap+xml; charset=UTF-8; action="urn:getQuote"[\r][\n]"

[2015-01-11 11:08:20,097] DEBUG - wire << "POST /services/SimpleStockQuoteService HTTP/1.1[\r][\n]"[2015-01-11 11:08:20,097] DEBUG - wire << "Content-Type: application/soap+xml; charset=UTF-8; action="urn:getQuote"[\r][\n]"

[2015-01-11 11:08:20,103] DEBUG - wire >> "HTTP/1.1 200 OK[\r][\n]"[2015-01-11 11:08:20,103] DEBUG - wire >> "Content-Type: application/soap+xml; charset=UTF-8; action="urn:getQuoteResponse"[\r][\n]"

[2015-01-11 11:08:20,113] DEBUG - wire << "HTTP/1.1 200 OK[\r][\n]"[2015-01-11 11:08:20,114] DEBUG - wire << "Content-Type: application/soap+xml; charset=UTF-8; action="urn:getQuoteResponse"[\r][\n]"

Troubleshooting WSO2 ESB (WireLogs)

Wire Logs for Callout Mediator/Forwarding Message Processor

The callout mediator and the forwarding message processor uses the Axis2 CommonsHTTPSender to invoke services. It does not use the non-blocking nhttp/passthrough transports. Hence, to enable wire logs we have to follow a differentapproach as follows;

- Shutdown the ESB server- Open log4j.properties file from a text editor.- Put the following two entries into the log4j.properties.log4j.logger.httpclient.wire.header=DEBUGlog4j.logger.httpclient.wire.content=DEBUG- Start the ESB server

Troubleshooting WSO2 ESB (Timeouts) l Timeout related issues are the one of the main causes for most of the support

queriesl Lack of knowledge about timeout configurations made it hard to tune up the systemsl WSO2 ESB has several timeout configurations

Troubleshooting WSO2 ESB (Timeouts) l There are two connections involved in the scenario; ‘Client-to-Proxy connection’ and

‘Proxy-to-Backend connection’. l They are two separate connections that do not depend on each other. This means

that even if one connection times out, the other connection won’t time out at the same time.

Client-to-Proxy Connection Timeout Parameters

http.socket.timeout:In the client-to-proxy connection we have to consider only one configuration parameter. That is http.socket.timeout whichcan be configured from the passthru-http.properties file. This file is located at the <ESB_HOME>/repository/confdirectory. This represents the socket timeout value of the passthrough http/https transport listener.

If we use NIO transport instead of the default passthrough transport we need to use the below parameter in nhttp.properties file

ESB 4.7.0 or later - http.socket.timeout.receiverESB 4.6.0 or prior – http.socket.timeout

Troubleshooting WSO2 ESB (Timeouts)

Proxy-to-Backend Connection Timeout Parameters Timeout configuration for proxy-to-backend connection is somewhat advanced compared to the client-to-proxy connection parameters.http.socket.timeout:This represents the socket timeout value of the passthrough http/https transport sender. This is the same parameter used in transport listener (above).

Endpoint timeout:This is the timeout configuration parameter which can be configured at the endpoint level. It allows us to configuredifferent timeout values for different endpoints.

It is a must to configure the http.socket.timeout to a higher value than all the endpoint timeout values.

Endpoint timeout <= http.socket.timeout


synapse.global_timeout_interval:

Synapse is a complete asynchronous engine which does not block worker threads on network IO. It registers a callback and return. When the response is received, the registered callback is used to correlate it with the request and further processing is done. If the backend server does not respond back, it is required to clear the registered callbacks.

This is done by TimeoutHandler. The ‘synapse.global_timeout_interval’ parameter represents the time duration that a callback should be kept in the callback store.

If we have configured a timeout duration at the endpoint level, this global timeout value is not taken into consideration for that particular endpoint. For all the other endpoints that doesn’t have a timeout configuration, this global parametervalue is considered as the timeout duration.

synapse.global_timeout_interval can be configured from the synapse.properties file. This file is located at the<ESB_HOME>/repository/conf directory. The default value is 120 seconds.


TimeoutHandler is executed in an interval of 15 seconds. So the callbacks get cleared can be deviated up to 15 seconds from the configured value. The TimeoutHandler execution interval can be configured by defining the followingproperty at synapse.properties file: synapse.timeout_handler_interval

Timeout Configuration for Forwarding Message Processor/Callout MediatorTimeout configuration of the callout mediator and the forwarding message processor is completely different from the above described procedure.Timeouts can be be configured at the axis2_blocking_client.xml which is located at the <ESB_HOME>/repository/conf/axis2 folder.

SO_TIMEOUT and the CONNECTION_TIMEOUT are the parameters which should be configured for the http transport senders. The following is a sample configuration http transport;

<transportSender name="http" class="org.apache.axis2.transport.http.CommonsHTTPTransportSender"><parameter name="PROTOCOL">HTTP/1.1</parameter><parameter name="Transfer-Encoding">chunked</parameter><parameter name="cacheHttpClient">true</parameter><parameter name="defaultMaxConnectionsPerHost">200</parameter><parameter name="SO_TIMEOUT">120000</parameter><parameter name="CONNECTION_TIMEOUT">120000</parameter></transportSender>

Debugging WSO2 ESB

WSO2 source code is open and you can download the source from the public repositoryhttps://svn.wso2.org/repos/wso2/carbon/

Latest code will be available in GitHubhttps://github.com/wso2

You can debug the code section which is causing problems by following the steps given below.

1) Locate the component of the product which is causing the issue. You can get an idea about the component from the error log

2) Download the relevant source code component to your computer

3) Create a project from an IDE using the downloaded source code. WSO2 provides maven pom file to create maven projects from the source

4) Start the ESB server with the option -debug <Port number>

5) Connect to the process from the IDE with the above port number

6) Now you can debug the source code

Best practices (Error handling)l If a sequence explicitly defines a fault handler using the onError attribute, the ESB

invokes the specified fault handler whenever an error occurs in the sequence. This is true even if the sequence is invoked by a proxy service or in an API.

l If a request arrives through the main sequence and fails within a sequence that does not explicitly define a fault handler, the default fault sequence will be invoked.

l If a request arrives through a proxy service or an API, and fails within a sequence that does not explicitly define a fault handler, the fault sequence of the proxy service will be invoked. If the proxy service does not define a fault sequence, no fault handler will be invoked.

l When there is a fault handler engaged in the proxy service-level, and another error handler engaged in the sequence-level, the sequence-level error handler is invoked in the event of an error (as per rule 1). In this case, the proxy service fault sequence is ignored.

l If an error occurs in an API before a message hits any of its defined sequence, HTTP 200 will return. But, you can define a default error sequence called _resource_mismatch_handler_ in this case.

Best practices (Error handling)l Whenever an error occurs in WSO2 ESB, the mediation engine attempts to

provide the user as much information as possible about the error. This is done by initializing a set of property values on the erroneous message. Given below are the properties:

ERROR_CODEERROR_MESSAGEERROR_DETAILERROR_EXCEPTION

The above properties can be logged using the log mediator like below inside the fault sequence path.

<log level="custom"><property name="text" value="An unexpected error occured"/><property name="message" expression="get-property('ERROR_MESSAGE')"/><property name="code" expression="get-property('ERROR_CODE')"/><property name="detail" expression="get-property('ERROR_DETAIL')"/><property name="exception" expression="get-property('ERROR_EXCEPTION')"/></log>

Best practices (Error handling)

Sending Custom FaultsUse the following mediators to send custom faults messages to the client.

makeFaultGiven below is an example code for makeFault mediator:<makefault><code value="tns:Receiver" xmlns:tns="http://www.w3.org/2003/05/soap-envelope"/><reason expression="get-property('ERROR_MESSAGE')"/></makefault>

payloadFactoryGiven below is an example code for payloadFactory mediator:<sequence name="myErrorSequence"><payloadFactory><format><ns:MyResponse xmlns:ns="http://services.samples"><ns:Error/></ns:MyResponse></format></payloadFactory><send/></sequence>

Best practices (Security)

Use DMZ-based deployments.

Change default admin user name and password.

Change the default key stores.

Secure plain text passwords and configure user password policies.

In REST services deployments, secure your services with OAuth (Bearer and HMAC) and Basic authentication with HTTPS. This is recommended because:

l It adds authentication, authorization and validation in to the services.l It enables communication to be done in a confidential manner while achieving

non-repudiation andl integrity of the messages.l It helps you avoid attacks such as replay, DoS and to define rules for throttling the

service calls.l It helps you to control access to services in a fine-grained manner using role-

based, attribute-based and policy-based access control.

Questions

Thank You

Technology

Wso2 esb-maintenance-guide