Neolane v6.1 - Production Guide - Adobe Campaign … Description job Command line processing merge Form merge midSourcing Recovery of delivery information in mid-sourcing mode monitor

Production GuideNeolane v6.1

This document, and the software it describes, are provided subject to a License Agreement and may not be used or copied outside of theprovisions of the License Agreement. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in anyform or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of Neolane.

The information contained in this document is provided for informational purposes only and may be revised without notice. It does notconstitute a commitment on the part of Neolane. Neolane does not guarantee the accuracy nor the completeness of the informationcontained within this document. References to company names are intended to be ficticious and for illustrative purposes only and do notrefer to any real-world company.

Any brands cited are the property of their respective owners. Windows is the registered trademark of Microsoft Corporation in the UnitedStates and other countries. Java, MySQL and Open Office are trademarks of Oracle Corporation in the United States and in other countries.Linux is the registered trademark of Linus Torvalds in the United States and in other countries. This product includes software developedby Apache Software Foundation (http://www.apache.org/).

For any questions or queries, please send a message to the following address: [email protected].

Version number : 8142

Neolane18 rue Roger Simon Barboux, 94110 Arcueil - France+33 1 41 98 35 35www.neolane.com

Neolane v6.1 - Production Guide

Chapter 1. General architecture of Neolane . . . . . . . . . . . . . . . 5

Minimum architecture . . . . . . . . . . . . . . . . . . . . . . . . 5Distributed architecture . . . . . . . . . . . . . . . . . . . . . . . . 6List of open ports . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Chapter 2. Production procedures . . . . . . . . . . . . . . . . . . 9

Configuration principle . . . . . . . . . . . . . . . . . . . . . . . . 9Operating principle . . . . . . . . . . . . . . . . . . . . . . . . . 10Administration . . . . . . . . . . . . . . . . . . . . . . . . . . 12Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Changing the syslogd listening port . . . . . . . . . . . . . . . . . 13Configuring security zones . . . . . . . . . . . . . . . . . . . . . 13

Log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Monitoring Neolane processes . . . . . . . . . . . . . . . . . . . . 15

Manual monitoring . . . . . . . . . . . . . . . . . . . . . . . . 15SMTP Reports . . . . . . . . . . . . . . . . . . . . . . . . . 20Automatic monitoring . . . . . . . . . . . . . . . . . . . . . . . 24Automatic monitoring via Neolane scripts . . . . . . . . . . . . . . . 27

Usual commands . . . . . . . . . . . . . . . . . . . . . . . . . 30Monitoring commands . . . . . . . . . . . . . . . . . . . . . . 30Module launch commands . . . . . . . . . . . . . . . . . . . . . 31Shut down services . . . . . . . . . . . . . . . . . . . . . . . 31Restart services . . . . . . . . . . . . . . . . . . . . . . . . . 31The config command . . . . . . . . . . . . . . . . . . . . . . . 32

Chapter 3. Data processing . . . . . . . . . . . . . . . . . . . . . 33

Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Physical files . . . . . . . . . . . . . . . . . . . . . . . . . . 33Database . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Duplicating environments . . . . . . . . . . . . . . . . . . . . . . 34

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 34Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 35

Description of the database Cleanup workflow . . . . . . . . . . . . . . 38Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Neolane v6.1 - Production Guide | 3

Table of Contents

Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Tasks carried out by the Database cleanup workflow . . . . . . . . . . . . . . 40

Chapter 4. Updating Neolane . . . . . . . . . . . . . . . . . . . . . . . 49

Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49In Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50In Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Resolving upgrade conflicts . . . . . . . . . . . . . . . . . . . . . . . . 52Informing client consoles of the available update . . . . . . . . . . . . . . . . 53

Example of the Unicode switch of an existing instance . . . . . . . . . . . . . . 54

Chapter 5. Database maintenance . . . . . . . . . . . . . . . . . . . . . 57

List of tables to maintain . . . . . . . . . . . . . . . . . . . . . . . . . . 57Neolane tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Customer tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Types of maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Application maintenance . . . . . . . . . . . . . . . . . . . . . . . . . 61Technical maintenance . . . . . . . . . . . . . . . . . . . . . . . . . 61

RDBMS Specific recommendations . . . . . . . . . . . . . . . . . . . . . . 62PostgreSQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Chapter 6. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . 79

Temporary files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Database performances . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Platform configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 80Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . 80Specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Modules and frequent issues . . . . . . . . . . . . . . . . . . . . . . . . 81Log precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Forwarding of tracking logs . . . . . . . . . . . . . . . . . . . . . . . . . 83OpenOffice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Email delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Workflow execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Failure to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Connection thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . 86Dr Watson in Windows XP (client) . . . . . . . . . . . . . . . . . . . . . . 86Stack trace in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Encoding of the Oracle database . . . . . . . . . . . . . . . . . . . . . . . 88Reactivating the console update request . . . . . . . . . . . . . . . . . . . . 88Lost password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Abnormal JSP behavior . . . . . . . . . . . . . . . . . . . . . . . . . . 89

4 | © Neolane 2013

Neolane

Table of ContentsMinimum architecture . . . . . . . . . . . . . . . . . . . . . . . . . 5Distributed architecture . . . . . . . . . . . . . . . . . . . . . . . . 6List of open ports . . . . . . . . . . . . . . . . . . . . . . . . . . 6

This chapter details the general architecture of Neolane.

Minimum architecture

In a minimum configuration, Neolane operates with:

n the Neolane application server,

n the database.

Neolane v6.1 - Production Guide - General architecture of Neolane | 5

CHAPTER 1

General architecture ofNeolane

This diagram shows that the only traffic involved in the context of a minimum architecture is:

1 HTTP protocol traffic to the Neolane server via the Internet,

2 SMTP protocol traffic from and to the Neolane server via the Internet.

Distributed architecture

Neolane is made up of multiple modules which can be broken down over several machines. This operatingmode has several advantages:

n load balancing,

n setting up of module redundancy,

n building of an architecture broken down over several service providers (segmentation of the servicesprovided).

The distribution of modules over several machines provides great flexibility of use and improved adaptability.

Tip:

For more on the various architectures, refer to the Installation Guide.

List of open ports

ConfigurableConcerned Neolane module or ap-plication

Port number

YESWeb Servers (Apache/IIS)443/tcp or 80/tcp

NOOpenOffice.org server (Xvfb)6025/tcp

YESNeolane: Syslogd6666/udp (local)

YESNeolane: web module8005/tcp (local)

6 | © Neolane 2013

Neolane

ConfigurableConcerned Neolane module or ap-plication

Port number

YESNeolane: web module (tomcat)8080/tcp

YESStatistics server (stat server)7777

Neolane v6.1 - Production Guide - General architecture of Neolane | 7

General architecture of Neolane

8 | Neolane v6.1 - Production Guide

Neolane

Table of ContentsConfiguration principle . . . . . . . . . . . . . . . . . . . . . . . . . 9Operating principle . . . . . . . . . . . . . . . . . . . . . . . . . 10Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Changing the syslogd listening port . . . . . . . . . . . . . . . . . . . 13Configuring security zones . . . . . . . . . . . . . . . . . . . . . . 13

Log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Monitoring Neolane processes . . . . . . . . . . . . . . . . . . . . . . 15

Manual monitoring . . . . . . . . . . . . . . . . . . . . . . . . . 15SMTP Reports . . . . . . . . . . . . . . . . . . . . . . . . . . 20Automatic monitoring . . . . . . . . . . . . . . . . . . . . . . . . 24Automatic monitoring via Neolane scripts . . . . . . . . . . . . . . . . . 27

Usual commands . . . . . . . . . . . . . . . . . . . . . . . . . . 30Monitoring commands . . . . . . . . . . . . . . . . . . . . . . . 30Module launch commands . . . . . . . . . . . . . . . . . . . . . . 31Shut down services . . . . . . . . . . . . . . . . . . . . . . . . 31Restart services . . . . . . . . . . . . . . . . . . . . . . . . . . 31The config command . . . . . . . . . . . . . . . . . . . . . . . . 32

Configuration principle

The Neolane platform is based on the concept of instances, similar to that of virtual hosts used byApache. This mode of operation lets you share a server by assigning several instances to it. Instancesare completely separate from each other and operate with their own database and configurationfile.For a given server, there are two elements that are common to all Neolane instances:

n The internal password: this is the general administrator password. It is common to all instancesof a particular application server.

Warning:

To log on with the Internal identifier, you need to have defined a password beforehand. Formore on this, refer to the Installation guide.

n Multiple technical server configurations: these configurations can all be overloaded in the specificconfiguration of an instance.

Neolane v6.1 - Production Guide - Production procedures | 9

CHAPTER 2

Production procedures

installation-v6.1-en.pdf#nameddest=ASN-Internal_Passwd_Definition

The configuration files are saved in the conf directory of the installation directory. The configuration is brokendown into three files:

n serverConf.xml: overall configuration for all instances.

n serverConf.xml.diff: delta between the initial configuration and the current configuration. This file isgenerated automatically by the application and must not be modified manually. It is used to automaticallypropagate user modifications when updating a build version.

n config-<instance>.xml (where <instance> is the instance name): specific configuration of aninstance.

An instance configuration is loaded as follows:

1 The module loads the serverConf.xml file to obtain the parameters shared by all instances.

2 It then loads the config-<instance>.xml file. The values found in this file have priority over valuescontained in serverConf.xml.These two files have the same format. Any value in serverConf.xml can be overloaded for a giveninstance in the config-<instance>.xml file.

This operating mode provides great flexibility for configurations.

Operating principle

Technically, the Neolane platform is based on several modules.There are many Neolane modules. Some operate continuously, while others are started up occasionally toperform administrative tasks (e.g. to configure the database connection) or to run a recurrent task (e.g.consolidating tracking information).There are three types of Neolane modules:

n Multi-instance modules: a single process is run for all instances. This applies to the following modules:web, syslogd, trackinglogd and watchdog (activities from the config-default.xml file).

n Mono-instance modules: one process is run per instance. This applies to the following modules: mta,wfserver, inMail, sms and stat (activities from the config-<instance>.xml file).

n Utility modules: these are modules that are run occasionally to perform occasional or recurrent operations(cleanup, config, downloading tracking logs, etc.).

Module administration is performed using the command line tool nlserver installed in the bin directory ofthe installation folder.The general syntax of the nlserver tool is as follows:nlserver <command> <command arguments>

For the list of available modules, use the nlserver command.

The available modules are detailed in the following table:

DescriptionCommand

Standardizing enumeration valuesaliasCleansing

Sending the system activity report to <[email protected]>billing

Cleansing the database: deletes obsolete data from the database and runs an update of the statistics usedby the database engine optimizer.

cleanup

Modifying server configurationconfig

Copy of a databasecopybase

Exporting to command line: lets you send to the command line an export model created in the Neolaneclient console

export

Converting a set size filefileconvert

Importing to command line: lets you send to the command line an import model created in the Neolaneclient console.

import

Inbound mail analyzerinMail

Availability of the customer installation fileinstallsetup

Executing JavaScript scripts, with access to SOAP APIs.javascript

10 | © Neolane 2013

Neolane

DescriptionCommand

Command line processingjob

Form mergemerge

Recovery of delivery information in mid-sourcing modemidSourcing

XML Displaying of the status of server processes and scheduled tasks, by instance.monitor

Main Agent transfer messagemta

Importing or exporting entity package filespackage

Displaying server process statusespdump

Preparing a delivery actionprepareda

Partial server restartrestart

Execution of a workflow instancerunwf

Full system shutdownshutdown

SMS notification processingsms

SQL script executionsql

Additional startsstart

Maintains MTA connection statisticsstat

Partial system shutdownstop

Submitting a delivery actionsubmitda

Log and trace writing serversyslogd

Consolidating and retrieving tracking logstracking

Tracking log writing and purging servertrackinglogd

Startup and monitoring instancewatchdog

Application server (HTTP and SOAP)web

Workflow serverwfserver

Warning:

There is one last module: the tracking and relay module linked to the application server which, for the sakeof performance, is integrated via native mechanisms into an Apache or IIS web server via a dynamic library.There is no Neolane command enabling you to start or administer this module. You must therefore use thecommands of the Web server itself.

Module usage and the syntax of its parameters are displayed using the following command: nlserver[module] -?

Example:nlserver config -?

Usage: nlserver [-verbose:<verbose mode>] [-?|h|H] [-version] [-noconsole][-tracefile:<file>] [-tracefilter:<[type|!type],...>][-instance:<instance>] [-low] [-high] [-queryplans] [-detach][-internalpassword:<[password/newpassword]>] [-postupgrade][-nogenschema] [-force] [-allinstances][-addinstance:<instance/DNS masks[/language]>][-setdblogin:<[dbms:]account[:database][/password]@server>][-monoinstance][-addtrackinginstance:<instance/masks DNS[/databaseId/[/language[/password]]]>][-trackingpassword:<[password][/newpassword]>][-setproxy:<protocol/server:port[/login]>] [-reload][-applyxsl:<schema/file.xsl>] [-filter:<file>][-setactivationkey:<activation key>][-getactivationkey:<client identifier>]

-verbose : verbose mode-? : display this help message-version : display version number-noconsole : no longer display logs and traces on the console-tracefile : name of trace file to be generated (without extension)-tracefilter : filter for the traces to be generated e.g.: wdbc,soap,!xtkquery.



-instance : instance to be used (default instance if this option is not present).-low : start up with low priority-high : start up with high priority (not recommended)-queryplans : generate traces with the execution plans of SQL queries.-detach : detaches the process from its parent (internal option)-internalpassword : changes the password of the server internal account.-postupgrade : updates the database following upgrade to a higher version. -nogenschema : does not recompute the schemas during database update-force : updates the database even if this has already been done with the current build -allinstances : updates the database over all configured instances-addinstance : adds a new instance.-setdblogin : sets the parameters for connection to the database of an instance. The DBMScan be 'oracle', 'postgresql', 'mssql' or 'odbc' (default=postgresql)-monoinstance : initialises for a single instance ().-addtrackinginstance : adds a new tracking instance.-trackingpassword : changes the tracking password of an instance-setproxy : sets the parameters for connection to a proxy server. The protocol can be 'http', 'https' or 'all'.-reload : asks the server to reload the configuration of the instances. -applyxsl : applies an XSL stylesheet to all entities of a schema. -filter : applies the XTK filter contained in the file during loading of the schema entities.-setactivationkey : sets the activation key

Administration

Automatic startup of the Neolane modules (web, mta, wfserver, etc.) is provided by the nlserver server.Installing Neolane automatically configures the machine so that the nlserver service starts up during theboot sequence.The following commands are used to start up and shut down the Neolane service manually:

n In Windows:

n net start nlserver6

n net stop nlserver6

n In Linux (as root):

n /etc/init.d/nlserver6 start

n /etc/init.d/nlserver6 stop

Here is a list of the usual administration commands accessible in Linux (as Neolane):

n Display all started Neolane modules: /etc/init.d/nlserver6 pdump or /etc/init.d/nlserver6status

Note:

Adding the -who parameter to the pdump command lets you collect information on current connections(users and processes).

n Start/stop a multi-instance or mono-instance module (web, trackinglogd, syslogd, mta, wfserver,inmail):nlserver start <module>[@<instance>]

nlserver stop <module>[@<instance>][-immediate] [-noconsole]

You can also use the nlserver restart <module>[@<instance>] command to restart a module.

Example:nlserver start web

nlserver start mta@my_instance

nlserver stop syslogd

nlserver stop wfserver@my_instance

nlserver stop web -immediate

nlserver restart web

12 | © Neolane 2013

Neolane

Note:

n If the instance is not specified, the "default" instance will be used.

n In the event of an emergency, use the -immediate option to force an immediate halt to the process(equivalent to Unix command kill -9).

n Use the -noconsole option to ensure that the module launched will display nothing on the console.Its logs will be written to the disk via the syslogd module.

n Use the -verbose option to display additional information on process actions.

Example:nlserver restart web -verbosenlserver start mta@myinstance -verboseThis option adds additional logs. We recommend starting the processes again without the -verboseoption once you have found the desired information, to avoid overloading logs.

n Start up all Neolane processes (equivalent to starting up the nlserver6 service): nlserver watchdog-noconsole

n Shut down all Neolane processes (equivalent to shutting down the nlserver6 service): nlservershutdown

n Reload the nlserver web module configuration (and the web server extension module, where applicable)when the serverConf.xml and config-<instance>.xml files have been edited.nlserver config -reload

Note:

Some configuration changes are not taken into account dynamically; Neolane must be shut down andthen restarted.

Configuration

Changing the syslogd listening portBy default, the syslogd listening port is 666 (udp). You can alter it using an environment variable if necessary.Once it is configured, this variable is taken into account by all Neolane modules.

In LinuxEdit the customer.sh file and add the following line:

export TRACE_ADDR=localhost:<listening port>

In WindowsYou need to create the TRACE_ADDR environment variable with the localhost value: <listening port>.

Warning:

We recommend running some tests to make sure your platform is working after you create this environmentvariable.

Configuring security zonesEach operator needs to be linked to a zone to log on to an instance and the operator IP must be included inthe addresses or address sets defined in the security zone. Technical zone configuration is carried out in theconfiguration file of the Neolane server. The linking of an operator to a security zone has to be defined inthe console (Administration > Access management > Operators node).



Note:

For more on configuring security zones, refer to the Installation guide.

Log files

The log files are organized as follows:

Each nlserver module generates a log file saved in the following directory: <installation

directory>/var/<instance>/log/<module>.log.The nlserver syslogd module saves the logs to the disk. This module is similar to the Unix syslog daemon,but has been adapted for compatibility between Unix and Windows. The other Neolane modules do not savetheir logs to the disk; they delegate this task to the syslogd module by sending it UDP packets.By default, the Neolane platform has the syslogd module installed on it, but it is possible to use anothersyslog daemon. This module creates the log files in the log directory.The logs of multi-instance modules are stored in the following directory: <installation

directory>/var/default/log/. The same log file is shared by all instances (e.g. web.log).The logs of the other modules are stored in a subfolder named after the instance. Each instance has its ownlog files.Multi-instance log files are listed in the following table:

DescriptionFile

Web module logs (client console, reports, SOAP API, etc.)web.log

Logs from the redirection modulewebmdl.log

Logs from the Neolane process monitoring modulewatchdog.log

Tracking logstrackinglogd.log

The mono-instance log files are listed in the following table:

DescriptionFile

mta module logsmta.log

Message delivery processing logsmtachild.log

Logs of the workflow server modulewfserver.log

Workflow execution logsrunwf.log

Bounce mail module loginMail.log

Warning:

The redir directory only exists on redirection servers. The url subdirectory contains the matches of theURLs to be redirected, and the subdirectory log contains the tracking logs. To generate tracking logs, thetrackinglogd module must be running.

By default, the logs are limited to two 10 MB files per module and per instance. The second file is called:<ModuleName>_2.log. The size of the logs is therefore limited to 2*10MB per module and per instance.

14 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#nameddest=csturges-36892

You can, however, keep larger files. To enable this, change the value of the maxFileSizeMb="10" settingin the syslogd node of the conf/serverConf.xml file. This value represents the maximum size in MB of alog file.If you wish to maintain further levels of detail in the logs, you can start the Neolane modules with the-verbose parameter:nlserver start <MODULE>@<INSTANCE> -verbose

Monitoring Neolane processes

The application server and the redirection server (tracking) can be monitored manually or automatically.

Manual monitoringGo to the Monitoring universe and click the Overview link to display the Neolane process monitoring page.

The page displayed lets you view the status of the connected instance, i.e.:

n information on the instance (version, name, database engine, installed packages, server system indicators),

n the list of missing processes and execution information (start date, PID, etc.),

n a view of workflows and deliveries.



Log journalIt is possible to display the log journal related to a process. To do this, click on the process, mta for example,then click Open the log journal.

System indicatorsThe list of system indicators enables you to display information concerning the machine, such as its physicaland virtual memory, active processes and available disk space. Indicators are different for Linux and Windowsoperating systems. Go to the Instance Monitoring page and click the Display link to open the list ofindicators

16 | © Neolane 2013

Neolane

In Windows

n Pending events queued : indicator specific to Neolane Message Center. Refer to the Eventsmonitoring thresholds section for more information.

n Memory: information concerning the physical memory (RAM).Current value: actual memory consumption.Max Value: total amount of memory installed.Available: amount of available memory.Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.When the Warning and Alert indicators are displayed, you can solve the issue by adding RAM to themachine which the Neolane server is installed on. You can also decide to install the Neolane server on adedicated machine.

n Swap Memory: information related to the virtual memory that matches a paging file: an area on thehard disk that Windows uses as if it were RAM.Current value: actual memory consumption.Max Value: total amount of memory.Available: amount of available memory.Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.When the Warning and Alert indicators are displayed, you can solve the issue by increasing the sizeof the exchange file in the advanced Windows settings.

n Disk XXX: information concerning machine readers.Current value: disk space actually used.Max Value: total disk capacity.Available : disk space available



message-center-v6.1-en.pdf#nameddest=CST-22466


Used: percentage of disk used.Warning: this indicator is displayed when the available disk space reaches 80% of the total capacity.Alert: this indicator is displayed when the available disk space reaches 90% of the total capacity.

n Number of processes too old: information concerning Neolane processes that have been active formore than one day.Current value: number of processes currently active.Max Value: maximum number of authorized processes (1).Alert: this indicator is displayed if the number of processes equals 1.When the Alert indicator is displayed, it may be that the concerned process is locked by the SQL databaseengine or that it is stuck in an infinite loop. The watchdog process provided by Neolane automaticallyre-starts all processes every day and enables you to solve this issue. However, you can also stop theconcerned process yourself to force re-start.

In Linux

n Pending events queued : indicator specific to Neolane Message Center. Refer to the Eventsmonitoring thresholds section for more information.

n Load average (1/5/15 minutes): information concerning the load, i.e. the use rate of the processerby the processes running on the machine over the last minute, five minutes, or fifteen minutesCurrent value: actual load of the machine.

18 | © Neolane 2013

Neolane



Max value: maximum use load of the process(es) on the machineWarning: this indicator is displayed when the load reaches 80% of the maximum authorized value overthe last minute, five minutes or fifteen minutes.Alert: this indicator is displayed when the load reaches 90% of the maximum authorized value of thelast minute, five minutes, or fifteen minutes.

n Memory: information concerning the physical memory (RAM).Current value: actual memory consumption.Max Value: total amount of memory installed.Available: amount of available memory.Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.When the Warning and Alert indicators are displayed, you can solve the issue by adding RAM to themachine which the Neolane server is installed on. You can also decide to install the Neolane server on adedicated machine.

n Swap Memory: information related to the virtual memory that matches a paging file: an area on thehard disk that Windows uses as if it were RAM.Current value: actual memory consumption.Max Value: total amount of memory.Available: amount of available memory.Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.When the Warning and Alert indicators are displayed, you can solve the issue by increasing the sizeof the exchange file.

n Core Files: information concerning the files generated following the crash of a Neolane process. Thesefiles enable you to diagnose the reasons of the crash.Current Value: number of existing files.Max Value: maximum number of authorized files (1).Warning: this indicator is displayed when the number of files nears 1.Alert: this indicator is displayed when the number of files equals 1.When a process is missing due to a crash, it is shown in red on the list of processes and is re-startedautomatically by the watchdog process provided by Neolane.

n Number of shared memory segments: information concerning the memory segments shared by allNeolane processes.Current value: number of memory segments currently in use.Max Value: maximum number of memory segments authorized (2).Warning: this indicator is displayed when the number of memory segments reaches 1.Alert: this indicator is displayed when the number of memory segments reaches 2.

n Number of processes too old: information concerning processes that have been active for over oneday.Current value: number of processes currently active.Max Value: maximum number of authorized processes.Warning: this indicator is displayed when the number of processes reaches 80% of the authorizedthreshold.Alert: this indicator is displayed when the number of processes reaches 90% of the authorized threshold.

n File Handles: information concerning the file descriptors, i.e. the number of files opened per process.Current value: current number of file descriptors.Max Value: maximum number of file descriptors authorized by the operating system.Warning: this indicator is displayed when the number of authorized file descriptors reaches the 80%threshold.Alert: this indicator is displayed when the number of authorized file descriptors reaches the 90% threshold.

n Processes: information concerning the machine processes.Current value: number of processes currently active.Max Value: maximum number of authorized processes.Active Processes: number of active processes.



Inactive Processes: number of inactive processes.Warning: this indicator is displayed when the number of authorized processes reaches the 80% threshold.Alert: this indicator is displayed when the number of authorized processes reaches the 90% threshold.

n Zombie Processes: information concerning the processes that have been stopped but still have aprocess identifier (PID) and remain visible in the process table.Current value: number of zombie processes that are currently active.Max Value: maximum number of authorize zombie processes (2).Warning: this indicator is displayed when the number of zombie processes nears 2.Alert this indicator is displayed when the number of zombie processes reaches 2.

SMTP ReportsSMTP delivery monitoring reports are integrated into the Neolane platform. They can be accessed via theconsole or using Web access.These reports display SMTP delivery statistics and SMTP errors by domain.To access them, the operator must have Administration rights.They are grouped in the Monitoring universe, under the 'SMTP Monitoring' label.

Warning:

n Information related to SMTP Monitoring is only available if the email channel has been activated in thedeployment wizard.

n The SMTP sending statistics are only offered if the statistics server is launched on the instance.

20 | © Neolane 2013

Neolane

SMTP sending statisticsThe SMTP sending statistics report lets you control server activity. It displays a synthesis of each of themtachilds.

The list of indicators for this report is shown below the chart.

1 Total number of messages sent.

2 - Blue line: messages ready for sending which arrived in the Shaper, i.e. last stage before sending SMTP(coincides with the incoming data).- Green line: messages successfully sent (coincides with the outgoing data).- Red line: messages abandoned by the Shaper, returned to the mta (coincides with the data rejectedon this recovery).These values are expressed in number of messages per hour.

3 Represents two queues of the Shaper:- Blue curve: queue of active messages. These messages will be sent as soon as possible.- Kaki curve: the 'deferred' queue. These messages cannot be returned for the moment due to throttlingor because no connection to the target is available. Retries will take place every 5s, 10s, 20s, 40s, 2 min,etc. for the defined MaxAgeSec time before being abandoned.

4 This charts shows a detail of abandoned messages (red curve on the 2nd chart): it shows the proportionof messages abandoned without retries (mauve) compared with messages whose sending failed (red).This lets you view the proportion of messages not processed within the granted period due to limitationsby the statistics server (throttling) or due to remote server unavailability.

5 SMTP connections open or being opened.

6 Estimate of the number of mtachild.

Note:

This report is related to the status of the Email Traffic Shaper component.

SMTP errors per domainThis report lets you view the delivery errors, over a set period, broken down by domain.



Note:

The minConnectionsToLog, minErrorsToLog and minMessagesToLog options of the serverConf.xmlfile define the thresholds above which connection statistics are taken into account.

The list of indicators for this report is shown below the table.

n The Domain column contains the name of the domain to which the messages are sent (or the realdomain name, yahoo.com for yahoo.fr for example),

n The Cnx column displays the number of SMTP connections open for this domain,

n The Sent column corresponds to the number of messages sent to this domain,

n The Volume column displays the volume of messages that have been attempted to be sent to thisdomain (approximate value),

n The Errors column displays a volume indicator of errors on this domain over the period,

n The Last response column displays the last SMTP response message received for this domain,

n The Date column displays the date of the last SMTP response received for this domain.

Note:

The values displayed in the Cnx, Sent, and Volume columns are calculated with respect to the periodselected in the Period field.

Click on a domain name to view its errors.

22 | © Neolane 2013

Neolane

They are categorized by PublicId: this identifier corresponds to an IP address shared by several Neolanemtas behind a router. The statistics server uses this identifier to memorize the connection and deliverystatistics between this starting point and the target server.

The Owner of domain field lets you group various domain names under the same label. In the initial reportview, all MX domain names will be associated to this owner.Click on a PublicId identifier to view further detail.



Note:

The percentage of errors is represented by two charts. The first is a horizontal progress bar on a blackbackground. The second chart is chronological. The selected period is divided into twelve time intervals, eachrepresented by a vertical progress bar. In both representations, if no error has been detected, the bar isblack. The color of the bar depends on the percentage of errors encountered (yellow, then orange, and lastly,red). The color grey means that no significant data volume has been found. It is possible to display the exactpercentage of errors by putting the cursor on the chart.

Note:

For further information about SMTP errors and managing them in Neolane, please consult the Installationguide.

Automatic monitoringNeolane offers several automatic monitoring methods, which are presented below.

Command lineCommandnlserver monitor

Lets you list a set of indicators on the Neolane modules and the system.It generates output in an easily processed XML format.This command can also be run with the -missing parameter, which lists the processes that are missing fromthis instance when the configuration files say that they should be executing.

nlserver monitor -missing10:17:59 > Application server for Neolane Version 6.0.0 (build 6749) of 26/07/2011mta@prodstat@prodwfserver@prod

Information published by the server

/r/test

The http(s)://<Application server URL/r/test page is used to test the redirection server. Werecommend using this same method to test the frontal servers used for tracking. This page can also be usedto test a load dispatcher.It displays a line like this in XML format:

<redir status='OK' date='2011-07-27 08:18:11.112Z' build='6743' host='training.neolane.net' localHost='servername'/>

Frequency: this test does not use any load, and so it can be run very often (e.g. once every second).

/nl/jsp/ping.jsp

This http(s)://<Application server url/nl/jsp/ping.jsp page operates in the same way asits network counterpart: it tests a complete query going through apache/tomcat/web module/database anduploading to the client. If everything is working properly, it returns an "OK". We recommend running thistest on machines with access to the databases (mtas and surveys, for instance).Usage: a session token associated with an operator login must be passed as an argument in order to log inremotely (see the tip in Automatic monitoring via Neolane scripts [page 27]).

24 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#nameddest=CST-12423


For example:

The operator name and login need to be previously configured in the Neolane client console with databaserights.

Frequency: this is a test that uses very little bandwidth. It can therefore be run fairly often (e.g. once everyten seconds).



/nl/jsp/monitor.jsp

This is a test to check that an operator can access the Neolane server via a web page; the same web pageas the one accessed via the client console menus. We recommend including it in your surveillance tools(Tivoli, Nagios, etc.).

Usage: a session token associated with an operator login which lets you connect to the instance needs tobe used as an argument (see the tip in Automatic monitoring via Neolane scripts [page 27]).The operator and their login needs to be configured previously in the Neolane client console with theappropriate database rights and restrictions.Frequency: this is a full server test and doesn't need be run often (it can be carried out once every tenminutes, for example).

/nl/jsp/soaprouter.jsp

This jsp represents the point of entry of Neolane application APIs. It can therefore provide detailed monitoringof the application. It can also be used to monitor Neolane web services. It is used in our monitoring scripts,but note that it is for power users only.

Monitoring based on deployment typesNeolane enables various deployment configurations (for more on this, refer to Deployment types). Thissection details the various automatic monitoring techniques to be applied depending on your type of installation.

MonitoringDeployment type

n /r/test and /nl/jsp/monitor.jsp on the Neolane serverStand-alone

n /r/test and /nl/jsp/ping.jsp on the frontal servers

n /nl/jsp/monitor.jsp on the application server

Standard

n /r/test and /nl/jsp/ping.jsp on the frontal servers

n /r/test and /nl/jsp/monitor.jsp on the application server

Enterprise

26 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#nameddest=SBL-8977

MonitoringDeployment type

n /nl/jsp/monitor.jsp on the application serverMid-sourcing

Automatic monitoring via Neolane scriptsNeolane provides an instance monitoring tool (netreport) that lets you send a report by email regarding thedetected anomalies.

Required elementsThe following pre-installation precautions are required for automatic monitoring:

n You must have the netreport.tgz (Linux installation) or netreport.zip (Windows installation) files,

n We strongly advise you not to install monitoring on the machine to be monitored,

n it must be installed on a machine with a JRE or a JDK,

n in Linux, the machine to be monitored must have the bc package.

Installation procedureThe installation procedure is as follows:

1 In the console, create a new operator if necessary (the 'monitoring' user already exists), but do not assignany rights.

2 Run archive extraction.

3 Read the readme file.

4 Update the netconf.xml configuration file.

5 Update the netreport.bat (Windows) or netreport.sh (Linux) file.

Configuring the netconf.xml fileThe XML configuration file contains the following elements:

n 'Properties' element [page 28],

n 'Instance' element [page 29]

n 'Host' element [page 29],

n Sub-elements [page 30].

Here is a configuration example:

<?xml version="1.0" encoding="ISO-8859-1"?><netconf> <properties mailServer="mail.neolane.net" mailFrom="[email protected]" recipientList="[email protected]"> <nightMode start="00:00 am" end="07:00 am"/>



<buildRange minimum="7829" maximum="8180"/> <buildRange minimum="8300" maximum="8400"/> <sla/> </properties>

<instance name="dev61" recipientList="[email protected],[email protected]"> <host name="dev61rd.domain.com" alias="dev61rd" sessiontoken="monitoring" criticalLevel="1" filter="wkf;new"> <ncs instance="dev61rd" url="/nl/jsp/soaprouter.jsp" includeDead="false" isSecure="false"/> <redir url="/r/test"/> <http url="/nl/jsp/ping.jsp"/> </host> <host name="dev61trk.domain.com" alias="dev61trk" sessiontoken="monitoring" criticalLevel="0" filter="wkf;new"> <ncs instance="dev61rd" url="/nl/jsp/soaprouter.jsp" includeDead="true" isSecure="false"/> </host> </instance> <host name="dev-test" alias="dev-test" sessiontoken="monitoring" criticalLevel="2"> <ncs instance="dev" url="/nl/jsp/soaprouter.jsp" includeDead="false"/> </host></netconf>

Note:

You can specify various configurations by adding a suffix to the netconf.xml file, for example,netconf-dev.xml, netconf-prod.xml, etc. Then specify the configuration to use for executing the netreportin the netreport.bat or netreport.sh files by adding $JAVA_HOME/bin/java netreport dev or@%JAVA_HOME%\bin\java netreport prod for example.

Warning:

For the monitoring operator to work, the machine that the netreport is executed on must be in a securityzone that is in sessionTokenOnly mode. If no trusted IP mask has been specified for this operator, thesecurity zone must also be in allowEmptyPassword and allowUserPassword mode.

'Properties' element

This element is used to populate the configuration of emails, i.e.

n mailServer: SMTP server used to send emails (e.g.: smtp.neolane.net).

n mailFrom: email address of the report sender (e.g.: <[email protected]>).

n recipientList: the list of email addresses of monitoring recipients. Addresses must be separated bycommas (no spaces).

n 'night' mode (optional) is used to make sure no emails are sent between the time specified. Instead,the data is consolidated and an email concerning the night's activity is sent after the end time (7:00 bydefault).

n The buildRange sub-element (optional) lets you specify a minimum and maximum build number . Anerror will be generated for all machines whose build number does not fall into this range.

<buildRange minimum="0000" maximum="9999"/>

n You can add an <sla/> (optional) sub-element in the properties element. A log file will be generatedeverytime the netreport is executed. The name of the file contains the configuration name and the timeand date, for example dev_06_12_13_16_47_05.tmp. The file contains the following information: instancename, machine name, severity level, (0 to 3, from least critical to most critical), date (timestamp format),time elapsed (in milliseconds) between the query and the response, service used (http, ncs, ncsex, redir).This information is separated by tabulation marks and line breaks at the end of each service.

Note:

The persistHtmlFile attribute with the value "true" on the <property> element is used to record thelatest monitoring status in the file netreport.html. This file is saved in the installation directory.

28 | © Neolane 2013

Neolane

'Instance' element

This element lets you regroup several machines (hosts) into the same instance. The instance names appearin the first part of the monitoring email. You can click on the name of an instance to access detail regardingeach machine.

<instance name="instanceName" recipientList="[email protected],[email protected]"> <host name="dev61rd.domain.com" ...> ... </host> <host name="dev61trk.domain.com" ...> ... </host></instance>

n name: instance name that will appear in the first part of the email.

n recipientList (optional): lets you send a monitoring report regarding a particular instance by email.

'Host' element

This element configures the monitoring of a given server on the host, i.e.

n name: name of the machine to be monitored.

n alias (optional): name of the monitored machine as it will appear in the report.

n sessionToken: provides login authentication via an authorized session token.To configure the session token, select the monitoring operator in the Neolane console. In the Accessrights tab, specify the IP addresses of the machines authorized to monitor this instance. You will thenbe able to connect to the monitoring page from those machines using the monitoring identifier andwithout needing to specify a password.

n criticalLevel (optional): lets you sort errors to be displayed by level of severity. Possible values are '0'(all levels displayed), '1' (only high and critical errors displayed) and '2' (only critical errors displayed). Ifthis attribute is not provided, all error levels are displayed.



n filter (optional): lets you exclude certain workflow errors, for example filter="wkf;wkf1". Workflowlabels must be serarated by commas.

Sub-elements

n tcp: checks if the server is up or down. You must enter a port number.

n http: checks that the Web server exists (application server is operational).

n ncs: checks the processes on the instance entered in the 'instance' attribute (workflow errors, memoryusage, etc.). The includead (mandatory) gives you the option of displaying dead processes ('true' or'false' values).

n redir: checks the tracking.

In most cases, only the ncs and redir sub-elements can be kept.In any case, certain nodes can be overloaded in the sub-elements (e.g., the node port=75 to overload theport used for the http, ncs or redir connection).

ncs instance="clap40" url="/nl/jsp/soaprouter.jsp" includeDead="false" port="80"/

In the ncs, redir and http sub-elements, you can add the isSecure attribute (optional) to choose whetheror not to use the https protocol ('true' or 'false' values). If this attribute is not provided, the http protocol isused.

Configuring the netreport.bat or netreport.sh fileTo configure it, edit this file and indicate which directory the JRE or JDK is installed in.

Launching monitoringTo launch monitoring, execute the netreport.bat or netreport.sh file at regular intervals via a script. Areport is sent after the first execution, and then only in the event of a change of status.

Testing monitoringTo test the monitoring, execute the netreport.bat or netreport.sh file.An email is sent to the recipients specified in the recipientList of the netconf.xml file.

Usual commands

This section lists the usual commands in Neolane.The command nlserver is the input command for the whole Neolane application.This command has the following syntax: nlserver <command> <arguments>

The parameter <command> corresponds to the module.

Note:

n In any case, you can add the -noconsole argument to delete comments displayed once the modulesare started.

n Conversely, you can add the argument -verbose to display more information.

Monitoring commands

Note:

To list all modules, you need to use the nlserver pdump command

You can add the parameter -who to list the connections in progress (database and application).

nlserver pdump -who11:00:39 > Application server for Neolane Version 6.0.0 (build 6749) from 26/07/2011web@default (9984) - 50.1 Mowatchdog (2273) - 6.6 Mosyslogd@default (9931) - 7.0 Mo

30 | © Neolane 2013

Neolane

trackinglogd@default (9985) - 45.6 Momta@test (9986) - 9.6 Mowfserver@test (9987) - 8.8 Mo

Connections ------------------------------------------------------Last Access IP Instance Login 21/07/2011 10:00:25 127.0.0.1 default formation_fr|tracking26/07/2011 10:07:44 127.0.0.1 default internal|monitoring

Connection pool --------------------------------------------------Datasource Server Provider Login default xxxxx myserver myprovider test400

Another useful command is nlserver monitor. It lists the monitoring XML file (obtained in the Neolaneclient or via the monitor.jsp web page).You can add the parameter -missing to list the absent modules (error in modules, modules shut down, etc.)

nlserver monitor -missing11:00:39 > Application server for Neolane Version 6.0.0 (build 6749) from 26/07/2011inMail@testmta@testwfserver@test

This corresponds to the modules with automatic startup but which have not been launched.

Module launch commandsThe syntax to launch modules will still have the following format:

nlserver start <ModuleName>@<INSTANCE>

nlserver stop <NameModule>@<INSTANCE>

Note:

<INSTANCE> corresponds to the name of the instance as entered in the configuration files, or default formono-instance modules.

Shut down servicesTo stop Neolane services, use one of the following commands:

n If you have root or administrator access:

n In Linux:

/etc/init.d/nlserver6 stop

n In Windows:

net stop nlserver6

n If not, then in the Neolane account:

nlserver shutdown

Restart servicesSimilarly, in order to restart Neolane you can use one of the following commands:


n In Linux:

/etc/init.d/nlserver6 start

n In Windows:



net start nlserver6

n Otherwise, in the Neolane account: nlserver watchdog -svc -noconsole

The config commandThe config command lets you manage server configuration, including the reconfiguration of the databaseconnection.Use the config command of the nlserver executable file with the -setdblogin parameter.

nlserver config -setdblogin:<[dbms:]account[:database][/password]@server>

nlserver config -setdblogin:PostgreSQL:neolane:test6@dbserver

Enter the password.To change the internal password: nlserver config -internalpassword

Warning:

To log on with the Internal identifier, you need to have defined a password beforehand. For more on this,refer to the Installation guide.

Note:

n In general, instead of modifying the configuration files by hand, you can use the config command

n To get the list of parameters, use the -? parameter: nlserver config -?

n In the case of an Oracle database, you must not specify the account. The syntax will be as follows:

nlserver config -setdblogin:Oracle:test6@dbserver

32 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#nameddest=ASN-Internal_Passwd_Definition

Table of ContentsBackup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Physical files . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Duplicating environments . . . . . . . . . . . . . . . . . . . . . . . 34

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Description of the database Cleanup workflow . . . . . . . . . . . . . . . . 38Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 38Tasks carried out by the Database cleanup workflow . . . . . . . . . . . . . 40

Backup

Backing up is essential in order to avoid losing data in the event of a problem (whether physical orsystem-related) on a machine.Data is stored in two separate locations:

n physical files are stored in the Neolane directories,

n other data is stored in the database.

Most of the data is in the database. This represents 99% of the information to be backed up.

Physical filesFiles are divided into several categories:

n Configuration files, located in nl6/confThese enable you to reconfigure Neolane very quickly.

n Redirection files nl6/var/<instanceName>/redirThese are on the tracking (often called 'frontal') servers, and include all previous campaignredirections. They are still used by previous campaigns.

n Log files: nl6/var/<instanceName>/logThese can be used to trace problems.

n The pURL configuration files (Neolane Microsites): nl6/var/<instanceName>/relay

Neolane v6.1 - Production Guide - Data processing | 33

CHAPTER 3

Data processing

These files, which are essential to microsite operation, define the way pURLS are relayed to the Webapplications.

Important:

The directories to be backed up are therefore:

n nl6/conf

n nl6/var/<instanceName>/redir (for each instance)

n nl6/var/<instanceName>/log (optional)

n nl6/var/<instanceName>/relay (optional)

DatabaseThe database contains all of the information displayed in the Neolane rich client console, as well as all theline-of-business data.

Warning:

It is essential to back up the database.

Your hosting company, and their database administrators in particular, are responsible for this operation.

Restoration

On a clean server, the restoration procedure is as follows:

n on an installed and configured operating system (networks),

n install third-party applications: Web server, JDK (if necessary),

n installation of Neolane binaries with the same build as the source system,

n copy configuration files, tracking logs and redirection files,

n create and rebuild the database,

n start Neolane.

Note:

For more information, refer to the Installation Guide.

Duplicating environments

Introduction

OverviewUsing Neolane requires installing and configuring one or more environments: development, test, pre-production,production, etc.Each environment contains a Neolane instance and each Neolane instance is linked to one or more databases.The application server can execute one or more processes: almost all of these have direct access to theinstance database.This section details the processes to be applied to duplicate a Neolane environment, i.e. to restore a sourceenvironment to a target environment, resulting in two identical work environments.To do this, apply the following steps:

1 create a copy of the databases on all instances in the source environment,

2 restore these copies on all instances of the target environment,

34 | © Neolane 2013

Neolane

3 Run the nms:freezeInstance.js cauterization script on the target environment before starting it up.This process doesn't impact the servers and their configuration.

Note:

In the context of Neolane, a cauterization combines actions that let you stop all processes interactingwith the outside: logs, tracking, deliveries, campaign workflows, etc.This step is necessary to avoid delivering messages several times (once from the nominal environmentand one from the duplicated environment).

Warning:

One environment can contain several instances. Each Neolane instance is subjected to a license contract.Check your license agreement to see how many environments you can have.The procedure below lets you transfer an environment without impacting the number of environmentsand instances you have installed.

Before you start

Warning:

We strongly recommend running a full backup of the databases for all instances of the source and targetenvironments before starting the transfer process. This way if a problem occurs, you will be able to restorethe backups and return to your initial configuration.

In order for this process to work, the source and target environments must have the same number ofinstances, the same purpose (marketing instance, delivery instance) and similar configurations. The technicalconfiguration must comply with software prerequisites. The same components must be installed on bothenvironments.

Implementation

Transfer procedureThis section will help you understand the steps required for transferring a source environment to a targetenvironment via a case study: our aim here is to restore a production environment (prod instance) to adevelopment environment (dev instance) to work in a context that is as close as possible to the 'live' platform.The following steps must be performed with great care: some processes may still be in progress when thesource environment databases are copied. Cauterization (step 3 below) prevents messages from being senttwice and maintains data consistency.

Warning:

n The following procedure is valid in PostgreSQL language. If the SQL language is different (Oracle, MySQL,etc.), the SQL queries must be adapted.

n The commands below apply within the context of a prod instance and a dev instance under PostgreSQL.

Step 1 - Run a backup of the source environment data (prod)Copy the databasesStart by copying all databases of the source environment. The operation depends on the database engineand is the responsibility of the database administrator.Under PostgreSQL, the command is:

pg_dump mydatabase > mydatabase.sql

Step 2 - Export the target environment configuration (dev)Most configuration elements are different for each environment: external accounts (mid-sourcing, routing,etc.), technical options (platform name, DatabaseId, email addresses and default URLs, etc.).


Data processing

Before saving the source database on the target database, you need to export the target environment (dev)configuration. To do this, export the content of these two tables: xtkoption and nmsextaccount.This export lets you keep the dev configuration and only refresh dev data (workflows, templates, Webapplications, recipients, etc.).To do this, perform a package export for the following two elements:

n Export the xtk:option table into an 'options_dev.xml' file, without the records with the following internalnames: 'WdbcTimeZone', 'NmsServer_LastPostUpgrade' and 'NmsBroadcast_RegexRules'.

n In an 'extaccount_dev.xml' file, export the nms:extAccount table for all records whose ID is not 0 (@id<> 0).

Check that the number of exported options/accounts is equal to the number of lines to export in each file.

Note:

The number of lines to export in a package export is 1000 lines. If the number of options or external accountsis more than 1000, you must carry out several exports.For more information, refer to the Export packages section.

Step 3 - Stop the target environment (dev)You need to stop Neolane processes on all servers of the target environment. This operation depends onyour operating system.You can stop all processes, or only those that write to the database.To stop all processes, use the following commands:

n In Windows:

net stop nlserver6

n In Linux:

/etc/init.d/nlserver6 stop

Use the following command to check that all processes are stopped:

nlserver pdump

Note:

In Windows, the webmdl process can still be active without impacting other operations.

You can also check that no system processes remain running.To do this, use the following process:

n In Windows: open the Task manager and check that there are no nlserver.exe processes.

n In Linux: run the ps aux | grep nlserver command and check that there are no nlserver processes.

Step 4 - Restore the databases in the target environment (dev)To restore the source databases onto the target environment, use the following command:

psql mydatabase < mydatabase.sql

Step 5 - Cauterize the target environment (dev)To avoid malfunctions, the processes linked to delivery sending and workflow execution must not be executedautomatically when the target environment is activated.To do this, run the following command:

nlserver javascript nms:freezeInstance.js -instance:<dev> -arg:run

Step 6 - Check cauterization

1 Check that the only deliverypart is the one whose ID is set to 0:

SELECT * FROM neolane.nmsdeliverypart;

36 | © Neolane 2013

Neolane

platform-v6.1-en.pdf#nameddest=ACY-48931

2 Check that the delivery status update is correct:

SELECT iState, count(*) FROM neolane.nmsdelivery GROUP BY iState;

3 Check that the workflow status update is correct:

SELECT iState, count(*) FROM neolane.xtkworkflow GROUP BY iState;SELECT iStatus, count(*) FROM neolane.xtkworkflow GROUP BY iStatus;

Step 7 - Restart the target environment Web process (dev)On the target environment, re-start the Neolane processes for all servers.

Note:

Before re-starting Adobe Campaign on the dev environment, you can apply an additional safety procedure:start the web module only.To do this, edit your instance's configuration file (config-dev.xml), then add the "_" character before theautoStart="true" options for each module (mta, stat, etc.).

Run the following command to start the Web process:

nlserver start web

Use the following command to check that only the web process has started:

nlserver pdump

Check that access to the client console functions.

Step 8 - Import options and external accounts into the target environment (dev)

Warning:

Warning: only the web process should be started at this step. If this is not the case, stop other runningprocesses before continuting.

Above all, check the values of several lines of the files before importing (for example: 'NmsTracking_Pointer'for the options table and the delivery or mid-sourcing accounts for the external account table).To import the configuration from the target environment database (dev):

1 Open the admin console of the database and purge the external accounts (table nms:extAccount) whoseID is not 0 (@id <> 0).

2 In the Neolane console, import the options_dev.xml package previously created via the import packagefunctionality.Check that the options have indeed been updated in the Administration > Platform > Options node.

3 In the Neolane console, import the extaccount_dev.xml previously created via the import packagefonctionality.Check that external databases have indeed been imported in the Administration > Platform > Externalaccounts.

Step 9 - Restart all processes and change users (dev)To start the Neolane processes, use the following commands:

n In Windows:

net start nlserver6

n In Linux:

/etc/init.d/nlserver6 start

Use the following command to check that the processes are started:

nlserver pdump


Data processing

Change users to find the users that already existed on the dev platform.

Description of the database Cleanup workflow

IntroductionThe Database cleanup workflow accessible via the Administration > Production > Technical workflowsnode, lets you delete obsolete data to avoid exponential growth of the database. The workflow is triggeredautomatically without user intervention.

ConfigurationThe database cleanup is configured on two levels: in the workflow scheduler and in the deployment wizard.

The scheduler

Note:

For more on the scheduler, view the relevant section of the Workflows guide.

By default, the Database cleanup workflow is configured to start daily at 4AM. The scheduler lets youchange the workflow triggering frequency. The following frequencies are available:

n Several times a day

n Daily

n Weekly

38 | © Neolane 2013

Neolane

workflow-v6.1-en.pdf#nameddest=ACY-7548

n Once

Warning:

In order for the Database cleanup workflow to start at the date and time defined in the scheduler, theworkflow engine (wfserver) must be started. If this isn't the case, database cleansing won't take place untilnext time the workflow engine is started.


Data processing

Deployment wizardThe Deployment wizard, accessed via the Tools > Advanced menu, lets you configure how long datais saved for. Values are expressed in days. If these values aren't altered, the workflow will use the defaultvalues.

The fields of the Purge of data window coincide with the following options. These are used by some of thetasks executed by the Database cleanup workflow:

n Consolidated tracking: NmsCleanup_TrackingStatPurgeDelay (refer to Cleanup of tracking logs[page 45])

n Delivery logs: NmsCleanup_BroadLogPurgeDelay (refer to Cleanup of delivery logs [page 45])

n Tracking logs: NmsCleanup_TrackingLogPurgeDelay (refer to Cleanup of tracking logs [page 45])

n Deleted deliveries: NmsCleanup_RecycledDeliveryPurgeDelay (refer to Cleanup of deliveries to bedeleted or recycled [page 41])

n Import rejects: NmsCleanup_RejectsPurgeDelay (refer to Cleanup of rejects generated by imports[page 44])

n Visitor profiles: NmsCleanup_VisitorPurgeDelay (refer to Cleanup of visitors [page 45])

n Offer propositions: NmsCleanup_PropositionPurgeDelay (refer to Cleanup of propositions [page 47])

Note:

The Offer propositions field is only available when the Interaction module is installed.

n Events: NmsCleanup_EventPurgeDelay (refer to Cleansing expired events [page 48])

n Archived events: NmsCleanup_EventHistoPurgeDelay (refer to Cleansing expired events [page 48])

Note:

The Events and Archived events fields are only available if the Message Center module is installed.

All tasks executed by the Database cleanup workflow are described in the following section.

Tasks carried out by the Database cleanup workflowAt the date and time defined in the workflow scheduler (refer to The scheduler [page 38]), the workflowengine starts the database cleanup process. The Database cleanup connects to the database and executesthe tasks in the sequence shown below.

40 | © Neolane 2013

Neolane

Warning:

If one of these tasks fails, the following ones will not be executed.

Note:

SQL queries with a LIMIT attribute will be executed repeatedly until all information is processed.

Lists to delete cleanupThe first task executed by the Database cleanup workflow deletes all groups with the deleteStatus !=0 attribute from the NmsGroup. Records linked to these groups and which exist in other tables are alsodeleted.

1 Lists to be deleted are recovered using the following SQL query:

SELECT iGroupId, sLabel, iType FROM NmsGroup WHERE iDeleteStatus <> 0 OR tsExpirationDate <= GetDate()

2 Each list has several links to other tables. All of these links are deleted in bulk using the following query:

DELETE FROM $(relatedTable) WHERE iGroupId=$(l) IN (SELECT iGroupId FROM $(relatedTable) WHERE iGroupId=$(l) LIMIT 5000)

where $(relatedTable) is a table related to NmsGroup and $(l) is the list identifier.

3 When the list is a 'List' type list, the associated table is deleted using the following query:

DROP TABLE grp$(l)

4 Every Select type list recovered by the operation is deleted using the following query:

DELETE FROM NmsGroup WHERE iGroupId=$(l)

where $(l) is the list identifier.

Cleanup of deliveries to be deleted or recycledThis task purges all deliveries to be deleted or recycled.

1 The Database cleanup workflow selects all deliveries for which the deleteStatus field has the valueYes or Recycled and whose delete date is earlier than the period defined in the Deleted deliveries(NmsCleanup_RecycledDeliveryPurgeDelay) field of the deployment wizard. For more on this, referto Deployment wizard [page 40]. This period is calculated in relation to the current server date.

2 For each mid-sourcing server, the task selects the list of deliveries to be deleted.

3 The Database cleanup workflow deletes delivery logs, attachments, mirror page information and allother related data.

4 Before deleting the delivery for good, the workflow purges linked information from the following tables:

n In the delivery exclusion table (NmsDlvExclusion), the following query is used:

DELETE FROM NmsDlvExclusion WHERE iDeliveryId=$(l)

where $(l) is the identifier of the delivery.

n In the coupon table (NmsCouponValue), the following query is used (with mass-deletions):

DELETE FROM NmsCouponValue WHERE iMessageId IN (SELECT iMessageId FROM NmsCouponValue WHERE EXISTS (SELECT B.iBroadLogId FROM $(BroadLogTableName) B WHERE B.iDeliveryId = $(l) AND B.iBroadLogId = iMessageId ) LIMIT 5000)


n In the delivery log tables (NmsBroadlogXxx), mass-deletions are executed in batches of 10,000records.

n In the offer proposition tables (NmsPropositionXxx), mass-deletions are executed in batches of10,000 records.

n In the tracking log tables (NmsTrackinglogXxx), mass-deletions are executed in batches of 5,000records.


Data processing

n In the delivery fragment table (NmsDeliveryPart), mass-deletions are executed in batches of 5,000records. This table contains personalization information on the remaining messages to be delivered.

n In the mirror page data fragment table (NmsMirrorPageInfo), mass-deletions are executed inbatches of 5,000 records. This table contains personalization information on all messages used forgenerating mirror pages.

n In the mirror page search table (NmsMirrorPageSearch), mass-deletions are executed in batchesof 5,000 records. This table is a search index which provides access to personalization informationstored in the NmsMirrorPageInfo table.

n In the batch process log table (XtkJobLog), mass-deletions are executed in batches of 5,000 records.This table contains the log of deliveries to be deleted.

n In the delivery URL tracking table (NmsTrackingUrl), the following query is used:

DELETE FROM NmsTrackingUrl WHERE iDeliveryId=$(l)


This table contains the URLs found in the deliveries to be deleted to enable their tracking.

5 The delivery is deleted from the delivery table (NmsDelivery):

DELETE FROM NmsDelivery WHERE iDeliveryId = $(l)


Deliveries using mid-sourcing

The Database cleanup workflow also deletes deliveries on the mid-sourcing server(s).

1 To do this, the workflow checks that each delivery is inactive (based on its status). If a delivery is active,it will be stopped before it is deleted. The check is carried out by executing the following query:

SELECT iState FROM NmsDelivery WHERE iDeliveryId = $(l) AND iState <> 100;


2 If the value of the status is Start pending, In progress, Recovery pending, Recovery in progress,Pause requested, Pause in progress, or Paused (values 51, 55, 61, 62, 71, 72, 75), the delivery isstopped and the task purges the linked information.

Cleanup of expired deliveriesThis task stops deliveries whose validity period has expired.

1 The Database cleanup workflow creates the list of deliveries which have expired. This list includes allexpired deliveries with a status other than Finished, as well as recently stopped deliveries with over10,000 non-processed messages. The following query is used:

SELECT iDeliveryId, iState FROM NmsDelivery WHERE iDeleteStatus=0 AND iIsModel=0 AND iDeliveryMode=1 AND ( (iState >= 51 AND iState < 85 AND tsValidity IS NOT NULL AND tsValidity < $(currentDate) ) OR (iState = 85 AND DateMinusDays(15) < tsLastModified AND iToDeliver - iProcessed >= 10000 ))

where delivery mode 1 matches the Mass delivery mode, state 51 matches the Start pendingstate, state 85 matches the Stopped state, and the highest number of delivery logs mass-updatedon the delivery server equals 10,000.

2 The workflow then includes the list of recently expired deliveries which use mid-sourcing. Deliveries forwhich no delivery logs have yet been recovered via the mid-sourcing server are excluded.The following query is used:

SELECT iDeliveryId, tsValidity, iMidRemoteId, mData FROM NmsDelivery WHERE (iDeliveryMode = 4 AND (iState = 85 OR iState = 95) AND tsValidity IS NOT NULL AND (tsValidity < SubDays(GetDate() , 15) OR tsValidity < $(DateOfLastLogPullUp)) AND tsLastModified > SubDays(GetDate() , 15))

3 The following query is used to detect whether or not the external account is still active, for filteringdeliveries by date:

SELECT iExtAccountId FROM NmsExtAccount WHERE iActive<>0 AND sName=$(providerName)

4 In the list of expired deliveries, delivery logs whose status is Pending, switch to Delivery cancelled,and all deliveries in this list switch to Finished.

42 | © Neolane 2013

Neolane

The following queries are used:

UPDATE $(BroadLogTableName) SET tsLastModified=$(curdate), iStatus=7, iMsgId=$(bl) WHERE iDeliveryId=$(dl) AND iStatus=6

where $(curdate) is the current date of the database server, $(bl) is the identifier of the deliverylogs message, $(dl) is the delivery identifier, delivery status 6 matches the Pending status anddelivery status 7 matches the Delivery cancelled status.

UPDATE NmsDelivery SET iState = 95, tsLastModified = $(curdate), tsBroadEnd = tsValidity WHERE iDeliveryId = $(dl)

where delivery state 95 matches the Finished status, and $(dl) is the identifier of the delivery.

5 All fragments (deliveryParts) of obsolete deliveries are deleted and all obsolete fragments of notificationdeliveries in progress are deleted. Mass-deletion is used for both these tasks.The following queries are used:

DELETE FROM NmsDeliveryPart WHERE iDeliveryPartId IN (SELECT iDeliveryPartId FROM NmsDeliveryPart WHERE iDeliveryId IN (SELECT iDeliveryId FROM NmsDelivery WHERE iState=95 OR iState=85) LIMIT 5000)

DELETE FROM NmsDeliveryPart WHERE iDeliveryPartId IN (SELECT iDeliveryPartId FROM NmsDeliveryPart WHERE tsValidity < $(curDate) LIMIT 500000)

where delivery state 95matches the Finished status, delivery state 85matches the Stoppedstatus, and $(curDate) is the current server date.

Cleanup of mirror pagesThis task deletes the web resources (mirror pages) used by deliveries.

1 First of all, the list of deliveries to be purged is recovered using the following query:

SELECT iDeliveryId, iNeedMirrorPage FROM NmsDelivery WHERE iWebResPurged = 0 AND tsWebValidity IS NOT NULL AND tsWebValidity < $(curdate)"

where $(curDate) is the current server date.

2 The NmsMirrorPageInfo table is then purged, if necessary using the identifier of the previouslyrecovered delivery. Mass-deletion is used to generate the following queries:

DELETE FROM NmsMirrorPageInfo WHERE iMirrorPageInfoId IN (SELECT iMirrorPageInfoId FROM NmsMirrorPageInfo WHERE iDeliveryId = $(dl)) LIMIT 5000)

DELETE FROM NmsMirrorPageSearch WHERE iMessageId IN (SELECT iMessageId FROM NmsMirrorPageSearch WHERE iDeliveryId = $(dl)) LIMIT 5000)

where $(dl) is the identifier of the delivery.

3 An entry is then added to the delivery log.

4 Purged deliveries are then identified, to avoid having to reprocess them later. The following query isexecuted:

UPDATE NmsDelivery SET iWebResPurged = 1 WHERE iDeliveryId IN ($(strIn))

where $(strIn) is the list of delivery identifiers.

Cleanup of work tablesThis task deletes from the database, all work tables which match deliveries whose status is Being edited,Stopped or Deleted.

1 The list of tables with names beginning with wkDlv_ is recovered first with the following query(postgresql):

SELECT relname FROM pg_class WHERE relname LIKE Lower('wkDlv_') ESCAPE E'\\\\' AND relkind IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'

2 The tables used by workflows in progress are then excluded. To do this, the list of deliveries in progressis recovered using the following query:

SELECT iDeliveryId FROM NmsDelivery WHERE iDeliveryId<>0 AND iDeleteStatus=0 AND iState NOT IN (0,85,100);


Data processing

where 0 is the value which matches the Being edited delivery status, 85 matches the Stopped statusand 100 matches the Deleted status.

3 Tables that are no longer used will be deleted using the following query:

DROP TABLE wkDlv_15487_1;

Cleanup of rejects generated by importsThis step lets you delete records for which all data wasn't processed during import.

1 Mass-deletion is carried out on the XtkReject table with the following query:

DELETE FROM XtkReject WHERE iRejectId IN (SELECT iRejectId FROM XtkReject WHERE tsLog < $(curDate)) LIMIT $(l))

where $(curDate) is the current server date from which we subtract the period defined for theNmsCleanup_RejectsPurgeDelay option (refer to Deployment wizard [page 40]) and $(l) is themaximum number of records to be mass deleted.

2 All orphan rejects are then deleted using the following query:

DELETE FROM XtkReject WHERE iJobId NOT IN (SELECT iJobId FROM XtkJob)

Cleanup of workflow instancesThis task purges each workflow instance using its identifer (lWorkflowId) and history (lHistory). Itdeletes inactive tables by running the worktable cleanup task again.

1 To recover the list of workflows to be deleted, the following query is used:

SELECT iWorkflowId, iHistory FROM XtkWorkflow WHERE iWorkflowId<>0

2 This query generates the list of workflows which will be used to delete all linked logs, finished tasks andfinished events, using the following queries:

DELETE FROM XtkWorkflowLog WHERE iWorkflowId=$(lworkflow) AND tsLog < DateMinusDays($(lhistory))

DELETE FROM XtkWorkflowTask WHERE iWorkflowId=$(lworkflow) AND iStatus<>0 AND tsCompletion < DateMinusDays($(lhistory))

DELETE FROM XtkWorkflowEvent WHERE iWorkflowId=$(l) AND iStatus>2 AND tsProcessing < DateMinusDays($(lHistory))

where $(lworkflow) is the identifier of the workflow and $(lhistory) is the identifier of the history.

3 All unused tables are deleted. For this purpose, all tables are collected thanks to a wkf% type maskusing the following query (postgresql):

SELECT relname FROM pg_class WHERE relname LIKE Lower('wkf%') ESCAPE E'\\\\' AND relkind IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'

4 Then all tables used by a pending workflow instance are excluded. The list of active workflows is recoveredusing the following query:

SELECT iWorkflowId FROM XtkWorkflow WHERE iWorkflowId<>0 AND iState<>20

5 Each workflow identifier is then recovered to find the name of the tables used by workflows in progress.These names are excluded from the list of previously recovered tables.

6 "incremental query" type activity history tables are excluded using the following queries:

SELECT relname FROM pg_class WHERE relname LIKE Lower('wkfhisto%') ESCAPE E'\\\\' AND relkind IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'

SELECT iWorkflowId FROM XtkWorkflow WHERE iWorkflowId IN ($(strCondition))

where $(strcondition) is the list of tables which match the wkfhisto% mask.

7 The remaining tables are deleted using the following query:

DROP TABLE wkf15487_12;

44 | © Neolane 2013

Neolane

Cleanup of workflow loginsThis task deletes workflow logins using the following query:

DELETE FROM XtkWorkflowLogin WHERE iWorkflowId NOT IN (SELECT iWorkflowId FROM XtkWorkflow)

Cleanup of orphan work tablesThis task deletes orphan work tables linked to groups. The NmsGroup table stores the groups to be cleansed(with a type different from 0). The prefix of the table names is grp. To identify the groups to be cleansed,the following query is used:

SELECT iGroupId FROM NmsGroup WHERE iType>0"

Cleanup of visitorsThis task deletes obsolete records from the visitor table using mass-deletion. Obsolete records are those forwhich the last modification is earlier than the conservation period defined in the deployment wizard (referto Deployment wizard [page 40]). The following query is used:

DELETE FROM NmsVisitor WHERE iVisitorId IN (SELECT iVisitorId FROM NmsVisitor WHERE iRecipientId = 0 AND tsLastModified < $(tsDate) LIMIT 5000)

where $(tsDate) is the current server date, from which we subtract the period defined for theNmsCleanup_VisitorPurgeDelay option.

Cleanup of NPAIThis task lets you delete records which match valid addresses from the NmsAddress table. The followingquery is used to perform mass-deletion:

DELETE FROM NmsAddress WHERE iAddressId IN (SELECT iAddressId FROM NmsAddress WHERE iStatus=2 AND tsLastModified < $(tsDate1) AND tsLastModified >= $(tsDate2) LIMIT 5000)

where status 2 matches the Valid status, $(tsDate1) is the current server date, and $(tsDate2)matches the NmsCleanup_LastCleanup option.

Cleanup of subscriptionsThis task purges all subscriptions deleted by the user from the NmsSubscription table, using mass-deletion.The following query is used:

DELETE FROM NmsSubscription WHERE iDeleteStatus <>0

Cleanup of tracking logsThis task deletes obsolete records from the tracking and webtracking log tables. Obsolete records are thosewhich are earlier than the conservation period defined in the deployment wizard (refer to Deployment wizard[page 40]).

1 First, the list of tracking log tables is recovered using the following query:

SELECT distinct(sTrackingLogSchema) FROM NmsDeliveryMapping WHERE sTrackingLogSchema IS NOT NULL;

2 Mass-deletion is used to purge all tables in the list of previously recovered tables. The following query isused:

DELETE FROM XtkTrackingLogRcp WHERE iTrackingLogId IN (SELECT iTrackingLogId FROM XtkTrackingLogRcp WHERE tsLog < $(tsDate) LIMIT 5000)

where $(tsDate) is the current server date from which we subtract the period defined for theNmsCleanup_TrackingLogPurgeDelay option.

3 The tracking statistics table is purged using mass-deletion. The following query is used:

DELETE FROM NmsTrackingStats WHERE iTrackingStatsId IN (SELECT iTrackingStatsId FROM NmsTrackingStats WHERE tsStart < $(tsDate) LIMIT 5000)

where $(tsDate) is the current server date from which we subtract the period defined for theNmsCleanup_TrackingStatPurgeDelay option.

Cleanup of delivery logsThis task lets you purge the delivery logs stored in various tables.


Data processing

1 For this purpose, the list of delivery log schemas is recovered using the following query:

SELECT distinct(sBroadLogSchema) FROM NmsDeliveryMapping WHERE sBroadLogSchema IS NOT NULL UNION SELECT distinct(sBroadLogExclSchema) FROM NmsDeliveryMapping WHERE sBroadLogExclSchema IS NOT NULL

2 When using mid-sourcing, the NmsBroadLogMid table isn't referenced in delivery mappings. Thenms:broadLogMid schema is added to the list recovered by the previous query.

3 The Database cleanup workflow then purges obsolete data from previously recovered tables. Thefollowing query is used:

DELETE FROM $(tableName) WHERE iBroadLogId IN (SELECT iBroadLogId FROM $(tableName) WHERE tsLastModified < $(option) LIMIT 5000)

where $(tableName) is the name of each table in the list of schemas, and $(option) is the datedefined for the NmsCleanup_BroadLogPurgeDelay option (refer to Deployment wizard [page 40]).

4 Finally, the workflow checks whether the NmsProviderMsgId table exists. If so, all obsolete data isdeleted using the following query:

DELETE FROM NmsProviderMsgId WHERE iBroadLogId IN (SELECT iBroadLogId FROM NmsProviderMsgId WHERE tsCreated < $(option) LIMIT 5000)

where $(option) matches the date defined for the NmsCleanup_BroadLogPurgeDelay option(refer to Deployment wizard [page 40]).

Cleanup of the NmsEmailErrorStat tableThis task cleanses the NmsEmailErrorStat table. The main program (coalesceErrors) defines two dates:

n Start date: date of the next process which matches the NmsLastErrorStatCoalesce option or themost recent date in the table.

n End date: current server date.

If the start date is greater than or equal to the end date, no process will take place. In this case, thecoalesceUpToDate message appears.

If the start date is earlier than the end date, the NmsEmailErrorStat table is cleansed.The total number of errors in the NmsEmailErrorStat table, between the start and end dates, is recoveredusing the following query:

"SELECT COUNT(*) FROM NmsEmailErrorStat WHERE tsDate>= $(start) AND tsDate< $(end)"

where $end and $start are the start and end dates defined previously.

If the total is greater than 0:

1 The following query is executed in order to keep only errors beyond a certain threshold (which equals20):

"SELECT iMXIP, iPublicId, SUM(iTotalConnections), SUM(iTotalErrors), SUM(iMessageErrors), SUM(iAbortedConnections), SUM(iFailedConnections), SUM(iRefusedConnections), SUM(iTimeoutConnections) FROM NmsEmailErrorStat WHERE tsDate>=$(start ) AND tsDate<$(end ) GROUP BY iMXIP, iPublicId HAVING SUM(iTotalErrors) >= 20"

2 The coalescingErrors message is displayed.

3 A new connection is created to delete all errors which occurred between the start and end dates. Thefollowing query is used:

"DELETE FROM NmsEmailErrorStat WHERE tsDate>=$(start) AND tsDate<$(end)"

4 Each error is saved in the NmsEmailErrorStat table using the following query:

"INSERT INTO NmsEmailErrorStat(iMXIP, iPublicId, tsDate, iTotalConnections, iTotalErrors, iTimeoutConnections, iRefusedConnections, iAbortedConnections, iFailedConnections, iMessageErrors) VALUES($(lmxip ), $(lpublicId ), $(tsstart ), $(lconnections ), $(lconnectionErrors ),$(ltimeoutConnections ), $(lrefusedConnections ), $(labortedConnections ), $(lfailedConnections ), $(lmessageErrors))"

where each variable matches a value recovered by the previous query.

5 The start variable is updated with the values of the previous process to finish the loop.

The loop and the task stop.

46 | © Neolane 2013

Neolane

Cleanups are executed on the NmsEmailError and cleanupNmsMxDomain tables.

Cleanup of the NmsEmailError tableThe following query is used:

DELETE FROM NmsEmailError WHERE iMXIP NOT IN (SELECT DISTINCT iMXIP FROM NmsEmailErrorStat)

This query deletes all lines without linked records in the NmsEmailErrorStat from the NmsEmailErrortable.

Cleanup of the NmsMxDomain tableThe following query is used:

DELETE FROM NmsMxDomain WHERE iMXIP NOT IN (SELECT DISTINCT iMXIP FROM NmsEmailErrorStat)

This query deletes all lines without a linked record in the NmsEmailErrorStat table from the NmsMxDomaintable.

Cleanup of propositionsIf the Interaction module is installed, this task is executed to purge the NmsPropositionXxx tables.The list of propositions tables is recovered and mass-deletion is carried out on each one, using the followingquery:

DELETE FROM NmsPropositionXxx WHERE iPropositionId IN (SELECT iPropositionId FROM NmsPropositionXxx WHERE tsLastModified < $(option) LIMIT 5000)

where $(option) is the date defined for the NmsCleanup_PropositionPurgeDelay option (refer toDeployment wizard [page 40]).

Cleanup of simulation tablesThis task cleanses orphan simulation tables (that are no longer linked to an offer simulation or a deliverysimulation).

1 To recover the list of simulations that require cleanup, the following query is used:

SELECT iSimulationId FROM NmsSimulation WHERE iSimulationId<>0

2 The name of the tables to delete is made up of the wkSimu_ prefix followed by the identifier of thesimulation (for instance: wkSimu_456831_aggr):

DROP TABLE wkSimu_456831_aggr

Statistics updateIf the XtkCleanup_NoStats option does not exist or if its value is 0 (i.e. the option is disabled), this taskis executed to update the statistics in the database. If the value of the option is 1 (i.e. the option is activated),statistics updating isn't executed. This task calls up a database procedure.

Subscription cleanup (NMAC)This task deletes any subscriptions related to deleted services or mobile applications.

1 To recover the list of broadlog schemas, the following query is used:

SELECT distinct(sBroadLogSchema) FROM NmsDeliveryMapping WHERE sBroadLogSchema IS NOT NULL

2 The task then recovers the names of the tables linked to the appSubscription link and deletes thesetables.

Cleansing session informationThis task cleanses information from the sessionInfo table, the following query is used:

DELETE FROM XtkSessionInfo WHERE tsexpiration < $(curdate)


Data processing

Cleansing expired eventsThis task cleanses the events received and stored on the execution instances and the events archived on acontrol instance.

Cleansing reactionsThis task cleanses the reactions (NmsRemaMatchRcp table) in which the hypotheses have themselvesbeen deleted.

48 | © Neolane 2013

Neolane

Table of ContentsUpgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

In Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . 50In Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Resolving upgrade conflicts . . . . . . . . . . . . . . . . . . . . . . 52Informing client consoles of the available update . . . . . . . . . . . . . . 53

Example of the Unicode switch of an existing instance . . . . . . . . . . . . . 54

This section presents the procedure to be applied to upgrade Neolane, client side and server side,and describes the switching to Unicode of an existing instance.The upgrade must be applied to all servers where Neolane is installed.Neolane is based on several processes executed on the server side which you will need to manipulateduring updates, in particular:

n Application server (nlserver web)

n Delivery server (nlserver mta)

n Redirection server (webmdl)

Note:

For more on the various Neolane processes, refer to the Installation guide.When using the Power Booster or Power Cluster type architecture, you must apply this process toall Power Booster/Cluster servers.

If the new version involves an alteration of the database structure, we recommend restarting theservers in the following order:

1 Application server (nlserver web),

2 Redirection server (webmdl),

3 Delivery server (nlserver mta).

Upgrade

Note:

In the following commands, the XXXX characters represent the Neolane build number.

Neolane v6.1 - Production Guide - Updating Neolane | 49

CHAPTER 4

Updating Neolane

installation-v6.1-en.pdf#nameddest=SBL-9856

Warning:

We strongly recommend making a database backup on each instance before updating. For more information,refer to Backup [page 33].

In WindowsTo update Neolane in a new version when delivering a new build, the following procedure should be appliedin Windows:

n Shut down services [page 50],

n Upgrade the Neolane server application [page 50],

n Synchronizing resources [page 50],

n Restart services [page 51].

To find out how to update the client console, refer to Client console availability.

Shut down servicesIn order to replace all files with the new version, you need to shut down all instances of the nlserver service.

1 Shut down the following services:

n Web services (IIS):iisreset /stop

n Neolane service: net stop nlserver6

Warning:

You also need to make sure the redirection server (webmdl) is stopped, so that the nlsrvmod.dll fileused by IIS can be replaced with the new version.

2 Check that no tasks are active by running the nlserver pdump command. The following should comeup:

C:\Program Files\Neolane\Neolane v6\bin>nlserver pdump11:33:39 > Application Server for Neolane Version 6.00 (build 6745) dated 11/07/2011No tasks

You can use the Windows Task Manager to make sure all processes are stopped.

Upgrade the Neolane server applicationTo run the upgrade file, apply the following steps:

1 Run setup.exe.To download this file, go to the Neolane Support page (http://support.neolane.net/) via the DownloadCenter link.

2 Select the installation mode: choose Update or repair

3 Click Next.

4 Click Finish.The installation program then copies the new files.

5 Once the operation is complete, click Finish.

Synchronizing resourcesUse the following command linenlserver config -postupgrade -allinstances

To carry out the following operations:

n Synchronize resources,

n update schemas,

n update the database.

50 | © Neolane 2013

Neolane


http://support.neolane.net/

Note:

This operation should be performed only once, and only on an (nlserver web) application server.

Then check whether the synchronization has generated errors or warnings. For more on this, refer to Resolvingupgrade conflicts [page 52].

Tip:

To synchronize one database only, use the following command: nlserver config -postupgrade-instance:<instance_name>.

Restart servicesThe services to be restarted are:

n Web services (IIS):iisreset /start

n Neolane service: net start nlserver6

In LinuxTo update Neolane in a new version when a new build is delivered, the procedure for Linux is as follows:

n Obtain updated packages [page 51],

n Performing an update [page 51],

n Rebooting the web server [page 51].

To find out how to update the client console, refer to Client console availability.

Obtain updated packagesStart by recovering both updated packages of Neolane.The files are:

n nlthirdparty6-XXXX-linux-2.6-amd64.deb or nlthirdparty6-XXXX-0.x86_64.rpm

n nlserver6-XXXX-linux-2.6-amd64.deb or nlserver6-XXXX-0.x86_64.rpm

Performing an update

n RPM based distribution (RedHat, SuSe)To install them, execute as root:

$rpm -Uvh nlthirdparty6-XXXX-0.x86_64.rpm nlserver6-XXXX-0.x86_64.rpm

n DEB based distribution (Debian)To install them, execute as root:

dpkg -i nlthirdparty6-XXXX-linux-2.6-amd64.deb nlserver6-XXXX-linux-2.6-amd64.deb

Note:

Resources are synchronized automatically, however you need to make sure no errors occurred. For more onthis, refer to Resolving upgrade conflicts [page 52].

Rebooting the web serverYou must shut down Apache for the new library to become applicable.To do this, execute the following command:

/etc/init.d/apache stop


Updating Neolane


Warning:

n Your script might be called httpd instead of apache.

n You MUST execute this command until you obtain the following reply:

httpd (no pid file) not running

This operation is required in order for Apache to apply the new library.

Then restart Apache:

/etc/init.d/apache start

Resolving upgrade conflictsDuring resource synchronization, the postupgrade command enables you to detect whether synchronizationhas generated errors or warnings.

View the synchronization resultThere are two ways of viewing the synchronization result:

n In the command-line interface, errors are materialized by a triple chevron >>> and synchronization isstopped automatically. Warnings are materialized by a double chevron >> and must be resolved oncesynchronization is complete. At the end of the postupgrade, a summary is displayed in the commandprompt. It can look like this:

2013-04-09 07:48:39.749Z 00002E7A 1 info log =========Summary of the update==========2013-04-09 07:48:39.749Z 00002E7A 1 info log neolanetest instance, 6 warning(s) and 0 error(s) during the update.2013-04-09 07:48:39.749Z 00002E7A 1 warning log The document with identifier 'mobileAppDeliveryFeedback' and type 'xtk:report' is in conflict with the new version.2013-04-09 07:48:39.749Z 00002E7A 1 warning log The document with identifier 'opensByUserAgent' and type 'xtk:report' is in conflict with the new version.2013-04-09 07:48:39.750Z 00002E7A 1 warning log The document with identifier 'deliveryValidation' and type 'nms:webApp' is in conflict with the new version.2013-04-09 07:48:39.750Z 00002E7A 1 warning log Document of identifier 'nms:includeView' and type 'xtk:srcSchema' updated in the database and found in the file system. You will have to merge the two versions manually.

If the warning concerns a conflict of resources, user attention is required to resolve it.

n The postupgrade_<server version number>_<time of postupgrade>.log log file contains thesynchronization result. It is available by default in the following directory: <installationdirectory/var/<instance/postupgrade. Errors and warnings are indicated by the error and warningattributes.

Resolving conflictsTo resolve conflicts, apply the following process:

1 In the Neolane tree, go to Administration>Configuration>Package management>Edit conflicts.

2 Select the conflict you want to resolve in the list.

There are three ways to resolve a conflict:

n Declare as resolved: requires user intervention beforehand.

n Accept the new version: recommended if the resources provided with Neolane have not been changedby the user.

n Keep the current version: means that the update is rejected.

Warning:

If you select this resolution mode, you may not benefit from corrections in the new version.

52 | © Neolane 2013

Neolane

If you chose to resolve the conflict manually, proceed as follows:

1 In the lower section of the window, search for the _conflict_ string to locate the entities with conflicts.The entity installed with the new version contains the new argument, the entity that matches the previousversion contains the cus argument.

2 Delete the version you don't want to keep. Delete the _conflict_argument_ string of the entity youare keeping.

3 Go to the conflict you have resolved. Click the Actions icon and select Declare as resolved.

4 Save your changes: the conflict is now resolved.

Informing client consoles of the available update

In WindowsOn the machine where the (nlserver web) Neolane application server is installed, download and copy thefilesetup-client-6.XXXX.exe

in [path of the application]\datakit\nl\en"\jsp

The next time client consoles are connected, a window will inform users about the availability of an updateand offer them the possibility of downloading and installing it.

Note:

Make sure the IIS_XPG user has the appropriate read rights for this installation file and refer to theInstallation Guide for more information.

In LinuxOn the machine where the Neolane application server (nlserver web) is installed, retrieve the followingpackage:setup-client-6.XXXX.exe


Updating Neolane

and copy it, saving as /usr/local/neolane/nl6/datakit/nl/en/jsp:

cp setup-client-6.XXXX.exe /usr/local/neolane/nl6/datakit/nl/en/jsp

The next time client consoles are connected, a window will inform users about the availability of an updateand offer them the possibility of downloading and installing it.

Note:

Make sure the Apache user has the appropriate read rights for this installation file and refer to the InstallationGuide for more information.

Example of the Unicode switch of an existing instance

For an existing prod instance in Linux/PostgreSQL, the steps for switching to unicode are as follows:

1 Stop the processes writing to the database:

su - neolanenlserver shutdown

2 Dump the database:

su - postgrespg_dump mydatabase > mydatabase.sql

3 Create a Unicode database:

createdb -E UNICODE mydatabase_unicode

4 Restore the database:

psql mydatabase_unicode < mydatabase.sql

5 Update the option indicating that the database is Unicode:

psql mydatabase_unicodeupdate XtkOption set sStringValue = 'u'||sStringValue where sName='XtkDatabaseId' and sStringValue not like 'u%';

6 On the tracking servers:

su - neolanecd nl6/confvi config-prod.xml

Add the u character in front of the value relating to the database identifier (databaseId):

<web> <redirection databaseId="u7F0000010554364C" trackingPassword="myPassword="/></web>

7 On servers calling the database:

su - neolanecd nl6/confvi config-prod.xml

Modify the database reference:

<dataSource name="default"> <dbcnx encrypted="1" login="neolane:mydatabase_unicode" password="xxxx=" provider="postgresql" server="yyyy"/></dataSource>

8 Reboot all the machines:

54 | © Neolane 2013

Neolane

/etc/init.d/apache stop/etc/init.d/nlserver6 stop/etc/init.d/nlserver6 start/etc/init.d/apache start

9 Confirm the switch. To do this, connect via the Neolane console and:

n check that the data is displayed correctly, in particular the accentuated characters:

n launch a delivery and check that the tracking retrieval works.


Updating Neolane


Neolane

Table of ContentsList of tables to maintain . . . . . . . . . . . . . . . . . . . . . . . 57

Neolane tables . . . . . . . . . . . . . . . . . . . . . . . . . . 58Customer tables . . . . . . . . . . . . . . . . . . . . . . . . . 60

Types of maintenance . . . . . . . . . . . . . . . . . . . . . . . . 61Application maintenance . . . . . . . . . . . . . . . . . . . . . . . 61Technical maintenance . . . . . . . . . . . . . . . . . . . . . . . 61

RDBMS Specific recommendations . . . . . . . . . . . . . . . . . . . . 62PostgreSQL . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 68MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Neolane is a highly transactional system (OLTP database). This means that the underlying databasewill be frequently updated, leading to a degradation of performance over time. To avoid this typeof issue, regular database maintenance is necessary.

Warning:

A database will only perform optimally if it is maintained on a regular basis. The automaticmaintenance offered by some RDBMSs isn't sufficient and does not replace in-depth maintenance.This isn't specific to Neolane but concerns all relational database transactional management systems.

This chapter details how to create maintenance plans to suit your needs.

Warning:

The procedures described in this document are recommendations. The person in charge of databasemaintenance plans is the database administrator. Please contact this person and not Neolane incase of problems.

List of tables to maintain

The list of tables to maintain depends on your version of Neolane, how you use it and on thedatamodel configuration.The following list contains only the tables most subject to fragmentation. The impacts are as follows:

Neolane v6.1 - Production Guide - Database maintenance | 57

CHAPTER 5

Database maintenance

n overconsumption of disk space, thus affecting database access,

n indexes which have not been regularly updated, which slows down query performance.

Neolane tables

From Neolane 4.05 upward

CommentsMain type of activitySizeTable name

There is one record per deliveryaction. A single record can beupdated several times to reflectdelivery progress, so indexes onthis table tend to fragment rap-idly.

UpdatesSmallNmsDelivery

Work table which records are in-serted into during delivery prepar-ation. They are then updatedduring delivery and finally deletedonce the delivery is complete.

This table tends to fragment rap-idly even though its average sizeis fairly limited.

Insertions, updates, deletionsMediumNmsDeliveryPart

This table contains the informa-tion needed to generate personal-ized mirror pages. It contains amemo (CLOB) field, and as suchwill tend to be very large. Thevolume is directly proportional tothe history of mirror pages kept.

Insertions, deletionsLargeNmsMirrorPageInfo

This table contains statistics onthe delivery process. Its recordsare regularly updated.

Insertions, updates, deletionsMediumNmsDeliveryStat

This table contains informationon email addresses. It is fre-quently updated as part of thequarantine process (records arecreated at the first delivery error,updated when the counterschange and deleted once deliveryis successful).

Updates, insertionsMediumNmsAddress

There is one record per workflowinstance, so very few records.However the table it is regularlyupdated to reflect status andprogress.

UpdatesSmallXtkWorkflow

Each execution of a workflowactivity leads to the creation of arecord in this table. The purgemechanism deletes them oncethey are expired.

Insertions, updates, deletionsSmallXtkWorkflowTask

Each transition activated betweentasks in a workflow leads to thecreation of a record in this table.The purge mechanism deletesthem once they are expired.

Insertions, updates, deletionsSmallXtkWorkflowEvent

58 | © Neolane 2013

Neolane


This table is specific to the work-flow engine. It enables the send-ing of commands to workflows(Start, Stop, Pause, for instance).Although it is small, this table istaken into account during thepurge of transactional tableslinked to workflows.

Insertions, updates, deletionsVery smallXtkWorkflowJob

From versions 4.05 and 5.00 upward


This is the largest table in thesystem. There is one record permessage sent, and these recordsare inserted, updated to track thedelivery status, and deleted whenthe history is purged.

Insertions, updates, deletionsLargestNmsBroadLog

Tracking logs are inserted anddeleted when the history ispurged, but they are not updated.

Insertions, deletionsLarge volumeNmsTrackingLog

From Neolane 5.10 upward


This table contains informationused for qualifying SMTP errors.It is fairly small, but will bemassively updated, so indexes onthis table tend to fragment rap-idly.

UpdatesSmallNmsBroadlogMsg

This table contains the aggreg-ates on SMTP errors sorted bydomain. It initially contains de-tailed information which is aggreg-ated by the cleanup task once itbecomes outdated.

Insertions, updates, deletionsMediumNmsEmailErrorStat

Only when the Neolane 5.10 (orlater) instance is used as a mid-sourcing instance. This is one ofthe largest table in the database.There is one record per messagesent, and these records are inser-ted, updated to track the deliverystatus, and deleted when thehistory is purged. When usingmid-sourcing, the recommenda-tion is to limit the history (usuallyless than two months), so thistable remains reasonable in termsof size (less than 30 Go for 60million rows, data+index), but itis very important to rebuild itfrom time to time.

Insertions, updates, deletionsLargeNmsBroadLogMid (on a mid-sourcing instance)




This is the largest table in thesystem. There is one record permessage sent, and these recordsare inserted, updated to track thedelivery status, and deleted whenthe history is purged. Note thatin 5.10, this table is smaller thanthe equivalent in 4.05 (NmsBroad-Log) since the SMTP messagetext is factorized in the NmsBroad-LogMsg table in the 5.10 version.However, it remains essential tore-index this table regularly(every other week to start with),and completely rebuild it fromtime to time (once a month, orwhen performance is impacted).

Insertions, updates, deletionsLargeNmsBroadLogRcp (when the Nm-sRecipient table is used)

Same as NmsBroadLogRcp butwith an external recipient table.Please adapt Yyy and Xxx withthe values in your delivery map-ping.

Insertions, updates, deletionsLargeYyyBroadLogXxx (when an extern-al recipient table is used)

Tracking logs are inserted anddeleted when the history ispurged, but they are not updated.The volume depends on thelength of data retention.

Insertions, deletionsLargeNmsTrackingLogRcp (when theNmsRecipient table is used)

Same as NmsTrackingLogRcp butwith an external recipient table.Please adapt Yyy and Xxx withthe values used in your deliverymapping.

Insertions, deletionsLargeYyyTrackingLogXxx (when theexternal recipient table is used)

From version 6.1

CommentsMain activitySizeTable name

Tables that include mobile applic-ations and their configuration.

Insertions, updates, deletionsVery small volumeNmsMobileApp

Table that includes the identifiersof mobile devices (addresses)used to send the notification(similar to a recipient table).

Insertions, updatesLarge volumeNmsAppSubscriptionRcp

Similar to the other broadlogtables, but with the NmsappSub-scriptionRcp instead of NmsRecip-ient.

Insertions, updates, deletionsLarge volumeNmsBroadLogAppSubRcp

Similar to the other trackingLogtables, but with the NmsappSub-scriptionRcp table instead of Nm-sRecipient.

Insertions, deletionsLarge volumeNmsTrackingLogAppSubRcp

Table that includes user sessions.The number of insertions anddeletions is very important.

Insertions, deletionsSmall volumeXtkSessionInfo

Customer tablesIn addition to the list above, tables containing created by customers (which do not exist in the Neolanedatamodel) during platform setup can also be subject to fragmentation, specially if they are frequentlyupdated during data loading or synchronization procedures. These tables can be part of the default Neolanedata model (for instance NmsRecipient). In this case, it is up to the administrator of the Neolane platformto conduct an audit of its specific database model to find these custom tables. These tables aren't necessarilymentioned explicitly in our maintenance procedures.

60 | © Neolane 2013

Neolane

Types of maintenance

Application maintenanceNeolane provides an out-of-the-box workflow which lets you schedule certain database maintenance tasks.This feature is the database cleanup workflow, which carries out the following tasks:

n deletion of expired records,

n deletion of orphaned records and status reinitialization for expired objects,

n updating the database statistics.

Warning:

Please note that the cleanup task deals mostly with application level maintenance, not with RDBMS levelmaintenance (with the exception of statistics update). However maintenance operations will be required onthe database. Just because the cleanup runs successfully does not mean that the database is optimally tuned.

Technical maintenanceThe database cleanup workflow does not include any database maintenance tool: it is up to you to organizemaintenance. To do this, you can either:

n ask your DBA to set up database maintenance without using the tools available in the Neolane platform,

n use the Neolane workflow engine to schedule and track these maintenance activities.

These maintenance procedures must be carried out on a regular basis and should include the following:

n re-index frequently updated tables,

n compact/rebuild the tables to avoid fragmentation.

Maintenance schedulingThe difficulty is finding appropriate slots for performing these maintenance activities, since they can heavilyimpact the database performance while running or even block the application (due to locking).These tasks are typically run once a week during a period of low activity that does not collide with backups,data reloading or aggregate calculation. Some systems which are highly solicited require more frequentmaintenance.More in-depth maintenance, such as full table rebuilds, can be performed once a month, preferably withapplications fully stopped since the system is unusable anyway.

Rebuilding a tableSeveral strategies are available:

DrawbacksBenefitsDescriptionOperations

Depending on the database,these defragmentation methodscan be provided as an RDBMSoption (Oracle) and aren't alwaysthe most efficient way of dealingwith larger tables.

Simply use the database defrag-mentation method. These meth-ods typically take care of integrityissues by locking the data duringdefragmentation.

Most database engines providedefragmentation methods.

Online defragmentation

Since the table is deleted and re-created, the application cannotbe left online, even in read onlymode (the table is not availableduring the restore phase).

This is the easiest way to defrag-ment a table. Also the only solu-tion when the database is almostfull.

Dump the table to a file, deletethe table in the database and re-store from the dump.

Dump and restore



DrawbacksBenefitsDescriptionOperations

Requires twice the amount ofspace.

All active processes writing to thetable during the process must bestopped. However, reading pro-cesses will not be affected, sincethe table is swapped at the lastmoment once rebuilt.

this method is faster than the firstapproach since it generates lessIOs (no copy as a file and readfrom this file).

This creates a copy of a table andits indexes, then drops the exist-ing one and renames the copy totake its place.

Duplicate, rename and drop

RDBMS Specific recommendations

To help you set up maintenance plans, this section lists some recommendations/best practices adapted tothe various RDBMS engines that Neolane supports. However, these are only recommendations. It is up toyou to adapt them to your needs, in keeping with your internal procedure and constraints. Your DBA is theperson who should ultimately be responsible for these plans.

PostgreSQL

Detecting large tables

1 You can add the following view to your database:

create or replace view uvSpace as SELECT c1.relname AS tablename, c2.relname AS indexname, c2.relpages * 8 / 1024 AS size_mbytes, c2.relfilenode AS filename, 0 AS row_count FROM pg_class c1, pg_class c2, pg_index i WHERE c1.oid = i.indrelid AND i.indexrelid = c2.oid UNION SELECT pg_class.relname AS tablename, NULL::"unknown" AS indexname, pg_class.relpages * 8 / 1024 AS size_mbytes, pg_class.relfilenode AS filename, cast(pg_class.reltuples as integer) AS row_count FROM pg_class WHERE pg_class.relkind = 'r'::"char" ORDER BY 3 DESC, 1, 2 DESC;

2 Running the following command allows you to spot large tables and indexes:

select * from uvSpace;

Simple maintenanceUnder PostgreSQL, the typical commands you can use are vacuum full and reindex.

Here is a typical example of an SQL maintenance plan to be executed on a regular basis using these twocommands:

vacuum full nmsdelivery; reindex table nmsdelivery;

vacuum full nmsdeliverystat; reindex table nmsdeliverystat;

vacuum full xtkworkflow; reindex table xtkworkflow;

vacuum full xtkworkflowevent; reindex table xtkworkflowevent;

vacuum full xtkworkflowjob; reindex table xtkworkflowjob;

vacuum full xtkworkflowlog; reindex table xtkworkflowlog;

vacuum full xtkworkflowtask;

62 | © Neolane 2013

Neolane

reindex table xtkworkflowtask;

vacuum full xtkjoblog; reindex table xtkjoblog;

vacuum full xtkjob; reindex table xtkjob;

vacuum full nmsaddress; reindex table nmsaddress;

vacuum full nmsdeliverypart; reindex table nmsdeliverypart;

vacuum full nmsmirrorpageinfo; reindex table nmsmirrorpageinfo;

Note:

n We recommend starting with smaller tables, this way if the process fails on large tables (where the riskof failure is highest), at least part of the maintenance has been completed.

n Also add the tables specific to your data model which can be subject to significant updates. This can bethe case for NmsRecipient if you have large daily data replication flows.

n The vacuum and re-index commands will lock the table, which pauses some processes whilemaintenance is carried out.

n For very large tables (typically above 5 Gb), vacuum full can become quite inefficient and take a verylong time. We do not recommend using it for the YyyNmsBroadLogXxx table.

n This maintenance operation can be implemented by a Neolane workflow, using an SQL activity (for moreon this, refer to the Workflows guide). Make sure you schedule maintenance for a low activity time whichdoes not collide with your backup window.

Rebuilding a databasePostgreSQL does not provide an easy way of performing an online table rebuild since vacuum full locksthe table, thus preventing regular production. This means that maintenance has to be performed when thetable is not used. You can either:

n perform maintenance when the Neolane platform is stopped,

n stop the various Neolane sub-services likely to write in the table being rebuilt (nlserver stopwfserver instance_name to stop the workflow process).

Here is an example of table defragmentation using specific functions to generate the necessary DDL. Thefollowing SQL lets you create two new functions: GenRebuildTablePart1 and GenRebuildTablePart2,which can be used to generate the necessary DDL to recreate a table.

n The second function then deletes the original table and renames the work table and its indexes.

n The second function then deletes the original table and renames the work table and its indexes.

n Using two functions instead of one means that if the first one fails, you don't run the risk of deleting theoriginal table.

-- ---------------------------------------------------------------------------- Generate the CREATE TABLE DDL for a table-- --------------------------------------------------------------------------create or replace function GenTableDDL(text) returns text as $$declarevstrTable text;vrecFld RECORD;vstrDDL text;vstrFields text;vstrNsTable text;vstrTableSpace text;

beginvstrTable = lower($1);

vstrDDL = ;

SELECTpg_catalog.quote_ident(n.nspname) || '.' || pg_catalog.quote_ident(c.relname),



workflow-v6.1-en.pdf#nameddest=ACY-7261

pg_catalog.quote_ident(t.spcname)INTOvstrNsTable, vstrTableSpace

FROMpg_namespace n, pg_class c left outer join pg_tablespace t on c.reltablespace = t.oid

WHEREn.oid = c.relnamespace ANDc.relname = vstrTable;

vstrDDL = 'CREATE TABLE ' || vstrNsTable || '_tmp(';

vstrFields = ;FOR vrecFld INSELECT

pg_catalog.quote_ident(a.attname) ||' ' || t.typname ||case when t.typname='varchar' then '(' || cast(a.atttypmod-4 as text) || ')'

when t.typname='numeric' then '(' || cast((a.atttypmod-4)/65536 as text) || ',' || cast((a.atttypmod-4)%65536 as text) || ')'

else end ||case when a.attnotnull then ' not null' else end ||case when a.atthasdef then ' default '|| d.adsrc else end as DDL

FROMpg_type t, pg_class c, pg_attribute a LEFT OUTER JOIN pg_attrdef d ON

d.adrelid=a.attrelid and d.adnum=a.attnumWHERE

a.attnum > 0 ANDa.attrelid = c.oid ANDt.oid = a.atttypid ANDc.relname = vstrTable

ORDER BYa.attnum

LOOPIF vstrFields <> THEN

vstrFields = vstrFields || ',' || chr(10) || ' ';ELSE

vstrFields = vstrFields || chr(10) || ' ';END IF;vstrFields = vstrFields || vrecFld.DDL;

END LOOP;

vstrDDL = vstrDDL || vstrFields || chr(10) || ')';if vstrTableSpace <> thenvstrDDL = vstrDDL || ' TABLESPACE ' || vstrTableSpace;

end if;vstrDDL = vstrDDL || ';' || chr(10);

return vstrDDL;END;$$ LANGUAGE plpgsql;

-- ---------------------------------------------------------------------------- Generate the CREATE INDEX DDL for a table-- --------------------------------------------------------------------------create or replace function GenIndexDDL(text) returns text as $$declarevstrTable text;vrecIndex RECORD;vstrDDL text;viFld integer;vstrFld text;


vstrDDL = ;

FOR vrecIndex INSELECT

i.indkey, i.indisunique,pg_catalog.quote_ident(c.relname) as tablename,

64 | © Neolane 2013

Neolane

pg_catalog.quote_ident(ic.relname) as indexname,pg_catalog.quote_ident(t.spcname) as tablespace

FROMpg_class c, pg_index i, pg_class ic left outer join pg_tablespace t on

ic.reltablespace = t.oidWHERE

i.indexrelid = ic.oid ANDi.indrelid = c.oid ANDc.relname = vstrTable

LOOP

vstrDDL = vstrDDL || 'CREATE ';if vrecIndex.indisunique then

vstrDDL = vstrDDL || 'UNIQUE ';end if;vstrDDL = vstrDDL ||

'INDEX ' ||vrecIndex.indexname || '_tmp ON ' ||vrecIndex.tablename || '_tmp(';

FOR viFld IN array_lower(vrecIndex.indkey, 1) .. array_upper(vrecIndex.indkey, 1) LOOP

SELECT pg_catalog.quote_ident(a.attname) INTO vstrFld FROM

pg_attribute a, pg_class cWHERE

a.attnum = vrecIndex.indkey[viFld] ANDa.attrelid = c.oid AND c.relname=vstrTable;

vstrDDL = vstrDDL || vstrFld;if viFld <> array_upper(vrecIndex.indkey, 1) then

vstrDDL = vstrDDL || ', ';end if;

END LOOP;vstrDDL = vstrDDL || ')';

if vrecIndex.tablespace <> thenvstrDDL = vstrDDL || 'TABLESPACE ' || vrecIndex.tablespace;

end if;vstrDDL = vstrDDL || ';' || chr(10);

END LOOP;


-- ---------------------------------------------------------------------------- Generate the ALTER INDEX RENAME for a table-- --------------------------------------------------------------------------create or replace function GenRenameIndexDDL(text) returns text as $$declarevstrTable text;vrecIndex RECORD;vstrDDL text;


vstrDDL = ;

FOR vrecIndex INSELECT

pg_catalog.quote_ident(n.nspname) as namespace,pg_catalog.quote_ident(ic.relname) as indexname

FROMpg_namespace n, pg_class c, pg_index i, pg_class ic

WHEREi.indexrelid = ic.oid ANDn.oid = ic.relnamespace ANDi.indrelid = c.oid ANDc.relname = vstrTable

LOOP



vstrDDL = vstrDDL || 'ALTER INDEX ' || vrecIndex.namespace || '.' || vrecIndex.indexname||

'_tmp RENAME TO ' || vrecIndex.indexname ||';' || chr(10);

END LOOP;


-- ---------------------------------------------------------------------------- Build a copy of a table, with index-- --------------------------------------------------------------------------create or replace function GenRebuildTablePart1(text) returns text as $$declarevstrTable text;vstrTmp text;vstrDDL text;


vstrDDL = ;

SELECT GenTableDDL(vstrTable) INTO vstrTmp;vstrDDL = vstrDDL|| vstrTmp || chr(10);

vstrDDL = vstrDDL|| 'INSERT INTO ' || vstrTable || '_tmp SELECT * FROM ' || vstrTable || ';'||chr(10);

SELECT GenIndexDDL(vstrTable) INTO vstrTmp;

vstrDDL = vstrDDL|| vstrTmp || chr(10);vstrDDL = vstrDDL|| 'VACUUM ANALYSE '|| vstrTable || '_tmp;' ||chr(10);

return vstrDDL;end;$$ LANGUAGE plpgsql;

-- ---------------------------------------------------------------------------- Drop the original table and rename the copy-- --------------------------------------------------------------------------create or replace function GenRebuildTablePart2(text) returns text as $$declarevstrTable text;vstrTmp text;vstrDDL text;


vstrDDL = 'DROP TABLE ' || vstrTable||';'|| chr(10);vstrDDL = vstrDDL|| 'ALTER TABLE ' || vstrTable || '_tmp RENAME TO ' || vstrTable ||';'||

chr(10);

SELECT GenRenameIndexDDL(vstrTable) INTO vstrTmp;vstrDDL = vstrDDL|| vstrTmp || chr(10);

return vstrDDL;end;$$ LANGUAGE plpgsql;

The following example can be used in a workflow to rebuild the required tables rather than using thevacuum/rebuild command:

function sqlGetMemo(strSql) { var res = sqlSelect("s, m:memo", strSql); return res.s.m.toString(); }

function RebuildTable(strTable)

66 | © Neolane 2013

Neolane

{ // Rebuild a table_tmp var strSql = sqlGetMemo("select GenRebuildTablePart1('"+strTable+"')"); logInfo("Rebuilding table '"+strTable+"'..."); // logInfo(strSql); sqlExec(strSql);

// If fails, there is an exception thrown and so we do not delete the original table strSql = sqlGetMemo("select GenRebuildTablePart2('"+strTable+"')"); logInfo("Swapping table '"+strTable+"'..."); //logInfo(strSql); sqlExec(strSql); }

RebuildTable('nmsrecipient'); RebuildTable('nmsrcpgrlrel'); // ... other tables here

Oracle

Compaction of tables and indexesThis can be performed using the shrink space option available with the alter table/index command.Start by enabling the row movement option of each table as shown below:

alter table nmsdelivery enable row movement; alter table nmsdeliverystat enable row movement; alter table xtkworkflow enable row movement; alter table xtkworkflowevent enable row movement; alter table xtkworkflowjob enable row movement; alter table xtkworkflowlog enable row movement; alter table xtkworkflowtask enable row movement; alter table xtkjoblog enable row movement; alter table xtkjob enable row movement; alter table nmsaddress enable row movement; alter table nmsdeliverypart enable row movement; alter table nmsmirrorpageinfo enable row movement;

All versions from Neolane 5.10 up, versions 6 included

alter table nmsbroadlogrcp enable row movement; alter table nmsbroadlogmid enable row movement; alter table nmsbroadlogmsg enable row movement; alter table nmsemailerrorstat enable row movement;

Note:

If the enable row movement command fails, you must kill all Oracle sessions using the table. Log in assystem and run the following command:

As system: alter system kill session 'sid,serial#';

You can then compact the tables as shown below:

alter table nmsdelivery shrink space compact; alter table nmsdelivery shrink space;

alter table nmsdeliverystat shrink space compact; alter table nmsdeliverystat shrink space;

alter table xtkworkflow shrink space compact; alter table xtkworkflow shrink space;

alter table xtkworkflowevent shrink space compact; alter table xtkworkflowevent shrink space;

alter table xtkworkflowjob shrink space compact; alter table xtkworkflowjob shrink space;

alter table xtkworkflowlog shrink space compact;



alter table xtkworkflowlog shrink space;

alter table xtkworkflowtask shrink space compact; alter table xtkworkflowtask shrink space;

alter table xtkjoblog shrink space compact; alter table xtkjoblog shrink space;

alter table nmsaddress shrink space compact; alter table nmsaddress shrink space;

alter table nmsmirrorpageinfo shrink space compact; alter table nmsmirrorpageinfo shrink space;

alter table nmsdeliverypart shrink space compact; alter table nmsdeliverypart shrink space;

All versions from Neolane 5.10 up, versions 6 included

alter table nmsbroadlogrcp shrink space compact; alter table nmsbroadlogrcp shrink space; alter table nmsbroadlogmid shrink space compact; alter table nmsbroadlogmid shrink space; alter table nmsbroadlogmsg shrink space compact; alter table nmsbroadlogmsg shrink space; alter table nmsemailerrorstat shrink space compact; alter table nmsemailerrorstat shrink space;

Do the same for indexes as shown below (please adapt the list to suit the needs of your data model):

alter index NMSBROADLOGRCP_LASTMODIFIED shrink space compact; alter index NMSBROADLOGRCP_LASTMODIFIED shrink space; ...

Note:

If you are using an enterprise version of Oracle, an alternative is to use the alter index xxx rebuildonline command to rebuild indexes.

DB2Please contact your database administrator to find out about the procedures best suited to your version ofDB2.

SQL Server

Note:

For the SQL Server, you can use the maintenance plan detailed here:http://ola.hallengren.com/sql-server-index-and-statistics-maintenance.html.

The example shown here concerns SQL Server 2005. If you are using SQL Server 2008, please contact yourdatabase administrator to find out about maintenance procedures for this version.If you are using Microsoft SQL Server 2005, most maintenance task can be carried out thanks to the wizard.

1 First, connect to Microsoft SQL Server Management Studio, with a login with administrator rights.

2 Go to the Management>Maintenance Plans folder, right-click on it and choose Maintenance PlanWizard

68 | © Neolane 2013

Neolane

http://ola.hallengren.com/sql-server-index-and-statistics-maintenance.html

3 Click Next when the first page comes up.

4 Select the type of maintenance plan you want to create (separate schedules for each task or singleschedule for the whole plan), then click the Change... button.



5 In the Job schedule properties window, select the desired execution settings and click OK, then clickNext.

6 Select the maintenance tasks you want to perform, then click Next.

Note:

We recommend performing at least the maintenance tasks shown below. You may also select the statisticsupdate task, although it is already carried out by the database cleanup workflow.

70 | © Neolane 2013

Neolane

7 In the drop-down list, select the database which you want to run the Database Check Integrity taskon.

8 Select the database and click OK, then click Next.



9 Configure the maximum size allocated to your database, then click Next.

Note:

If the size of the database exceeds this threshold, the maintenance plan will try to delete unused datato free up space.

10 Reorganize or rebuild the index:

n If the index fragmentation rate is between 10% and 40%, a reorganization is recommended.Choose which databases and objects (tables or views) you want to reorganize, then click Next.

72 | © Neolane 2013

Neolane

Note:

Depending on your configuration you can choose either the previously selected tables, or all tablesin your database.

n If the index fragmentation rate is higher than 40%, a rebuild is recommended.Select the options you want to apply to the index rebuild task, then click Next.

Note:

The rebuild index process is more constricting in terms of processor use and it locks the databaseresources. Tick the Keep index online while reindexing option if you want the index to be availableduring the rebuild.



11 Select the options you want to display in the activity report, then click Next.

12 Check the list of tasks configured for the maintenance plan, then click Finish.

74 | © Neolane 2013

Neolane

A summary of the maintenance plan and the statuses of its various steps is displayed.

13 Once the maintenance plan is complete, click Close.

14 In the SQL Server explorer, double-click the Management>Maintenance Plans folder.



15 Select the Neolane maintenance plan: the various steps are detailed in a workflow.

Note that an object has been created in the SQL Server Agent>Jobs folder. This object lets you startthe maintenance plan. In our example there is only one object since all the maintenance tasks are partof the same plan.

Warning:

For this object to run, the SQL Server agent must be enabled.

MySQL

Using the OPTIMIZE TABLE commandThe OPTIMIZE TABLE command can be used with MySQL 5.0 and 5.1. However, when used on InnoDbtables (which is the case with Neolane), it has a limited impact:

n it only updates index statistics and frees unused space in the index cluster; it does not rebuild the indexesor the table (for more on this, refer to http://dev.mysql.com/doc/refman/5.0/en/optimize-table.html).

n this command will lock the table, and may therefore be performed during system activity. This meansthat maintenance operations must be performed either during low activity, or with the Neolane systemoffline (with the associated risk of operations not completing).

76 | © Neolane 2013

Neolane

http://dev.mysql.com/doc/refman/5.0/en/optimize-table.html

So a first level of maintenance is to run the OPTIMIZE TABLE command on the transactional tablesevery other week to start with, and increasing the frequency to every week for heavily used databases,as shown below:

OPTIMIZE TABLE NmsDelivery;OPTIMIZE TABLE NmsDeliveryStat;OPTIMIZE TABLE XtkWorkflow;OPTIMIZE TABLE XtkWorkflowEvent;OPTIMIZE TABLE XtkWorkflowJob;OPTIMIZE TABLE XtkWorkflowLog;OPTIMIZE TABLE XtkWorkflowTask;OPTIMIZE TABLE XtkJobLog;OPTIMIZE TABLE XtkJob;OPTIMIZE TABLE NmsBroadLogMsg;OPTIMIZE TABLE NmsEmailErrorStat;OPTIMIZE TABLE NmsAddress;OPTIMIZE TABLE NmsDeliveryPart;OPTIMIZE TABLE NmsMirrorPageInfo;

Defragmenting tablesA full table defragmentation can be achieved by running a dump/restore operation on the concerned tables.This is performed thanks to shell script commands as shown below (repeat for each table):

/etc/init.d/nlserver6 stop mysqldump database_name NmsDelivery > NmsDelivery.sql mysql database_name < NmsDelivery.sql ... /etc/init.d/nlserver6 start

Note:

Note that this operation must be performed while the Neolane platform is offline, otherwise there is a riskof data loss. For more information, refer to http://dev.mysql.com/doc/refman/5.0/en/rebuilding-tables.html.

The ALTER TABLE command can also be used:

ALTER TABLE NmsRecipient ENGINE=InnoDB; ALTER TABLE NmsDeliveryStat ENGINE=InnoDB; ALTER TABLE XtkWorkflow ENGINE=InnoDB; ALTER TABLE XtkWorkflowEvent ENGINE=InnoDB; ALTER TABLE XtkWorkflowJob ENGINE=InnoDB; ALTER TABLE XtkWorkflowLog ENGINE=InnoDB; ALTER TABLE XtkWorkflowTask ENGINE=InnoDB; ALTER TABLE XtkJobLog ENGINE=InnoDB; ALTER TABLE XtkJob ENGINE=InnoDB; ALTER TABLE NmsBroadLogMsg ENGINE=InnoDB; ALTER TABLE NmsEmailErrorStat ENGINE=InnoDB; ALTER TABLE NmsAddress ENGINE=InnoDB; ALTER TABLE NmsDeliveryPart ENGINE=InnoDB; ALTER TABLE NmsMirrorPageInfo ENGINE=InnoDB;

This command can be started with the system still running, but will lead to heavy locking and some operationsmay not complete. We recommend using it during low activity periods.



http://dev.mysql.com/doc/refman/5.0/en/rebuilding-tables.html


Neolane

Table of ContentsTemporary files . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Database performances . . . . . . . . . . . . . . . . . . . . . . . . 80

Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 80Platform configuration . . . . . . . . . . . . . . . . . . . . . . . 80Database maintenance . . . . . . . . . . . . . . . . . . . . . . . 80Specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Modules and frequent issues . . . . . . . . . . . . . . . . . . . . . . 81Log precision . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Forwarding of tracking logs . . . . . . . . . . . . . . . . . . . . . . . 83OpenOffice . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Email delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Workflow execution . . . . . . . . . . . . . . . . . . . . . . . . . 84Failure to connect . . . . . . . . . . . . . . . . . . . . . . . . . . 85Connection thresholds . . . . . . . . . . . . . . . . . . . . . . . . 86Dr Watson in Windows XP (client) . . . . . . . . . . . . . . . . . . . . 86Stack trace in Linux . . . . . . . . . . . . . . . . . . . . . . . . . 87Encoding of the Oracle database . . . . . . . . . . . . . . . . . . . . . 88Reactivating the console update request . . . . . . . . . . . . . . . . . . 88Lost password . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Abnormal JSP behavior . . . . . . . . . . . . . . . . . . . . . . . . 89

This section lists some common errors and explains how to solve them.

Temporary files

If error messages such as the following appear (particularly in delivery logs) when the system isput into production:Unable to rename file '/tmp/tmp0000.tmp' to/usr/local/neolane/nl6/bin/..//var/XXX/mta/86510470.xml ;(errno=18, Invalidcross-device link) (iRc=-52)

The cause is as follows:Neolane generates temporary files under /tmp, and then renames them to move them to/usr/local/neolane/nl6/var. This error occurs when both folders (/tmp and/usr/local/neolane/nl6/var, which is in fact a symbolic link to /var/nl6) correspond todifferent devices. The df command is used for verification.

Neolane v6.1 - Production Guide - Troubleshooting | 79

CHAPTER 6

Troubleshooting

To correct this problem, the temporary files must be generated in the same device as the destination. Forexample, by executing:

$ cd ~/nl6/var$ mkdir tmp$ vi ~/nl6/customer.sh

and then adding:

export TMPDIR=/usr/local/neolane/nl6/var/tmp

Database performances

Most performance issues are linked to database maintenance. Here are four main leads to help you find thecause of slow performance:

n Configuration,

n Installation and configuration of the Neolane platform,

n database maintenance,

n real-time diagnosis.

ConfigurationCheck that the initial Neolane platform configuration is still valid and if necessary, reassess your customer'sneeds in terms of deliverability or database size. We also recommend running a full hardware check (CPU,RAM, IO system).

Platform configurationInappropriate configuration may affect platform performance. We recommend that you check networkconfiguration, platform deliverability options as well as MTA configuration in the serverConf.xml file.


Database cleanup taskPlease make sure the database cleanup task is operational. To do this, view the log files to see if they containany errors.

Maintenance plansMake sure database maintenance is correctly scheduled and executed. To do this, please contact yourdatabase administrator to learn more about:

n their maintenance schedule,

n previously executed maintenance plans,

n view the script logs.

Warning:

If you are using a mid-sourcing configuration, it is essential for databases to be maintained on a regularbasis. When analyzing a delivery on the marketing platform, the marketing instance sends information tothe mid-sourcing instance. If the process is slowed down, the marketing instance will be impacted.

For more on this, refer to Database maintenance [page 57].

Managing work tablesPlease check the number and size of work tables. When they exceed a certain size, database performanceis affected. These tables are created by workflows and deliveries. They remain in the database while workflowsand deliveries are active. To limit the size of work tables, you can carry out the following operations:

80 | © Neolane 2013

Neolane

n stop or delete deliveries with the following statuses: Failed, In progress, Ready for delivery, orPaused.

n stop or delete workflows which are paused due to an error,

n stop all workflows used for tests which do not contain an End activity and whose status therefore remainsPaused.

Warning:

If the operation takes a long time and frees up a lot of space, this means that in-depth maintenance isnecessary (index rebuilding, etc.). For more on this, refer to Database maintenance [page 57].

Neolane process monitoringDepending on Neolane installation settings, two tools can be used for platform monitoring:

n the instance production page. For more on this, refer to Manual monitoring [page 15].

n the netreport script. For more on this, refer to Automatic monitoring via Neolane scripts [page 27].

SpecificsIt may become necessary to run a real-time diagnosis to identify the cause of the issue. Start by checkingthe process and platform log files, then monitor database activity while recreating the issue. Pay particularattention to the following:

n the maintenance execution plan,

n SQL queries being executed,

n whether or not external processes are running at the same time (cleansing, imports, aggregate calculation,etc.).

Modules and frequent issues

Here is a list of modules impacted by frequent issues:

TroubleshootingExecution scopeModule

The operator who scheduled this export needs to re-launch it. Either delta orfull re-launch.

Execution of an export processexport

The operator who scheduled this export needs to re-launch it. Check the databasefor duplicates.

Execution of an import processimport

Check this module if bounce mails are no longer forwarded.Reading of the bounce mail boxinMail

Check this module if mails are no longer being sent.Delivers emailsmta

Check this module if mails are no longer being sent.Maintains MTA connection statist-ics

stat

In case of issues, re-launch the module. If necessary, apply the process to in-crease log precision. For more on this, refer to Log precision [page 82].

Execution of a workflow instancerunwf

If there are no logs in the log file: check that the module uses the 6666 port.For more on this, refer to List of open ports [page 6].

Log writingsyslogd

Check this module if tracking logs aren't forwarded anymore.Consolidating and retrievingtracking logs

tracking

Check this module if tracking logs aren't forwarded anymore and there is notrace of logs in the files on the server. For more on this, refer to Forwarding oftracking logs [page 83].

Tracking log writing and purgingserver

trackinglogd

Check this module if no processes are starting.Startup and monitoring instancewatchdog

Check this module if the console and web connections don't work and triggeran xtk:session type error

Application server (HTTP andSOAP)

web


Troubleshooting

TroubleshootingExecution scopeModule

Check this module if workflows aren't executing.Workflow serverwfserver

Log precision

You can apply this process to all Neolane modules to increase log precision.It involves relaunching the processes with a higher level of logs.

Warning:

This procedure cancels the services in progress on this module.

Neolane can operate with two levels of log:

1 The Verbose mode is the first level after the standard level. The following command activates it:

nlserver restart <MODULE_NAME> -verbose

Check that the error actually occurred, and then restart the process in the normal way:

nlserver restart <MODULE_NAME> -noconsole

2 The TraceFilter mode, which lets you save the greatest number of logs. It is activated by the followingcommand:

nlserver stop <MODULE_NAME>; nlserver <MODULE_NAME> -verbose -tracefilter:*

Note:

If you use tracefilter:*, all log types are activated: ncm, rdr, nms, jst, timing, wdbc, ldap, soap, xtk,xtkquery, session, xtkwriter, network, pop3, inmailThe most useful log types are: wdbc (displays all SQL queries), soap (displays all SOAP calls), ldap(displays all LDAP queries after authentication), xtkquery (displays the list of all the querydef).You can use them individually (tracefilter:soap,wdbc for example). You can also activate them all andchoose to exclude certain others: -tracefilter:*,!soap

Check that the error actually occurred, and then restart the process in the normal way:

nlserver restart <MODULE_NAME> -noconsole

Warning:

The logs of these commands are stored in the log file of the module.

Here is an example specific to the Web module. The other modules operate as indicated above.Before sending this command, check that no job in progress will be affected.

nlserver pdump -who

Next, shut down and restart the module in TraceFilter mode.

nlserver stop web; LD_PRELOAD=libjsig.so nlserver web -tomcat -verbose -tracefilter:* -tracefile:web_debug@default

Another example:

nlserver stop mta@<INSTANCE_NAME>; nlserver mta -instance:<INSTANCE_NAME> -tracefilter:* -tracefile:mta_debug@<INSTANCE_NAME>

82 | © Neolane 2013

Neolane

Note:

The Tracefile mode lets you save the logs. In the examples above, the logs are saved in thevar/<INSTANCE-NAME>/mta_debug.log and var/default/web_debug.log files.

Warning:

In Windows, do not add the LD_PRELOAD option. The following command suffices:

nlserver web -tomcat -verbose -tracefilter:*

Check that the problem occurs again, and then restart the module:

nlserver restart web -tomcat -noconsole

All information is available in the file /usr/local/neolane/nl6/var/default/log/web.log.

Forwarding of tracking logs

There can be multiple reasons for tracking logs not being forwarded. We recommend that you check thefollowing information:

n Does the Tracking workflow have errors?

n Is the module trackinglogd running on the machine?

n Have changes been made? They can trigger a loss of connection to the servers using the tracking alias.


Troubleshooting

OpenOffice

If the error message seems to indicate that the connection to the OpenOffice.org server has failed, werecommend that you verify the following points:

n Check that OpenOffice is properly configured. To do this, launch it manually once, using an account withadministrator rights in Windows, or using the Neolane user account in Linux

n In Windows, check that the Neolane service can interact with the Desktop. To do this, open the list ofWindows services and edit the properties of the Neolane service, and then go to the Connection taband select the option Allow System to Interact with Desktop.For more on this, refer to the relevant section of the Installation guide.

Email delivery

If your email campaigns aren't being sent and their status is Pending, your MTA module might not be startedTo check this and to start the module if necessary, apply the following steps:

1 Check that your mta@<instance> modules are launched on your MTA servers.

nlserver pdump11:00:39 > Application server for Neolane Version 6.X (build XXXX) of DD/MM/YYYY[...]mta@<INSTANCENAME> (9268) - 23.0 Mb[...]

2 If the MTA is not listed, start it via the following command:

nlserver start mta@<INSTANCENAME>

Note:

Replace <INSTANCENAME> with the name of your instance (production, development, etc.). The instancename is identified via the configuration files:

[path of application]nl6/conf/config-<INSTANCENAME>.xml

Also check the configuration of traffic management (IPAffinity). For more on this, refer to the Installationguide.

Workflow execution

If workflows aren't executing and their status is Start in progress, this might mean that the workflowmodule isn't launched.To check this and to start the module if necessary, apply the following steps:

1 Check that your wfserver@<instance> modules are launched on your main application server.

nlserver pdump11:00:39 > Application server for Neolane Version 6.X (build XXXX) of DD/MM/YYYY[...]wfserver@<INSTANCENAME> (9340) - 11.3 Mb[...]

2 If the workflow server is not listed, start it using the following command:

nlserver start wfserver@<INSTANCENAME>

84 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#page=1

installation-v6.1-en.pdf#nameddest=ASN-Delivery_Parameters_opt

installation-v6.1-en.pdf#nameddest=ASN-Delivery_Parameters_opt

Note:

Replace <INSTANCENAME> with the name of your instance (production, development, etc.). The instancename is identified via the configuration files:

[path of application]nl6/conf/config-<INSTANCENAME>.xml

Failure to connect

The reasons for this can be multiple and depend on various contexts.Check the general configuration of the security zones.

Note:

For more on configuring security zones, refer to the Installation guide.

Check the following information:

1 Network checks

n Do you have Internet access from your computer?Check that you can connect to websites on the Internet (for example). If you cannot connect, theproblem is on your machine. Contact your system administrator.

n Can you connect to the server hosting Neolane via another service?Connect to the server via SSH or any other means. If this is not possible, your host company has aproblem. Contact their system administrator.

2 Checks on Web server side (IIS/apache/etc.)

n Does the Web server respond?Connect to the Neolane server access URL using a Web browser: http(s)://<URLSERVER. If itdoes not respond, the web server is stopped on the machine. Contact the system administrator ofyour host company in order to restart the service.

n Has Neolane been correctly integrated?Log on to the: http(s)://<URLSERVER>/r/test URL. The server should return the followingtype of message

<redir status='OK' date='YYYY/MM/DD HH:MM:SS' build='XXXX' host='support.neolane.net' localHost='server'/>

If you do not obtain this result, check in your Web server configuration that integration is taken intoaccount. For further information about this procedure, refer to the Installation Guide.

3 Checks on the Neolane side

n Has the Neolane Web module been launched?Connect to the following URL: http(s)://<URLSERVER/nl/jsp/logon.jsp

n If you obtain a Tomcat Java error:Is the JAVA integration correctly performed? Neolane requires a SUN JDK.It is integrated in the file [path of application]/nl6/customer.sh

n If you obtain a blank page:Has the Neolane Web module started up? You should obtain:

nlserver pdump11:00:39 > Application server for Neolane Version 6.X (build XXXX) dated DD/MM/YYYY[...]web@default (27515) - 55.2 Mb[...]

If not, restart it using the following command:


Troubleshooting

installation-v6.1-en.pdf#nameddest=csturges-36892

nlserver start web

Note:

If you obtain a response of the following type when you list the Neolane modules:nlserver pdump

11:00:39 > Application server for Neolane Version 6.X (build XXXX) dated DD/MM/YYYYNo tasks

You must restart the entire Neolane application. To do this, use the following commands:


n In Linux: /etc/init.d/nlserver6 start

n In Windows: net start nlserver6

n If not, then in the Neolane account:nlserver watchdog -svc -noconsole

Connection thresholds

For heavily loaded servers, the connection threshold might be exceeded. In any event, it is useful to find outwhy.There are three different thresholds:

1 The Web connection threshold, configured in your web server. To modify it, contact your systemadministrator.

2 The database connection threshold. To modify it, contact your database administrator.

3 The Neolane connection threshold, available in two places:

1 Tomcat side: all queries actually arriving on the Neolane Tomcat client.This threshold is configured in the nl6/tomcat-7/conf/server.xml file. The maxProcessorsattribute lets you increase the threshold of the number of queries processed at a time. It can bechanged to 250, for instance.

<Connector className="org.apache.coyote.tomcat7.CoyoteConnector" port="8080" minProcessors="5" maxProcessors="75"

enableLookups="true" redirectPort="8443"acceptCount="100" debug="0" connectionTimeout="20000"useURIValidationHack="false" disableUploadTimeout="true" />

</Connector>

2 Database: set of all connections open at the same time on the database.This threshold is configured in the file nl6/tomcat-4/conf/server.xml. The Pool attribute locatedin dataSource lets you increase the threshold of queries processed simultaneously. For example, itcan be increased to 90:

<pool aliveTestDelaySec="600" freeCnx="0" maxCnx="90" maxIdleDelaySec="1200"/

Dr Watson in Windows XP (client)

The Dr Watson tool can block the re-starting of Neolane in Windows: you need to disable thisTo do this:

1 Open a DOS command window via the following menu: Start>Run in Windows.

86 | © Neolane 2013

Neolane

2 Enter the following command: start Drwtsn32

3 Disable notifications:

Stack trace in Linux

A stack trace represents a trace contained in a core type file. This file is generated in the event of a machineerror. It can identify the origin of the error.

Note:

n A core file is named core.<NUM>.

n gdb - The GNU Debugger must be installed on the machine.

Neolane technical support can ask you for this stack trace. To obtain it, enter the following commands inLinux:

su - neolanegdb nlserver <coreFile//You get lots of output but the stack trace (or Back trace) can be obtained with : (gdb) bt// and forward all the output : #0 0x0836c189 in ObjectValue::SetPropertyTarget ()#1 0x082623b3 in CXtkScriptProperty::VDeclareProperties ()#2 0x0826a835 in CXtkScriptContext::OnGetProperty ()#3 0x08370b3d in JavaScriptGetProperty ()#4 0x557b8aa7 in js_Interpret () from /usr/local/neolane/nl6/lib/libmozjs.so#5 0x557afb97 in js_Execute () from /usr/local/neolane/nl6/lib/libmozjs.so#6 0x5578921f in JS_ExecuteScript () from /usr/local/neolane/nl6/lib/libmozjs.so#7 0x08370468 in JavaScriptSource::Evaluate ()#8 0x0848fa03 in JSTDeliveryContext::ProcessScript ()#9 0x0848fcb6 in JSTDeliveryContext::ProcessTemplate ()#10 0x08460d2b in CDeliveryToolbox::CreateMailMessage ()#11 0x080d51fe in CMtaQueue::PrepareMessages ()#12 0x080d2b07 in CMtaQueue::Prepare ()#13 0x080dca38 in CMtaChild::OnRun ()#14 0x0814792c in ThreadStartRoutine ()#15 0x55575b63 in start_thread () from /lib/tls/libpthread.so.0#16 0x5565918a in clone () from /lib/tls/libc.so.6

Neolane technical support might ask you to run this command using a specific executable (to be supplied byus).


Troubleshooting

In this case, simply run the following command:

gdb nlserver <coreFile>

by replacing nlserver with the executable provided by Neolane

For example:

gdb nlserver.1823 <coreFile>

Encoding of the Oracle database

Neolane uses the Oracle default encodings. If you use different ones, an incompatibility can occur. For moreon this, refer to the Installation guide.

Reactivating the console update request

If you selected the Do not request console update option and you wish to reactivate the update request,apply the following procedure:

1 Open the editor of the registry database using the regedit command in the Windows Start>Executemenu.

2 In the tree, display the options of the \HKEY_CURRENT_USER\Software\Neolane\NL_6\nlclientnode.

3 Delete the confAdvisedUpgrade entry and close the Registry editor.

88 | © Neolane 2013

Neolane

installation-v6.1-en.pdf#nameddest=ACY-20669

Lost password

You can change or recover a lost passwordThere are two possible scenarios:

n Password lost by a Neolane operator.In this case, you can change the password of the operator concerned. To do this, connect via an operatorwith administrator rights and replace the operator password.

n Internal password loss.If the internal password is lost, you must reinitialize it. To do this, apply the following procedure:

1 Edit the /usr/local/neolane/nl6/conf/serverConf.xml file.

2 Go to the internalPassword line.

<xtk internalPassword="myPassword"/>

3 Delete the string in quotes, in this case: myPassword

You thus obtain the following line:

!-- XTK authentication mode internalPassword : Password of internal account --><xtk internalPassword=""/

4 Save changes and close the file.

5 Configure the new password. To do this, enter the following commands:

nlserver config -internalpassword11:00:39 > Application server for Neolane Version 6.X (build XXXX) of dd/mm/yyyyEnter current password.Password: (empty)Enter the new password.Password: Confirmation

6 You can now use your new password to connect in Internal mode.

Abnormal JSP behavior

If certain jsp jobs are not successfully executed, you must force them to recompile.For this, enter the following commands:

nlserver stop webcd nl6/tomcat-7rm -r work/nlserver start web


Troubleshooting

The jsp jobs are regenerated the next time you connect.

90 | © Neolane 2013

Neolane

Documents

Neolane v6.1 - Production Guide - Adobe Campaign … Description job Command line processing merge Form merge midSourcing Recovery of delivery information in mid-sourcing mode monitor