SAP Event Stream Processor: SDK Guide - SAP Help Portal

PUBLIC

SAP Event Stream Processor 5.1 SP09Document Version: 1.1 – 2015-03-20

SAP Event Stream Processor: SDK Guide

Content

1 Changing Studio Local Cluster Credentials for External Access. . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Server Certificate Verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.1 Enabling Certificate Verification for the Java SDK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Enabling Certificate Verification for the C/C++ and .NET SDKs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 C SDK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1 C SDK: Entity Lifecycles and Access Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 C SDK: Required Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113.3 C SDK: Starting and Stopping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4 C SDK: Publishing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

C SDK: Publishing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.5 C SDK: Subscribing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

C SDK: Subscribing with Guaranteed Delivery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18C SDK: Subscribing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.6 C SDK: Failover Handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.7 C SDK: Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .223.8 C SDK: Versioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.9 C SDK: API Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4 Java SDK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .244.1 Java SDK: Entity Lifecycles and Access Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.2 Java SDK: Required Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.3 Java SDK: Starting and Stopping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .284.4 Java SDK: Publishing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Java SDK: Publishing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.5 Java SDK: Subscribing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Java SDK: Subscribing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Java SDK: Subscribing with Guaranteed Delivery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.6 Java SDK: Failover Handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.7 Java SDK: Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.8 Java SDK: Versioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.9 Java SDK: API Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5 .NET SDK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.1 .NET SDK: Entity Lifecycles and Access Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

.NET SDK: Starting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

.NET SDK: Connecting to a Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2P U B L I C© 2015 SAP SE or an SAP affiliate company. All rights reserved.

SAP Event Stream Processor: SDK GuideContent

.NET SDK: Getting and Connecting to a Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.2 .NET SDK: Required Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .425.3 .NET SDK: Publishing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44

.NET SDK: Publishing in Direct Access Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

.NET SDK: Publishing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465.4 .NET SDK: Subscribing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

.NET SDK: Subscribing to a Stream in Callback Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

.NET SDK: Subscribing Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50

.NET SDK: Subscribing with Guaranteed Delivery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.5 .NET SDK: Stopping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.6 .NET SDK: Failover Handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.7 .NET SDK: Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.8 .NET SDK: API Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.9 .NET SDK: Versioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

SAP Event Stream Processor: SDK GuideContent

P U B L I C© 2015 SAP SE or an SAP affiliate company. All rights reserved. 3

1 Changing Studio Local Cluster Credentials for External Access

Before connecting to the SAP® Event Stream Processor Studio local cluster with a component from outside of Studio (such as command line tools, external adapters, or custom applications), you will first have to change the system-generated local cluster password.

Procedure

1. In Studio, open the SAP ESP Run-Test perspective.2. In the Server View, stop any projects running in the local cluster.3. Right-click the local cluster, and select Disconnect and Stop Local Server.4. Right-click the local cluster, and select Change Password.5. Enter the new values for the local cluster password.

Ensure that you make note of the new values as you will be prompted to enter them every time you connect to the local cluster with a component from outside of Studio.

6. Right-click the local cluster, and select Connect Server.


SAP Event Stream Processor: SDK GuideChanging Studio Local Cluster Credentials for External Access

2 Server Certificate Verification

For security reasons, when you are using Secure Socket Layers (SSL) communication, SAP® recommends enabling the SDK clients to verify the certificate authenticity and host name information of any SAP® Event Stream Processor server to which the SDK clients connect. Before enabling the SDKs for certificate verification, ensure that SSL is enabled on the server side, required SSL keys are generated, and appropriate root certificates are available to the client.

● First, generate an SSL key in the Java KeyStore. See Generating an SSL Key in the Java KeyStore in the SAP Event Stream Processor: Security Guide for detailed instructions.

● Then, convert this SSL key to generate a PEM formatted server certificate and key. See Generating PEM Format Private Keys in the SAP Event Stream Processor: Security Guide for detailed instructions.

● Finally, extract the root certificate for your client and make sure it is available to the client. See Generating a TrustStore for a Client in the SAP Event Stream Processor: Security Guide for detailed instructions.

2.1 Enabling Certificate Verification for the Java SDK

If your Java SDK client connects to the SAP Event Stream Processor server using SSL communication, set the javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword properties to ensure the client automatically verifies the server certificate before connecting.

Prerequisites

● Enable SSL on the server side.● Generate an SSL key in the Java KeyStore and convert it into PEM format.● Have a Java TrustStore which contains an extracted root certificate.

See Server Certificate Verification [page 5] for additional details.

Procedure

● To set these properties from the command line:

java -Djavax.net.ssl.trustStore=c:\truststore.jks -D javax.net.ssl.trustStorePassword=<trustStore password> com.test.JavaClient

where <trustStore password> is the password you used to create the TrustStore for the Java SDK client.

SAP Event Stream Processor: SDK GuideServer Certificate Verification


● To set these properties programatically, do the following in your Java SDK client program before connecting to the SAP Event Stream Processor server:

System.setProperty(“javax.net.ssl.trustStore” “c:/truststore.jks”); System.setProperty(“javax.net.ssl. trustStorePassword” “<trustStore Password>”);

where <trustStore password> is the password you used to create the TrustStore for the Java SDK client.

2.2 Enabling Certificate Verification for the C/C++ and .NET SDKs

If your C/C++ or .NET SDK client connects to the SAP Event Stream Processor server using SSL communication, set the STREAM_CACERT environment variable to ensure the client automatically verifies the server certificate before connecting.

Prerequisites

● Enable SSL on the server side.● Generate an SSL key in the Java KeyStore and convert it into PEM format.● Have a PEM formatted root certificate available to the client.

See Server Certificate Verification [page 5] for additional details.

Context

Set the STREAM_CACERT environment variable to the location of the cert.pem file:

Procedure

● You can set this from the command line and then run your client:a. (Windows) If cert.pem is located in the c:\certs\ folder: c:>set STREAM_CACERT=c:\certs

\cert.pem b. (Linux) If cert.pem is located in the /tmp/certs/ folder: export STREAM_CACERT=/tmp/certs/

cert.pem ● To set the environment variable programatically:

a. For a C/C++ client, run:

setenv(“STREAM_CACERT”, “c:/certs/cert.pem”);



b. For a .NET SDK client, run:

Environment.SetEnvironmentVariable(“STREAM_CACERT”, “c:/certs/cert.pem”);



3 C SDK

Information on starting and stopping, and subscribing and publishing using the C SDK. Also includes references information and examples.

3.1 C SDK: Entity Lifecycles and Access Modes

In the SAP® Event Stream Processor C SDK, all entities exposed by the SDK have a common life cycle and multiple access modes.

User interaction in the Event Stream Processor SDK is handled through entities the SDK exposes. The main entities are Server, Project, Publisher, and Subscriber. These entities correspond to the functional areas of the SDK. The Server object represents a running instance of a cluster, the Project corresponds to a single project deployed to the cluster, the Publisher object deals with publishing data to a running project, and the Subscriber object subscribes to data streams.

On initial retrieval, an entity is considered to be open. When an entity is open, you can retrieve certain static information about it. To accomplish its assigned tasks, an entity has to connect to the corresponding component in the cluster. A server connects to a running instance of a cluster, and EspProject, EspPublisher, and EspSubscriber all connect to running instances of a project in a cluster.

In the connected state, an entity can interact with the cluster components. Once an entity is disconnected, it can no longer interact with the cluster but is still an active object in the SDK, and can be reconnected to the cluster. Once an entity is closed, it is no longer available for interaction and is reclaimed by the SDK. To reuse an entity that has closed, retrieve a fresh copy of the entity.

For example, you can retrieve a Project object and connect it to a project in the cluster. If the back-end project dies, the SDK Project receives a disconnected event. You can attempt to reconnect manually, or, if you are using callback mode and your configuration supports it, the SDK tries to reconnect automatically. Upon successful reconnection, the SDK generates a connected event. If you actively close the entity, it disconnects from the back-end project and the SDK reclaims the Project object. To reconnect, you first need to retrieve a new Project object.

The SDK provides great flexibility in structuring access to the entities exposed by the API. There are three modes that can be used to access entities: direct, callback, and select.

Direct access is the default mode when retrieving an entity. In this mode, all operations on an entity return when an error occurs or the operation completes successfully. There are no events generated later, so there is no need to have an associated event handler.

In callback access, an event handler must be associated with the request. Most calls to the entity return immediately, but completion of the request is indicated by the generation of the corresponding event. The SDK has two internal threads to implement the callback mechanism. The update thread monitors all entities currently registered for callbacks for applicable updates. If an update is found, an appropriate event is created and queued to the dispatch thread. The dispatch thread calls the registered handlers for the user code to process them.

You can register multiple callbacks on each publisher or subscriber by calling ESPAPICALL int32_t esp_publisher_set_callback(EspPublisher * publisher, uint32_t events,


SAP Event Stream Processor: SDK GuideC SDK

PUBLISHER_CALLBACK_T callback, void * user_data, EspError * error); or ESPAPICALL int32_t esp_subscriber_set_callback(EspSubscriber * subscriber, uint32_t events, SUBSCRIBER_CALLBACK_T callback, void * user_data, EspError * error); multiple times. Each registered handler gets the same events.

The following example shows how an EspProject could be accessed in callback mode. If you are working in callback mode and want to receive the callback events, register your callback handlers before you call connect on the entity you are interested in:

EspProjectOptions * options = esp_project_options_create(error); int rc = esp_project_options_set_access_mode(options, CALLBACK_ACCESS, error); const char * temp = "esp://host.domain.com/workspace/project"; EspUri * uri = esp_uri_create_string(temp, error); // Create credentials to authenticate with project. Assume cluster is setup to use user password authentication EspCredentials * creds = esp_credentials_create(ESP_CREDENTIALS_USER_PASSWORD, error); esp_credentials_set_user(creds, “user”, error); esp_credentials_set_password(creds, “password”, error); EspProject * project = esp_project_get(uri, creds, options, error); // If you are not going to reuse the credentials, you need to free them esp_credentials_free(creds, error); rc = esp_project_set_callback(project, ESP_PROJECT_EVENT_ALL, project_callback, NULL, error); rc = esp_project_connect(project, error); // The callback handler void project_callback(const EspProjectEvent * event, void * data) { EspProject * project = NULL; const EspError * error = NULL; int rc; uint32_t type; rc = esp_project_event_get_type(event, &type, NULL); switch (type) { case ESP_PROJECT_EVENT_CONNECTED: project = esp_project_event_get_project(event, NULL); break; case ESP_PROJECT_EVENT_DISCONNECTED: project = esp_project_event_get_project(event, NULL); esp_project_close(project, NULL); // you can call close inside a callback break; case ESP_PROJECT_EVENT_CLOSED: case ESP_PROJECT_EVENT_STALE: case ESP_PROJECT_EVENT_UPTODATE: break; case ESP_PROJECT_EVENT_ERROR: error = esp_project_event_get_error(event, NULL); break; } }

The select access mode lets you multiplex various entities in a single thread—somewhat similar to the select and poll mechanisms available on many systems—to monitor file descriptors. An entity is registered with an EspSelector together with the events to monitor for. Then, call the esp_selector_select(...) method, which blocks until a monitored update occurs in the background. The function returns a list of EspEvent objects. First determine the category (server, project, publisher, subscriber) of the event, then handle the appropriate event type. In select mode, the SDK uses one background update thread to monitor for updates. If detected, the appropriate event is created and pushed to the EspSelector. The event is then handled in your own thread.



The following example uses a single selector to multiplex different entities.

// Assuming the EspServer, EspProject, EspPublisher, EspSubscriber have been created with the correct options // Not doing error checking, etc for clarity EspSelector * selector = esp_selector_create("server-select", error); rc = esp_server_select_with(server, selector, ESP_SERVER_EVENT_ALL, error); EspList * list = esp_list_create(ESP_LIST_EVENT_T, error); rc = esp_server_connect(m_server, error); uint32_t type; const void * ev; int c; int done = 0; while (!done) { esp_list_clear(list, error); rc = esp_selector_select(selector, list, error); c = esp_list_get_count(list, error); for (int i = 0; i < c; i++) { ev = esp_list_get_event(list, i, error); int cat = esp_event_get_category(ev, error); switch ( cat ) { case ESP_EVENT_SERVER: srvevent = (EspServerEvent*) ev; esp_server_event_get_type(srvevent, &type, error); switch (type) { // process server events case ESP_SERVER_EVENT_CONNECTED: break; // ..... } default: break; case ESP_EVENT_PROJECT: prjevent = (EspProjectEvent*) ev; esp_project_event_get_type(prjevent, &type, error); switch (type) { // process project events case ESP_PROJECT_EVENT_CONNECTED: break; } case ESP_EVENT_PUBLISHER: { pubevent = (EspPublisherEvent*) ev; esp_publisher_event_get_type(pubevent, &type, error); switch (type) { case ESP_PUBLISHER_EVENT_CONNECTED: break; } } break; case ESP_EVENT_SUBSCRIBER: { subevent = (EspSubscriberEvent*) ev; esp_subscriber_event_get_type(subevent, &type, error); switch (type) { case ESP_SUBSCRIBER_EVENT_CONNECTED: break; } break; } } } }



3.2 C SDK: Required Libraries

A list of the required libraries in order to run the C/C++ SDK.

All file paths are relative to the STREAMING_HOME directory.

Table 1: Required Libraries

Operating System Files

Linux, Solaris ● lib/libstreamingclient_lib.so● lib/libstreamingsystem_lib.so● lib/libgcc_s.so.1● lib/libicui18n.so.44● lib/libicuuc.so.44● lib/libicudata.so.44● lib/libsasl2.so.2● lib/libstdc++.so.6

Windows ● bin\streamingclient_lib.dll● bin\streamingsystem_lib.dll● bin\icudt44.dll● bin\icuin44.dll● bin\icuuc44.dll● bin\libeay32.dll● bin\msvcp100.dll● bin\msvcr100.dll● bin\ssleay32.dll● bin\xerces-c_2_6.dll

For 32bit SDK support:

● bin\x86x\streamingclient_lib.dll● bin\x86\streamingsystem_lib.dll● bin\86x\icudt44.dll● bin\86x\icuin44.dll● bin\86x\xerces-c_2_6.dll

NoteTo support legacy applications and legacy third-party libraries that are 32-bit, the 64-bit Windows installer includes 32-bit versions of the C/C++ SDK.



3.3 C SDK: Starting and Stopping

Start the C SDK before performing operations.

Initializing the C SDK prompts it to start its internal threads and register required resources. This call can be made from any thread, but it must be made before any other SDK functionality is used.

Example:

int rc; EspError * error = esp_error_create();rc = esp_sdk_start(error);if (rc) { }

Once the application using C SDK is ready to exit or its functionality is no longer needed, stop the C SDK. This stops its internal threads and releases any held resources.

Example:

rc = esp_sdk_stop(error); if (rc) { }

Multiple SDK start calls may be made. The C SDK requires corresponding number of stop calls to properly shutdown.

3.4 C SDK: Publishing

The SDK provides several options for publishing data to a project.

The steps involved in publishing data are:

1. Create an EspPublisher for the project to publish to. You can create an EspPublisher directly or from a previously retrieved and connected EspProject object.

2. Create an EspMessageWriter for the stream to publish to. You can create multiple EspMessageWriters from a single EspPublisher.

3. Create an EspRelativeRowWriter.4. Format the data buffer to publish using EspRelativeRowWriter methods.5. Publish the data.

While EspPublisher is thread-safe, EspMessageWriter and EspRelativeRowWriter are not. Therefore, ensure that you synchronize access to the latter two.

The SDK provides a number of options to tune the behavior of an EspPublisher. Specify these options using EspPublisherOptions when creating the EspPublisher. Once created, options cannot be changed. Like all other entities in the SDK, publishing also supports the direct, callback, and select access modes.

In addition to access modes, the SDK supports internal buffering. When publishing is buffered, the data is first written to an internal queue. This is picked up by a publishing thread and then written to the ESP project. Buffering is possible only in direct access mode. Direct and buffered publishing potentially provides the best throughput.



Two other settings influence publishing: batching mode and sync mode. Batching controls how data rows are written to the socket. They can be written individually or grouped together in either envelope or transaction batches. Envelopes group individual rows together to send to the ESP project and are read together from the socket by the project. This improves network throughput. Transaction batches, like envelope batches, are also written and read in groups. However, with transaction batches, the platform only processes the group if all the rows in the batch are processed successfully. If one fails, the whole batch is rolled back.

NoteWhen using shine-through to preserve previous values for data that are null in an update record, publish rows individually or in envelopes, rather than in transaction batches.

When publishing is buffered, you can specify how the SDK batches rows in EspPublisherOptions. EXPLICIT_BLOCKING lets you control the batches by using start transaction and end block calls. AUTO_BLOCKING ignores these calls and batches rows internally. The default mode is NO_BLOCKING.

Sync mode settings control the publishing handshake between the SDK and the ESP project. By default, the SDK keeps sending data to the ESP project without waiting for acknowledgement. If sync mode is set to true, the SDK waits for acknowledgement from the ESP project before sending the next batch of data. This provides an application level delivery guarantee, but it reduces throughput.

There are certain considerations to keep in mind when using callback or select mode publishing. These modes are driven by the ESP_PUBLISHER_EVENT_READY event, which indicates that the publisher is ready to accept more data. In response, you can publish data or issue a commit, but only one such action is permitted in response to a single ESP_PUBLISHER_EVENT_READY event.

Publishing in async mode improves throughput, but does not provide an application level delivery guarantee. Since TCP does not provide an application level delivery guarantee either, data in the TCP buffer could be lost when a client exits. Therefore, a commit must be executed before a client exit when publishing in async mode.

In general terms, the return code from a Publish call indicates whether or not the row was successfully transmitted. Any error that occurs during processing on the ESP project (such as a duplicate insert) will not get returned. The precise meaning of the return code from a Publish call depends on the access mode and the choice of synchronous or asynchronous transmission.

When consistent recovery mode is enabled, a project treats commits issued by publishers as a system wide checkpoint request. When the publisher issues commits when consistent recovery is disabled, the project only syncs to disk input windows that are assigned to log stores and does not perform a full checkpoint. In both cases, when the commit call returns from the project, the publisher notifies its source that the data in question has been processed. With respect to system performance, frequent commits have a negative impact in both cases. The impact is more prominent when consistent recovery is enabled because a full checkpoint operation is more expensive compared to a log store sync operation.

NoteIn both consistent recovery and non-consistent recovery modes, all recoverable windows are recovered to the last checkpointed state. Additionally, in the case of non-consistent recovery mode, events in input windows that are only synced to disk but not yet checkpointed are also recovered. See Data Recovery in the SAP Event Stream Processor: Developer Guide for additional information on data recovery

When using callback or select access mode, the return only indicates whether or not the SDK was able to queue the data. The indication of whether or not the data was actually written to the socket will be returned by the appropriate event. The callback and select access modes do not currently support synchronous publishing.



When using direct access mode, the type of transmission used determines what the return from the Publish call indicates. If publishing in asynchronous mode, the return only indicates that the SDK has written the data to the socket. If publishing in synchronous mode, the return from the Publish call indicates the response code the ESP project sent.

Like all entities, if you intend to work in callback mode with a Publisher and want to get notified, register the callback handler before the event is triggered. For example:

esp_publisher_options_set_access_mode(options, CALLBACK_ACCESS, error); esp_publisher_set_callback(publisher, events, callback, NULL, error); esp_publisher_connect(publisher, error);

The following code snippets show different ways of publishing data. The sample code provided here is for illustration only; it does not comprise a complete, working example.

The first example shows publishing in direct access mode with transaction blocks.

EspCredentials * creds = esp_credentials_create(ESP_CREDENTIALS_USER_PASSWORD, error); esp_credentials_set_user(creds, “user”, error); esp_credentials_set_password(creds, “password”, error); // Create publisher with default options from an existing EspProject publisher = esp_project_create_publisher(project, creds, error); esp_credentials_free(creds, error); // Connect the publisher int rc = esp_publisher_connect(publisher, error); // Retrieve the EspStream we want to publish to const EspStream * stream = esp_project_get_stream(project, "Stream1", error); // Determine its schema const EspSchema * schema = esp_stream_get_schema(stream, error); // Create EspMessageWriter to publish to "Stream1" EspMessageWriter * writer = esp_publisher_get_writer(publisher, stream, error); EspRelativeRowWriter * row_writer = esp_message_writer_get_relative_rowwriter(writer, error); // Number of columns in "Stream1" int32_t numcols; esp_schema_get_numcolumns(schema, &numcols, error); int32_t intvalue = 10; bool inblock = false; // Your logic to determine how long to publish while (....) { // Your logic to determine whether to start a transaction if (!inblock) { esp_message_writer_start_transaction(writer, 0, NULL); inblock = true; } // Start a data row esp_relative_rowwriter_start_row(row_writer, NULL); int32_t coltype; for (int i = 0; i < numcols; ++i) { esp_schema_get_column_type(schema, i, &coltype, error); switch (coltype) { case ESP_DATATYPE_INTEGER: esp_relative_rowwriter_set_integer(row_writer, intvalue++, error); break; // ... // Code to fill in other data types goes here // ...



// NOTE - you must fill in all data fields, with NULLs needed default: esp_relative_rowwriter_set_null(row_writer, error); break; } } // End the data row esp_relative_rowwriter_end_row(row_writer, error); // Determine if the batch is to be ended, we code for 60 rows per block if ((nrows % 60) == 0) { // End the batch started in esp_message_writer_start_transaction() esp_message_writer_end_block(writer, error); // Publish the batch esp_publisher_publish(publisher, writer, error); inblock = false; } } // Done with publishing esp_publisher_close(publisher, error);

This example shows publishing in callback access mode.

// Create EspPublisherOptions int rc; EspPublisherOptions * options = esp_publisher_options_create(error); // Set access mode rc = esp_publisher_options_set_access_mode(options, CALLBACK_ACCESS, error); // Create EspPublisher using the options above from existing EspProject publisher = esp_project_create_publisher(project, options, error); // Free EspPublisherOptions esp_publisher_options_free(options, error); // Set callback handler rc = esp_publisher_set_callback(publisher, ESP_PUBLISHER_EVENT_ALL, publish_callback, NULL, m_error); // Connect publisher rc = esp_publisher_connect(publisher, error); ... ... ... // Handler function void publish_callback(const EspPublisherEvent * event, void * user_data) { EspPublisher * publisher = NULL; EspMessageWriter * mwriter = NULL; EspRelativeRowWriter * row_writer = NULL; EspProject * project = NULL; const EspStream * stream = NULL; const EspSchema * schema = NULL; EspError * error = esp_error_create(); int rc; uint32_t type; publisher = esp_publisher_event_get_publisher(event, error); rc = esp_publisher_event_get_type(event, &type, error); switch (type) { case ESP_PUBLISHER_EVENT_CONNECTED: // EspProject, EspStream, EspSchema can be retrieved from the EspPublisherEvent // if required



project = esp_publisher_get_project(publisher, error); stream = esp_project_get_stream(project, "Stream1", error); schema = esp_stream_get_schema(stream, error); break; case ESP_PUBLISHER_EVENT_READY: // Populate EspMessageWriter with data to publish rc = esp_publisher_publish(publisher, mwriter, error); break; case ESP_PUBLISHER_EVENT_DISCONNECTED: esp_publisher_close(publisher, error); break; case ESP_PUBLISHER_EVENT_CLOSED: break; } if (error) esp_error_free(error); }

3.4.1 C SDK: Publishing Examples

The previous sample code on publishing is provided for illustration purposes, but does not comprise a full, working example.

SAP Event Stream Processor ships with fully functioning examples you can use as a starting point for your own projects. Examples for publishing are located in:

%STREAMING_HOME%\examples\cpp\SDK (Windows)

$STREAMING_HOME/examples/cpp/SDK (Linux and Solaris)

3.5 C SDK: Subscribing

The SDK provides various options for subscribing to a project.

Subscribing to data using the SDK involves the following steps:

1. Create an EspSubscriber object. This can be created directly or retrieved from EspProject.2. Connect the EspSubscriber.3. Subscribe to one or more streams. Call esp_subscriber_subscribe(EspSubscriber * subscriber,

const EspStream * stream, EspError * error); for each stream you are connecting to.4. In direct access mode, retrieve events using esp_subscriber_get_next_event(). In callback and select

access modes, the event is generated by the SDK and passed back to user code.5. For data events, retrieve EspMessageReader. This encapsulates a single message from the ESP project. It

may consist of a single data row or a block with multiple data rows.6. Retrieve one or more EspRowReaders. Use the methods in EspRowReader to read in individual fields.

The sample code provided here is for illustration only; it does not comprise a complete, working example. This example shows subscribing to a stream using direct access mode with default options:

EspError * error = esp_error_create(); esp_sdk_start(error);



EspUri * project_uri = esp_uri_create_string("esp://server:port//default/vwap", error);EspCredentials * creds = esp_credentials_create(ESP_CREDENTIALS_USER_PASSWORD, error);esp_credentials_set_user(creds, “user”, error);esp_credentials_set_password(creds, “password”, error);EspProject * project = esp_project_get(project_uri, creds, NULL, error);rc = esp_project_connect(project, error);// Reusing credentials for the subscriberEspSubscriber * subscriber = esp_project_create_subscriber(project, creds, error);// Now free credentialsesp_credentials_free(creds, error);rc = esp_subscriber_connect(subscriber, error);EspStream * stream = esp_project_get_stream(project, "Trades", error);rc = esp_subscriber_subsribe(subscriber, stream, error);while (true) { EspSubscriberEvent * event = esp_subscriber_get_next_event(subscriber, error); // process event data // delete event esp_subscriber_event_free(event);}esp_subscriber_close(subscriber, error); esp_sdk_close();

If the event is an ESP_SUBSCRIBER_EVENT_DATA event, it contains field data. This is a typical example of reading data from a subscribe event:

// stream for this event const EspStream * stream = esp_subscriber_event_get_stream(event, error); // get message reader EspMessageReader * reader = esp_subscriber_event_get_reader(event, error); int rc = esp_message_reader_is_block(reader, &flag, error); // get the stream schema if you do not have it const EspSchema * schema = esp_stream_get_schema(stream, error); EspRowReader * row_reader; int32_t int_value; int numcolumns = 0, numrows = 0; int type; // need to know how many columns are there rc = esp_schema_get_numcolumns(schema, &numcolumns, error); // loop until we finish all rows while ((row_reader = esp_message_reader_next_row(reader, error)) != NULL) { for (int i = 0; i < numcolumns; ++i) { // if column is null, skip rc = esp_row_reader_is_null(row_reader, i, &flag, error); if ( flag ) continue; rc = esp_schema_get_column_type(schema, i, &type, error); switch ( type ) { case ESP_DATATYPE_INTEGER: rc = esp_row_reader_get_integer(row_reader, i, &int_value, error); break; case ESP_DATATYPE_LONG: rc = esp_row_reader_get_long(row_reader, i, &long_value, error); break; case ESP_DATATYPE_FLOAT: rc = esp_row_reader_get_float(row_reader, i, &double_value, error); // ... // other data types



// ... } } }

3.5.1 C SDK: Subscribing with Guaranteed Delivery

Use guaranteed delivery (GD) to ensure that events are still delivered to the subscriber if the connection is temporarily lost or the server is restarted.

Prerequisites

Enable guaranteed delivery in a window and attach a log store in the CCL. To receive checkpoint messages from the server on streams using GD with checkpoint, set the Auto Checkpoint parameter in the project configuration file. The client may also receive checkpoint messages if the consistent recovery option is turned on and a publisher commits a message.

Context

Guaranteed delivery is a delivery mechanism that preserves events produced by a window, keeps data in a log store, and tracks events consumed by GD subscribers. For more information on guaranteed delivery, see the SAP Event Stream Processor: Developer Guide.

A CCL project can be set to checkpoint after a number of messages pass through it. Once the configured number of messages pass through the project, the server commits the log store and sends a checkpoint message to the subscriber. This indicates that all messages up to the checkpoint sequence number are safely logged in the system.

A subscriber must indicate to the server when it has processed the messages and can recover them without the server. The subscriber can call esp_publisher_commit_gd at any time to tell the server the sequence number of the last message that has been processed. The commit call ensures that the server will not resend messages up to and including the last sequence number committed, and allows it to reclaim resources consumed by these messages. The subscriber should not commit sequence numbers higher than the sequence number received via the last checkpoint message. This ensures that no data is lost if the server restarts.

Procedure

1. Request a GD subscription by calling esp_subscriber_options_set_gd_session(EspSubscriberOptions * options, char * session_name, EspError * error) and creating the Subscriber object.

2. Create and connect a Publisher object.



3. Check if streams have GD or GD with checkpoint enabled by calling esp_stream_is_gd_enabled(const EspStream * stream, EspError * error) and esp_stream_is_checkpoint_enabled(const EspStream * stream, EspError * error).

4. Retrieve active and inactive GD sessions by calling esp_project_get_active_gd_sessions(EspProject * Project, EspList * gd_sessions, EspError * error) and esp_project_get_inactive_gd_sessions(EspProject * Project, EspList * gd_sessions, EspError * error).

5. Retrieve the checkpoint sequence number for the last checkpointed data by calling esp_subscriber_event_get_checkpoint_sequence_number(const EspSubscriberEvent * event, int64_t * seq_val, EspError * error).

6. Tell the server that the subscriber has committed messages up to a given sequence number and no longer needs them by calling esp_publisher_commit_gd(EspPublisher * publisher, char * gd_name, int32_t stream_ids[], int64_t seq_nos[], int32_t len, EspError * error).

7. Cancel the GD session by closing the subscriber or by calling esp_project_cancel_gd_subscriber_session(EspProject * project, char * gd_session, EspError * error).

ExampleExample

int rc = esp_project_connect(project, g_error); if (rc != 0) { print_error_and_exit(g_error, __LINE__); } EspSubscriberOptions *options = esp_subscriber_options_create(g_error); // This will set the GD session. Provide a unique GD session name (GD999 in the example). // The GD session will only be created once esp_project_create_subscriber(…) is called. esp_subscriber_options_set_gd_session(options, "GD999", g_error); EspSubscriber * subscriber = esp_project_create_subscriber(project, options, g_error); if (NULL == subscriber) { print_error_and_exit(g_error, __LINE__); } // Here this example expects that the CCL has a window called "In1" with an attached log store and // guaranteed delivery enabled. You can use stream name from your CCL. const EspStream * stream = esp_project_get_stream(project, "In1", g_error); if (NULL == stream) { print_error_and_exit(g_error, __LINE__); } // This call checks whether stream in the CCL has GD enabled. rc = esp_stream_is_gd_enabled(stream, g_error); if (rc == 0) { printf("%s\n", "stream has GD enabled!”); } // This call checks whether the checkpoint is set in the CCL project. rc = esp_stream_is_checkpoint_enabled(stream, g_error); if (rc == 0) { printf("%s\n", "chkpointEnable...OK"); } rc = esp_subscriber_connect(subscriber, g_error); if (rc != 0) { print_error_and_exit(g_error, __LINE__); } // This call checks whether it is a GD subscriber. rc = esp_subscriber_is_gd_enabled(subscriber, g_error); if (rc == 0) { printf("%s\n", "GD subscriber!"); } EspList * gd_sessions = esp_list_create(ESP_LIST_STRING_T, g_error); // This gets all the active gd_sessions and populates the gd_sessions list. rc = esp_project_get_active_gd_sessions(project, gd_sessions, g_error);



const char * gds = esp_list_get_string(gd_sessions, 0, g_error); printf("%s", gds); while (!done) { event = esp_subscriber_get_next_event(subscriber, g_error); if (esp_error_get_code(g_error) != 0) { print_error_and_exit(g_error, __LINE__); } rc = esp_subscriber_event_get_type(event, &eventType, g_error); if (rc != 0) { esp_subscriber_event_free(event, g_error); print_error_and_exit(g_error, __LINE__); } switch(eventType) { case ESP_SUBSCRIBER_EVENT_DATA: // Process events break; // This event type indicates that the event is a check point event. Check point events are received only // if CCR file has “auto-cp-trans” set. // You can issue GD commit at any time after you have processed the events and can recover it without the ESP server. // This example assumes that CCL project is check point enabled. case ESP_SUBSCRIBER_CHECKPOINT: printf("<checkpoint-is-received/>\n"); int64_t seq_val; // Get the check point sequence number from the check point event. rc = esp_subscriber_event_get_checkpoint_sequence_number(event, &seq_val, g_error); if (rc == 0) { printf("SeqNo # %ld\n", seq_val); int32_t ids[1]; esp_stream_get_id(stream, &ids[0], g_error); printf("stream id = %d\n", ids[0]); int64_t sq[1]; sq[0] = seq_val; printf("seq = %d\n", sq[0]); // Do extra steps to make sure that all the events up to this seq_val are consumed // by third party applications or safely logged so that they can be recovered without // ESP server if needed. // Code for logging or handing off to 3rd party applications goes here… // The following call sends commit message to server indicating that it no longer needs // events up to seq_val. In this scenario (GD + check point), you should not commit sequence // numbers higher than the sequence number received from the server to safely recover data when // server is restarted. rc = esp_publisher_commit_gd(publisher, "GD999", ids, sq, 1, g_error); printf("rc = %d\n", rc); } break; default: break; } esp_subscriber_event_free(event, g_error); } // This cancels the GD subscriber session. When you close a subscriber, this call is internally called and GD session is



// cancelled. This call is here to show how to use the API. rc = esp_project_cancel_gd_subscriber_session(project, "GD999", g_error); if (rc == 0) { printf("cancelled.... SUCCESS!\n"); }

3.5.2 C SDK: Subscribing Examples

The previous sample code on subscribing is provided for illustration purposes, but does not comprise a full, working example.

SAP Event Stream Processor ships with fully functioning examples you can use as a starting point for your own projects. Examples for subscribing are located in:

%STREAMING_HOME%\examples\cpp\SDK (Windows)

$STREAMING_HOME/examples/cpp/SDK (Linux and Solaris)

3.6 C SDK: Failover Handling

The SDK supports either fully transparent or automatic failover in a number of situations.

Cluster failovers

the URIs used to connect to a back-end component can include a list of cluster manager specifications. The SDK maintains connections to these transparently. So, if any one manager in the cluster goes down, the SDK tries to reconnect to another instance. The SDK returns an error only if connections to all known instances fail. If working in callback or select access modes, you can configure the SDK with an additional level of tolerance for loss of connectivity. In this case, the SDK does not disconnect an EspServer instance even if all known manager instances are down. Instead, it generates an ESP_SERVER_EVENT_STALE event. If it manages to reconnect after a (configurable) number of attempts, it generates an ESP_SERVER_EVENT_UPTODATE. Otherwise, it disconnects and generates an ESP_SERVER_EVENT_DISCONNECTED event.

Project failovers

the Event Stream Processor cluster allows a project to be deployed with failover. Based on the configuration settings, a cluster restarts a project if it detects that it has exited (however, projects are not restarted if they are explicitly closed by the user). To support this, you can have EspProject instances monitor the cluster for project restarts and then reconnect. This works only in callback or select modes. An ESP_PROJECT_EVENT_STALE event is generated when the SDK detects that the project has gone down. If it is able to reconnect, it generates an ESP_PROJECT_EVENT_UPTODATE event. Otherwise, it generates an ESP_PROJECT_EVENT_DISCONNECTED event.

When the SDK reconnects, entities obtained from the project are no longer valid. This includes publishers, subscribers, message readers/writers, and row readers/writers. After reconnecting, recreate these objects from the project.

In direct access mode, the SDK does not monitor the cluster for restarts. If a communication error occurs, the project object and all project-related entities are invalidated. Close the project,



which also closes any elements it contains, then create a new project object and reconnect. The following example shows one way of doing this:

// Encountered communication error esp_project_close(project, error);project = esp_project_get(uri, creds, NULL, error); rc = esp_project_connect(project, error); // if the project has been successfully restarted this will succeedif (!rc) {// exit or loop} // create publisher or subscriber and proceed

Active-active deployments

you can deploy a project in active-active mode. In this mode, the cluster starts two instances of the project: a primary instance and a secondary instance. Any data published to the primary instance is automatically mirrored to the secondary instance. The SDK supports such active-active deployments. When connected to an active-active deployment, if the currently connected instance goes down, EspProject tries to reconnect to the alternate instance. Unlike failovers, this happens transparently. Therefore, if the reconnection is successful, there is no indication given to the user. In addition to EspProject, there is support for this mode when publishing and subscribing. If subscribed to a project in an active-active deployment, the SDK does not disconnect the subscription if the instance goes down. Instead, it generates an ESP_SUBSCRIBER_EVENT_DATA_LOST event. It then tries to reconnect to the peer instance. If it is able to reconnect, the SDK resubscribes to the same streams. Subscription clients then receive an ESP_SUBSCRIBER_EVENT_SYNC_START event, followed by the data events, and finally an ESP_SUBSCRIBER_EVENT_SYNC_END event. Clients can use this sequence to maintain consistency with their view of the data if needed. Reconnection during publishing is also supported but only if publishing in synchronous mode. It is not possible for the SDK to guarantee data consistency otherwise. Reconnection during publishing happens transparently; there are no external user events generated.

3.7 C SDK: Examples

ESP includes several working examples for the C SDK.

Table 2:

PublisherExample Demonstrates the basics of SDK use

PublisherAnySchemaExample Publishes using stream metadata

SubscriberCallbackExample Subscribes using the callback mechanism

SubscriberExample Displays published data

Subscriber_gd_example Subscribes using the guaranteed delivery mechanism

UpdateShineThroughExample Publishes updates using ShineThrough

These examples and a readme file with instructions for running them are located at STREAMING_HOME\examples\cpp\sdk.



3.8 C SDK: Versioning

All SDK versions are designed to function seamlessly with previous and future server versions.

The SDK has a version number specific to a support package release. Version numbers ensure compatibility and flexibility between newer and older versions of the SDK and the Event Stream Processor server. If the SDK is working with an older or newer version of the server, it uses the appropriate protocols for that version, and vice versa. In the event that that one of the components in the relationship is too old or too new for a desired functionality, a message prompts clarifying this scenario and provides a solution.

Setting the failifNewer parameter to "true" in ServerOptions allows only connections to concurrent cluster versions. Setting the parameter to "false" allows all connections automatically, and a message prompts only if one component uses a function not supported by the other. This flag is optional, and it defaults to false.

3.9 C SDK: API Reference

Detailed information on methods, functions, and other programming building blocks is provided in the API level documentation.

To access the API level documentation:

1. Navigate to <Install_Dir>/ESP-5_1/doc/sdk/c.2. Launch index.html.



4 Java SDK

Information on starting and stopping, and subscribing and publishing using the Java SDK. Also includes references information and examples.

4.1 Java SDK: Entity Lifecycles and Access Modes

In the SAP® Event Stream Processor Java SDK, all entities share a common lifecycle and set of access modes.

User interaction in the Event Stream Processor SDK is handled through entities the SDK exposes. The main entities are Server, Project, Publisher, and Subscriber. These entities correspond to the functional areas of the SDK. For example, the Server object represents a running instance of a cluster, the Project corresponds to a single project deployed to the cluster, the Publisher object deals with publishing data to a running project, and the Subscriber object subscribes to data streams.

On initial retrieval an entity is considered to be open. When an entity is open, you can retrieve certain static information about it. To accomplish its assigned tasks, an entity connects to the corresponding component in the cluster. A server connects to a running instance of a cluster, and Project, Publisher and Subscriber all connect to running instances of a project in a cluster.




Direct access is the default mode when retrieving an entity. In this mode, all calls return when an error occurs or the operation completes successfully. There are no events generated later, so there is no need to have an associated event handler.


You can register multiple callbacks on each publisher or subscriber by calling the setCallback(...) method multiple times. Each registered handler gets the same events.


SAP Event Stream Processor: SDK GuideJava SDK

The following sample illustrates accessing a project in callback mode. If you are working in callback mode and want to receive the callback events, register your callback handlers before you call connect on the entity you are interested in.

ProjectOptions opts = new ProjectOptions.Builder().setAccessMode(AccessMode.CALLBACK).create(); Project project = SDK.getInstance().getProject(projectUri, creds, opts); project.setCallback(EnumSet.allOf(ProjectEvent.Type.class), new ProjectHandler("Handler")); project.connect(60000); // // Wait or block. Rest of the project lifecycle is handled in the project callback handler // // Project handler class public class ProjectHandler implements Callback { String m_name; ProjectHandler(String name) { m_name = name; } public String getName() { return m_name; } public void processEvent(ProjectEvent pevent) { Project p = pevent.getProject(); try { switch ( pevent.getType() ) { // Project has connected - can retrieve streams, deployment etc. case CONNECTED: String[] streams = pevent.getProject().getModelledStreamNames(); break; // Project disconnected - only call possible connect again case DISCONNECTED: break; // Project closed - this object should not be accessed anymore by user code case CLOSED: break; case STALE: case UPTODATE: break; case ERROR: break; } } catch (IOException e) { } } }

The select access mode lets you multiplex various entities in a single user thread—somewhat similar to the select and poll mechanisms available on many systems—to monitor file descriptors. Register an entity using a Selector together with the events to monitor for. Then, call Selector.select(), which blocks until a monitored update occurs in the background. The function returns a list of SdkEvent objects. First determine the category (server, project, publisher, subscriber) of the event, then handle the appropriate event type. In this mode, the SDK uses a single background update thread to monitor for updates. If detected, the appropriate event is created and pushed to the Selector. The event is then handled in your own thread.



This example shows multiplexing of different entities.

Uri cUri = new Uri.Builder(REAL_CLUSTER_URI).create(); Selector selector = SDK.getInstance().getDefaultSelector(); ServerOptions srvopts = new ServerOptions.Builder().setAccessMode(AccessMode.SELECT).create(); Server server = SDK.getInstance().getServer(cUri, creds, srvopts); ProjectOptions prjopts = new ProjectOptions.Builder().setAccessMode(AccessMode.SELECT).create(); Project project = null; //SDK.getInstance().getProject(cUri, creds, prjopts); SubscriberOptions subopts = new SubscriberOptions.Builder().setAccessMode(AccessMode.SELECT).create(); Subscriber subscriber = null; //SDK.getInstance().getProject(cUri, creds, prjopts); PublisherOptions pubopts = new PublisherOptions.Builder().setAccessMode(AccessMode.SELECT).create(); Publisher publisher = null; //SDK.getInstance().getProject(cUri, creds, prjopts); server.connect(); server.selectWith(selector, EnumSet.allOf(ServerEvent.Type.class)); // Your logic to exit the loop goes here ... while (true) { List<sdkevent> events = selector.select(); for (SdkEvent event : events) { switch (event.getCategory()) { // Server events case SERVER: ServerEvent srvevent = (ServerEvent) event; switch (srvevent.getType()) { // Server has connected - can now perform operations, such as adding removing // applications. case CONNECTED: case MANAGER_LIST_CHANGE: Manager[] managers = srvevent.getServer().getManagers(); for (Manager m : managers) System.out.println("Manager:" + m); break; case CONTROLLER_LIST_CHANGE: Controller[] controllers = srvevent.getServer().getControllers(); for (Controller cn : controllers) System.out.println("Controller:" + cn); break; case WORKSPACE_LIST_CHANGE: break; // This indicates that the Server has updated its state with the latest running application // information. Project objects can now be retrieved case APPLICATION_LIST_CHANGE: case DISCONNECTED: case CLOSED: case ERROR: break; } break;



// Project events case ESP_PROJECT: ProjectEvent prjevent = (ProjectEvent) event; switch (prjevent.getType()) { case CONNECTED: case DISCONNECTED: case CLOSED: case ERROR: case WARNING: break; } break; // Publisher events case PUBLISHER: PublisherEvent pubevent = (PublisherEvent) event; switch (pubevent.getType()) { case CONNECTED: // The publisher is read. This event is to be used to publish data in callback mode case READY: case DISCONNECTED: case CLOSED: break; } break; // Subscriber events case SUBSCRIBER: SubscriberEvent subevent = (SubscriberEvent) event; switch (subevent.getType()) { case CONNECTED: case SUBSCRIBED: case SYNC_START: // There is data. This event is to be used to retrieve the subscribed data. case DATA: case SYNC_END: case DISCONNECTED: case CLOSED: case UNSUBSCRIBED: case DATA_INVALID: case ERROR: case STREAM_EXIT: case DATA_LOST: break; } break; } } }

4.2 Java SDK: Required Libraries

A list of the required librariesin order to run the Java SDK.





Operating System Files

Windows ● libj\streaming-client.jar● libj\streaming-system.jar● libj\xmlrpc-common-3.1.3.jar● libj\xmlrpc-client-3.1.3.jar● libj\ws-commons-util-1.0.2.jar● libj\commons-logging-1.1.jar● libj\commons-codec-1.3.jar● libj\commons-collections-3.2.1.jar● libj\commons-configuration-1.6.jar● libj\commons-lang-2.6.jar● libj\postgresql.jar● libj\sylapi.jar● libj\log4j-1.2.16.jar

Linux, Solaris ● libj/streaming-client.jar● libj/streaming-system.jar● libj/xmlrpc-common-3.1.3.jar● libj/xmlrpc-client-3.1.3.jar● libj/ws-commons-util-1.0.2.jar● libj/commons-logging-1.1.jar● libj/commons-codec-1.3.jar● libj/commons-collections-3.2.1.jar● libj/commons-configuration-1.6.jar● libj/commons-lang-2.6.jar● libj/postgresql.jar● libj/sylapi.jar● libj/log4j-1.2.16.jar

4.3 Java SDK: Starting and Stopping

Start the Java SDK before performing operations.

Initializing the SDK prompts it to start its internal threads and register required resources. This call can be made from any thread, but it must be made before any other SDK functionality is used.

Example:

static final SDK sdk = SDK.getInstance();// retrieve the SDK singleton sdk.start(); // start the SDK

Stop the Java SDK once the application using Java SDK is ready to exit or its functionality is no longer needed. This stops its internal threads and releases any held resources.



Example:

sdk.stop(); = esp_sdk_stop(error); // stop the SDK

Multiple SDK start calls may be made. The Java SDK requires a corresponding number of stop calls to properly shut down.

4.4 Java SDK: Publishing

The SDK provides a number of different options to publish data to a project.

The steps involved in publishing data are.

1. Retrieve a Publisher for the project to which you need to publish. You can create a Publisher directly or from a previously retrieved and connected Project object.

2. Create a MessageWriter for the stream to publish to. You can create multiple MessageWriters from a single Publisher.

3. Create a RelativeRowWriter method.4. Format the data buffer to publish using RelativeRowWriter methods.5. Publish the data.

While Publisher is thread-safe, MessageWriter and RelativeRowWriter are not. Therefore, ensure that you synchronize access to the latter two.

The SDK supports automatic publisher switchover in high availability (HA) configurations, except when publishing asynchronously. Switching over automatically in this instance risks dropping or duplicating records because the SDK does not know which records have been published.

The SDK provides a number of options to tune the behavior of a Publisher. Specify these options using the PublisherOptions object when the Publisher is created. Once created, options cannot be changed. Like all other entities in the SDK, publishing supports the direct, callback, and select access modes.

In addition to access modes, the SDK supports internal buffering. When publishing is buffered, the data is first written to an internal queue. This is picked up by a publishing thread and then written to the ESP project. Note that buffering is possible only in direct access mode. Direct access mode together with buffered publishing potentially provides the best throughput.

Two other settings influence publishing: batching mode and sync mode. Batching controls how data rows are written to the socket. They can be written individually, or grouped together in envelope or transaction batches. Envelopes group individual rows together to send to the ESP project and are read together from the socket by the project. This improves network throughput. Transaction batches, like envelopes, are also written and read in groups. However, with transaction batches, the ESP project only processes the group if all the rows in the batch are processed successfully. If one fails, the whole batch is rolled back.

NoteWhen using shine-through to preserve previous values for data that are null in an update record, publish rows individually or in envelopes, rather than in transaction batches.



When publishing is buffered, you can specify how the SDK batches rows in PublisherOptions. EXPLICIT lets you control the batches by using start transaction and end block calls. AUTOBLOCKING ignores these calls and batches rows internally. The default mode is NONE.

Sync mode settings control the publishing handshake between the SDK and the ESP project. By default, the SDK keeps sending data to the ESP project without waiting for an acknowledgement. If sync mode is set to true, the SDK waits for an acknowledgement from the ESP project each time it sends data before sending more. Issue an explicit Publisher.commit() call when publishing in asynchronous mode before the client application exits or closes the Publisher. This is to guarantee that all data written is read by the ESP project.

In general terms, the return code from a Publish call indicates whether or not the row was successfully transmitted. Any error that occurs during processing on the ESP project (such as a duplicate insert) will not get returned. The precise meaning of the return code from a Publish call depends on the access mode and the choice of synchronous or asynchronous transmission.

When consistent recovery mode is enabled, a project treats commits issued by publishers as a system wide checkpoint request. When the publisher issues commits when consistent recovery is disabled, the project only syncs to disk input windows that are assigned to log stores and does not perform a full checkpoint. In both cases, when the commit call returns from the project, the publisher notifies its source that the data in question has been processed. With respect to system performance, frequent commits have a negative impact in both cases. The impact is more prominent when consistent recovery is enabled because a full checkpoint operation is more expensive compared to a log store sync operation.



When using direct access mode, the type of transmission used determines what the return from the Publish call indicates. If publishing in asynchronous mode, the return only indicates that the SDK has written the data to the socket. If publishing in synchronous mode, the return from the Publish call indicates the response code the ESP project sent.

There are certain considerations to keep in mind when using callback or select mode publishing. These modes are driven by the PublisherEvent.READY event, which indicates that the publisher is ready to accept more data. In response, users can publish data or issue a commit, but only one such action is permitted in response to a single PublisherEvent.READY event.


PublisherOptions opts = new PublisherOptions.Builder().setAccessMode(AccessMode.CALLBACK).create(); Publisher publisher = project.createPublisher(opts); PublisherHandler handler = new PublisherHandler(); publisher.setCallback(EnumSet.allOf(PublisherEvent.Type.class), handler); publisher.connect();



Below are some code snippets that show different ways of publishing data. The sample code provided here is for illustration only; it does not comprise a complete, working example.

This example demonstrates publishing in direct access mode with transaction blocks.

// The Project must be connected first project.connect(60000); Publisher publisher = project.createPublisher(); publisher.connect(); Stream stream = project.getStream("Stream"); MessageWriter mw = publisher.getMessageWriter(s); RelativeRowWriter writer = mw.getRelativeRowWriter(); // It is more efficient to cache this DataType[] types = stream.getEffectiveSchema().getColumnTypes(); // Your logic to loop over data to publish while (true) { // Logic to determine if to start a transaction block if ( ...) mw.startTransaction(0); // Loop over rows in a single block while (true) { // Loop over columns in a row for (i = ...) { writer.startRow(); writer.setOperation(Operation.INSERT); switch (types[i]) { case DATE: writer.setDate(datevalue); break; case DOUBLE: writer.setDouble(doublevalue); break; case INTEGER: writer.setInteger(intvalue); break; case LONG: writer.setLong(longvalue); break; case MONEY: break; case STRING: writer.setString(stringvalue); break; case TIMESTAMP: writer.setTimestamp(tsvalue); break; // // Other data types // } } writer.endRow(); } // Logic to determine if to end block if ( ...) mw.endBlock(); }



publisher.commit();

This example demonstrates publishing in callback access mode. Notice that the access mode is set before the publisher is connected; this is a requirement for both callback and select access modes.

p.connect(60000); PublisherOptions opts = new PublisherOptions.Builder() .setAccessMode(AccessMode.CALLBACK) .create(); Publisher pub = p.createPublisher(opts); PublisherHandler handler = new PublisherHandler(); pub.setCallback(EnumSet.allOf(PublisherEvent.Type.class), handler); pub.connect(); // Block/wait. Publishing happens in the callback handler // .... // // Publisher callback handler // public class PublisherHandler implements Callback<PublisherEvent> { Stream m_stream; MessageWriter m_mwriter; RelativeRowWriter m_rowwriter; public PublisherHandler() throws IOException { } public String getName() { return "PublishHandler"; } public void processEvent(PublisherEvent event) { switch (event.getType()) { case CONNECTED: // It is advisable to create and cache these try { m_stream = event.getPublisher().getProject().getStream("Stream"); m_mwriter = event.getPublisher().getMessageWriter(m_stream); m_rowwriter = m_mwriter.getRelativeRowWriter(); } catch (IOException e) { e.printStackTrace(); } break; case READY: // Publishing code goes here. // NOTE: Only a single publish or a commit call can be made in one READY callback break; case ERROR: case DISCONNECTED: case CLOSED: break; } } }



4.4.1 Java SDK: Publishing Examples



%STREAMING_HOME%\examples\java\com\sybase\esp\sdk\examples (Windows)

$STREAMING_HOME/examples/java/com/sybase/esp/sdk/examples (Linux and Solaris)

4.5 Java SDK: Subscribing

The SDK provides several options for subscribing to data in a project.

Subscribing to data using the SDK involves:

1. Create a Subscriber object. Create the object directly or retrieve it from a Project object.2. Connect the Subscriber object.3. Subscribe to one or more streams. Call Subscriber.subscribeStream(Stream stream) for each

stream you are connecting to, or use Subscriber.subscriberStreams(Stream[] stream) to specify multiple streams.

4. In direct access mode, retrieve events using the Subscriber.getNextEvent() object. In callback and select access modes, the SDK generates events and passes them back to user code.

5. For data events, retrieve MessageReader. This encapsulates a single message from the ESP project. It may consist of a single data row or a block with multiple data rows.

6. Retrieve one or more RowReaders. Use the methods in RowReader to read in individual fields.

The sample code provided here is for illustration only; it does not comprise a complete, working example. This example demonstrates subscribing to a stream in direct access mode with default options:

p.connect(60000); subscriber = p.createSubscriber(); String strName = "WIN1"; subscriber.subscribeStream("WIN1"); subscriber.connect(); // Various data type we will be reading BigDatetime bigdatetime = null; Money m = null; byte[] binary = null; // Logic to exit loop goes here while (true) { SubscriberEvent event = subscriber.getNextEvent(); switch (event.getType()) { case SYNC_START: break; case SYNC_END: break; // There is data to read case DATA: while ( reader.hasNextRow() ) {



RowReader rr = reader.nextRowReader(); for (int j = 0; j < rr.getSchema().getColumnCount(); ++j) { if ( rr.isNull(j)) continue; // This is legal but it is better to cache the data types array switch ( rr.getSchema().getColumnTypes()[j]) { case INTEGER: rr.getInteger(j); break; case LONG: rr.getLong(j); break; case STRING: rr.getString(j); break; case TIMESTAMP: rr.getTimestamp(j)); break; case MONEY01: m = rr.getMoney(j); break; // ... // process other data types // ... } } } break; } } subscriber.disconnect();

4.5.1 Java SDK: Subscribing Examples



%STREAMING_HOME%\examples\java\com\sybase\esp\sdk\examples (Windows)

$STREAMING_HOME/examples/java/com/sybase/esp/sdk/examples (Linux and Solaris)



4.5.2 Java SDK: Subscribing with Guaranteed Delivery


Prerequisites


Context



A subscriber must indicate to the server when it has processed the messages and can recover them without the server. The subscriber can call esp_publisher_commit_gd at any time to tell the server the sequence number of the last message that has been processed. The commit call ensures that the server will not resend messages up to and including the last sequence number committed, and allows it to reclaim resources consumed by these messages. The subscriber should not commit sequence numbers higher than the sequence number received via the last checkpoint message. This ensures that no data is lost if the server restarts.

Procedure

1. Request a GD subscription by setting setGDSession(String gdName) in SubscriberOptions and creating the subscriber.

2. Create and connect a Publisher object.3. Check if streams have GD or GD with checkpoint enabled, either from the project interface by calling

project.isGDStream(String streamName) and project.isGDCheckPointStream(String streamName) or from the stream interface by calling stream.isGDEnabled() and stream.isGDCheckPointEnabled().

4. Retrieve active and inactive GD sessions by calling project.getActiveGDSessions() and project.getInactiveGDSessions().



5. Tell the server that the subscriber has committed messages up to a given sequence number and no longer needs them by calling publisher.commitGD(String gdName, int[] streamIDs, long[] seqNos) throws IOException.

6. Cancel the GD session by closing the subscriber or with project.cancelGDSession(String gdName).

ExampleExample

// create a GD subscriber SDK sdk = SDK.getInstance(); SubscriberOptions.Builder optBuilder = new SubscriberOptions.Builder(); optBuilder.setGDSession(gdName); SubscriberOptions options = optBuilder.create(); Subscriber subscriber = sdk.createSubscriber(projectUri, creds, options); subscriber.connect(); subscriber.subscribeStream(gdStream); // create an ESP project ProjectOptions opts = new ProjectOptions.Builder().create(); ProjectImpl project = (ProjectImpl) SDK.getInstance().getProject(projectUri, creds, opts); project.connect(60000); // create a publisher to commitGD message Publisher publisher = project.createPublisher(); publisher.connect(); Stream stream = project.getStream(gdStream); // check GD status/mode boolean gdStatus = project.isGDStream(gdStream); boolean gdCPStatus = project.isGDCheckPointStream(gdStream); boolean isGD = subscriber.isGD(); // retrieve GD sessions String[] actGD = project.getActiveGDSessions(); String[] inactGD = project.getInactiveGDSessions(); int streamId = stream.getStreamId(); long seqNo = 0L; // sequence number for GD record SubscriberEvent event = null; boolean done = false; while (!done) { event = subscriber.getNextEvent(); switch (event.getType()) { case DATA: MessageReader reader = event.getMessageReader(); Stream s = reader.getStream(); String str = s.getName(); while (reader.hasNextRow()) { // use nextRowReader(isGD) to read GD and non-GD records RowReader row = reader.nextRowReader(isGD); System.out.println(row.toXmlString(str)); } break; case CHECKPOINT: // retrieve the sequence number returned by CHECKPOINT message seqNo = event.getSeqNo(); System.out.println("<checkpoint>" + seqNo + "</checkpoint>"); int[] streamIds = new int[1]; streamIds[0] = streamId; long[] seqNos = new long[1]; seqNos[0] = seqNo; // commitGD message for a single stream with the corresponding // last checkpointed sequence number publisher.commitGD(gdName, streamIds, seqNos);



System.out.println("<commitGD>" + seqNo + "</commitGD>"); break; case CLOSED: System.out.println("<closed/>"); done = true; break; case DISCONNECTED: System.out.println("<disconnected/>"); done = true; break; } } // cancel GD session and disconnect subscriber and publisher before exit project.cancelGDSession(gdName); subscriber.disconnect(); publisher.disconnect();

4.6 Java SDK: Failover Handling


Cluster Failovers

the URIs used to connect to a back-end component can include a list of cluster manager specifications. The SDK maintains connections to these transparently. If any one manager in the cluster goes down, the SDK tries to reconnect to another instance. If connections to all known instances fail, the SDK returns an error. If working in callback or select access modes, you can configure the SDK with an additional level of tolerance for loss of connectivity. In this case, the SDK does not disconnect a Server instance even if all known manager instances are down. Instead, it generates a ServerEvent.STALE event. If it manages to reconnect after a (configurable) number of attempts, it generates a ServerEvent.UPTODATE event. Otherwise, it disconnects and generates a ServerEvent.DISCONNECTED event.

Project Failovers

the Event Stream Processor cluster lets you deploy projects with failover. Based on the configuration settings, a cluster restarts a project if it detects that it has exited (however, projects are not restarted if they are explicitly closed by the user). To support this, you can have Project instances monitor the cluster for project restarts and then reconnect. This works only in callback or select modes. When the SDK detects that a project has gone down, it generates a ProjectEvent.STALE event. If it is able to reconnect, it generates a ProjectEvent.UPTODATE event, otherwise it generates a ProjectEvent.DISCONNECTED event.


In direct access mode, the SDK does not monitor the cluster for restarts. If a communication error occurs, the project object and all project-related entities are invalidated. Close the project, which also closes any elements it contains, then create a new project object and reconnect. The following example shows one way of doing this:

// Encountered communication error project.close();



Project project = s_sdk.getProject(uri, creds); project.connect(WAIT_TIME_60000_MS);// will succeed if cluster has restarted project. // Create publisher or subscriber and proceed

Active-Active Deployments

you can deploy a project in active-active mode. In this mode, a cluster starts two instances of the project: a primary instance and a secondary instance. Any data published to the primary is automatically mirrored to the secondary instance. The SDK supports active-active deployments. When connected to an active-active deployment, if the currently connected instance goes down, the Project tries to reconnect to the alternate instance. Unlike failovers, this happens transparently. Therefore, if the reconnection is successful, there is no indication given to the user. In addition to the Project, there is support for this mode when publishing and subscribing. If subscribed to a project in an active-active deployment, the SDK does not disconnect the subscription if the instance goes down. Instead, it generates a SubscriberEvent.DATA_LOST event. It then tries to reconnect to the peer instance. If it is able to reconnect, the SDK resubscribes to the same streams. Subscription clients then receive a SubscriberEvent.SYNC_START event, followed by the data events, and finally a SubscriberEvent.SYNC_END event. Clients can use this sequence to maintain consistency with their view of the data if needed. Reconnection during publishing is also supported, but only if publishing in synchronous mode. It is not possible for the SDK to guarantee data consistency otherwise. Reconnection during publishing happens transparently; there are no external user events generated.

4.7 Java SDK: Examples

SAP Event Stream Processor includes several working examples for the Java SDK.

Table 4:

ESP_JDBC_Example Shows how to perform ad-hoc inserts, updates, and deletes for ESP windows

ESP_SQL_Projection Shows how to subscribe to a subset of data

ProjectDeployment Shows how to deploy a project to the cluster with the SDK

PublisherAnySchemaExample Publishes using stream metadata


SubscriberCallbackExample Subscribes using the callback mechanism


UpdateShineThroughExample Publishes updates using ShineThrough

WorkspaceProjectProperties Shows the type of information about the cluster and projects that can be retrieved. Similar to streamingclusteradmin functionality

These examples and a readme file with instructions for running them are located at STREAMING_HOME\examples\java.



4.8 Java SDK: Versioning

All SDK versions are designed to function seamlessly with previous and future server versions.

The SDK has a version number specific to a support package release. Version numbers ensure compatibility and flexibility between newer and older versions of the SDK and the Event Stream Processor server. If the SDK is working with an older or newer version of the server, it uses the appropriate protocols for that version, and vice versa. In the event that that one of the components in the relationship is too old or too new for a desired functionality, a message prompts clarifying this scenario and provides a solution.


4.9 Java SDK: API Reference


To access the API level documentation:

1. Navigate to <Install_Dir>/ESP-5_1/doc/sdk/java.2. Launch index.html.



5 .NET SDK

Information on starting and stopping, and subscribing and publishing using the .NET SDK. Also includes references information and examples.

5.1 .NET SDK: Entity Lifecycles and Access Modes

The SAP® Event Stream Processor .NET SDK offers the same functionality and uses the same concepts as the C SDK. All entities exposed by the SDK have a common lifecycle.

User interaction in the Event Stream Processor SDK is handled through entities the SDK exposes. The main entities are Server, Project, Publisher, and Subscriber. These entities correspond to the functional areas of the SDK. The Server object represents a running instance of a cluster, the Project corresponds to a single project deployed to the cluster, the Publisher object deals with publishing data to a running project, and the Subscriber object subscribes to data streams.

On initial retrieval, an entity is considered to be open. When an entity is open, you can retrieve certain static information about it. To accomplish its assigned tasks, an entity has to connect to the corresponding component in the cluster. A server connects to a running instance of a cluster, and NetEspProject, NetEspPublisher, and NetEspSubscriber all connect to running instances of a project in a cluster.




Direct access is the default mode when retrieving an entity. In this mode, all calls return when an error occurs or the operation completes successfully. There are no events generated later, so there is no need to have an associated event handler.


You can register multiple callbacks on each publisher or subscriber by calling int32_t NetEspPublisher::set_callback (uint32_t events, PUBLISHER_EVENT_CALLBACK^ callback,


SAP Event Stream Processor: SDK Guide.NET SDK

IntPtr^ user_data, NetEspError^ error) or int32_t NetEspSubscriber::set_callback (uint32_t events, SUBSCRIBER_EVENT_CALLBACK^ callback, IntPtr^ user_data, NetEspError^ error) multiple times. Each registered handler gets the same events.

The select access mode lets you multiplex various entities in a single user thread—somewhat similar to the select and poll mechanisms available on many systems—to monitor file descriptors. To register an entity, call the select_with(...) method on the entity you want to monitor (NetEspServer, NetEspPublisher, NetEspSubscriber, or NetEspProject), passing in the NetEspSelector instance together with the events to monitor for. Then, call the select(...) method on the NetEspSelector instance, which blocks until a monitored update occurs in the background. The function returns a list of NetEspEvent objects. First determine the category (server, project, publisher, subscriber) of the event, then handle the appropriate event type. In this mode, the SDK uses a single background update thread to monitor for updates. If detected, the appropriate event is created and pushed to the NetEspSelector. The event is then handled in your own thread.

5.1.1 .NET SDK: Starting

Before performing operations, start the SDK.

Procedure

1. Create an error message store for the following:

NetEspError error = new NetEspError();

2. Get an instance of the .NET SDK and invoke the start method:

NetEspSdk s_sdk = NetEspSdk.get_sdk(); s_sdk.start(espError);

5.1.2 .NET SDK: Connecting to a Server

When you have started the SDK, connect to a server.

Prerequisites

Start the SDK.



Procedure

1. Create a URI object:

NetEspUri uri = new NetEspUri(); uri.set_uri("esp://myserver:19011", error);

2. Create your credentials. The type of credentials depends on which security method is configured with the cluster:

NetEspCredentials creds = new NetEspCredentials(NetEspCredentials.NET_ESP_CREDENTIALS_T.NET_ESP_CREDENTIALS_SERVER_RSA); creds.set_user("auser");creds.set_password("1234");creds.set_keyfile("..\\test_data\\keys\\client.pem");

3. Set options:

NetEspServerOptions options = new NetEspServerOptions(); options.set_mode(NetEspServerOptions.NET_ESP_ACCESS_MODE_T.NET_CALLBACK_ACCESS);

4. Connect to the server:

server = new NetEspServer(uri, creds, options); int rc = server.connect(error);

5.1.3 .NET SDK: Getting and Connecting to a Project

To publish or subscribe to data, get and connect to a project instance.

Procedure

1. Get the project:

NetEspProject project = server.get_project(“workspacename”, “projectname”, error);

2. Connect to the project:

project.connect(error);

5.2 .NET SDK: Required Libraries

A list of the required libraries in order to run the .NET SDK.





SDK Files

.NET 3.5 ● net\interfaces\sdk\net35\streamingnetsdk_lib.dll● net\interfaces\sdk\net35\streamingsystem_lib.dll● net\interfaces\sdk\net35\icudt44.dll● net\interfaces\sdk\net35\icuin44.dll● net\interfaces\sdk\net35\icuuc44.dll● net\interfaces\sdk\net35\xerces-c_2_6.dll● net\interfaces\sdk\net35\libeay32.dll● net\interfaces\sdk\net35\ssleay32.dll

For 32bit SDK support:

● net\interfaces\sdk\net35\x86\streamingnetsdk_lib.dll● net\interfaces\sdk\net35\x86\streamingsystem_lib.dll● net\interfaces\sdk\net35\x86\icudt44.dll● net\interfaces\sdk\net35\x86\icuin44.dll● net\interfaces\sdk\net35\x86\icuuc44.dll● net\interfaces\sdk\net35\x86\xerces-c_2_6.dll

.NET 4.0 ● net\interfaces\sdk\streamingnetsdk_lib.dll● net\interfaces\sdk\streamingsystem_lib.dll● net\interfaces\sdk\icudt44.dll● net\interfaces\sdk\icuin44.dll● net\interfaces\sdk\icuuc44.dll● net\interfaces\sdk\xerces-c_2_6.dll● net\interfaces\sdk\libeay32.dll● net\interfaces\sdk\ssleay32.dll

For 32bit SDK Support:

● net\interfaces\sdk\x86\streamingnetsdk_lib.dll● net\interfaces\sdk\x86\streamingsystem_lib.dll● net\interfaces\sdk\x86\icudt44.dll● net\interfaces\sdk\x86\icuin44.dll● net\interfaces\sdk\x86\icuuc44.dll● net\interfaces\sdk\x86\xerces-c_2_6.dll

NoteTo support legacy applications and legacy third-party libraries that are 32-bit, the 64-bit Windows installer includes 32-bit versions of the C/C++ SDK.



5.3 .NET SDK: PublishingThe SDK provides various options for publishing data to a project.

The steps involved in publishing data are:

1. Create a NetEspPublisher from a previously connected NetEspProject instance.2. Create a NetEspMessageWriter for the stream to publish to. You can create multiple NetEspMessageWriters

from a single NetEspPublisher.3. Create a NetEspRelativeRowWriter.4. Format the data buffer to publish using NetEspRelativeRowWriter methods.5. Publish the data.

While NetEspPublisher is thread-safe, NetEspMessageWriter and NetEspRelativeRowWriter are not. Therefore, ensure that you synchronize access to the latter two.

The SDK provides a number of options to tune the behavior of a NetEspPublisher. Specify these options using NetEspPublisherOptions when creating the NetEspPublisher. Once created, options cannot be changed. Like all other entities in the SDK, publishing also supports the direct, callback, and select access modes.

In addition to access modes, the SDK supports internal buffering. When publishing is buffered, the data is first written to an internal queue. This is picked up by a publishing thread and then written to the ESP project. Buffering is possible only in direct access mode. Direct and buffered publishing potentially provides the best throughput.

Two other settings influence publishing: batching mode and sync mode. Batching controls how data rows are written to the socket. They can be written individually or grouped together in either envelope or transaction batches. Envelopes group individual rows together to send to the ESP project and are read together from the socket by the project. This improves network throughput. Transaction batches, like envelope batches, are also written and read in groups. However, with transaction batches, the ESP project only processes the group if all the rows in the batch are processed successfully. If one fails, the whole batch is rolled back.

When publishing is buffered, you can specify how the SDK batches rows in NetEspPublisherOptions. NET_EXPLICIT_BLOCKING lets you control the batches by using start transaction and end block calls. NET_AUTO_BLOCKING ignores these calls and batches rows internally. The default mode is NET_NO_BLOCKING.

Sync mode settings control the publishing handshake between the SDK and the ESP project. By default, the SDK sends data to the ESP project without waiting for acknowledgement. If sync mode is set to true, the SDK waits for acknowledgement from the ESP project before sending the next batch of data. This provides an application level delivery guarantee, but it reduces throughput.

Publishing in async mode improves throughput, but does not provide an application level delivery guarantee. Since TCP does not provide an application level delivery guarantee either, data in the TCP buffer could be lost when a client exits. Therefore, a commit must be executed before a client exit when publishing in async mode.

In general terms, the return code from a Publish call indicates whether or not the row was successfully transmitted. Any error that occurs during processing on the Event Stream Processor project (such as a duplicate insert) will not get returned. The precise meaning of the return code from a Publish call depends on the access mode and the choice of synchronous or asynchronous transmission.

When consistent recovery mode is enabled, a project treats commits issued by publishers as a system wide checkpoint request. When the publisher issues commits when consistent recovery is disabled, the project only syncs to disk input windows that are assigned to log stores and does not perform a full checkpoint. In both cases, when the commit call returns from the project, the publisher notifies its source that the data in question has been processed. With respect to system performance, frequent commits have a negative impact in both cases. The



impact is more prominent when consistent recovery is enabled because a full checkpoint operation is more expensive compared to a log store sync operation.



When using direct access mode, the type of transmission used determines what the return from the Publish call indicates. If publishing in asynchronous mode, the return only indicates that the SDK has written the data to the socket. If publishing in synchronous mode, the return from the Publish call indicates the response code Event Stream Processor sent.

There are certain considerations to keep in mind when using callback or select mode publishing. These modes are driven by the NET_ESP_PUBLISHER_EVENT_READY event, which indicates that the publisher is ready to accept more data. In response, you can publish data or issue a commit, but only one such action is permitted in response to a single NET_ESP_PUBLISHER_EVENT_READY event.


net_esp_publisher_options_set_access_mode(options, CALLBACK_ACCESS, error); net_esp_publisher_set_callback(publisher, events, callback, NULL, error); net_esp_publisher_connect(publisher, error);

5.3.1 .NET SDK: Publishing in Direct Access Mode

Publishing in direct access mode is a multistep process that involves creating and connecting to a publisher, then identifying the stream to publish to and the data to publish.

Context

The following code snippets are provided to illustrate one way of publishing and not as a complete, working example. Adapt this sample as necessary to suit your specific publishing scenario.

Procedure

1. Create a publisher:

NetEspCredentials creds = new NetEspCredentials



(NetEspCredentials.NET_ESP_CREDENTIALS_T.NET_ESP_CREDENTIALS_USER_PASSWORD);creds.set_user("user");creds.set_password("password"); NetEspPublisher publisher = project.create_publisher(creds, error);

2. Connect to the publisher:

Publisher.connect(error);

3. Get a stream:

NetEspStream stream = project.get_stream("WIN2", error);

4. Get the Message Writer:

NetEspMessageWriter writer = publisher.get_message_writer(stream, error);

5. Get and start the Row Writer, and set an opcode to insert one row:

NetEspRelativeRowWriter rowwriter = writer.get_relative_row_writer(error); rowwriter.start_row(error); rowwriter.set_opcode(1, error);

6. Set the column values sequentially, starting from the first column. Call the appropriate set method for the data type of the column. For example, if the column type is string:

rc = rowwriter.set_string(“some value”, error);

7. When you have set all column values, end the row:

rc = rowwriter.end_row(error);

8. Publish the data:

rc = publisher.publish(writer, error);

5.3.2 .NET SDK: Publishing Examples



%STREAMING_HOME%\examples\net\PublisherExample (Windows)

$STREAMING_HOME/examples/net/PublisherExample (Linux and Solaris)



5.4 .NET SDK: Subscribing

The SDK provides various options for subscribing to a project.

The steps involved in subscribing to data using the SDK are:

1. Create a NetEspSubscriber from a previously connected NetEspProject instance.2. Subscribe to one or more streams. Call int32_t NetEspSubscriber::subscribe_stream

(NetEspStream^ stream, NetEspError^ error ) for each stream you are connecting to.3. In direct access mode, retrieve events using NetEspSubscriber.get_next_event(). In callback and

select access modes, the event is generated by the SDK and passed back to user code.4. For data events, retrieve NetEspMessageReader. This encapsulates a single message from the ESP project. It

may consist of a single data row or a block with multiple data rows.5. Retrieve one or more NetEspRowReader. Use the methods in NetEspRowReader to read in individual fields.

5.4.1 .NET SDK: Subscribing to a Stream in Callback Mode

Subscribing in callback mode is a multistep process that involves creating a subscriber and callback registry, connecting to the subscriber, and then subscribing to a stream.

Context

The following code snippets are provided to illustrate one way of subscribing and not as a complete, working example. Adapt this sample as necessary to suit your particular subscription scenario.

Procedure

1. Create a subscriber:

NetEspSubscriberOptions options = new NetEspSubscriberOptions(); options.set_mode(NetEspSubscriberOptions.NET_ESP_ACCESS_MODE_T.NET_CALLBACK_ACCESS); NetEspSubscriber subscriber = project.create_subscriber(options,error);

2. Create the callback registry:

NetEspSubscriber.SUBSCRIBER_EVENT_CALLBACK callbackInstance = new NetEspSubscriber.SUBSCRIBER_EVENT_CALLBACK(subscriber_callback); subscriber.set_callback(NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_ALL, callbackInstance, null, error);

3. Connect to the subscriber:

subscriber.connect(error);



4. Subscribe to a stream:

subscriber.subscribe_stream(stream, error);

○ Callback function implementation:

Public static void subscriber_callback(NetEspSubscriberEvent event, ValueType data) { switch (evt.getType()) { case (uint)(NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_CONNECTED): Console.WriteLine("the callback happened: connected!"); break;(uint)( NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_DATA): //handleData ... break; default: break; } }//end subscriber_callback

○ handleData implementation:

NetEspRowReader row_reader = null; while ((row_reader = evt.getMessageReader().next_row(error)) != null) { for (int i = 0; i < schema.get_numcolumns(); ++i) { if ( row_reader.is_null(i) == 1) { Console.Write("null, "); continue; } switch (NetEspStream.getType(schema.get_column_type((uint)i, error))) { case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_INTEGER: ivalue = row_reader.get_integer(i, error); Console.Write(ivalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_LONG: lvalue = row_reader.get_long(i, error); Console.Write(lvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_FLOAT: fvalue = row_reader.get_float(i, error); Console.Write(fvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_STRING: svalue = row_reader.get_string(i, error); Console.Write(svalue); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_SECONDDATE: dvalue = row_reader.get_seconddate(i, error); Console.Write(dvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MSDATE: tvalue = row_reader.get_msdate(i, error);



Console.Write(tvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_BOOLEAN: boolvalue = row_reader.get_boolean(i, error); Console.Write(boolvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_BINARY: uint buffersize = 256; binvalue = row_reader.get_binary(i, buffersize, error); Console.Write(System.Text.Encoding.Default.GetString(binvalue) + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_INTERVAL: intervalue = row_reader.get_interval(i, error); Console.Write(intervalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MONEY01: mon = row_reader.get_money(i, error); Console.Write(mon.get_long(error) + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MONEY02: lvalue = row_reader.get_money_as_long(i, error); Console.Write(lvalue + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MONEY03: mon = row_reader.get_money(i, error); Console.Write(mon.get_long(error) + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MONEY10: mon = row_reader.get_money(i, error); Console.Write(mon.get_long(error) + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_MONEY15: mon = row_reader.get_money(i, error); Console.Write(mon.get_long(error) + ", "); break; case NetEspStream.NET_DATA_TYPE_T.NET_ESP_DATATYPE_BIGDATETIME: bdt2 = row_reader.get_bigdatetime(i, error); long usecs = bdt2.get_microseconds(error); Console.Write(usecs + ", "); break; } } rc = subscriber.disconnect(error); }



5.4.2 .NET SDK: Subscribing Examples



%STREAMING_HOME%\examples\net\SubscriberExample (Windows)

$STREAMING_HOME/examples/net/SubscriberExample (Linux and Solaris)

5.4.3 .NET SDK: Subscribing with Guaranteed Delivery


Prerequisites


Context



A subscriber must indicate to the server when it has processed the messages and can recover them without the server. The subscriber can call NetEspPublisher.commit_gd at any time to tell the server the sequence number of the last message that has been processed. The commit call ensures that the server will not resend messages up to and including the last sequence number committed, and allows it to reclaim resources consumed by these messages. The subscriber should not commit sequence numbers higher than the sequence number received via the last checkpoint message. This ensures that no data is lost if the server restarts.



Procedure

1. Request a GD subscription by calling NetEspSubscriberOptions.set_gd_session(string session_name) and creating the NetEspSubscriber object.

2. Create and connect a NetEspPublisher object.3. Check if streams have GD or GD with checkpoint enabled by calling

NetEspStream.is_gd_enabled(NetEspError^ error) and NetEspStream.is_checkpoint_enabled(NetEspError^ error).

4. Retrieve active and inactive GD sessions by calling NetEspProject.get_active_gd_sessions(NetEspError^ error) and NetEspProject.get_inactive_gd_sessions NetEspError^ error).

5. Retrieve the checkpoint sequence number for the last checkpointed data by calling NetEspSubscriberEvent.get_checkpoint_sequence_number(NetEspError^ error).

6. Tell the server that the subscriber has committed messages up to a given sequence number and no longer needs them by calling NetEspPublisher.commit_gd(String^ session_name, array<int32_t>^ stream_ids, array<int64_t>^ seq_nos, NetEspError^ error).

7. Cancel the GD session by closing the subscriber or by calling NetEspProject.cancel_gd_subscriber_session(String^ gd_session, NetEspError^ error).

ExampleExample

// To connect to ESP server NetEspError espError = new NetEspError(); NetEspSdk s_sdk = SYBASE.Esp.SDK.NetEspSdk.get_sdk(); s_sdk.start(espError); NetEspCredentials creds = new NetEspCredentials(NetEspCredentials.NET_ESP_CREDENTIALS_T.NET_ESP_CREDENTIALS_USER_PASSWORD); creds.set_user("sybase"); creds.set_password("sybase"); NetEspUri uri = new NetEspUri(); uri.set_uri("esp://localhost:19011", espError); NetEspServerOptions soptions = new NetEspServerOptions(); NetEspServer server = s_sdk.get_server(uri, creds, soptions, espError); server.connect(espError); // To connect to an ESP project NetEspProject project = new NetEspProject(); NetEspProjectOptions projoptions = new NetEspProjectOptions(); project = server.get_project("workspace", "gd", projoptions, espError); project.connect(espError); // To create a GD subscriber NetEspSubscriberOptions suboptions = new NetEspSubscriberOptions(); suboptions.set_gd_session("GD999"); NetEspSubscriber subscriber = project.create_subscriber(suboptions, espError); // To create a publisher to commitGD message NetEspPublisher publisher = project.create_publisher(null, espError); publisher.connect(espError); //check GD status/mode NetEspStream stream1 = project.get_stream("In1", espError); subscriber.subscribe_stream(stream1, espError); subscriber.connect(espError); stream1.is_gd_enabled(espError); stream1.is_checkpoint_enabled(espError); subscriber.is_gd_enabled(); // retrieve GD sessions



project.get_active_gd_sessions(espError); project.get_inactive_gd_sessions(espError); NetEspSubscriberEvent event1; Boolean done = false; while (!done) { event1 = subscriber.get_next_event(espError); switch (event1.getType()) { case (uint)NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_DATA: NetEspMessageReader reader = event1.getMessageReader(); NetEspRowReader rows = reader.next_row(espError); while (rows != null) { int intcol1 = rows.get_integer(1, espError); Console.Out.Write(intcol1); string stringco2 = rows.get_string(2, espError); Console.Out.WriteLine(" " + stringco2); rows = reader.next_row(espError); } break; case (uint)NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_CLOSED: done = true; break; case (uint)NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_EVENT_DISCONNECTED: done = true; break; case (uint)NetEspSubscriber.NET_ESP_SUBSCRIBER_EVENT.NET_ESP_SUBSCRIBER_CHECKPOINT: // retrieve the sequence number returned by NET_ESP_SUBSCRIBER_CHECKPOINT message long seq_val = event1.get_checkpoint_sequence_number(espError); if (seq_val > 0) { Console.Out.WriteLine("SeqNo #" + seq_val); int[] idArray = new int[1]; long[] seqArray = new long[1]; idArray[0] = stream1.get_id(); seqArray[0] = seq_val; // commitGD message for a single stream with the corresponding last checkpointed sequence number publisher.commit_gd("GD999", idArray, seqArray, espError); } break; default: break; } } // cancel GD session and disconnect subscriber and publisher before exit project.cancel_gd_subscriber_session("GD999", espError); subscriber.disconnect(espError); server.disconnect(espError);



5.5 .NET SDK: Stopping

When your operations are complete, stop the .NET SDK to free up resources.

Procedure

To stop the .NET SDK, use:

s_sdk.stop(espError);

5.6 .NET SDK: Failover Handling


Cluster failovers

the URIs used to connect to a back-end component can include a list of cluster manager specifications. The SDK maintains connections to these transparently. If any one manager in the cluster goes down, the SDK tries to reconnect to another instance. If connections to all known instances fail, the SDK returns an error. If working in callback or select access modes, you can configure the SDK with an additional level of tolerance for loss of connectivity. In this case, the SDK does not disconnect a NetEspServer instance even if all known manager instances are down. Instead, it generates a NET_ESP_SERVER_EVENT_STALE event. If it manages to reconnect after a (configurable) number of attempts, it generates a NET_ESP_SERVER_EVENT_UPTODATE event. Otherwise, it disconnects and generates a NET_ESP_SERVER_EVENT_DISCONNECTED event.

Project failovers

the Event Stream Processor cluster allows a project to be deployed with failover. Based on the configuration settings, a cluster restarts a project if it detects that it has exited (however, projects are not restarted if they are explicitly closed by the user). To support this, you can have NetEspProject instances monitor the cluster for project restarts and then reconnect. This works only in callback or select modes. A NET_ESP_PROJECT_EVENT_STALE is generated when the SDK detects that the project has gone down. If it is able to reconnect, it generates a NET_ESP_PROJECT_EVENT_UPTODATE event. Otherwise, it generates a NET_ESP_PROJECT_EVENT_DISCONNECTED event.


In direct access mode, the SDK does not monitor the cluster for restarts. If a communication error occurs, the project object and all project-related entities are invalidated. Close the project, which also closes any elements it contains, then create a new project object and reconnect. The following example shows one way of doing this:

// encountered error



project.close(esperror);project = server.get_project(workspace, pname, projoptions, esperror);rc = project.connect(esperror);// if the project has been successfully restarted this will succeedif (!rc) {// exit or loop} // create publisher or subscriber and proceed

Active-active deployments

You can deploy a project in active-active mode. In this mode, the cluster starts two instances of the project: a primary instance and a secondary instance. Any data published to the primary instance is automatically mirrored to the secondary instance. The SDK supports active-active deployments. When connected to an active-active deployment, if the currently connected instance goes down, NetEspProject tries to reconnect to the alternate instance. Unlike failovers, this happens transparently. Therefore, if the reconnection is successful, there is no indication given to the user. In addition to NetEspProject, there is support for this mode when publishing and subscribing. If subscribed to a project in an active-active deployment, the SDK does not disconnect the subscription if the instance goes down. Instead, it generates a NET_ESP_SUBSCRIBER_EVENT_DATA_LOST event. It then tries to reconnect to the peer instance. If it is able to reconnect, the SDK resubscribes to the same streams. Subscription clients then receive a NET_ESP_SUBSCRIBER_EVENT_SYNC_START event, followed by the data events, and finally a NET_ESP_SUBSCRIBER_EVENT_SYNC_END event. Clients can use this sequence to maintain consistency with their view of the data if needed. Reconnection during publishing is also supported but only if publishing in synchronous mode. It is not possible for the SDK to guarantee data consistency otherwise. Reconnection during publishing happens transparently; there are no external user events generated.

5.7 .NET SDK: Examples

SAP Event Stream Processor includes several working examples for the .NET SDK.

Table 6:


PublisherAnySchema Publishes using stream metadata

SubscriberCallback Subscribes using the callback mechanism


SubscriberGdExample Subscribes using the guaranteed delivery mechanism

UpdateShineThrough Publishes updates using ShineThrough

These examples and a readme file with instructions for running them are located at STREAMING_HOME\examples\net.



5.8 .NET SDK: API Reference


Context

API documentation is packaged and installed with ESP. To access the API level documentation:

Procedure

1. Navigate to <Install_Dir>\ESP-5_1\doc\sdk\net.

2. Launch index.html.

5.9 .NET SDK: Versioning

All SDK versions are designed to function seamlessly with current and previous server versions.

The SDK has a version number specific to a support package release. Version numbers ensure compatibility and flexibility between newer and older versions of the SDK and the Event Stream Processor server. If the SDK is working with an older or newer version of the server, it uses the appropriate protocols for that version and vice versa. In the event that that one of the components in the relationship is too old or too new for a desired functionality, a message prompts clarifying this scenario and provides a solution.




Important Disclaimers and Legal Information

Coding SamplesAny software coding and/or code lines / strings ("Code") included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, unless damages were caused by SAP intentionally or by SAP's gross negligence.

AccessibilityThe information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date of publication; it is in no way intended to be a binding guideline on how to ensure accessibility of software products. SAP in particular disclaims any liability in relation to this document. This disclaimer, however, does not apply in cases of wilful misconduct or gross negligence of SAP. Furthermore, this document does not result in any direct or indirect contractual obligations of SAP.

Gender-Neutral LanguageAs far as possible, SAP documentation is gender neutral. Depending on the context, the reader is addressed directly with "you", or a gender-neutral noun (such as "sales person" or "working days") is used. If when referring to members of both sexes, however, the third-person singular cannot be avoided or a gender-neutral noun does not exist, SAP reserves the right to use the masculine form of the noun and pronoun. This is to ensure that the documentation remains comprehensible.

Internet HyperlinksThe SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. SAP does not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. SAP shall not be liable for any damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. All links are categorized for transparency (see: http://help.sap.com/disclaimer).


SAP Event Stream Processor: SDK GuideImportant Disclaimers and Legal Information

http://help.sap.com/disclaimer/

SAP Event Stream Processor: SDK GuideImportant Disclaimers and Legal Information


www.sap.com/contactsap

© 2015 SAP SE or an SAP affiliate company. All rights reserved.No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice.Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.Please see http://www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

http://www.sap.com/contactsap

http://www.sap.com/corporate-en/legal/copyright/index.epx

http://www.sap.com/corporate-en/legal/copyright/index.epx

Documents

SAP Event Stream Processor: SDK Guide - SAP Help Portal